Column Filtering in Logical Replication
Hi,
Filtering of columns at the publisher node will allow for selective
replication of data between publisher and subscriber. In case the updates
on the publisher are targeted only towards specific columns, the user will
have an option to reduce network consumption by not sending the data
corresponding to new columns that do not change. Note that replica
identity values will always be sent irrespective of column filtering settings.
The column values that are not sent by the publisher will be populated
using local values on the subscriber. For insert command, non-replicated
column values will be NULL or the default.
If column names are not specified while creating or altering a publication,
all the columns are replicated as per current behaviour.
The proposal for syntax to add table with column names to publication is as
follows:
Create publication:
CREATE PUBLICATION <pub_name> [ FOR TABLE [ONLY] table_name [(colname
[,…])] | FOR ALL TABLES]
Alter publication:
ALTER PUBLICATION <pub_name> ADD TABLE [ONLY] table_name [(colname [, ..])]
Please find attached a patch that implements the above proposal.
While the patch contains basic implementation and tests, several
improvements
and sanity checks are underway. I will post an updated patch with those
changes soon.
Kindly let me know your opinion.
Thank you,
Rahila Syed
Attachments:
0001-Add-column-filtering-to-logical-replication.patchapplication/octet-stream; name=0001-Add-column-filtering-to-logical-replication.patchDownload
From a70ec52e079d628b3e72ec130f548fb9040b41b0 Mon Sep 17 00:00:00 2001
From: rahila <rahilasyed.90@gmail.com>
Date: Mon, 7 Jun 2021 16:27:21 +0530
Subject: [PATCH] Add column filtering to logical replication
Add capability to specifiy column names at while linking
the table to a publication at the time of CREATE or ALTER
publication. This will allows replicating only the specified
columns. Rest of the columns on the subscriber will be populated
locally. If column names are not specified all columns are
replicated. REPLICA IDENTITY columns are always replicated
irrespective of column names specification.
Add a tap test for the same in subscription folder.
---
src/backend/catalog/pg_publication.c | 20 +++--
src/backend/commands/publicationcmds.c | 32 +++++--
src/backend/parser/gram.y | 23 +++++-
src/backend/replication/logical/proto.c | 22 +++--
src/backend/replication/pgoutput/pgoutput.c | 87 ++++++++++++++++++--
src/include/catalog/pg_publication.h | 9 +-
src/include/catalog/pg_publication_rel.h | 4 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 6 ++
src/include/replication/logicalproto.h | 4 +-
src/test/subscription/t/021_column_filter.pl | 52 ++++++++++++
11 files changed, 224 insertions(+), 36 deletions(-)
create mode 100644 src/test/subscription/t/021_column_filter.pl
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 86e415af89..89b4edf5a4 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -141,18 +141,20 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
* Insert new publication / relation mapping.
*/
ObjectAddress
-publication_add_relation(Oid pubid, Relation targetrel,
+publication_add_relation(Oid pubid, PublicationRelationInfo *targetrel,
bool if_not_exists)
{
Relation rel;
HeapTuple tup;
Datum values[Natts_pg_publication_rel];
bool nulls[Natts_pg_publication_rel];
- Oid relid = RelationGetRelid(targetrel);
+ Oid relid = RelationGetRelid(targetrel->relation);
Oid prrelid;
Publication *pub = GetPublication(pubid);
ObjectAddress myself,
referenced;
+ ListCell *lc;
+ List *target_cols = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -172,10 +174,10 @@ publication_add_relation(Oid pubid, Relation targetrel,
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_OBJECT),
errmsg("relation \"%s\" is already member of publication \"%s\"",
- RelationGetRelationName(targetrel), pub->name)));
+ RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel);
+ check_publication_add_relation(targetrel->relation);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -188,6 +190,14 @@ publication_add_relation(Oid pubid, Relation targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ foreach(lc, targetrel->columns)
+ {
+ char *colname;
+ colname = strVal(lfirst(lc));
+ target_cols = lappend(target_cols, colname);
+ }
+ values[Anum_pg_publication_rel_prrel_attr - 1] =
+ PointerGetDatum(strlist_to_textarray(target_cols));
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -209,7 +219,7 @@ publication_add_relation(Oid pubid, Relation targetrel,
table_close(rel, RowExclusiveLock);
/* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcache(targetrel);
+ CacheInvalidateRelcache(targetrel->relation);
return myself;
}
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 95c253c8e0..17c2e041e9 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -515,10 +515,13 @@ OpenTableList(List *tables)
*/
foreach(lc, tables)
{
- RangeVar *rv = castNode(RangeVar, lfirst(lc));
+ PublicationTable *t = lfirst(lc);
+ RangeVar *rv = castNode(RangeVar, t->relation);
bool recurse = rv->inh;
Relation rel;
Oid myrelid;
+ PublicationRelationInfo *pub_rel;
+ ListCell *lc1;
/* Allow query cancel in case this takes a long time */
CHECK_FOR_INTERRUPTS();
@@ -539,7 +542,17 @@ OpenTableList(List *tables)
continue;
}
- rels = lappend(rels, rel);
+ pub_rel = palloc(sizeof(PublicationRelationInfo));
+ pub_rel->relation = rel;
+ pub_rel->relid = myrelid;
+ foreach(lc1, t->columns)
+ {
+ char *colname;
+ colname = strVal(lfirst(lc1));
+ elog(LOG, "String value %s\n", colname);
+ }
+ pub_rel->columns = t->columns;
+ rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
/*
@@ -572,7 +585,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
- rels = lappend(rels, rel);
+ pub_rel = palloc(sizeof(PublicationRelationInfo));
+ pub_rel->relation = rel;
+ pub_rel->relid = childrelid;
+ pub_rel->columns = t->columns;
+ rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
}
@@ -593,9 +610,9 @@ CloseTableList(List *rels)
foreach(lc, rels)
{
- Relation rel = (Relation) lfirst(lc);
+ PublicationRelationInfo *pub_rel = (PublicationRelationInfo *)lfirst(lc);
- table_close(rel, NoLock);
+ table_close(pub_rel->relation, NoLock);
}
}
@@ -612,7 +629,8 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
foreach(lc, rels)
{
- Relation rel = (Relation) lfirst(lc);
+ PublicationRelationInfo *pub_rel = (PublicationRelationInfo *)lfirst(lc);
+ Relation rel = pub_rel->relation;
ObjectAddress obj;
/* Must be owner of the table or superuser. */
@@ -620,7 +638,7 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(rel->rd_rel->relkind),
RelationGetRelationName(rel));
- obj = publication_add_relation(pubid, rel, if_not_exists);
+ obj = publication_add_relation(pubid, pub_rel, if_not_exists);
if (stmt)
{
EventTriggerCollectSimpleCommand(obj, InvalidObjectAddress,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index eb24195438..26e58a7264 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -426,14 +426,14 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
transform_element_list transform_type_list
TriggerTransitions TriggerReferencing
vacuum_relation_list opt_vacuum_relation_list
- drop_option_list
+ drop_option_list publication_table_list
%type <node> opt_routine_body
%type <groupclause> group_clause
%type <list> group_by_list
%type <node> group_by_item empty_grouping_set rollup_clause cube_clause
%type <node> grouping_sets_clause
-%type <node> opt_publication_for_tables publication_for_tables
+%type <node> opt_publication_for_tables publication_for_tables publication_table
%type <list> opt_fdw_options fdw_options
%type <defelt> fdw_option
@@ -9612,7 +9612,7 @@ opt_publication_for_tables:
;
publication_for_tables:
- FOR TABLE relation_expr_list
+ FOR TABLE publication_table_list
{
$$ = (Node *) $3;
}
@@ -9622,7 +9622,22 @@ publication_for_tables:
}
;
+publication_table_list:
+ publication_table
+ { $$ = list_make1($1); }
+ | publication_table_list ',' publication_table
+ { $$ = lappend($1, $3); }
+ ;
+publication_table: relation_expr opt_column_list
+ {
+ PublicationTable *n = makeNode(PublicationTable);
+ n->relation = $1;
+ n->columns = $2;
+ $$ = (Node *) n;
+ }
+ ;
+
/*****************************************************************************
*
* ALTER PUBLICATION name SET ( options )
@@ -9643,7 +9658,7 @@ AlterPublicationStmt:
n->options = $5;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name ADD_P TABLE relation_expr_list
+ | ALTER PUBLICATION name ADD_P TABLE publication_table_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 1cf59e0fb0..d783d8e7c3 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -31,7 +31,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary, Bitmapset *att_list);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -140,7 +140,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *att_list)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -152,7 +152,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_list);
}
/*
@@ -184,7 +184,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary, Bitmapset *att_list)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -205,11 +205,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, att_list);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_list);
}
/*
@@ -278,7 +278,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -491,7 +491,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary, Bitmapset *att_list)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -542,6 +542,12 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
continue;
}
+ if (att_list != NULL && !(bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, att_list)))
+ {
+ pq_sendbyte(out, LOGICALREP_COLUMN_UNCHANGED);
+ continue;
+ }
+
typtup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(att->atttypid));
if (!HeapTupleIsValid(typtup))
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abd5217ab1..1b6231dcf8 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,12 +15,14 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
@@ -70,6 +72,8 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx);
+static Bitmapset *get_tuple_columns_map(TupleDesc tuple_desc, List *columns);
+static int get_att_num_by_name(TupleDesc desc, const char *attname);
/*
* Entry in the map used to remember which relation schemas we sent.
@@ -115,6 +119,7 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+ List *columns;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -534,6 +539,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
RelationSyncEntry *relentry;
TransactionId xid = InvalidTransactionId;
Relation ancestor = NULL;
+ Bitmapset *att_list;
if (!is_publishable_relation(relation))
return;
@@ -579,6 +585,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INSERT:
{
HeapTuple tuple = &change->data.tp.newtuple->tuple;
+ TupleDesc tuple_desc;
/* Switch relation if publishing via root. */
if (relentry->publish_as_relid != RelationGetRelid(relation))
@@ -590,10 +597,11 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (relentry->map)
tuple = execute_attr_map_tuple(tuple, relentry->map);
}
-
+ tuple_desc = RelationGetDescr(relation);
+ att_list = get_tuple_columns_map(tuple_desc, relentry->columns);
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, att_list);
OutputPluginWrite(ctx, true);
break;
}
@@ -602,6 +610,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
HeapTuple oldtuple = change->data.tp.oldtuple ?
&change->data.tp.oldtuple->tuple : NULL;
HeapTuple newtuple = &change->data.tp.newtuple->tuple;
+ TupleDesc tuple_desc;
/* Switch relation if publishing via root. */
if (relentry->publish_as_relid != RelationGetRelid(relation))
@@ -619,10 +628,12 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
relentry->map);
}
}
+ tuple_desc = RelationGetDescr(relation);
+ att_list = get_tuple_columns_map(tuple_desc, relentry->columns);
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, att_list);
OutputPluginWrite(ctx, true);
break;
}
@@ -1031,11 +1042,11 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
- entry->map = NULL; /* will be set by maybe_send_schema() if
- * needed */
+ entry->columns = NIL;
+ entry->map = NULL; /* will be set by maybe_send_schema() if needed */
}
- /* Validate the entry */
+ /* Validate thel entry */
if (!entry->replicate_valid)
{
List *pubids = GetRelationPublications(relid);
@@ -1116,15 +1127,36 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ int nelems, i;
+ bool isnull;
+ Datum *elems;
+ HeapTuple pub_rel_tuple;
+ Datum pub_rel_cols;
+
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP, ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP, pub_rel_tuple, Anum_pg_publication_rel_prrel_attr, &isnull);
+ if (!isnull)
+ {
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ deconstruct_array(DatumGetArrayTypePCopy(pub_rel_cols),
+ TEXTOID, -1, false, 'i',
+ &elems, NULL, &nelems);
+ for (i = 0; i < nelems; i++)
+ entry->columns = lappend(entry->columns, TextDatumGetCString(elems[i]));
+ MemoryContextSwitchTo(oldctx);
+ }
+ ReleaseSysCache(pub_rel_tuple);
+ }
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
+
}
- if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
- break;
}
list_free(pubids);
@@ -1136,6 +1168,41 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
return entry;
}
+static Bitmapset *
+get_tuple_columns_map(TupleDesc tuple_desc, List *columns)
+{
+ ListCell *cell;
+ Bitmapset *att_list = NULL;
+ foreach(cell, columns)
+ {
+ const char *attname = lfirst(cell);
+ int attnum = get_att_num_by_name(tuple_desc, attname);
+
+ if (!bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, att_list))
+ att_list = bms_add_member(att_list,
+ attnum - FirstLowInvalidHeapAttributeNumber);
+
+ }
+ return att_list;
+}
+
+static int
+get_att_num_by_name(TupleDesc desc, const char *attname)
+{
+ int i;
+
+ for (i = 0; i < desc->natts; i++)
+ {
+ if (TupleDescAttr(desc, i)->attisdropped)
+ continue;
+
+ if (namestrcmp(&(TupleDescAttr(desc, i)->attname), attname) == 0)
+ return TupleDescAttr(desc, i)->attnum;
+ }
+
+ return FirstLowInvalidHeapAttributeNumber;
+}
+
/*
* Cleanup list of streamed transactions and update the schema_sent flag.
*
@@ -1220,6 +1287,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ list_free(entry->columns);
+ entry->columns = NIL;
if (entry->map)
{
/*
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index f332bad4d4..7bdc9bb9b8 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -83,6 +83,13 @@ typedef struct Publication
PublicationActions pubactions;
} Publication;
+typedef struct PublicationRelationInfo
+{
+ Oid relid;
+ Relation relation;
+ List *columns;
+} PublicationRelationInfo;
+
extern Publication *GetPublication(Oid pubid);
extern Publication *GetPublicationByName(const char *pubname, bool missing_ok);
extern List *GetRelationPublications(Oid relid);
@@ -108,7 +115,7 @@ extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern bool is_publishable_relation(Relation rel);
-extern ObjectAddress publication_add_relation(Oid pubid, Relation targetrel,
+extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelationInfo *targetrel,
bool if_not_exists);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index b5d5504cbb..d3ef8fdb18 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN
+ text prrel_attr[1]; /* Variable length field starts here */
+#endif
} FormData_pg_publication_rel;
/* ----------------
@@ -40,6 +43,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
*/
typedef FormData_pg_publication_rel *Form_pg_publication_rel;
+DECLARE_TOAST(pg_publication_rel, 8895, 8896);
DECLARE_UNIQUE_INDEX_PKEY(pg_publication_rel_oid_index, 6112, PublicationRelObjectIndexId, on pg_publication_rel using btree(oid oid_ops));
DECLARE_UNIQUE_INDEX(pg_publication_rel_prrelid_prpubid_index, 6113, PublicationRelPrrelidPrpubidIndexId, on pg_publication_rel using btree(prrelid oid_ops, prpubid oid_ops));
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index d9e417bcd7..2037705f45 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -491,6 +491,7 @@ typedef enum NodeTag
T_PartitionRangeDatum,
T_PartitionCmd,
T_VacuumRelation,
+ T_PublicationTable,
/*
* TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index def9651b34..a17c1aa9f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3623,6 +3623,12 @@ typedef struct AlterTSConfigurationStmt
bool missing_ok; /* for DROP - skip error if missing? */
} AlterTSConfigurationStmt;
+typedef struct PublicationTable
+{
+ NodeTag type;
+ RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
+} PublicationTable;
typedef struct CreatePublicationStmt
{
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 55b90c03ea..879c58c497 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -134,11 +134,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *att_list);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *att_list);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
diff --git a/src/test/subscription/t/021_column_filter.pl b/src/test/subscription/t/021_column_filter.pl
new file mode 100644
index 0000000000..496f5e35e2
--- /dev/null
+++ b/src/test/subscription/t/021_column_filter.pl
@@ -0,0 +1,52 @@
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test TRUNCATE
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+# setup
+
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, b int, c int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, b int, c int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub1 FOR TABLE tab1(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+);
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab1 VALUES (1,2,3)");
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'column c not replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab1 SET c = 5 where a = 1");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'column c not replicated');
--
2.17.2 (Apple Git-113)
On Thu, Jul 1, 2021 at 1:06 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
Hi,
Filtering of columns at the publisher node will allow for selective replication of data between publisher and subscriber. In case the updates on the publisher are targeted only towards specific columns, the user will have an option to reduce network consumption by not sending the data corresponding to new columns that do not change. Note that replica identity values will always be sent irrespective of column filtering settings. The column values that are not sent by the publisher will be populated using local values on the subscriber. For insert command, non-replicated column values will be NULL or the default.
If column names are not specified while creating or altering a publication,
all the columns are replicated as per current behaviour.The proposal for syntax to add table with column names to publication is as follows:
Create publication:CREATE PUBLICATION <pub_name> [ FOR TABLE [ONLY] table_name [(colname [,…])] | FOR ALL TABLES]
Alter publication:
ALTER PUBLICATION <pub_name> ADD TABLE [ONLY] table_name [(colname [, ..])]
Please find attached a patch that implements the above proposal.
While the patch contains basic implementation and tests, several improvements
and sanity checks are underway. I will post an updated patch with those changes soon.Kindly let me know your opinion.
I haven't looked into the patch yet but +1 for the idea.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Thu, Jul 1, 2021 at 1:06 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
Hi,
Filtering of columns at the publisher node will allow for selective replication of data between publisher and subscriber. In case the updates on the publisher are targeted only towards specific columns, the user will have an option to reduce network consumption by not sending the data corresponding to new columns that do not change. Note that replica identity values will always be sent irrespective of column filtering settings. The column values that are not sent by the publisher will be populated using local values on the subscriber. For insert command, non-replicated column values will be NULL or the default.
If column names are not specified while creating or altering a publication,
all the columns are replicated as per current behaviour.The proposal for syntax to add table with column names to publication is as follows:
Create publication:CREATE PUBLICATION <pub_name> [ FOR TABLE [ONLY] table_name [(colname [,…])] | FOR ALL TABLES]
Alter publication:
ALTER PUBLICATION <pub_name> ADD TABLE [ONLY] table_name [(colname [, ..])]
Please find attached a patch that implements the above proposal.
While the patch contains basic implementation and tests, several improvements
and sanity checks are underway. I will post an updated patch with those changes soon.Kindly let me know your opinion.
This idea gives more flexibility to the user, +1 for the feature.
Regards,
Vignesh
Hello, here are a few comments on this patch.
The patch adds a function get_att_num_by_name; but we have a lsyscache.c
function for that purpose, get_attnum. Maybe that one should be used
instead?
get_tuple_columns_map() returns a bitmapset of the attnos of the columns
in the given list, so its name feels wrong. I propose
get_table_columnset(). However, this function is invoked for every
insert/update change, so it's going to be far too slow to be usable. I
think you need to cache the bitmapset somewhere, so that the function is
only called on first use. I didn't look very closely, but it seems that
struct RelationSyncEntry may be a good place to cache it.
The patch adds a new parse node PublicationTable, but doesn't add
copyfuncs.c, equalfuncs.c, readfuncs.c, outfuncs.c support for it.
Maybe try a compile with WRITE_READ_PARSE_PLAN_TREES and/or
COPY_PARSE_PLAN_TREES enabled to make sure everything is covered.
(I didn't verify that this actually catches anything ...)
The new column in pg_publication_rel is prrel_attr. This name seems at
odds with existing column names (we don't use underscores in catalog
column names). Maybe prattrs is good enough? prrelattrs? We tend to
use plurals for columns that are arrays.
It's not super clear to me that strlist_to_textarray() and related
processing will behave sanely when the column names contain weird
characters such as commas or quotes, or just when used with uppercase
column names. Maybe it's worth having tests that try to break such
cases.
You seem to have left a debugging "elog(LOG)" line in OpenTableList.
I got warnings from "git am" about trailing whitespace being added by
the patch in two places.
Thanks!
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
Hi, I was wondering if/when a subset of cols is specified then does
that mean it will be possible for the table to be replicated to a
*smaller* table at the subscriber side?
e.g Can a table with 7 cols replicated to a table with 2 cols?
table tab1(a,b,c,d,e,f,g) --> CREATE PUBLICATION pub1 FOR TABLE
tab1(a,b) --> table tab1(a,b)
~~
I thought maybe that should be possible, but the expected behaviour
for that scenario was not very clear to me from the thread/patch
comments. And the new TAP test uses the tab1 table created exactly the
same for pub/sub, so I couldn't tell from the test code either.
------
Kind Regards,
Peter Smith.
Fujitsu Australia
Hi Peter,
Hi, I was wondering if/when a subset of cols is specified then does
that mean it will be possible for the table to be replicated to a
*smaller* table at the subscriber side?
e.g Can a table with 7 cols replicated to a table with 2 cols?
table tab1(a,b,c,d,e,f,g) --> CREATE PUBLICATION pub1 FOR TABLE
tab1(a,b) --> table tab1(a,b)~~
I thought maybe that should be possible, but the expected behaviour
for that scenario was not very clear to me from the thread/patch
comments. And the new TAP test uses the tab1 table created exactly the
same for pub/sub, so I couldn't tell from the test code either.
Currently, this capability is not included in the patch. If the table on
the subscriber
server has lesser attributes than that on the publisher server, it throws
an error at the
time of CREATE SUBSCRIPTION.
About having such a functionality, I don't immediately see any issue with
it as long
as we make sure replica identity columns are always present on both
instances.
However, need to carefully consider situations in which a server subscribes
to multiple
publications, each publishing a different subset of columns of a table.
Thank you,
Rahila Syed
Hi Alvaro,
Thank you for comments.
The patch adds a function get_att_num_by_name; but we have a lsyscache.c
function for that purpose, get_attnum. Maybe that one should be used
instead?Thank you for pointing that out, I agree it makes sense to reuse the
existing function.
Changed it accordingly in the attached patch.
get_tuple_columns_map() returns a bitmapset of the attnos of the columns
in the given list, so its name feels wrong. I propose
get_table_columnset(). However, this function is invoked for every
insert/update change, so it's going to be far too slow to be usable. I
think you need to cache the bitmapset somewhere, so that the function is
only called on first use. I didn't look very closely, but it seems that
struct RelationSyncEntry may be a good place to cache it.Makes sense, changed accordingly.
The patch adds a new parse node PublicationTable, but doesn't add
copyfuncs.c, equalfuncs.c, readfuncs.c, outfuncs.c support for it.
Maybe try a compile with WRITE_READ_PARSE_PLAN_TREES and/or
COPY_PARSE_PLAN_TREES enabled to make sure everything is covered.
(I didn't verify that this actually catches anything ...)
I will test this and include these changes in the next version.
The new column in pg_publication_rel is prrel_attr. This name seems at
odds with existing column names (we don't use underscores in catalog
column names). Maybe prattrs is good enough? prrelattrs? We tend to
use plurals for columns that are arrays.Renamed it to prattrs as per suggestion.
It's not super clear to me that strlist_to_textarray() and related
processing will behave sanely when the column names contain weird
characters such as commas or quotes, or just when used with uppercase
column names. Maybe it's worth having tests that try to break such
cases.Sure, I will include these tests in the next version.
You seem to have left a debugging "elog(LOG)" line in OpenTableList.
Removed.
I got warnings from "git am" about trailing whitespace being added by
the patch in two places.Should be fixed now.
Thank you,
Rahila Syed
Attachments:
v1-0001-Add-column-filtering-to-logical-replication.patchapplication/octet-stream; name=v1-0001-Add-column-filtering-to-logical-replication.patchDownload
From 482dcd54aa6f2d46c96036ce325e75a4750d9e7a Mon Sep 17 00:00:00 2001
From: rahila <rahilasyed.90@gmail.com>
Date: Mon, 7 Jun 2021 16:27:21 +0530
Subject: [PATCH] Add column filtering to logical replication
Add capability to specifiy column names at while linking
the table to a publication at the time of CREATE or ALTER
publication. This will allows replicating only the specified
columns. Rest of the columns on the subscriber will be populated
locally. If column names are not specified all columns are
replicated. REPLICA IDENTITY columns are always replicated
irrespective of column names specification.
Add a tap test for the same in subscription folder.
---
src/backend/catalog/pg_publication.c | 20 ++++--
src/backend/commands/publicationcmds.c | 25 +++++--
src/backend/parser/gram.y | 23 ++++--
src/backend/replication/logical/proto.c | 22 +++---
src/backend/replication/pgoutput/pgoutput.c | 62 +++++++++++++---
src/include/catalog/pg_publication.h | 9 ++-
src/include/catalog/pg_publication_rel.h | 4 ++
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 6 ++
src/include/replication/logicalproto.h | 4 +-
src/test/subscription/t/021_column_filter.pl | 76 ++++++++++++++++++++
11 files changed, 216 insertions(+), 36 deletions(-)
create mode 100644 src/test/subscription/t/021_column_filter.pl
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 86e415af89..0948998f5e 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -141,18 +141,20 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
* Insert new publication / relation mapping.
*/
ObjectAddress
-publication_add_relation(Oid pubid, Relation targetrel,
+publication_add_relation(Oid pubid, PublicationRelationInfo *targetrel,
bool if_not_exists)
{
Relation rel;
HeapTuple tup;
Datum values[Natts_pg_publication_rel];
bool nulls[Natts_pg_publication_rel];
- Oid relid = RelationGetRelid(targetrel);
+ Oid relid = RelationGetRelid(targetrel->relation);
Oid prrelid;
Publication *pub = GetPublication(pubid);
ObjectAddress myself,
referenced;
+ ListCell *lc;
+ List *target_cols = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -172,10 +174,10 @@ publication_add_relation(Oid pubid, Relation targetrel,
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_OBJECT),
errmsg("relation \"%s\" is already member of publication \"%s\"",
- RelationGetRelationName(targetrel), pub->name)));
+ RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel);
+ check_publication_add_relation(targetrel->relation);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -188,6 +190,14 @@ publication_add_relation(Oid pubid, Relation targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ foreach(lc, targetrel->columns)
+ {
+ char *colname;
+ colname = strVal(lfirst(lc));
+ target_cols = lappend(target_cols, colname);
+ }
+ values[Anum_pg_publication_rel_prattrs - 1] =
+ PointerGetDatum(strlist_to_textarray(target_cols));
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -209,7 +219,7 @@ publication_add_relation(Oid pubid, Relation targetrel,
table_close(rel, RowExclusiveLock);
/* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcache(targetrel);
+ CacheInvalidateRelcache(targetrel->relation);
return myself;
}
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 95c253c8e0..c8abdbe1d6 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -515,10 +515,12 @@ OpenTableList(List *tables)
*/
foreach(lc, tables)
{
- RangeVar *rv = castNode(RangeVar, lfirst(lc));
+ PublicationTable *t = lfirst(lc);
+ RangeVar *rv = castNode(RangeVar, t->relation);
bool recurse = rv->inh;
Relation rel;
Oid myrelid;
+ PublicationRelationInfo *pub_rel;
/* Allow query cancel in case this takes a long time */
CHECK_FOR_INTERRUPTS();
@@ -539,7 +541,11 @@ OpenTableList(List *tables)
continue;
}
- rels = lappend(rels, rel);
+ pub_rel = palloc(sizeof(PublicationRelationInfo));
+ pub_rel->relation = rel;
+ pub_rel->relid = myrelid;
+ pub_rel->columns = t->columns;
+ rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
/*
@@ -572,7 +578,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
- rels = lappend(rels, rel);
+ pub_rel = palloc(sizeof(PublicationRelationInfo));
+ pub_rel->relation = rel;
+ pub_rel->relid = childrelid;
+ pub_rel->columns = t->columns;
+ rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
}
@@ -593,9 +603,9 @@ CloseTableList(List *rels)
foreach(lc, rels)
{
- Relation rel = (Relation) lfirst(lc);
+ PublicationRelationInfo *pub_rel = (PublicationRelationInfo *)lfirst(lc);
- table_close(rel, NoLock);
+ table_close(pub_rel->relation, NoLock);
}
}
@@ -612,7 +622,8 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
foreach(lc, rels)
{
- Relation rel = (Relation) lfirst(lc);
+ PublicationRelationInfo *pub_rel = (PublicationRelationInfo *)lfirst(lc);
+ Relation rel = pub_rel->relation;
ObjectAddress obj;
/* Must be owner of the table or superuser. */
@@ -620,7 +631,7 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(rel->rd_rel->relkind),
RelationGetRelationName(rel));
- obj = publication_add_relation(pubid, rel, if_not_exists);
+ obj = publication_add_relation(pubid, pub_rel, if_not_exists);
if (stmt)
{
EventTriggerCollectSimpleCommand(obj, InvalidObjectAddress,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index eb24195438..3615ef4a46 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -426,14 +426,14 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
transform_element_list transform_type_list
TriggerTransitions TriggerReferencing
vacuum_relation_list opt_vacuum_relation_list
- drop_option_list
+ drop_option_list publication_table_list
%type <node> opt_routine_body
%type <groupclause> group_clause
%type <list> group_by_list
%type <node> group_by_item empty_grouping_set rollup_clause cube_clause
%type <node> grouping_sets_clause
-%type <node> opt_publication_for_tables publication_for_tables
+%type <node> opt_publication_for_tables publication_for_tables publication_table
%type <list> opt_fdw_options fdw_options
%type <defelt> fdw_option
@@ -9612,7 +9612,7 @@ opt_publication_for_tables:
;
publication_for_tables:
- FOR TABLE relation_expr_list
+ FOR TABLE publication_table_list
{
$$ = (Node *) $3;
}
@@ -9622,6 +9622,21 @@ publication_for_tables:
}
;
+publication_table_list:
+ publication_table
+ { $$ = list_make1($1); }
+ | publication_table_list ',' publication_table
+ { $$ = lappend($1, $3); }
+ ;
+
+publication_table: relation_expr opt_column_list
+ {
+ PublicationTable *n = makeNode(PublicationTable);
+ n->relation = $1;
+ n->columns = $2;
+ $$ = (Node *) n;
+ }
+ ;
/*****************************************************************************
*
@@ -9643,7 +9658,7 @@ AlterPublicationStmt:
n->options = $5;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name ADD_P TABLE relation_expr_list
+ | ALTER PUBLICATION name ADD_P TABLE publication_table_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 1cf59e0fb0..d783d8e7c3 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -31,7 +31,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary, Bitmapset *att_list);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -140,7 +140,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *att_list)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -152,7 +152,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_list);
}
/*
@@ -184,7 +184,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary, Bitmapset *att_list)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -205,11 +205,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, att_list);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_list);
}
/*
@@ -278,7 +278,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -491,7 +491,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary, Bitmapset *att_list)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -542,6 +542,12 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
continue;
}
+ if (att_list != NULL && !(bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, att_list)))
+ {
+ pq_sendbyte(out, LOGICALREP_COLUMN_UNCHANGED);
+ continue;
+ }
+
typtup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(att->atttypid));
if (!HeapTupleIsValid(typtup))
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abd5217ab1..a04e307f4d 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,12 +15,14 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
@@ -70,6 +72,7 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx);
+static Bitmapset* get_table_columnset(Oid relid, List *columns, Bitmapset *att_list);
/*
* Entry in the map used to remember which relation schemas we sent.
@@ -115,6 +118,7 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+ Bitmapset *att_list;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -590,10 +594,9 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (relentry->map)
tuple = execute_attr_map_tuple(tuple, relentry->map);
}
-
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->att_list);
OutputPluginWrite(ctx, true);
break;
}
@@ -619,10 +622,9 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
relentry->map);
}
}
-
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->att_list);
OutputPluginWrite(ctx, true);
break;
}
@@ -1031,8 +1033,8 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
- entry->map = NULL; /* will be set by maybe_send_schema() if
- * needed */
+ entry->att_list = NULL;
+ entry->map = NULL; /* will be set by maybe_send_schema() if needed */
}
/* Validate the entry */
@@ -1116,15 +1118,38 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ int nelems, i;
+ bool isnull;
+ Datum *elems;
+ HeapTuple pub_rel_tuple;
+ Datum pub_rel_cols;
+ List *columns = NIL;
+
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP, ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP, pub_rel_tuple, Anum_pg_publication_rel_prattrs, &isnull);
+ if (!isnull)
+ {
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ deconstruct_array(DatumGetArrayTypePCopy(pub_rel_cols),
+ TEXTOID, -1, false, 'i',
+ &elems, NULL, &nelems);
+ for (i = 0; i < nelems; i++)
+ columns = lappend(columns, TextDatumGetCString(elems[i]));
+ entry->att_list = get_table_columnset(publish_as_relid, columns, entry->att_list);
+ MemoryContextSwitchTo(oldctx);
+ }
+ ReleaseSysCache(pub_rel_tuple);
+ }
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
+
}
- if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
- break;
}
list_free(pubids);
@@ -1136,6 +1161,23 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
return entry;
}
+static Bitmapset*
+get_table_columnset(Oid relid, List *columns, Bitmapset *att_list)
+{
+ ListCell *cell;
+ foreach(cell, columns)
+ {
+ const char *attname = lfirst(cell);
+ int attnum = get_attnum(relid, attname);
+
+ if (!bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, att_list))
+ att_list = bms_add_member(att_list,
+ attnum - FirstLowInvalidHeapAttributeNumber);
+
+ }
+ return att_list;
+}
+
/*
* Cleanup list of streamed transactions and update the schema_sent flag.
*
@@ -1220,6 +1262,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->att_list);
+ entry->att_list = NULL;
if (entry->map)
{
/*
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index f332bad4d4..7bdc9bb9b8 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -83,6 +83,13 @@ typedef struct Publication
PublicationActions pubactions;
} Publication;
+typedef struct PublicationRelationInfo
+{
+ Oid relid;
+ Relation relation;
+ List *columns;
+} PublicationRelationInfo;
+
extern Publication *GetPublication(Oid pubid);
extern Publication *GetPublicationByName(const char *pubname, bool missing_ok);
extern List *GetRelationPublications(Oid relid);
@@ -108,7 +115,7 @@ extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern bool is_publishable_relation(Relation rel);
-extern ObjectAddress publication_add_relation(Oid pubid, Relation targetrel,
+extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelationInfo *targetrel,
bool if_not_exists);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index b5d5504cbb..d1d4eec2c0 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN
+ text prattrs[1]; /* Variable length field starts here */
+#endif
} FormData_pg_publication_rel;
/* ----------------
@@ -40,6 +43,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
*/
typedef FormData_pg_publication_rel *Form_pg_publication_rel;
+DECLARE_TOAST(pg_publication_rel, 8895, 8896);
DECLARE_UNIQUE_INDEX_PKEY(pg_publication_rel_oid_index, 6112, PublicationRelObjectIndexId, on pg_publication_rel using btree(oid oid_ops));
DECLARE_UNIQUE_INDEX(pg_publication_rel_prrelid_prpubid_index, 6113, PublicationRelPrrelidPrpubidIndexId, on pg_publication_rel using btree(prrelid oid_ops, prpubid oid_ops));
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index d9e417bcd7..2037705f45 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -491,6 +491,7 @@ typedef enum NodeTag
T_PartitionRangeDatum,
T_PartitionCmd,
T_VacuumRelation,
+ T_PublicationTable,
/*
* TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index def9651b34..a17c1aa9f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3623,6 +3623,12 @@ typedef struct AlterTSConfigurationStmt
bool missing_ok; /* for DROP - skip error if missing? */
} AlterTSConfigurationStmt;
+typedef struct PublicationTable
+{
+ NodeTag type;
+ RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
+} PublicationTable;
typedef struct CreatePublicationStmt
{
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 55b90c03ea..879c58c497 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -134,11 +134,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *att_list);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *att_list);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
diff --git a/src/test/subscription/t/021_column_filter.pl b/src/test/subscription/t/021_column_filter.pl
new file mode 100644
index 0000000000..b5c73bfc7d
--- /dev/null
+++ b/src/test/subscription/t/021_column_filter.pl
@@ -0,0 +1,76 @@
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test TRUNCATE
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+# setup
+
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, b int, c int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, b int, c int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int)");
+
+#Test create publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub1 FOR TABLE tab1(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+);
+#Initial sync
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab1 VALUES (1,2,3)");
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'insert on column c is not replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab1 SET c = 5 where a = 1");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'update on column c is not replicated');
+
+#Test alter publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION"
+);
+
+$node_publisher->wait_for_catchup('sub1');
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 VALUES (1,'abc',3)");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2");
+is($result, qq(1|abc|), 'insert on column c is not replicated');
--
2.17.2 (Apple Git-113)
On 7/12/21 10:32 AM, Rahila Syed wrote:
Hi Peter,
Hi, I was wondering if/when a subset of cols is specified then does
that mean it will be possible for the table to be replicated to a
*smaller* table at the subscriber side?e.g Can a table with 7 cols replicated to a table with 2 cols?
table tab1(a,b,c,d,e,f,g) --> CREATE PUBLICATION pub1 FOR TABLE
tab1(a,b) --> table tab1(a,b)~~
I thought maybe that should be possible, but the expected behaviour
for that scenario was not very clear to me from the thread/patch
comments. And the new TAP test uses the tab1 table created exactly the
same for pub/sub, so I couldn't tell from the test code either.
Currently, this capability is not included in the patch. If the table on
the subscriber
server has lesser attributes than that on the publisher server, it
throws an error at the
time of CREATE SUBSCRIPTION.
That's a bit surprising, to be honest. I do understand the patch simply
treats the filtered columns as "unchanged" because that's the simplest
way to filter the *data* of the columns. But if someone told me we can
"filter columns" I'd expect this to work without the columns on the
subscriber.
About having such a functionality, I don't immediately see any issue
with it as long
as we make sure replica identity columns are always present on both
instances.
Yeah, that seems like an inherent requirement.
However, need to carefully consider situations in which a server
subscribes to multiple
publications, each publishing a different subset of columns of a table.
Isn't that pretty much the same situation as for multiple subscriptions
each with a different set of I/U/D operations? IIRC we simply merge
those, so why not to do the same thing here and merge the attributes?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 7/12/21 11:38 AM, Rahila Syed wrote:
Hi Alvaro,
Thank you for comments.
The patch adds a function get_att_num_by_name; but we have a lsyscache.c
function for that purpose, get_attnum. Maybe that one should be used
instead?Thank you for pointing that out, I agree it makes sense to reuse the
existing function.
Changed it accordingly in the attached patch.
get_tuple_columns_map() returns a bitmapset of the attnos of the columns
in the given list, so its name feels wrong. I propose
get_table_columnset(). However, this function is invoked for every
insert/update change, so it's going to be far too slow to be usable. I
think you need to cache the bitmapset somewhere, so that the function is
only called on first use. I didn't look very closely, but it seems that
struct RelationSyncEntry may be a good place to cache it.Makes sense, changed accordingly.
To nitpick, I find "Bitmapset *att_list" a bit annoying, because it's
not really a list ;-)
FWIW "make check" fails for me with this version, due to segfault in
OpenTableLists. Apparenly there's some confusion - the code expects the
list to contain PublicationTable nodes, and tries to extract the
RangeVar from the elements. But the list actually contains RangeVar, so
this crashes and burns. See the attached backtrace.
I'd bet this is because the patch uses list of RangeVar in some cases
and list of PublicationTable in some cases, similarly to the "row
filtering" patch nearby. IMHO this is just confusing and we should
always pass list of PublicationTable nodes.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
On 2021-Jul-12, Tomas Vondra wrote:
FWIW "make check" fails for me with this version, due to segfault in
OpenTableLists. Apparenly there's some confusion - the code expects the
list to contain PublicationTable nodes, and tries to extract the
RangeVar from the elements. But the list actually contains RangeVar, so
this crashes and burns. See the attached backtrace.I'd bet this is because the patch uses list of RangeVar in some cases
and list of PublicationTable in some cases, similarly to the "row
filtering" patch nearby. IMHO this is just confusing and we should
always pass list of PublicationTable nodes.
+1 don't make the code guess what type of list it is. Changing all the
uses of that node to deal with PublicationTable seems best.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Cuando no hay humildad las personas se degradan" (A. Christie)
Hi Tomas,
Thank you for your comments.
Currently, this capability is not included in the patch. If the table on
the subscriber
server has lesser attributes than that on the publisher server, it
throws an error at the
time of CREATE SUBSCRIPTION.That's a bit surprising, to be honest. I do understand the patch simply
treats the filtered columns as "unchanged" because that's the simplest
way to filter the *data* of the columns. But if someone told me we can
"filter columns" I'd expect this to work without the columns on the
subscriber.OK, I will look into adding this.
However, need to carefully consider situations in which a server
subscribes to multiple
publications, each publishing a different subset of columns of a table.
Isn't that pretty much the same situation as for multiple subscriptions
each with a different set of I/U/D operations? IIRC we simply merge
those, so why not to do the same thing here and merge the attributes?
Yeah, I agree with the solution to merge the attributes, similar to how
operations are merged. My concern was also from an implementation point
of view, will it be a very drastic change. I now had a look at how remote
relation
attributes are acquired for comparison with local attributes at the
subscriber.
It seems that the publisher will need to send the information about the
filtered columns
for each publication specified during CREATE SUBSCRIPTION.
This will be read at the subscriber side which in turn updates its cache
accordingly.
Currently, the subscriber expects all attributes of a published relation to
be present.
I will add code for this in the next version of the patch.
To nitpick, I find "Bitmapset *att_list" a bit annoying, because it's
not really a list ;-)
I will make this change with the next version
FWIW "make check" fails for me with this version, due to segfault in
OpenTableLists. Apparenly there's some confusion - the code expects the
list to contain PublicationTable nodes, and tries to extract the
RangeVar from the elements. But the list actually contains RangeVar, so
this crashes and burns. See the attached backtrace.
Thank you for the report, This is fixed in the attached version, now all
publication
function calls accept the PublicationTableInfo list.
Thank you,
Rahila Syed
Attachments:
v2-0001-Add-column-filtering-to-logical-replication.patchapplication/octet-stream; name=v2-0001-Add-column-filtering-to-logical-replication.patchDownload
From 93c08736f33a8b25484aec9abdf8c7775fb86486 Mon Sep 17 00:00:00 2001
From: rahila <rahilasyed.90@gmail.com>
Date: Mon, 7 Jun 2021 16:27:21 +0530
Subject: [PATCH] Add column filtering to logical replication
Add capability to specifiy column names at while linking
the table to a publication at the time of CREATE or ALTER
publication. This will allows replicating only the specified
columns. Rest of the columns on the subscriber will be populated
locally. If column names are not specified all columns are
replicated. REPLICA IDENTITY columns are always replicated
irrespective of column names specification.
Add a tap test for the same in subscription folder.
---
src/backend/catalog/pg_publication.c | 20 ++++--
src/backend/commands/publicationcmds.c | 46 +++++++++---
src/backend/parser/gram.y | 27 +++++--
src/backend/replication/logical/proto.c | 22 +++---
src/backend/replication/pgoutput/pgoutput.c | 62 +++++++++++++---
src/include/catalog/pg_publication.h | 9 ++-
src/include/catalog/pg_publication_rel.h | 4 ++
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 6 ++
src/include/replication/logicalproto.h | 4 +-
src/test/subscription/t/021_column_filter.pl | 76 ++++++++++++++++++++
11 files changed, 235 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/021_column_filter.pl
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 86e415af89..0948998f5e 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -141,18 +141,20 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
* Insert new publication / relation mapping.
*/
ObjectAddress
-publication_add_relation(Oid pubid, Relation targetrel,
+publication_add_relation(Oid pubid, PublicationRelationInfo *targetrel,
bool if_not_exists)
{
Relation rel;
HeapTuple tup;
Datum values[Natts_pg_publication_rel];
bool nulls[Natts_pg_publication_rel];
- Oid relid = RelationGetRelid(targetrel);
+ Oid relid = RelationGetRelid(targetrel->relation);
Oid prrelid;
Publication *pub = GetPublication(pubid);
ObjectAddress myself,
referenced;
+ ListCell *lc;
+ List *target_cols = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -172,10 +174,10 @@ publication_add_relation(Oid pubid, Relation targetrel,
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_OBJECT),
errmsg("relation \"%s\" is already member of publication \"%s\"",
- RelationGetRelationName(targetrel), pub->name)));
+ RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel);
+ check_publication_add_relation(targetrel->relation);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -188,6 +190,14 @@ publication_add_relation(Oid pubid, Relation targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ foreach(lc, targetrel->columns)
+ {
+ char *colname;
+ colname = strVal(lfirst(lc));
+ target_cols = lappend(target_cols, colname);
+ }
+ values[Anum_pg_publication_rel_prattrs - 1] =
+ PointerGetDatum(strlist_to_textarray(target_cols));
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -209,7 +219,7 @@ publication_add_relation(Oid pubid, Relation targetrel,
table_close(rel, RowExclusiveLock);
/* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcache(targetrel);
+ CacheInvalidateRelcache(targetrel->relation);
return myself;
}
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 95c253c8e0..7388143258 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -394,7 +394,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
foreach(newlc, rels)
{
- Relation newrel = (Relation) lfirst(newlc);
+ PublicationRelationInfo *newpubrel = (PublicationRelationInfo *) lfirst(newlc);
+ Relation newrel = newpubrel->relation;
if (RelationGetRelid(newrel) == oldrelid)
{
@@ -407,8 +408,19 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
{
Relation oldrel = table_open(oldrelid,
ShareUpdateExclusiveLock);
-
- delrels = lappend(delrels, oldrel);
+ /*
+ * Copy relation info into PublicationRelationInfo as
+ * PublicationDropTables accepts a list of PublicationRelationInfo
+ */
+ PublicationRelationInfo *pubrel = palloc(sizeof(PublicationRelationInfo));
+ pubrel->relation = oldrel;
+ pubrel->relid = oldrelid;
+ /*
+ * Dummy initialization as won't need this info to delete a table
+ * from publication
+ */
+ pubrel->columns = NIL;
+ delrels = lappend(delrels, pubrel);
}
}
@@ -515,10 +527,12 @@ OpenTableList(List *tables)
*/
foreach(lc, tables)
{
- RangeVar *rv = castNode(RangeVar, lfirst(lc));
+ PublicationTable *t = lfirst(lc);
+ RangeVar *rv = castNode(RangeVar, t->relation);
bool recurse = rv->inh;
Relation rel;
Oid myrelid;
+ PublicationRelationInfo *pub_rel;
/* Allow query cancel in case this takes a long time */
CHECK_FOR_INTERRUPTS();
@@ -539,7 +553,11 @@ OpenTableList(List *tables)
continue;
}
- rels = lappend(rels, rel);
+ pub_rel = palloc(sizeof(PublicationRelationInfo));
+ pub_rel->relation = rel;
+ pub_rel->relid = myrelid;
+ pub_rel->columns = t->columns;
+ rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
/*
@@ -572,7 +590,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
- rels = lappend(rels, rel);
+ pub_rel = palloc(sizeof(PublicationRelationInfo));
+ pub_rel->relation = rel;
+ pub_rel->relid = childrelid;
+ pub_rel->columns = t->columns;
+ rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
}
@@ -593,9 +615,9 @@ CloseTableList(List *rels)
foreach(lc, rels)
{
- Relation rel = (Relation) lfirst(lc);
+ PublicationRelationInfo *pub_rel = (PublicationRelationInfo *)lfirst(lc);
- table_close(rel, NoLock);
+ table_close(pub_rel->relation, NoLock);
}
}
@@ -612,7 +634,8 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
foreach(lc, rels)
{
- Relation rel = (Relation) lfirst(lc);
+ PublicationRelationInfo *pub_rel = (PublicationRelationInfo *)lfirst(lc);
+ Relation rel = pub_rel->relation;
ObjectAddress obj;
/* Must be owner of the table or superuser. */
@@ -620,7 +643,7 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(rel->rd_rel->relkind),
RelationGetRelationName(rel));
- obj = publication_add_relation(pubid, rel, if_not_exists);
+ obj = publication_add_relation(pubid, pub_rel, if_not_exists);
if (stmt)
{
EventTriggerCollectSimpleCommand(obj, InvalidObjectAddress,
@@ -644,7 +667,8 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
foreach(lc, rels)
{
- Relation rel = (Relation) lfirst(lc);
+ PublicationRelationInfo *pubrel = (PublicationRelationInfo *) lfirst(lc);
+ Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index eb24195438..cd94da2dd4 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -426,14 +426,14 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
transform_element_list transform_type_list
TriggerTransitions TriggerReferencing
vacuum_relation_list opt_vacuum_relation_list
- drop_option_list
+ drop_option_list publication_table_list
%type <node> opt_routine_body
%type <groupclause> group_clause
%type <list> group_by_list
%type <node> group_by_item empty_grouping_set rollup_clause cube_clause
%type <node> grouping_sets_clause
-%type <node> opt_publication_for_tables publication_for_tables
+%type <node> opt_publication_for_tables publication_for_tables publication_table
%type <list> opt_fdw_options fdw_options
%type <defelt> fdw_option
@@ -9612,7 +9612,7 @@ opt_publication_for_tables:
;
publication_for_tables:
- FOR TABLE relation_expr_list
+ FOR TABLE publication_table_list
{
$$ = (Node *) $3;
}
@@ -9622,6 +9622,21 @@ publication_for_tables:
}
;
+publication_table_list:
+ publication_table
+ { $$ = list_make1($1); }
+ | publication_table_list ',' publication_table
+ { $$ = lappend($1, $3); }
+ ;
+
+publication_table: relation_expr opt_column_list
+ {
+ PublicationTable *n = makeNode(PublicationTable);
+ n->relation = $1;
+ n->columns = $2;
+ $$ = (Node *) n;
+ }
+ ;
/*****************************************************************************
*
@@ -9643,7 +9658,7 @@ AlterPublicationStmt:
n->options = $5;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name ADD_P TABLE relation_expr_list
+ | ALTER PUBLICATION name ADD_P TABLE publication_table_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
@@ -9651,7 +9666,7 @@ AlterPublicationStmt:
n->tableAction = DEFELEM_ADD;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name SET TABLE relation_expr_list
+ | ALTER PUBLICATION name SET TABLE publication_table_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
@@ -9659,7 +9674,7 @@ AlterPublicationStmt:
n->tableAction = DEFELEM_SET;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name DROP TABLE relation_expr_list
+ | ALTER PUBLICATION name DROP TABLE publication_table_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 1cf59e0fb0..d783d8e7c3 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -31,7 +31,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary, Bitmapset *att_list);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -140,7 +140,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *att_list)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -152,7 +152,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_list);
}
/*
@@ -184,7 +184,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary, Bitmapset *att_list)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -205,11 +205,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, att_list);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_list);
}
/*
@@ -278,7 +278,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -491,7 +491,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary, Bitmapset *att_list)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -542,6 +542,12 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
continue;
}
+ if (att_list != NULL && !(bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, att_list)))
+ {
+ pq_sendbyte(out, LOGICALREP_COLUMN_UNCHANGED);
+ continue;
+ }
+
typtup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(att->atttypid));
if (!HeapTupleIsValid(typtup))
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abd5217ab1..a04e307f4d 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,12 +15,14 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
@@ -70,6 +72,7 @@ static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
LogicalDecodingContext *ctx);
+static Bitmapset* get_table_columnset(Oid relid, List *columns, Bitmapset *att_list);
/*
* Entry in the map used to remember which relation schemas we sent.
@@ -115,6 +118,7 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+ Bitmapset *att_list;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -590,10 +594,9 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (relentry->map)
tuple = execute_attr_map_tuple(tuple, relentry->map);
}
-
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->att_list);
OutputPluginWrite(ctx, true);
break;
}
@@ -619,10 +622,9 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
relentry->map);
}
}
-
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->att_list);
OutputPluginWrite(ctx, true);
break;
}
@@ -1031,8 +1033,8 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
- entry->map = NULL; /* will be set by maybe_send_schema() if
- * needed */
+ entry->att_list = NULL;
+ entry->map = NULL; /* will be set by maybe_send_schema() if needed */
}
/* Validate the entry */
@@ -1116,15 +1118,38 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ int nelems, i;
+ bool isnull;
+ Datum *elems;
+ HeapTuple pub_rel_tuple;
+ Datum pub_rel_cols;
+ List *columns = NIL;
+
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP, ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP, pub_rel_tuple, Anum_pg_publication_rel_prattrs, &isnull);
+ if (!isnull)
+ {
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ deconstruct_array(DatumGetArrayTypePCopy(pub_rel_cols),
+ TEXTOID, -1, false, 'i',
+ &elems, NULL, &nelems);
+ for (i = 0; i < nelems; i++)
+ columns = lappend(columns, TextDatumGetCString(elems[i]));
+ entry->att_list = get_table_columnset(publish_as_relid, columns, entry->att_list);
+ MemoryContextSwitchTo(oldctx);
+ }
+ ReleaseSysCache(pub_rel_tuple);
+ }
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
+
}
- if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
- break;
}
list_free(pubids);
@@ -1136,6 +1161,23 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
return entry;
}
+static Bitmapset*
+get_table_columnset(Oid relid, List *columns, Bitmapset *att_list)
+{
+ ListCell *cell;
+ foreach(cell, columns)
+ {
+ const char *attname = lfirst(cell);
+ int attnum = get_attnum(relid, attname);
+
+ if (!bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, att_list))
+ att_list = bms_add_member(att_list,
+ attnum - FirstLowInvalidHeapAttributeNumber);
+
+ }
+ return att_list;
+}
+
/*
* Cleanup list of streamed transactions and update the schema_sent flag.
*
@@ -1220,6 +1262,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->att_list);
+ entry->att_list = NULL;
if (entry->map)
{
/*
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index f332bad4d4..7bdc9bb9b8 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -83,6 +83,13 @@ typedef struct Publication
PublicationActions pubactions;
} Publication;
+typedef struct PublicationRelationInfo
+{
+ Oid relid;
+ Relation relation;
+ List *columns;
+} PublicationRelationInfo;
+
extern Publication *GetPublication(Oid pubid);
extern Publication *GetPublicationByName(const char *pubname, bool missing_ok);
extern List *GetRelationPublications(Oid relid);
@@ -108,7 +115,7 @@ extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern bool is_publishable_relation(Relation rel);
-extern ObjectAddress publication_add_relation(Oid pubid, Relation targetrel,
+extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelationInfo *targetrel,
bool if_not_exists);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index b5d5504cbb..d1d4eec2c0 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN
+ text prattrs[1]; /* Variable length field starts here */
+#endif
} FormData_pg_publication_rel;
/* ----------------
@@ -40,6 +43,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
*/
typedef FormData_pg_publication_rel *Form_pg_publication_rel;
+DECLARE_TOAST(pg_publication_rel, 8895, 8896);
DECLARE_UNIQUE_INDEX_PKEY(pg_publication_rel_oid_index, 6112, PublicationRelObjectIndexId, on pg_publication_rel using btree(oid oid_ops));
DECLARE_UNIQUE_INDEX(pg_publication_rel_prrelid_prpubid_index, 6113, PublicationRelPrrelidPrpubidIndexId, on pg_publication_rel using btree(prrelid oid_ops, prpubid oid_ops));
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index d9e417bcd7..2037705f45 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -491,6 +491,7 @@ typedef enum NodeTag
T_PartitionRangeDatum,
T_PartitionCmd,
T_VacuumRelation,
+ T_PublicationTable,
/*
* TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index def9651b34..a17c1aa9f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3623,6 +3623,12 @@ typedef struct AlterTSConfigurationStmt
bool missing_ok; /* for DROP - skip error if missing? */
} AlterTSConfigurationStmt;
+typedef struct PublicationTable
+{
+ NodeTag type;
+ RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
+} PublicationTable;
typedef struct CreatePublicationStmt
{
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 55b90c03ea..879c58c497 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -134,11 +134,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *att_list);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *att_list);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
diff --git a/src/test/subscription/t/021_column_filter.pl b/src/test/subscription/t/021_column_filter.pl
new file mode 100644
index 0000000000..b5c73bfc7d
--- /dev/null
+++ b/src/test/subscription/t/021_column_filter.pl
@@ -0,0 +1,76 @@
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test TRUNCATE
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+# setup
+
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, b int, c int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, b int, c int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int)");
+
+#Test create publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub1 FOR TABLE tab1(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+);
+#Initial sync
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab1 VALUES (1,2,3)");
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'insert on column c is not replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab1 SET c = 5 where a = 1");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'update on column c is not replicated');
+
+#Test alter publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION"
+);
+
+$node_publisher->wait_for_catchup('sub1');
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 VALUES (1,'abc',3)");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2");
+is($result, qq(1|abc|), 'insert on column c is not replicated');
--
2.17.2 (Apple Git-113)
On Tue, Jul 13, 2021 at 7:44 PM Rahila Syed <rahilasyed90@gmail.com> wrote:
Hi Tomas,
Thank you for your comments.
Currently, this capability is not included in the patch. If the table on
the subscriber
server has lesser attributes than that on the publisher server, it
throws an error at the
time of CREATE SUBSCRIPTION.That's a bit surprising, to be honest. I do understand the patch simply
treats the filtered columns as "unchanged" because that's the simplest
way to filter the *data* of the columns. But if someone told me we can
"filter columns" I'd expect this to work without the columns on the
subscriber.OK, I will look into adding this.
However, need to carefully consider situations in which a server
subscribes to multiple
publications, each publishing a different subset of columns of atable.
Isn't that pretty much the same situation as for multiple subscriptions
each with a different set of I/U/D operations? IIRC we simply merge
those, so why not to do the same thing here and merge the attributes?Yeah, I agree with the solution to merge the attributes, similar to how
operations are merged. My concern was also from an implementation point
of view, will it be a very drastic change. I now had a look at how remote
relation
attributes are acquired for comparison with local attributes at the
subscriber.
It seems that the publisher will need to send the information about the
filtered columns
for each publication specified during CREATE SUBSCRIPTION.
This will be read at the subscriber side which in turn updates its cache
accordingly.
Currently, the subscriber expects all attributes of a published relation
to be present.
I will add code for this in the next version of the patch.To nitpick, I find "Bitmapset *att_list" a bit annoying, because it's
not really a list ;-)
I will make this change with the next version
FWIW "make check" fails for me with this version, due to segfault in
OpenTableLists. Apparenly there's some confusion - the code expects the
list to contain PublicationTable nodes, and tries to extract the
RangeVar from the elements. But the list actually contains RangeVar, so
this crashes and burns. See the attached backtrace.Thank you for the report, This is fixed in the attached version, now all
publication
function calls accept the PublicationTableInfo list.Thank you,
Rahila Syed
The patch does not apply, and an rebase is required
Hunk #8 succeeded at 1259 (offset 99 lines).
Hunk #9 succeeded at 1360 (offset 99 lines).
1 out of 9 hunks FAILED -- saving rejects to file
src/backend/replication/pgoutput/pgoutput.c.rej
patching file src/include/catalog/pg_publication.h
Changing the status to "Waiting on Author"
--
Ibrar Ahmed
Hello,
I think this looks good regarding the PublicationRelationInfo API that was
discussed.
Looking at OpenTableList(), I think you forgot to update the comment --
it says "open relations specified by a RangeVar list", but the list is
now of PublicationTable. Also I think it would be good to say that the
returned tables are PublicationRelationInfo, maybe such as "In the
returned list of PublicationRelationInfo, the tables are locked ..."
In AlterPublicationTables() I was confused by some code that seemed
commented a bit too verbosely (for a moment I thought the whole list was
being copied into a different format). May I suggest something more
compact like
/* Not yet in list; open it and add it to the list */
if (!found)
{
Relation oldrel;
PublicationRelationInfo *pubrel;
oldrel = table_open(oldrelid, ShareUpdateExclusiveLock);
/* Wrap it in PublicationRelationInfo */
pubrel = palloc(sizeof(PublicationRelationInfo));
pubrel->relation = oldrel;
pubrel->relid = oldrelid;
pubrel->columns = NIL; /* not needed */
delrels = lappend(delrels, pubrel);
}
Thanks!
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
One thing I just happened to notice is this part of your commit message
: REPLICA IDENTITY columns are always replicated
: irrespective of column names specification.
... for which you don't have any tests -- I mean, create a table with a
certain REPLICA IDENTITY and later try to publish a set of columns that
doesn't include all the columns in the replica identity, then verify
that those columns are indeed published.
Having said that, I'm not sure I agree with this design decision; what I
think this is doing is hiding from the user the fact that they are
publishing columns that they don't want to publish. I think as a user I
would rather get an error in that case:
ERROR: invalid column list in published set
DETAIL: The set of published commands does not include all the replica identity columns.
or something like that. Avoid possible nasty surprises of security-
leaking nature.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
"On the other flipper, one wrong move and we're Fatal Exceptions"
(T.U.X.: Term Unit X - http://www.thelinuxreview.com/TUX/)
Hi,
Currently, this capability is not included in the patch. If the table on
the subscriber
server has lesser attributes than that on the publisher server, it
throws an error at the
time of CREATE SUBSCRIPTION.That's a bit surprising, to be honest. I do understand the patch simply
treats the filtered columns as "unchanged" because that's the simplest
way to filter the *data* of the columns. But if someone told me we can
"filter columns" I'd expect this to work without the columns on the
subscriber.OK, I will look into adding this.
This has been added in the attached patch. Now, instead of
treating the filtered columns as unchanged and sending a byte
with that information, unfiltered columns are not sent to the subscriber
server at all. This along with saving the network bandwidth, allows
the logical replication to even work between tables with different numbers
of
columns i.e with the table on subscriber server containing only the
filtered
columns. Currently, replica identity columns are replicated irrespective of
the presence of the column filters, hence the table on the subscriber side
must
contain the replica identity columns.
The patch adds a new parse node PublicationTable, but doesn't add
copyfuncs.c, equalfuncs.c, readfuncs.c, outfuncs.c support for it.
Thanks, added this.
Looking at OpenTableList(), I think you forgot to update the comment --
it says "open relations specified by a RangeVar list",
Thank you for the review, Modified this.
To nitpick, I find "Bitmapset *att_list" a bit annoying, because it's
not really a list ;-)
Changed this.
It's not super clear to me that strlist_to_textarray() and related
processing will behave sanely when the column names contain weird
characters such as commas or quotes, or just when used with uppercase
column names. Maybe it's worth having tests that try to break such
cases.
Added a few test cases for this.
In AlterPublicationTables() I was confused by some code that seemed
commented a bit too verbosely
Modified this as per the suggestion.
: REPLICA IDENTITY columns are always replicated
: irrespective of column names specification.
... for which you don't have any tests
I have added these tests.
Having said that, I'm not sure I agree with this design decision; what I
think this is doing is hiding from the user the fact that they are
publishing columns that they don't want to publish. I think as a user I
would rather get an error in that case:
ERROR: invalid column list in published set
DETAIL: The set of published commands does not include all the replica
identity columns.
or something like that. Avoid possible nasty surprises of security-
leaking nature.
Ok, Thank you for your opinion. I agree that giving an explicit error in
this case will be safer.
I will include this, in case there are no counter views.
Thank you for your review comments. Please find attached the rebased and
updated patch.
Thank you,
Rahila Syed
Attachments:
v3-0001-Add-column-filtering-to-logical-replication.patchapplication/octet-stream; name=v3-0001-Add-column-filtering-to-logical-replication.patchDownload
From 4711ee538a35c7a5c4eb4f23258e3bd8a3ab0bb4 Mon Sep 17 00:00:00 2001
From: rahila <rahilasyed.90@gmail.com>
Date: Mon, 7 Jun 2021 16:27:21 +0530
Subject: [PATCH] Add column filtering to logical replication
Add capability to specifiy column names while linking
the table to a publication, at the time of CREATE or ALTER
publication. This will allow replicating only the specified
columns. Rest of the columns on the subscriber will be populated
locally. This facilitates replication to a table on subscriber
containing only the subscribed/filtered columns.
If column names are not specified, all the columns are
replicated. REPLICA IDENTITY columns are always replicated
irrespective of the column filters.
Add a tap test for the same in src/test/subscription.
---
src/backend/catalog/pg_publication.c | 20 +++-
src/backend/commands/copyfromparse.c | 1 -
src/backend/commands/publicationcmds.c | 50 +++++---
src/backend/nodes/copyfuncs.c | 13 +++
src/backend/nodes/equalfuncs.c | 12 ++
src/backend/nodes/outfuncs.c | 12 ++
src/backend/nodes/readfuncs.c | 16 +++
src/backend/parser/gram.y | 27 ++++-
src/backend/replication/logical/proto.c | 86 +++++++++++---
src/backend/replication/logical/relation.c | 1 -
src/backend/replication/logical/tablesync.c | 96 ++++++++++++++-
src/backend/replication/pgoutput/pgoutput.c | 95 ++++++++++++---
src/include/catalog/pg_publication.h | 9 +-
src/include/catalog/pg_publication_rel.h | 4 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 6 +
src/include/replication/logicalproto.h | 6 +-
src/test/subscription/t/021_column_filter.pl | 116 +++++++++++++++++++
18 files changed, 499 insertions(+), 72 deletions(-)
create mode 100644 src/test/subscription/t/021_column_filter.pl
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 2a2fe03c13..ad04ffe04b 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -141,18 +141,20 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
* Insert new publication / relation mapping.
*/
ObjectAddress
-publication_add_relation(Oid pubid, Relation targetrel,
+publication_add_relation(Oid pubid, PublicationRelationInfo *targetrel,
bool if_not_exists)
{
Relation rel;
HeapTuple tup;
Datum values[Natts_pg_publication_rel];
bool nulls[Natts_pg_publication_rel];
- Oid relid = RelationGetRelid(targetrel);
+ Oid relid = RelationGetRelid(targetrel->relation);
Oid prrelid;
Publication *pub = GetPublication(pubid);
ObjectAddress myself,
referenced;
+ ListCell *lc;
+ List *target_cols = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -172,10 +174,10 @@ publication_add_relation(Oid pubid, Relation targetrel,
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_OBJECT),
errmsg("relation \"%s\" is already member of publication \"%s\"",
- RelationGetRelationName(targetrel), pub->name)));
+ RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel);
+ check_publication_add_relation(targetrel->relation);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -188,6 +190,14 @@ publication_add_relation(Oid pubid, Relation targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ foreach(lc, targetrel->columns)
+ {
+ char *colname;
+ colname = strVal(lfirst(lc));
+ target_cols = lappend(target_cols, colname);
+ }
+ values[Anum_pg_publication_rel_prattrs - 1] =
+ PointerGetDatum(strlist_to_textarray(target_cols));
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -209,7 +219,7 @@ publication_add_relation(Oid pubid, Relation targetrel,
table_close(rel, RowExclusiveLock);
/* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcache(targetrel);
+ CacheInvalidateRelcache(targetrel->relation);
return myself;
}
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index fdf57f1556..515728df67 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -839,7 +839,6 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("extra data after last expected column")));
-
fieldno = 0;
/* Loop to read the user attributes on the line. */
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 8487eeb7e6..aee5645e31 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -393,7 +393,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
foreach(newlc, rels)
{
- Relation newrel = (Relation) lfirst(newlc);
+ PublicationRelationInfo *newpubrel = (PublicationRelationInfo *) lfirst(newlc);
+ Relation newrel = newpubrel->relation;
if (RelationGetRelid(newrel) == oldrelid)
{
@@ -401,13 +402,20 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
break;
}
}
-
+ /* Not yet in the list, open it and add to the list */
if (!found)
{
Relation oldrel = table_open(oldrelid,
ShareUpdateExclusiveLock);
-
- delrels = lappend(delrels, oldrel);
+ /*
+ * Wrap relation into PublicationRelationInfo
+ */
+ PublicationRelationInfo *pubrel = palloc(sizeof(PublicationRelationInfo));
+ pubrel->relation = oldrel;
+ pubrel->relid = oldrelid;
+ /* This is not needed to delete a table */
+ pubrel->columns = NIL;
+ delrels = lappend(delrels, pubrel);
}
}
@@ -498,9 +506,9 @@ RemovePublicationRelById(Oid proid)
}
/*
- * Open relations specified by a RangeVar list.
- * The returned tables are locked in ShareUpdateExclusiveLock mode in order to
- * add them to a publication.
+ * Open relations specified by a PublicationTable list.
+ * In the returned list of PublicationRelationInfo, tables are locked
+ * in ShareUpdateExclusiveLock mode in order to add them to a publication.
*/
static List *
OpenTableList(List *tables)
@@ -514,10 +522,12 @@ OpenTableList(List *tables)
*/
foreach(lc, tables)
{
- RangeVar *rv = lfirst_node(RangeVar, lc);
+ PublicationTable *t = lfirst(lc);
+ RangeVar *rv = castNode(RangeVar, t->relation);
bool recurse = rv->inh;
Relation rel;
Oid myrelid;
+ PublicationRelationInfo *pub_rel;
/* Allow query cancel in case this takes a long time */
CHECK_FOR_INTERRUPTS();
@@ -538,7 +548,11 @@ OpenTableList(List *tables)
continue;
}
- rels = lappend(rels, rel);
+ pub_rel = palloc(sizeof(PublicationRelationInfo));
+ pub_rel->relation = rel;
+ pub_rel->relid = myrelid;
+ pub_rel->columns = t->columns;
+ rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
/*
@@ -571,7 +585,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
- rels = lappend(rels, rel);
+ pub_rel = palloc(sizeof(PublicationRelationInfo));
+ pub_rel->relation = rel;
+ pub_rel->relid = childrelid;
+ pub_rel->columns = t->columns;
+ rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
}
@@ -592,9 +610,9 @@ CloseTableList(List *rels)
foreach(lc, rels)
{
- Relation rel = (Relation) lfirst(lc);
+ PublicationRelationInfo *pub_rel = (PublicationRelationInfo *)lfirst(lc);
- table_close(rel, NoLock);
+ table_close(pub_rel->relation, NoLock);
}
}
@@ -611,7 +629,8 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
foreach(lc, rels)
{
- Relation rel = (Relation) lfirst(lc);
+ PublicationRelationInfo *pub_rel = (PublicationRelationInfo *)lfirst(lc);
+ Relation rel = pub_rel->relation;
ObjectAddress obj;
/* Must be owner of the table or superuser. */
@@ -619,7 +638,7 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(rel->rd_rel->relkind),
RelationGetRelationName(rel));
- obj = publication_add_relation(pubid, rel, if_not_exists);
+ obj = publication_add_relation(pubid, pub_rel, if_not_exists);
if (stmt)
{
EventTriggerCollectSimpleCommand(obj, InvalidObjectAddress,
@@ -643,7 +662,8 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
foreach(lc, rels)
{
- Relation rel = (Relation) lfirst(lc);
+ PublicationRelationInfo *pubrel = (PublicationRelationInfo *) lfirst(lc);
+ Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 29020c908e..0763802502 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4951,6 +4951,16 @@ _copyForeignKeyCacheInfo(const ForeignKeyCacheInfo *from)
return newnode;
}
+static PublicationTable *
+_copyPublicationTable(const PublicationTable *from)
+{
+ PublicationTable *newnode = makeNode(PublicationTable);
+
+ COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(columns);
+
+ return newnode;
+}
/*
* copyObjectImpl -- implementation of copyObject(); see nodes/nodes.h
@@ -5866,6 +5876,9 @@ copyObjectImpl(const void *from)
case T_PartitionCmd:
retval = _copyPartitionCmd(from);
break;
+ case T_PublicationTable:
+ retval = _copyPublicationTable(from);
+ break;
/*
* MISCELLANEOUS NODES
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 8a1762000c..b0f37b2ceb 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -3114,6 +3114,15 @@ _equalValue(const Value *a, const Value *b)
return true;
}
+static bool
+_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
+{
+ COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(columns);
+
+ return true;
+}
+
/*
* equal
* returns whether two nodes are equal
@@ -3862,6 +3871,9 @@ equal(const void *a, const void *b)
case T_PartitionCmd:
retval = _equalPartitionCmd(a, b);
break;
+ case T_PublicationTable:
+ retval = _equalPublicationTable(a, b);
+ break;
default:
elog(ERROR, "unrecognized node type: %d",
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 87561cbb6f..f04eb536c9 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -3821,6 +3821,15 @@ _outPartitionRangeDatum(StringInfo str, const PartitionRangeDatum *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outPublicationTable(StringInfo str, const PublicationTable *node)
+{
+ WRITE_NODE_TYPE("PUBLICATIONTABLE");
+
+ WRITE_NODE_FIELD(relation);
+ WRITE_NODE_FIELD(columns);
+}
+
/*
* outNode -
* converts a Node into ascii string and append it to 'str'
@@ -4520,6 +4529,9 @@ outNode(StringInfo str, const void *obj)
case T_PartitionRangeDatum:
_outPartitionRangeDatum(str, obj);
break;
+ case T_PublicationTable:
+ _outPublicationTable(str, obj);
+ break;
default:
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 77d082d8b4..6b2d8efb01 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -2702,6 +2702,20 @@ _readPartitionRangeDatum(void)
READ_DONE();
}
+/*
+ * _readPublicationTable
+ */
+static PublicationTable *
+_readPublicationTable(void)
+{
+ READ_LOCALS(PublicationTable);
+
+ READ_NODE_FIELD(relation);
+ READ_NODE_FIELD(columns);
+
+ READ_DONE();
+}
+
/*
* parseNodeString
*
@@ -2973,6 +2987,8 @@ parseNodeString(void)
return_value = _readPartitionBoundSpec();
else if (MATCH("PARTITIONRANGEDATUM", 19))
return_value = _readPartitionRangeDatum();
+ else if (MATCH("PUBLICATIONTABLE", 16))
+ return_value = _readPublicationTable();
else
{
elog(ERROR, "badly formatted node string \"%.32s\"...", token);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 39a2849eba..2c9af95db8 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -426,14 +426,14 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
transform_element_list transform_type_list
TriggerTransitions TriggerReferencing
vacuum_relation_list opt_vacuum_relation_list
- drop_option_list
+ drop_option_list publication_table_list
%type <node> opt_routine_body
%type <groupclause> group_clause
%type <list> group_by_list
%type <node> group_by_item empty_grouping_set rollup_clause cube_clause
%type <node> grouping_sets_clause
-%type <node> opt_publication_for_tables publication_for_tables
+%type <node> opt_publication_for_tables publication_for_tables publication_table
%type <list> opt_fdw_options fdw_options
%type <defelt> fdw_option
@@ -9620,7 +9620,7 @@ opt_publication_for_tables:
;
publication_for_tables:
- FOR TABLE relation_expr_list
+ FOR TABLE publication_table_list
{
$$ = (Node *) $3;
}
@@ -9630,6 +9630,21 @@ publication_for_tables:
}
;
+publication_table_list:
+ publication_table
+ { $$ = list_make1($1); }
+ | publication_table_list ',' publication_table
+ { $$ = lappend($1, $3); }
+ ;
+
+publication_table: relation_expr opt_column_list
+ {
+ PublicationTable *n = makeNode(PublicationTable);
+ n->relation = $1;
+ n->columns = $2;
+ $$ = (Node *) n;
+ }
+ ;
/*****************************************************************************
*
@@ -9651,7 +9666,7 @@ AlterPublicationStmt:
n->options = $5;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name ADD_P TABLE relation_expr_list
+ | ALTER PUBLICATION name ADD_P TABLE publication_table_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
@@ -9659,7 +9674,7 @@ AlterPublicationStmt:
n->tableAction = DEFELEM_ADD;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name SET TABLE relation_expr_list
+ | ALTER PUBLICATION name SET TABLE publication_table_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
@@ -9667,7 +9682,7 @@ AlterPublicationStmt:
n->tableAction = DEFELEM_SET;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name DROP TABLE relation_expr_list
+ | ALTER PUBLICATION name DROP TABLE publication_table_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 52b65e9572..8bfecf44ca 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,9 +29,9 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *att_map);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary, Bitmapset *att_map);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +398,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *att_map)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +410,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_map);
}
/*
@@ -442,7 +442,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary, Bitmapset *att_map)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -463,11 +463,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, att_map);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_map);
}
/*
@@ -536,7 +536,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -651,7 +651,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *att_map)
{
char *relname;
@@ -673,7 +673,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, att_map);
}
/*
@@ -749,20 +749,37 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary,
+ Bitmapset *att_map)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
bool isnull[MaxTupleAttributeNumber];
int i;
uint16 nliveatts = 0;
+ Bitmapset *idattrs = NULL;
+ bool replidentfull;
+ Form_pg_attribute att;
desc = RelationGetDescr(rel);
+ replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ if (!replidentfull)
+ idattrs = RelationGetIdentityKeyBitmap(rel);
+
for (i = 0; i < desc->natts; i++)
{
+ att = TupleDescAttr(desc, i);
if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
continue;
+ /*
+ * Do not increment count of attributes if not a part of column filters
+ * except for replica identity columns or if replica identity is full.
+ */
+ if (att_map != NULL && !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, att_map)
+ && !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, idattrs)
+ && !replidentfull)
+ continue;
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -800,6 +817,16 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
continue;
}
+ /*
+ * Do not send attribute data if it is not a part of column filters,
+ * except if it is a part of REPLICA IDENTITY or REPLICA IDENTITY is
+ * full, send the data.
+ */
+ if (att_map != NULL && !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, att_map)
+ && !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, idattrs)
+ && !replidentfull)
+ continue;
+
typtup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(att->atttypid));
if (!HeapTupleIsValid(typtup))
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
@@ -904,7 +931,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *att_map)
{
TupleDesc desc;
int i;
@@ -914,20 +941,34 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
desc = RelationGetDescr(rel);
+ /* fetch bitmap of REPLICATION IDENTITY attributes */
+ replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ if (!replidentfull)
+ idattrs = RelationGetIdentityKeyBitmap(rel);
+
/* send number of live attributes */
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ /* REPLICA IDENTITY FULL means all columns are sent as part of key. */
+ if (replidentfull || bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs))
+ {
+ nliveatts++;
+ continue;
+ }
+ /* Skip sending if not a part of column filter */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map))
continue;
nliveatts++;
}
pq_sendint16(out, nliveatts);
- /* fetch bitmap of REPLICATION IDENTITY attributes */
- replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
- if (!replidentfull)
- idattrs = RelationGetIdentityKeyBitmap(rel);
-
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -937,6 +978,13 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ /* Exlude filtered columns, REPLICA IDENTITY COLUMNS CAN'T BE EXCLUDED */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map) && !bms_is_member(att->attnum
+ - FirstLowInvalidHeapAttributeNumber, idattrs)
+ && !replidentfull)
+ continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
@@ -944,7 +992,6 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
flags |= LOGICALREP_IS_REPLICA_IDENTITY;
pq_sendbyte(out, flags);
-
/* attribute name */
pq_sendstring(out, NameStr(att->attname));
@@ -953,6 +1000,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* attribute mode */
pq_sendint32(out, att->atttypmod);
+
}
bms_free(idattrs);
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index c37e2a7e29..d7a7b00841 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -354,7 +354,6 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)
attnum = logicalrep_rel_att_by_name(remoterel,
NameStr(attr->attname));
-
entry->attrmap->attnums[i] = attnum;
if (attnum >= 0)
missingatts = bms_del_member(missingatts, attnum);
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index f07983a43c..f336a384a1 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -111,6 +111,7 @@
#include "replication/origin.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -695,19 +696,27 @@ fetch_remote_table_info(char *nspname, char *relname,
LogicalRepRelation *lrel)
{
WalRcvExecResult *res;
+ WalRcvExecResult *res_pub;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
+ TupleTableSlot *slot_pub;
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid pubRow[] = {TEXTARRAYOID};
bool isnull;
- int natt;
+ int natt,i;
+ Datum *elems;
+ int nelems;
+ List *pub_columns = NIL;
+ ListCell *lc;
+ bool am_partition = false;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -737,6 +746,7 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetChar(slot_getattr(slot, 4, &isnull));
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
@@ -774,11 +784,78 @@ fetch_remote_table_info(char *nspname, char *relname,
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+
+ /*
+ * Now, fetch the values of publications' column filters
+ * For a partition, use pg_inherit to find the parent,
+ * as the pg_publication_rel contains only the topmost parent
+ * table entry in case of table partitioning.
+ *
+ * XXX Modify the join query to be able to fetch topmost parent,
+ * Currently it fetches immediate parent of the partition.
+ */
+ resetStringInfo(&cmd);
+ if (!am_partition)
+ appendStringInfo(&cmd, "SELECT prattrs from pg_publication_rel"
+ " WHERE prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd, "SELECT prattrs from pg_publication_rel pb, pg_inherits pinh"
+ " WHERE pb.prrelid = pinh.inhparent AND pinh.inhrelid = %u", lrel->remoteid);
+
+ res_pub = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(pubRow), pubRow);
+
+ if (res_pub->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch published columns info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, res_pub->err)));
+ slot_pub = MakeSingleTupleTableSlot(res_pub->tupledesc, &TTSOpsMinimalTuple);
+
+ while (tuplestore_gettupleslot(res_pub->tuplestore, true, false, slot_pub))
+ {
+ deconstruct_array(DatumGetArrayTypePCopy(slot_getattr(slot_pub, 1, &isnull)),
+ TEXTOID, -1, false, 'i',
+ &elems, NULL, &nelems);
+ for (i = 0; i < nelems; i++)
+ pub_columns = lappend(pub_columns, TextDatumGetCString(elems[i]));
+ ExecClearTuple(slot_pub);
+ }
+ ExecDropSingleTupleTableSlot(slot_pub);
+ walrcv_clear_result(res_pub);
+
+ /*
+ * Store the column names only if they are contained in column filter
+ * LogicalRepRelation will only contain attributes corresponding
+ * to those specficied in column filters.
+ */
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
+ char * rel_colname =
TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ bool found = false;
Assert(!isnull);
+ if (pub_columns != NIL)
+ {
+ foreach(lc, pub_columns)
+ {
+ char *pub_colname = lfirst(lc);
+ if(!strcmp(pub_colname, rel_colname))
+ {
+ found = true;
+ lrel->attnames[natt] = rel_colname;
+ break;
+ }
+ }
+ }
+ else
+ {
+ found = true;
+ lrel->attnames[natt] = rel_colname;
+ }
+ if (!found)
+ continue;
+
lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
@@ -829,8 +906,17 @@ copy_table(Relation rel)
/* Start copy on the publisher. */
initStringInfo(&cmd);
if (lrel.relkind == RELKIND_RELATION)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 14d737fd93..033f36e00c 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,12 +15,14 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
@@ -81,10 +83,12 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *att_map);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
+static Bitmapset* get_table_columnset(Oid relid, List *columns, Bitmapset *att_map);
/*
* Entry in the map used to remember which relation schemas we sent.
@@ -130,6 +134,7 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+ Bitmapset *att_map;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -570,11 +575,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
}
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->att_map);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->att_map);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -587,7 +592,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *att_map)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -609,14 +615,24 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
-
+ /*
+ * Do not send type information if attribute is
+ * not present in column filter.
+ * XXX Allow sending type information for REPLICA
+ * IDENTITY COLUMNS with user created type.
+ * even when they are not mentioned in column filters.
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map))
+ continue;
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, att_map);
OutputPluginWrite(ctx, false);
}
@@ -690,10 +706,9 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (relentry->map)
tuple = execute_attr_map_tuple(tuple, relentry->map);
}
-
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->att_map);
OutputPluginWrite(ctx, true);
break;
}
@@ -719,10 +734,9 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
relentry->map);
}
}
-
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->att_map);
OutputPluginWrite(ctx, true);
break;
}
@@ -1119,6 +1133,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
bool found;
+ Oid ancestor_id;
MemoryContext oldctx;
Assert(RelationSyncCache != NULL);
@@ -1139,8 +1154,8 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
- entry->map = NULL; /* will be set by maybe_send_schema() if
- * needed */
+ entry->att_map = NULL;
+ entry->map = NULL; /* will be set by maybe_send_schema() if needed */
}
/* Validate the entry */
@@ -1171,6 +1186,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
if (pub->alltables)
{
@@ -1181,7 +1197,6 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (!publish)
{
- bool ancestor_published = false;
/*
* For a partition, check if any of the ancestors are
@@ -1206,6 +1221,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
pub->oid))
{
ancestor_published = true;
+ ancestor_id = ancestor;
if (pub->pubviaroot)
publish_as_relid = ancestor;
}
@@ -1224,15 +1240,41 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ int nelems, i;
+ bool isnull;
+ Datum *elems;
+ HeapTuple pub_rel_tuple;
+ Datum pub_rel_cols;
+ List *columns = NIL;
+
+ if (ancestor_published)
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP, ObjectIdGetDatum(ancestor_id),
+ ObjectIdGetDatum(pub->oid));
+ else
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP, ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP, pub_rel_tuple, Anum_pg_publication_rel_prattrs, &isnull);
+ if (!isnull)
+ {
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ deconstruct_array(DatumGetArrayTypePCopy(pub_rel_cols),
+ TEXTOID, -1, false, 'i',
+ &elems, NULL, &nelems);
+ for (i = 0; i < nelems; i++)
+ columns = lappend(columns, TextDatumGetCString(elems[i]));
+ entry->att_map = get_table_columnset(publish_as_relid, columns, entry->att_map);
+ MemoryContextSwitchTo(oldctx);
+ }
+ ReleaseSysCache(pub_rel_tuple);
+ }
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
}
- if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
- break;
}
list_free(pubids);
@@ -1244,6 +1286,25 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
return entry;
}
+/*
+ * Return a bitmapset of attributes given the list of column names
+ */
+static Bitmapset*
+get_table_columnset(Oid relid, List *columns, Bitmapset *att_map)
+{
+ ListCell *cell;
+ foreach(cell, columns)
+ {
+ const char *attname = lfirst(cell);
+ int attnum = get_attnum(relid, attname);
+
+ if (!bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, att_map))
+ att_map = bms_add_member(att_map,
+ attnum - FirstLowInvalidHeapAttributeNumber);
+ }
+ return att_map;
+}
+
/*
* Cleanup list of streamed transactions and update the schema_sent flag.
*
@@ -1328,6 +1389,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->att_map);
+ entry->att_map = NULL;
if (entry->map)
{
/*
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index f332bad4d4..7bdc9bb9b8 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -83,6 +83,13 @@ typedef struct Publication
PublicationActions pubactions;
} Publication;
+typedef struct PublicationRelationInfo
+{
+ Oid relid;
+ Relation relation;
+ List *columns;
+} PublicationRelationInfo;
+
extern Publication *GetPublication(Oid pubid);
extern Publication *GetPublicationByName(const char *pubname, bool missing_ok);
extern List *GetRelationPublications(Oid relid);
@@ -108,7 +115,7 @@ extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern bool is_publishable_relation(Relation rel);
-extern ObjectAddress publication_add_relation(Oid pubid, Relation targetrel,
+extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelationInfo *targetrel,
bool if_not_exists);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index b5d5504cbb..d1d4eec2c0 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN
+ text prattrs[1]; /* Variable length field starts here */
+#endif
} FormData_pg_publication_rel;
/* ----------------
@@ -40,6 +43,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
*/
typedef FormData_pg_publication_rel *Form_pg_publication_rel;
+DECLARE_TOAST(pg_publication_rel, 8895, 8896);
DECLARE_UNIQUE_INDEX_PKEY(pg_publication_rel_oid_index, 6112, PublicationRelObjectIndexId, on pg_publication_rel using btree(oid oid_ops));
DECLARE_UNIQUE_INDEX(pg_publication_rel_prrelid_prpubid_index, 6113, PublicationRelPrrelidPrpubidIndexId, on pg_publication_rel using btree(prrelid oid_ops, prpubid oid_ops));
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 6a4d82f0a8..56d13ff022 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -490,6 +490,7 @@ typedef enum NodeTag
T_PartitionRangeDatum,
T_PartitionCmd,
T_VacuumRelation,
+ T_PublicationTable,
/*
* TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index e28248af32..bbdfaa2f45 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3624,6 +3624,12 @@ typedef struct AlterTSConfigurationStmt
bool missing_ok; /* for DROP - skip error if missing? */
} AlterTSConfigurationStmt;
+typedef struct PublicationTable
+{
+ NodeTag type;
+ RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
+} PublicationTable;
typedef struct CreatePublicationStmt
{
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 2e29513151..cb47341b6c 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -207,11 +207,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *att_map);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *att_map);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -228,7 +228,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *att_map);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/subscription/t/021_column_filter.pl b/src/test/subscription/t/021_column_filter.pl
new file mode 100644
index 0000000000..f78fdbf52f
--- /dev/null
+++ b/src/test/subscription/t/021_column_filter.pl
@@ -0,0 +1,116 @@
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test TRUNCATE
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+
+# setup
+
+my $node_publisher = PostgresNode->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgresNode->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int)");
+# Test with weird column names
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, B varchar, \"c'\" int)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, \"c'\" int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)");
+
+#Test create publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub1 FOR TABLE tab1(a, \"B\"), tab3(\"a'\",\"c'\"), test_part(b)");
+
+my $result = $node_publisher->safe_psql('postgres',
+ "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
+is($result, qq(tab1|{a,B}
+tab3|{a',c'}
+test_part|{b}), 'publication relation updated');
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+);
+#Initial sync
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab1 VALUES (1,2,3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 VALUES (1,2,3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (1,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (2,'bcd', '2021-07-03 11:12:13')");
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'insert on column c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab3");
+is($result, qq(1|3), 'insert on column b is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part");
+is($result, qq(1|abc\n2|bcd), 'insert on all columns is replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab1 SET c = 5 where a = 1");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'update on column c is not replicated');
+
+#Test alter publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION"
+);
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 VALUES (1,'abc',3)");
+#sleep(5);
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2");
+is($result, qq(1|abc), 'insert on column c is not replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET c = 5 where a = 1");
+is($result, qq(1|abc), 'update on column c is not replicated');
--
2.17.2 (Apple Git-113)
On Mon, Aug 9, 2021 at 1:36 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
Having said that, I'm not sure I agree with this design decision; what I
think this is doing is hiding from the user the fact that they are
publishing columns that they don't want to publish. I think as a user I
would rather get an error in that case:ERROR: invalid column list in published set
DETAIL: The set of published commands does not include all the replica identity columns.or something like that. Avoid possible nasty surprises of security-
leaking nature.Ok, Thank you for your opinion. I agree that giving an explicit error in this case will be safer.
+1 for an explicit error in this case.
Can you please explain why you have the restriction for including
replica identity columns and do we want to put a similar restriction
for the primary key? As far as I understand, if we allow default
values on subscribers for replica identity, then probably updates,
deletes won't work as they need to use replica identity (or PK) to
search the required tuple. If so, shouldn't we add this restriction
only when a publication has been defined for one of these (Update,
Delete) actions?
Another point is what if someone drops the column used in one of the
publications? Do we want to drop the entire relation from publication
or just remove the column filter or something else?
Do we want to consider that the columns specified in the filter must
not have NOT NULL constraint? Because, otherwise, the subscriber will
error out inserting such rows?
Minor comments:
================
pq_sendbyte(out, flags);
-
/* attribute name */
pq_sendstring(out, NameStr(att->attname));
@@ -953,6 +1000,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* attribute mode */
pq_sendint32(out, att->atttypmod);
+
}
bms_free(idattrs);
diff --git a/src/backend/replication/logical/relation.c
b/src/backend/replication/logical/relation.c
index c37e2a7e29..d7a7b00841 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -354,7 +354,6 @@ logicalrep_rel_open(LogicalRepRelId remoteid,
LOCKMODE lockmode)
attnum = logicalrep_rel_att_by_name(remoterel,
NameStr(attr->attname));
-
entry->attrmap->attnums[i] = attnum;
There are quite a few places in the patch that contains spurious line
additions or removals.
--
With Regards,
Amit Kapila.
On Mon, Aug 9, 2021 at 3:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Aug 9, 2021 at 1:36 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
Having said that, I'm not sure I agree with this design decision; what I
think this is doing is hiding from the user the fact that they are
publishing columns that they don't want to publish. I think as a user I
would rather get an error in that case:ERROR: invalid column list in published set
DETAIL: The set of published commands does not include all the replica identity columns.or something like that. Avoid possible nasty surprises of security-
leaking nature.Ok, Thank you for your opinion. I agree that giving an explicit error in this case will be safer.
+1 for an explicit error in this case.
Can you please explain why you have the restriction for including
replica identity columns and do we want to put a similar restriction
for the primary key? As far as I understand, if we allow default
values on subscribers for replica identity, then probably updates,
deletes won't work as they need to use replica identity (or PK) to
search the required tuple. If so, shouldn't we add this restriction
only when a publication has been defined for one of these (Update,
Delete) actions?Another point is what if someone drops the column used in one of the
publications? Do we want to drop the entire relation from publication
or just remove the column filter or something else?Do we want to consider that the columns specified in the filter must
not have NOT NULL constraint? Because, otherwise, the subscriber will
error out inserting such rows?
I noticed that other databases provide this feature [1]https://docs.oracle.com/en/cloud/paas/goldengate-cloud/gwuad/selecting-columns.html#GUID-9A851C8B-48F7-43DF-8D98-D086BE069E20 and they allow
users to specify "Columns that are included in Filter" or specify "All
columns to be included in filter except for a subset of columns". I am
not sure if want to provide both ways in the first version but at
least we should consider it as a future extensibility requirement and
try to choose syntax accordingly.
--
With Regards,
Amit Kapila.
Hi Amit,
Thanks for your review.
Can you please explain why you have the restriction for including
replica identity columns and do we want to put a similar restriction
for the primary key? As far as I understand, if we allow default
values on subscribers for replica identity, then probably updates,
deletes won't work as they need to use replica identity (or PK) to
search the required tuple. If so, shouldn't we add this restriction
only when a publication has been defined for one of these (Update,
Delete) actions?
Yes, like you mentioned they are needed for Updates and Deletes to work.
The restriction for including replica identity columns in column filters
exists because
In case the replica identity column values did not change, the old row
replica identity columns
are not sent to the subscriber, thus we would need new replica identity
columns
to be sent to identify the row that is to be Updated or Deleted.
I haven't tested if it would break Insert as well though. I will update
the patch accordingly.
Another point is what if someone drops the column used in one of the
publications? Do we want to drop the entire relation from publication
or just remove the column filter or something else?
Thanks for pointing this out. Currently, this is not handled in the patch.
I think dropping the column from the filter would make sense on the lines
of the table being dropped from publication, in case of drop table.
Do we want to consider that the columns specified in the filter must
not have NOT NULL constraint? Because, otherwise, the subscriber will
error out inserting such rows?I think you mean columns *not* specified in the filter must not have NOT
NULL constraint
on the subscriber, as this will break during insert, as it will try to
insert NULL for columns
not sent by the publisher.
I will look into fixing this. Probably this won't be a problem in
case the column is auto generated or contains a default value.
Minor comments:
================
pq_sendbyte(out, flags);
-
/* attribute name */
pq_sendstring(out, NameStr(att->attname));@@ -953,6 +1000,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* attribute mode */
pq_sendint32(out, att->atttypmod);
+
}bms_free(idattrs); diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index c37e2a7e29..d7a7b00841 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -354,7 +354,6 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)attnum = logicalrep_rel_att_by_name(remoterel,
NameStr(attr->attname));
-
entry->attrmap->attnums[i] = attnum;There are quite a few places in the patch that contains spurious line
additions or removals.
Thank you for your comments, I will fix these.
Thank you,
Rahila Syed
On Thu, Aug 12, 2021 at 8:40 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
Can you please explain why you have the restriction for including
replica identity columns and do we want to put a similar restriction
for the primary key? As far as I understand, if we allow default
values on subscribers for replica identity, then probably updates,
deletes won't work as they need to use replica identity (or PK) to
search the required tuple. If so, shouldn't we add this restriction
only when a publication has been defined for one of these (Update,
Delete) actions?Yes, like you mentioned they are needed for Updates and Deletes to work.
The restriction for including replica identity columns in column filters exists because
In case the replica identity column values did not change, the old row replica identity columns
are not sent to the subscriber, thus we would need new replica identity columns
to be sent to identify the row that is to be Updated or Deleted.
I haven't tested if it would break Insert as well though. I will update the patch accordingly.
Okay, but then we also need to ensure that the user shouldn't be
allowed to enable the 'update' or 'delete' for a publication that
contains some filter that doesn't have replica identity columns.
Another point is what if someone drops the column used in one of the
publications? Do we want to drop the entire relation from publication
or just remove the column filter or something else?Thanks for pointing this out. Currently, this is not handled in the patch.
I think dropping the column from the filter would make sense on the lines
of the table being dropped from publication, in case of drop table.
I think it would be tricky if you want to remove the column from the
filter because you need to recompute the entire filter and update it
again. Also, you might need to do this for all the publications that
have a particular column in their filter clause. It might be easier to
drop the entire filter but you can check if it is easier another way
than it is good.
Do we want to consider that the columns specified in the filter must
not have NOT NULL constraint? Because, otherwise, the subscriber will
error out inserting such rows?I think you mean columns *not* specified in the filter must not have NOT NULL constraint
on the subscriber, as this will break during insert, as it will try to insert NULL for columns
not sent by the publisher.
Right.
--
With Regards,
Amit Kapila.
Hi,
Another point is what if someone drops the column used in one of the
publications? Do we want to drop the entire relation from publication
or just remove the column filter or something else?
After thinking about this, I think it is best to remove the entire table
from publication,
if a column specified in the column filter is dropped from the table.
Because, if we drop the entire filter without dropping the table, it means
all the columns will be replicated,
and the downstream server table might not have those columns.
If we drop only the column from the filter we might have to recreate the
filter and check for replica identity.
That means if the replica identity column is dropped, you can't drop it
from the filter,
and might have to drop the entire publication-table mapping anyways.
Thus, I think it is cleanest to drop the entire relation from publication.
This has been implemented in the attached version.
Do we want to consider that the columns specified in the filter must
not have NOT NULL constraint? Because, otherwise, the subscriber will
error out inserting such rows?I think you mean columns *not* specified in the filter must not have NOT
NULL constraint
on the subscriber, as this will break during insert, as it will try to
insert NULL for columns
not sent by the publisher.
I will look into fixing this. Probably this won't be a problem in
case the column is auto generated or contains a default value.
I am not sure if this needs to be handled. Ideally, we need to prevent the
subscriber tables from having a NOT NULL
constraint if the publisher uses column filters to publish the values of
the table. There is no way
to do this at the time of creating a table on subscriber.
As this involves querying the publisher for this information, it can be
done at the time of initial table synchronization.
i.e error out if any of the subscribed tables has NOT NULL constraint on
non-filter columns.
This will lead to the user dropping and recreating the subscription after
removing the
NOT NULL constraint from the table.
I think the same can be achieved by doing nothing and letting the
subscriber error out while inserting rows.
Minor comments:
================
pq_sendbyte(out, flags);
-
/* attribute name */
pq_sendstring(out, NameStr(att->attname));@@ -953,6 +1000,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* attribute mode */
pq_sendint32(out, att->atttypmod);
+
}bms_free(idattrs); diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c index c37e2a7e29..d7a7b00841 100644 --- a/src/backend/replication/logical/relation.c +++ b/src/backend/replication/logical/relation.c @@ -354,7 +354,6 @@ logicalrep_rel_open(LogicalRepRelId remoteid, LOCKMODE lockmode)attnum = logicalrep_rel_att_by_name(remoterel,
NameStr(attr->attname));
-
entry->attrmap->attnums[i] = attnum;There are quite a few places in the patch that contains spurious line
additions or removals.Fixed these in the attached patch.
Having said that, I'm not sure I agree with this design decision; what I
think this is doing is hiding from the user the fact that they are
publishing columns that they don't want to publish. I think as a user I
would rather get an error in that case:
ERROR: invalid column list in published set
DETAIL: The set of published commands does not include all the replica
identity columns.
Added this.
Also added some more tests. Please find attached a rebased and updated
patch.
Thank you,
Rahila Syed
Attachments:
v4-0001-Add-column-filtering-to-logical-replication.patchapplication/octet-stream; name=v4-0001-Add-column-filtering-to-logical-replication.patchDownload
From 182a78d1b14616479421a433f1784f401e2ac294 Mon Sep 17 00:00:00 2001
From: rahila <rahilasyed.90@gmail.com>
Date: Mon, 7 Jun 2021 16:27:21 +0530
Subject: [PATCH] Add column filtering to logical replication
Add capability to specifiy column names while linking
the table to a publication, at the time of CREATE or ALTER
publication. This will allow replicating only the specified
columns. Other columns, if any, on the subscriber will be populated
locally or NULL will be inserted if no value is supplied for the column
by the upstream during INSERT.
This facilitates replication to a table on subscriber
containing only the subscribed/filtered columns.
If no filter is specified, all the columns are replicated.
REPLICA IDENTITY columns are always replicated.
Thus, prohibit adding relation to publication, if column filters
do not contain REPLICA IDENTITY.
Add a tap test for the same in src/test/subscription.
---
src/backend/access/common/relation.c | 21 +++
src/backend/catalog/pg_publication.c | 65 ++++++++-
src/backend/commands/copyfromparse.c | 1 -
src/backend/commands/publicationcmds.c | 50 +++++--
src/backend/nodes/copyfuncs.c | 13 ++
src/backend/nodes/equalfuncs.c | 12 ++
src/backend/nodes/outfuncs.c | 12 ++
src/backend/nodes/readfuncs.c | 16 +++
src/backend/parser/gram.y | 27 +++-
src/backend/replication/logical/proto.c | 86 ++++++++---
src/backend/replication/logical/tablesync.c | 97 ++++++++++++-
src/backend/replication/pgoutput/pgoutput.c | 75 ++++++++--
src/include/catalog/pg_publication.h | 9 +-
src/include/catalog/pg_publication_rel.h | 4 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 6 +
src/include/replication/logicalproto.h | 6 +-
src/include/utils/rel.h | 1 +
src/test/subscription/t/021_column_filter.pl | 143 +++++++++++++++++++
19 files changed, 574 insertions(+), 71 deletions(-)
create mode 100644 src/test/subscription/t/021_column_filter.pl
diff --git a/src/backend/access/common/relation.c b/src/backend/access/common/relation.c
index 632d13c1ea..59c1136f2e 100644
--- a/src/backend/access/common/relation.c
+++ b/src/backend/access/common/relation.c
@@ -21,12 +21,14 @@
#include "postgres.h"
#include "access/relation.h"
+#include "access/sysattr.h"
#include "access/xact.h"
#include "catalog/namespace.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/lmgr.h"
#include "utils/inval.h"
+#include "utils/lsyscache.h"
#include "utils/syscache.h"
@@ -215,3 +217,22 @@ relation_close(Relation relation, LOCKMODE lockmode)
if (lockmode != NoLock)
UnlockRelationId(&relid, lockmode);
}
+
+/*
+ * Return a bitmapset of attributes given the list of column names
+ */
+Bitmapset*
+get_table_columnset(Oid relid, List *columns, Bitmapset *att_map)
+{
+ ListCell *cell;
+ foreach(cell, columns)
+ {
+ const char *attname = lfirst(cell);
+ int attnum = get_attnum(relid, attname);
+
+ if (!bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, att_map))
+ att_map = bms_add_member(att_map,
+ attnum - FirstLowInvalidHeapAttributeNumber);
+ }
+ return att_map;
+}
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 2a2fe03c13..6687fcb12d 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -47,8 +47,12 @@
* error if not.
*/
static void
-check_publication_add_relation(Relation targetrel)
+check_publication_add_relation(Relation targetrel, List *targetcols)
{
+ bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ Oid relid = RelationGetRelid(targetrel);
+ Bitmapset *idattrs;
+
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
@@ -73,6 +77,35 @@ check_publication_add_relation(Relation targetrel)
errmsg("cannot add relation \"%s\" to publication",
RelationGetRelationName(targetrel)),
errdetail("Temporary and unlogged relations cannot be replicated.")));
+
+ /*
+ * Cannot specify column filter when REPLICA IDENTITY IS FULL
+ * or if column filter does not contain REPLICA IDENITY columns
+ */
+ if (targetcols != NIL)
+ {
+ if (replidentfull)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot add relation \"%s\" to publication",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter with REPLICA IDENTITY FULL")));
+ else
+ {
+ Bitmapset *filtermap = NULL;
+ idattrs = RelationGetIndexAttrBitmap(targetrel, INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ filtermap = get_table_columnset(relid, targetcols, filtermap);
+ if (!bms_is_subset(idattrs, filtermap))
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot add relation \"%s\" to publication",
+ RelationGetRelationName(targetrel)),
+ errdetail("Column filter must include REPLICA IDENTITY columns")));
+ }
+ }
+ }
}
/*
@@ -141,18 +174,20 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
* Insert new publication / relation mapping.
*/
ObjectAddress
-publication_add_relation(Oid pubid, Relation targetrel,
+publication_add_relation(Oid pubid, PublicationRelationInfo *targetrel,
bool if_not_exists)
{
Relation rel;
HeapTuple tup;
Datum values[Natts_pg_publication_rel];
bool nulls[Natts_pg_publication_rel];
- Oid relid = RelationGetRelid(targetrel);
+ Oid relid = RelationGetRelid(targetrel->relation);
Oid prrelid;
Publication *pub = GetPublication(pubid);
ObjectAddress myself,
referenced;
+ ListCell *lc;
+ List *target_cols = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -172,10 +207,18 @@ publication_add_relation(Oid pubid, Relation targetrel,
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_OBJECT),
errmsg("relation \"%s\" is already member of publication \"%s\"",
- RelationGetRelationName(targetrel), pub->name)));
+ RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel);
+ foreach(lc, targetrel->columns)
+ {
+ char *colname;
+
+ colname = strVal(lfirst(lc));
+ target_cols = lappend(target_cols, colname);
+
+ }
+ check_publication_add_relation(targetrel->relation, target_cols);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -188,6 +231,8 @@ publication_add_relation(Oid pubid, Relation targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ values[Anum_pg_publication_rel_prattrs - 1] =
+ PointerGetDatum(strlist_to_textarray(target_cols));
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -196,7 +241,15 @@ publication_add_relation(Oid pubid, Relation targetrel,
heap_freetuple(tup);
ObjectAddressSet(myself, PublicationRelRelationId, prrelid);
+ foreach(lc, target_cols)
+ {
+ int attnum;
+ attnum = get_attnum(relid, lfirst(lc));
+ /* Add dependency on the column */
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attnum);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -209,7 +262,7 @@ publication_add_relation(Oid pubid, Relation targetrel,
table_close(rel, RowExclusiveLock);
/* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcache(targetrel);
+ CacheInvalidateRelcache(targetrel->relation);
return myself;
}
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index fdf57f1556..515728df67 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -839,7 +839,6 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("extra data after last expected column")));
-
fieldno = 0;
/* Loop to read the user attributes on the line. */
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 8487eeb7e6..aee5645e31 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -393,7 +393,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
foreach(newlc, rels)
{
- Relation newrel = (Relation) lfirst(newlc);
+ PublicationRelationInfo *newpubrel = (PublicationRelationInfo *) lfirst(newlc);
+ Relation newrel = newpubrel->relation;
if (RelationGetRelid(newrel) == oldrelid)
{
@@ -401,13 +402,20 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
break;
}
}
-
+ /* Not yet in the list, open it and add to the list */
if (!found)
{
Relation oldrel = table_open(oldrelid,
ShareUpdateExclusiveLock);
-
- delrels = lappend(delrels, oldrel);
+ /*
+ * Wrap relation into PublicationRelationInfo
+ */
+ PublicationRelationInfo *pubrel = palloc(sizeof(PublicationRelationInfo));
+ pubrel->relation = oldrel;
+ pubrel->relid = oldrelid;
+ /* This is not needed to delete a table */
+ pubrel->columns = NIL;
+ delrels = lappend(delrels, pubrel);
}
}
@@ -498,9 +506,9 @@ RemovePublicationRelById(Oid proid)
}
/*
- * Open relations specified by a RangeVar list.
- * The returned tables are locked in ShareUpdateExclusiveLock mode in order to
- * add them to a publication.
+ * Open relations specified by a PublicationTable list.
+ * In the returned list of PublicationRelationInfo, tables are locked
+ * in ShareUpdateExclusiveLock mode in order to add them to a publication.
*/
static List *
OpenTableList(List *tables)
@@ -514,10 +522,12 @@ OpenTableList(List *tables)
*/
foreach(lc, tables)
{
- RangeVar *rv = lfirst_node(RangeVar, lc);
+ PublicationTable *t = lfirst(lc);
+ RangeVar *rv = castNode(RangeVar, t->relation);
bool recurse = rv->inh;
Relation rel;
Oid myrelid;
+ PublicationRelationInfo *pub_rel;
/* Allow query cancel in case this takes a long time */
CHECK_FOR_INTERRUPTS();
@@ -538,7 +548,11 @@ OpenTableList(List *tables)
continue;
}
- rels = lappend(rels, rel);
+ pub_rel = palloc(sizeof(PublicationRelationInfo));
+ pub_rel->relation = rel;
+ pub_rel->relid = myrelid;
+ pub_rel->columns = t->columns;
+ rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
/*
@@ -571,7 +585,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
- rels = lappend(rels, rel);
+ pub_rel = palloc(sizeof(PublicationRelationInfo));
+ pub_rel->relation = rel;
+ pub_rel->relid = childrelid;
+ pub_rel->columns = t->columns;
+ rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
}
@@ -592,9 +610,9 @@ CloseTableList(List *rels)
foreach(lc, rels)
{
- Relation rel = (Relation) lfirst(lc);
+ PublicationRelationInfo *pub_rel = (PublicationRelationInfo *)lfirst(lc);
- table_close(rel, NoLock);
+ table_close(pub_rel->relation, NoLock);
}
}
@@ -611,7 +629,8 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
foreach(lc, rels)
{
- Relation rel = (Relation) lfirst(lc);
+ PublicationRelationInfo *pub_rel = (PublicationRelationInfo *)lfirst(lc);
+ Relation rel = pub_rel->relation;
ObjectAddress obj;
/* Must be owner of the table or superuser. */
@@ -619,7 +638,7 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(rel->rd_rel->relkind),
RelationGetRelationName(rel));
- obj = publication_add_relation(pubid, rel, if_not_exists);
+ obj = publication_add_relation(pubid, pub_rel, if_not_exists);
if (stmt)
{
EventTriggerCollectSimpleCommand(obj, InvalidObjectAddress,
@@ -643,7 +662,8 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
foreach(lc, rels)
{
- Relation rel = (Relation) lfirst(lc);
+ PublicationRelationInfo *pubrel = (PublicationRelationInfo *) lfirst(lc);
+ Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 38251c2b8e..44f1a4e7e6 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4939,6 +4939,16 @@ _copyForeignKeyCacheInfo(const ForeignKeyCacheInfo *from)
return newnode;
}
+static PublicationTable *
+_copyPublicationTable(const PublicationTable *from)
+{
+ PublicationTable *newnode = makeNode(PublicationTable);
+
+ COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(columns);
+
+ return newnode;
+}
/*
* copyObjectImpl -- implementation of copyObject(); see nodes/nodes.h
@@ -5854,6 +5864,9 @@ copyObjectImpl(const void *from)
case T_PartitionCmd:
retval = _copyPartitionCmd(from);
break;
+ case T_PublicationTable:
+ retval = _copyPublicationTable(from);
+ break;
/*
* MISCELLANEOUS NODES
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 8a1762000c..b0f37b2ceb 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -3114,6 +3114,15 @@ _equalValue(const Value *a, const Value *b)
return true;
}
+static bool
+_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
+{
+ COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(columns);
+
+ return true;
+}
+
/*
* equal
* returns whether two nodes are equal
@@ -3862,6 +3871,9 @@ equal(const void *a, const void *b)
case T_PartitionCmd:
retval = _equalPartitionCmd(a, b);
break;
+ case T_PublicationTable:
+ retval = _equalPublicationTable(a, b);
+ break;
default:
elog(ERROR, "unrecognized node type: %d",
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 87561cbb6f..f04eb536c9 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -3821,6 +3821,15 @@ _outPartitionRangeDatum(StringInfo str, const PartitionRangeDatum *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outPublicationTable(StringInfo str, const PublicationTable *node)
+{
+ WRITE_NODE_TYPE("PUBLICATIONTABLE");
+
+ WRITE_NODE_FIELD(relation);
+ WRITE_NODE_FIELD(columns);
+}
+
/*
* outNode -
* converts a Node into ascii string and append it to 'str'
@@ -4520,6 +4529,9 @@ outNode(StringInfo str, const void *obj)
case T_PartitionRangeDatum:
_outPartitionRangeDatum(str, obj);
break;
+ case T_PublicationTable:
+ _outPublicationTable(str, obj);
+ break;
default:
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 0dd1ad7dfc..0be46165b4 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -2702,6 +2702,20 @@ _readPartitionRangeDatum(void)
READ_DONE();
}
+/*
+ * _readPublicationTable
+ */
+static PublicationTable *
+_readPublicationTable(void)
+{
+ READ_LOCALS(PublicationTable);
+
+ READ_NODE_FIELD(relation);
+ READ_NODE_FIELD(columns);
+
+ READ_DONE();
+}
+
/*
* parseNodeString
*
@@ -2973,6 +2987,8 @@ parseNodeString(void)
return_value = _readPartitionBoundSpec();
else if (MATCH("PARTITIONRANGEDATUM", 19))
return_value = _readPartitionRangeDatum();
+ else if (MATCH("PUBLICATIONTABLE", 16))
+ return_value = _readPublicationTable();
else
{
elog(ERROR, "badly formatted node string \"%.32s\"...", token);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 39a2849eba..2c9af95db8 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -426,14 +426,14 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
transform_element_list transform_type_list
TriggerTransitions TriggerReferencing
vacuum_relation_list opt_vacuum_relation_list
- drop_option_list
+ drop_option_list publication_table_list
%type <node> opt_routine_body
%type <groupclause> group_clause
%type <list> group_by_list
%type <node> group_by_item empty_grouping_set rollup_clause cube_clause
%type <node> grouping_sets_clause
-%type <node> opt_publication_for_tables publication_for_tables
+%type <node> opt_publication_for_tables publication_for_tables publication_table
%type <list> opt_fdw_options fdw_options
%type <defelt> fdw_option
@@ -9620,7 +9620,7 @@ opt_publication_for_tables:
;
publication_for_tables:
- FOR TABLE relation_expr_list
+ FOR TABLE publication_table_list
{
$$ = (Node *) $3;
}
@@ -9630,6 +9630,21 @@ publication_for_tables:
}
;
+publication_table_list:
+ publication_table
+ { $$ = list_make1($1); }
+ | publication_table_list ',' publication_table
+ { $$ = lappend($1, $3); }
+ ;
+
+publication_table: relation_expr opt_column_list
+ {
+ PublicationTable *n = makeNode(PublicationTable);
+ n->relation = $1;
+ n->columns = $2;
+ $$ = (Node *) n;
+ }
+ ;
/*****************************************************************************
*
@@ -9651,7 +9666,7 @@ AlterPublicationStmt:
n->options = $5;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name ADD_P TABLE relation_expr_list
+ | ALTER PUBLICATION name ADD_P TABLE publication_table_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
@@ -9659,7 +9674,7 @@ AlterPublicationStmt:
n->tableAction = DEFELEM_ADD;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name SET TABLE relation_expr_list
+ | ALTER PUBLICATION name SET TABLE publication_table_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
@@ -9667,7 +9682,7 @@ AlterPublicationStmt:
n->tableAction = DEFELEM_SET;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name DROP TABLE relation_expr_list
+ | ALTER PUBLICATION name DROP TABLE publication_table_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 9f5bf4b639..e5712d4f64 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,9 +29,9 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *att_map);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary, Bitmapset *att_map);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +398,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *att_map)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +410,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_map);
}
/*
@@ -442,7 +442,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary, Bitmapset *att_map)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -463,11 +463,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, att_map);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_map);
}
/*
@@ -536,7 +536,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -651,7 +651,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *att_map)
{
char *relname;
@@ -673,7 +673,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, att_map);
}
/*
@@ -749,20 +749,37 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary,
+ Bitmapset *att_map)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
bool isnull[MaxTupleAttributeNumber];
int i;
uint16 nliveatts = 0;
+ Bitmapset *idattrs = NULL;
+ bool replidentfull;
+ Form_pg_attribute att;
desc = RelationGetDescr(rel);
+ replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ if (!replidentfull)
+ idattrs = RelationGetIdentityKeyBitmap(rel);
+
for (i = 0; i < desc->natts; i++)
{
+ att = TupleDescAttr(desc, i);
if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
continue;
+ /*
+ * Do not increment count of attributes if not a part of column filters
+ * except for replica identity columns or if replica identity is full.
+ */
+ if (att_map != NULL && !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, att_map)
+ && !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, idattrs)
+ && !replidentfull)
+ continue;
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -800,6 +817,16 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
continue;
}
+ /*
+ * Do not send attribute data if it is not a part of column filters,
+ * except if it is a part of REPLICA IDENTITY or REPLICA IDENTITY is
+ * full, send the data.
+ */
+ if (att_map != NULL && !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, att_map)
+ && !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, idattrs)
+ && !replidentfull)
+ continue;
+
typtup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(att->atttypid));
if (!HeapTupleIsValid(typtup))
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
@@ -904,7 +931,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *att_map)
{
TupleDesc desc;
int i;
@@ -914,20 +941,34 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
desc = RelationGetDescr(rel);
+ /* fetch bitmap of REPLICATION IDENTITY attributes */
+ replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ if (!replidentfull)
+ idattrs = RelationGetIdentityKeyBitmap(rel);
+
/* send number of live attributes */
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ /* REPLICA IDENTITY FULL means all columns are sent as part of key. */
+ if (replidentfull || bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs))
+ {
+ nliveatts++;
+ continue;
+ }
+ /* Skip sending if not a part of column filter */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map))
continue;
nliveatts++;
}
pq_sendint16(out, nliveatts);
- /* fetch bitmap of REPLICATION IDENTITY attributes */
- replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
- if (!replidentfull)
- idattrs = RelationGetIdentityKeyBitmap(rel);
-
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -937,6 +978,13 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ /* Exlude filtered columns, REPLICA IDENTITY COLUMNS CAN'T BE EXCLUDED */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map) && !bms_is_member(att->attnum
+ - FirstLowInvalidHeapAttributeNumber, idattrs)
+ && !replidentfull)
+ continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
@@ -944,7 +992,6 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
flags |= LOGICALREP_IS_REPLICA_IDENTITY;
pq_sendbyte(out, flags);
-
/* attribute name */
pq_sendstring(out, NameStr(att->attname));
@@ -953,6 +1000,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* attribute mode */
pq_sendint32(out, att->atttypmod);
+
}
bms_free(idattrs);
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index f07983a43c..9bd834914b 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -111,6 +111,7 @@
#include "replication/origin.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -695,19 +696,27 @@ fetch_remote_table_info(char *nspname, char *relname,
LogicalRepRelation *lrel)
{
WalRcvExecResult *res;
+ WalRcvExecResult *res_pub;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
+ TupleTableSlot *slot_pub;
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid pubRow[] = {TEXTARRAYOID};
bool isnull;
- int natt;
+ int natt,i;
+ Datum *elems;
+ int nelems;
+ List *pub_columns = NIL;
+ ListCell *lc;
+ bool am_partition = false;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -737,6 +746,7 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetChar(slot_getattr(slot, 4, &isnull));
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
@@ -774,11 +784,79 @@ fetch_remote_table_info(char *nspname, char *relname,
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+
+ /*
+ * Now, fetch the values of publications' column filters
+ * For a partition, use pg_inherit to find the parent,
+ * as the pg_publication_rel contains only the topmost parent
+ * table entry in case the table is partitioned.
+ * Run a recursive query to iterate through all the parents
+ * of the partition and retreive the record for the parent
+ * that exists in pg_publication_rel.
+ */
+ resetStringInfo(&cmd);
+ if (!am_partition)
+ appendStringInfo(&cmd, "SELECT prattrs from pg_publication_rel"
+ " WHERE prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd, "WITH RECURSIVE t(inhparent) AS ( SELECT inhparent from pg_inherits where inhrelid = %u"
+ " UNION SELECT pg.inhparent from pg_inherits pg, t where inhrelid = t.inhparent)"
+ " SELECT prattrs from pg_publication_rel WHERE prrelid IN (SELECT inhparent from t)", lrel->remoteid);
+
+ res_pub = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(pubRow), pubRow);
+
+ if (res_pub->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch published columns info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, res_pub->err)));
+ slot_pub = MakeSingleTupleTableSlot(res_pub->tupledesc, &TTSOpsMinimalTuple);
+
+ while (tuplestore_gettupleslot(res_pub->tuplestore, true, false, slot_pub))
+ {
+ deconstruct_array(DatumGetArrayTypePCopy(slot_getattr(slot_pub, 1, &isnull)),
+ TEXTOID, -1, false, 'i',
+ &elems, NULL, &nelems);
+ for (i = 0; i < nelems; i++)
+ pub_columns = lappend(pub_columns, TextDatumGetCString(elems[i]));
+ ExecClearTuple(slot_pub);
+ }
+ ExecDropSingleTupleTableSlot(slot_pub);
+ walrcv_clear_result(res_pub);
+
+ /*
+ * Store the column names only if they are contained in column filter
+ * LogicalRepRelation will only contain attributes corresponding
+ * to those specficied in column filters.
+ */
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
+ char * rel_colname =
TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ bool found = false;
Assert(!isnull);
+ if (pub_columns != NIL)
+ {
+ foreach(lc, pub_columns)
+ {
+ char *pub_colname = lfirst(lc);
+ if(!strcmp(pub_colname, rel_colname))
+ {
+ found = true;
+ lrel->attnames[natt] = rel_colname;
+ break;
+ }
+ }
+ }
+ else
+ {
+ found = true;
+ lrel->attnames[natt] = rel_colname;
+ }
+ if (!found)
+ continue;
+
lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
@@ -829,8 +907,17 @@ copy_table(Relation rel)
/* Start copy on the publisher. */
initStringInfo(&cmd);
if (lrel.relkind == RELKIND_RELATION)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 14d737fd93..aa2a46a503 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,16 +15,19 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -81,11 +84,11 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *att_map);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
-
/*
* Entry in the map used to remember which relation schemas we sent.
*
@@ -130,6 +133,7 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+ Bitmapset *att_map;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -570,11 +574,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
}
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->att_map);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->att_map);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -587,7 +591,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *att_map)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -609,14 +614,24 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
-
+ /*
+ * Do not send type information if attribute is
+ * not present in column filter.
+ * XXX Allow sending type information for REPLICA
+ * IDENTITY COLUMNS with user created type.
+ * even when they are not mentioned in column filters.
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map))
+ continue;
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, att_map);
OutputPluginWrite(ctx, false);
}
@@ -693,7 +708,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->att_map);
OutputPluginWrite(ctx, true);
break;
}
@@ -722,7 +737,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->att_map);
OutputPluginWrite(ctx, true);
break;
}
@@ -1119,6 +1134,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
bool found;
+ Oid ancestor_id;
MemoryContext oldctx;
Assert(RelationSyncCache != NULL);
@@ -1139,8 +1155,8 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
- entry->map = NULL; /* will be set by maybe_send_schema() if
- * needed */
+ entry->att_map = NULL;
+ entry->map = NULL; /* will be set by maybe_send_schema() if needed */
}
/* Validate the entry */
@@ -1171,6 +1187,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
if (pub->alltables)
{
@@ -1181,7 +1198,6 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (!publish)
{
- bool ancestor_published = false;
/*
* For a partition, check if any of the ancestors are
@@ -1206,6 +1222,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
pub->oid))
{
ancestor_published = true;
+ ancestor_id = ancestor;
if (pub->pubviaroot)
publish_as_relid = ancestor;
}
@@ -1224,15 +1241,41 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ int nelems, i;
+ bool isnull;
+ Datum *elems;
+ HeapTuple pub_rel_tuple;
+ Datum pub_rel_cols;
+ List *columns = NIL;
+
+ if (ancestor_published)
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP, ObjectIdGetDatum(ancestor_id),
+ ObjectIdGetDatum(pub->oid));
+ else
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP, ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP, pub_rel_tuple, Anum_pg_publication_rel_prattrs, &isnull);
+ if (!isnull)
+ {
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ deconstruct_array(DatumGetArrayTypePCopy(pub_rel_cols),
+ TEXTOID, -1, false, 'i',
+ &elems, NULL, &nelems);
+ for (i = 0; i < nelems; i++)
+ columns = lappend(columns, TextDatumGetCString(elems[i]));
+ entry->att_map = get_table_columnset(publish_as_relid, columns, entry->att_map);
+ MemoryContextSwitchTo(oldctx);
+ }
+ ReleaseSysCache(pub_rel_tuple);
+ }
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
}
- if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
- break;
}
list_free(pubids);
@@ -1328,6 +1371,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->att_map);
+ entry->att_map = NULL;
if (entry->map)
{
/*
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index f332bad4d4..7bdc9bb9b8 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -83,6 +83,13 @@ typedef struct Publication
PublicationActions pubactions;
} Publication;
+typedef struct PublicationRelationInfo
+{
+ Oid relid;
+ Relation relation;
+ List *columns;
+} PublicationRelationInfo;
+
extern Publication *GetPublication(Oid pubid);
extern Publication *GetPublicationByName(const char *pubname, bool missing_ok);
extern List *GetRelationPublications(Oid relid);
@@ -108,7 +115,7 @@ extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern bool is_publishable_relation(Relation rel);
-extern ObjectAddress publication_add_relation(Oid pubid, Relation targetrel,
+extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelationInfo *targetrel,
bool if_not_exists);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index b5d5504cbb..d1d4eec2c0 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN
+ text prattrs[1]; /* Variable length field starts here */
+#endif
} FormData_pg_publication_rel;
/* ----------------
@@ -40,6 +43,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
*/
typedef FormData_pg_publication_rel *Form_pg_publication_rel;
+DECLARE_TOAST(pg_publication_rel, 8895, 8896);
DECLARE_UNIQUE_INDEX_PKEY(pg_publication_rel_oid_index, 6112, PublicationRelObjectIndexId, on pg_publication_rel using btree(oid oid_ops));
DECLARE_UNIQUE_INDEX(pg_publication_rel_prrelid_prpubid_index, 6113, PublicationRelPrrelidPrpubidIndexId, on pg_publication_rel using btree(prrelid oid_ops, prpubid oid_ops));
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 6a4d82f0a8..56d13ff022 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -490,6 +490,7 @@ typedef enum NodeTag
T_PartitionRangeDatum,
T_PartitionCmd,
T_VacuumRelation,
+ T_PublicationTable,
/*
* TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 7af13dee43..a9660e405c 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3624,6 +3624,12 @@ typedef struct AlterTSConfigurationStmt
bool missing_ok; /* for DROP - skip error if missing? */
} AlterTSConfigurationStmt;
+typedef struct PublicationTable
+{
+ NodeTag type;
+ RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
+} PublicationTable;
typedef struct CreatePublicationStmt
{
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 83741dcf42..709b4be916 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -207,11 +207,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *att_map);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *att_map);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -228,7 +228,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *att_map);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index b4faa1c123..b4c49fa32f 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -680,5 +680,6 @@ RelationGetSmgr(Relation rel)
/* routines in utils/cache/relcache.c */
extern void RelationIncrementReferenceCount(Relation rel);
extern void RelationDecrementReferenceCount(Relation rel);
+extern Bitmapset* get_table_columnset(Oid relid, List *columns, Bitmapset *att_map);
#endif /* REL_H */
diff --git a/src/test/subscription/t/021_column_filter.pl b/src/test/subscription/t/021_column_filter.pl
new file mode 100644
index 0000000000..334e14da95
--- /dev/null
+++ b/src/test/subscription/t/021_column_filter.pl
@@ -0,0 +1,143 @@
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test TRUNCATE
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 9;
+
+# setup
+
+my $node_publisher = PostgresNode->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgresNode->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int)");
+# Test with weird column names
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, B varchar, \"c'\" int)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+#Test replication with multi-level partition
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, \"c'\" int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+#Test create publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub1 FOR TABLE tab1(a, \"B\"), tab3(\"a'\",\"c'\"), test_part(a,b)");
+
+my $result = $node_publisher->safe_psql('postgres',
+ "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
+is($result, qq(tab1|{a,B}
+tab3|{a',c'}
+test_part|{a,b}), 'publication relation updated');
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+);
+#Initial sync
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab1 VALUES (1,2,3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 VALUES (1,2,3)");
+#Test for replication of partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (1,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (2,'bcd', '2021-07-03 11:12:13')");
+#Test for replication of multi-level partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (4,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (5,'bcd', '2021-07-03 11:12:13')");
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'insert on column c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab3");
+is($result, qq(1|3), 'insert on column b is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part");
+is($result, qq(1|abc\n2|bcd\n4|abc\n5|bcd), 'insert on all columns is replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab1 SET c = 5 where a = 1");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'update on column c is not replicated');
+
+#Test alter publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION"
+);
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 VALUES (1,'abc',3)");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2");
+is($result, qq(1|abc), 'insert on column c is not replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET c = 5 where a = 1");
+is($result, qq(1|abc), 'update on column c is not replicated');
+
+#Test error conditions
+my ($psql_rc, $psql_out, $psql_err) = $node_publisher->psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE test_part(b)");
+like($psql_err, qr/Column filter must include REPLICA IDENTITY columns/, 'Error when column filter does not contain REPLICA IDENTITY');
+
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE test_part DROP COLUMN b");
+$result = $node_publisher->safe_psql('postgres',
+ "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
+is($result, qq(tab1|{a,B}
+tab2|{a,b}
+tab3|{a',c'}), 'publication relation test_part removed');
--
2.17.2 (Apple Git-113)
On Thu, Sep 2, 2021 at 7:21 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
...
Also added some more tests. Please find attached a rebased and updated patch.
I fetched and applied the v4 patch.
It applied cleanly, and the build and make check was OK.
But I encountered some errors running the TAP subscription tests, as follows:
...
t/018_stream_subxact_abort.pl ...... ok
t/019_stream_subxact_ddl_abort.pl .. ok
t/020_messages.pl .................. ok
t/021_column_filter.pl ............. 1/9
# Failed test 'insert on column c is not replicated'
# at t/021_column_filter.pl line 126.
# got: ''
# expected: '1|abc'
# Failed test 'update on column c is not replicated'
# at t/021_column_filter.pl line 130.
# got: ''
# expected: '1|abc'
# Looks like you failed 2 tests of 9.
t/021_column_filter.pl ............. Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/9 subtests
t/021_twophase.pl .................. ok
t/022_twophase_cascade.pl .......... ok
t/023_twophase_stream.pl ........... ok
t/024_add_drop_pub.pl .............. ok
t/100_bugs.pl ...................... ok
Test Summary Report
-------------------
t/021_column_filter.pl (Wstat: 512 Tests: 9 Failed: 2)
Failed tests: 6-7
Non-zero exit status: 2
Files=26, Tests=263, 192 wallclock secs ( 0.57 usr 0.09 sys + 110.17
cusr 25.45 csys = 136.28 CPU)
Result: FAIL
make: *** [check] Error 1
------
Kind Regards,
Peter Smith.
Fujitsu Australia
On 2021-Sep-02, Rahila Syed wrote:
After thinking about this, I think it is best to remove the entire table
from publication,
if a column specified in the column filter is dropped from the table.
Hmm, I think it would be cleanest to give responsibility to the user: if
the column to be dropped is in the filter, then raise an error, aborting
the drop. Then it is up to them to figure out what to do.
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
"El destino baraja y nosotros jugamos" (A. Schopenhauer)
I think the WITH RECURSIVE query would be easier and more performant by
using pg_partition_tree and pg_partition_root.
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
"Porque Kim no hacía nada, pero, eso sí,
con extraordinario éxito" ("Kim", Kipling)
On Thu, Sep 2, 2021 at 2:19 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-02, Rahila Syed wrote:
After thinking about this, I think it is best to remove the entire table
from publication,
if a column specified in the column filter is dropped from the table.Hmm, I think it would be cleanest to give responsibility to the user: if
the column to be dropped is in the filter, then raise an error, aborting
the drop.
Do you think that will make sense if the user used Cascade (Alter
Table ... Drop Column ... Cascade)?
--
With Regards,
Amit Kapila.
On Thu, Sep 2, 2021 at 2:51 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
Do we want to consider that the columns specified in the filter must
not have NOT NULL constraint? Because, otherwise, the subscriber will
error out inserting such rows?I think you mean columns *not* specified in the filter must not have NOT NULL constraint
on the subscriber, as this will break during insert, as it will try to insert NULL for columns
not sent by the publisher.
I will look into fixing this. Probably this won't be a problem in
case the column is auto generated or contains a default value.I am not sure if this needs to be handled. Ideally, we need to prevent the subscriber tables from having a NOT NULL
constraint if the publisher uses column filters to publish the values of the table. There is no way
to do this at the time of creating a table on subscriber.As this involves querying the publisher for this information, it can be done at the time of initial table synchronization.
i.e error out if any of the subscribed tables has NOT NULL constraint on non-filter columns.
This will lead to the user dropping and recreating the subscription after removing the
NOT NULL constraint from the table.
I think the same can be achieved by doing nothing and letting the subscriber error out while inserting rows.
That makes sense and also it is quite possible that users don't have
such columns in the tables on subscribers. I guess we can add such a
recommendation in the docs instead of doing anything in the code.
Few comments:
============
1.
+
+ /*
+ * Cannot specify column filter when REPLICA IDENTITY IS FULL
+ * or if column filter does not contain REPLICA IDENITY columns
+ */
+ if (targetcols != NIL)
+ {
+ if (replidentfull)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot add relation \"%s\" to publication",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter with REPLICA IDENTITY FULL")));
Why do we want to have such a restriction for REPLICA IDENTITY FULL? I
think it is better to expand comments in that regards.
2.
@@ -839,7 +839,6 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("extra data after last expected column")));
-
fieldno = 0;
@@ -944,7 +992,6 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
flags |= LOGICALREP_IS_REPLICA_IDENTITY;
pq_sendbyte(out, flags);
-
/* attribute name */
pq_sendstring(out, NameStr(att->attname));
@@ -953,6 +1000,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* attribute mode */
pq_sendint32(out, att->atttypmod);
+
}
Spurious line removals and addition.
--
With Regards,
Amit Kapila.
On Sat, Sep 4, 2021 at 10:12 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Thu, Sep 2, 2021 at 2:51 AM Rahila Syed <rahilasyed90@gmail.com> wrote:
Do we want to consider that the columns specified in the filter must
not have NOT NULL constraint? Because, otherwise, the subscriber will
error out inserting such rows?I think you mean columns *not* specified in the filter must not have NOT NULL constraint
on the subscriber, as this will break during insert, as it will try to insert NULL for columns
not sent by the publisher.
I will look into fixing this. Probably this won't be a problem in
case the column is auto generated or contains a default value.I am not sure if this needs to be handled. Ideally, we need to prevent the subscriber tables from having a NOT NULL
constraint if the publisher uses column filters to publish the values of the table. There is no way
to do this at the time of creating a table on subscriber.As this involves querying the publisher for this information, it can be done at the time of initial table synchronization.
i.e error out if any of the subscribed tables has NOT NULL constraint on non-filter columns.
This will lead to the user dropping and recreating the subscription after removing the
NOT NULL constraint from the table.
I think the same can be achieved by doing nothing and letting the subscriber error out while inserting rows.That makes sense and also it is quite possible that users don't have
such columns in the tables on subscribers. I guess we can add such a
recommendation in the docs instead of doing anything in the code.Few comments:
============
Did you give any thoughts to my earlier suggestion related to syntax [1]/messages/by-id/CAA4eK1J9b_0_PMnJ2jq9E55bcbmTKdUmy6jPnkf1Zwy2jxah_g@mail.gmail.com?
[1]: /messages/by-id/CAA4eK1J9b_0_PMnJ2jq9E55bcbmTKdUmy6jPnkf1Zwy2jxah_g@mail.gmail.com
--
With Regards,
Amit Kapila.
On 2021-Sep-04, Amit Kapila wrote:
On Thu, Sep 2, 2021 at 2:19 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-02, Rahila Syed wrote:
After thinking about this, I think it is best to remove the entire table
from publication,
if a column specified in the column filter is dropped from the table.Hmm, I think it would be cleanest to give responsibility to the user: if
the column to be dropped is in the filter, then raise an error, aborting
the drop.Do you think that will make sense if the user used Cascade (Alter
Table ... Drop Column ... Cascade)?
... ugh. Since CASCADE is already defined to be a potentially-data-loss
operation, then that may be acceptable behavior. For sure the default
RESTRICT behavior shouldn't do it, though.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
On Sat, Sep 4, 2021 at 8:11 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-04, Amit Kapila wrote:
On Thu, Sep 2, 2021 at 2:19 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-02, Rahila Syed wrote:
After thinking about this, I think it is best to remove the entire table
from publication,
if a column specified in the column filter is dropped from the table.Hmm, I think it would be cleanest to give responsibility to the user: if
the column to be dropped is in the filter, then raise an error, aborting
the drop.Do you think that will make sense if the user used Cascade (Alter
Table ... Drop Column ... Cascade)?... ugh. Since CASCADE is already defined to be a potentially-data-loss
operation, then that may be acceptable behavior. For sure the default
RESTRICT behavior shouldn't do it, though.
That makes sense to me.
--
With Regards,
Amit Kapila.
Hi,
On Mon, Sep 6, 2021 at 8:53 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Sat, Sep 4, 2021 at 8:11 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:On 2021-Sep-04, Amit Kapila wrote:
On Thu, Sep 2, 2021 at 2:19 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:
On 2021-Sep-02, Rahila Syed wrote:
After thinking about this, I think it is best to remove the entire
table
from publication,
if a column specified in the column filter is dropped from thetable.
Hmm, I think it would be cleanest to give responsibility to the
user: if
the column to be dropped is in the filter, then raise an error,
aborting
the drop.
Do you think that will make sense if the user used Cascade (Alter
Table ... Drop Column ... Cascade)?... ugh. Since CASCADE is already defined to be a potentially-data-loss
operation, then that may be acceptable behavior. For sure the default
RESTRICT behavior shouldn't do it, though.That makes sense to me.
However, the default (RESTRICT) behaviour of DROP TABLE allows
removing the table from the publication. I have implemented the removal of
table from publication
on drop column (RESTRICT) on the same lines.
Although it does make sense to not allow dropping tables from publication,
in case of RESTRICT.
It makes me wonder how DROP TABLE (RESTRICT) allows cascading the drop
table to publication.
Did you give any thoughts to my earlier suggestion related to syntax [1]?
[1] -
/messages/by-id/CAA4eK1J9b_0_PMnJ2jq9E55bcbmTKdUmy6jPnkf1Zwy2jxah_g@mail.gmail.com
For future support to replicate all columns except (x,y,z), I think some
optional keywords like
COLUMNS NOT IN can be inserted between table name and (*columns_list*) as
follows.
ALTER PUBLICATION ADD TABLE tab_name [COLUMNS NOT IN] (x,y,z)
I think this should be possible as a future addition to proposed syntax in
the patch.
Please let me know your opinion.
Thank you,
Rahila Syed
On 2021-Sep-06, Rahila Syed wrote:
... ugh. Since CASCADE is already defined to be a
potentially-data-loss operation, then that may be acceptable
behavior. For sure the default RESTRICT behavior shouldn't do it,
though.That makes sense to me.
However, the default (RESTRICT) behaviour of DROP TABLE allows
removing the table from the publication. I have implemented the
removal of table from publication on drop column (RESTRICT) on the
same lines.
But dropping the table is quite a different action from dropping a
column, isn't it? If you drop a table, it seems perfectly reasonable
that it has to be removed from the publication -- essentially, when the
user drops a table, she is saying "I don't care about this table
anymore". However, if you drop just one column, that doesn't
necessarily mean that the user wants to stop publishing the whole table.
Removing the table from the publication in ALTER TABLE DROP COLUMN seems
like an overreaction. (Except perhaps in the special case were the
column being dropped is the only one that was being published.)
So let's discuss what should happen. If you drop a column, and the
column is filtered out, then it seems to me that the publication should
continue to have the table, and it should continue to filter out the
other columns that were being filtered out, regardless of CASCADE/RESTRICT.
However, if the column is *included* in the publication, and you drop
it, ISTM there are two cases:
1. If it's DROP CASCADE, then the list of columns to replicate should
continue to have all columns it previously had, so just remove the
column that is being dropped.
2. If it's DROP RESTRICT, then an error should be raised so that the
user can make a concious decision to remove the column from the filter
before dropping the column.
Did you give any thoughts to my earlier suggestion related to syntax [1]?
[1] /messages/by-id/CAA4eK1J9b_0_PMnJ2jq9E55bcbmTKdUmy6jPnkf1Zwy2jxah_g@mail.gmail.com
This is a great followup idea, after the current feature is committed.
There are a few things that have been reported in review comments; let's
get those addressed before adding more features on top.
I pushed the clerical part of this -- namely the addition of
PublicationTable node and PublicationRelInfo struct. I attach the part
of your v4 patch that I didn't include. It contains a couple of small
corrections, but I didn't do anything invasive (such as pgindent)
because that would perhaps cause you too much merge pain.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
Attachments:
v5-0001-Add-column-filtering-to-logical-replication.patchtext/x-diff; charset=utf-8Download
From 6a9e266cc8ce10f087a906bae2be7f6682ba19ac Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 6 Sep 2021 10:34:29 -0300
Subject: [PATCH v5] Add column filtering to logical replication
Add capability to specifiy column names while linking
the table to a publication, at the time of CREATE or ALTER
publication. This will allow replicating only the specified
columns. Other columns, if any, on the subscriber will be populated
locally or NULL will be inserted if no value is supplied for the column
by the upstream during INSERT.
This facilitates replication to a table on subscriber
containing only the subscribed/filtered columns.
If no filter is specified, all the columns are replicated.
REPLICA IDENTITY columns are always replicated.
Thus, prohibit adding relation to publication, if column filters
do not contain REPLICA IDENTITY.
Add a tap test for the same in src/test/subscription.
---
src/backend/access/common/relation.c | 21 +++++
src/backend/catalog/pg_publication.c | 56 +++++++++++-
src/backend/commands/publicationcmds.c | 8 +-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 3 +-
src/backend/replication/logical/proto.c | 90 ++++++++++++++-----
src/backend/replication/logical/tablesync.c | 97 +++++++++++++++++++--
src/backend/replication/pgoutput/pgoutput.c | 74 +++++++++++++---
src/include/catalog/pg_publication.h | 1 +
src/include/catalog/pg_publication_rel.h | 4 +
src/include/nodes/parsenodes.h | 1 +
src/include/replication/logicalproto.h | 6 +-
src/include/utils/rel.h | 1 +
14 files changed, 317 insertions(+), 47 deletions(-)
diff --git a/src/backend/access/common/relation.c b/src/backend/access/common/relation.c
index 632d13c1ea..59c1136f2e 100644
--- a/src/backend/access/common/relation.c
+++ b/src/backend/access/common/relation.c
@@ -21,12 +21,14 @@
#include "postgres.h"
#include "access/relation.h"
+#include "access/sysattr.h"
#include "access/xact.h"
#include "catalog/namespace.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/lmgr.h"
#include "utils/inval.h"
+#include "utils/lsyscache.h"
#include "utils/syscache.h"
@@ -215,3 +217,22 @@ relation_close(Relation relation, LOCKMODE lockmode)
if (lockmode != NoLock)
UnlockRelationId(&relid, lockmode);
}
+
+/*
+ * Return a bitmapset of attributes given the list of column names
+ */
+Bitmapset*
+get_table_columnset(Oid relid, List *columns, Bitmapset *att_map)
+{
+ ListCell *cell;
+ foreach(cell, columns)
+ {
+ const char *attname = lfirst(cell);
+ int attnum = get_attnum(relid, attname);
+
+ if (!bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, att_map))
+ att_map = bms_add_member(att_map,
+ attnum - FirstLowInvalidHeapAttributeNumber);
+ }
+ return att_map;
+}
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index d6fddd6efe..a78af8f807 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -47,8 +47,12 @@
* error if not.
*/
static void
-check_publication_add_relation(Relation targetrel)
+check_publication_add_relation(Relation targetrel, List *targetcols)
{
+ bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ Oid relid = RelationGetRelid(targetrel);
+ Bitmapset *idattrs;
+
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
@@ -73,6 +77,35 @@ check_publication_add_relation(Relation targetrel)
errmsg("cannot add relation \"%s\" to publication",
RelationGetRelationName(targetrel)),
errdetail("Temporary and unlogged relations cannot be replicated.")));
+
+ /*
+ * Cannot specify column filter when REPLICA IDENTITY IS FULL
+ * or if column filter does not contain REPLICA IDENITY columns
+ */
+ if (targetcols != NIL)
+ {
+ if (replidentfull)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot add relation \"%s\" to publication",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter with REPLICA IDENTITY FULL")));
+ else
+ {
+ Bitmapset *filtermap = NULL;
+ idattrs = RelationGetIndexAttrBitmap(targetrel, INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ filtermap = get_table_columnset(relid, targetcols, filtermap);
+ if (!bms_is_subset(idattrs, filtermap))
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot add relation \"%s\" to publication",
+ RelationGetRelationName(targetrel)),
+ errdetail("Column filter must include REPLICA IDENTITY columns")));
+ }
+ }
+ }
}
/*
@@ -153,6 +186,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
Publication *pub = GetPublication(pubid);
ObjectAddress myself,
referenced;
+ ListCell *lc;
+ List *target_cols = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -175,7 +210,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ foreach(lc, targetrel->columns)
+ {
+ char *colname;
+
+ colname = strVal(lfirst(lc));
+ target_cols = lappend(target_cols, colname);
+ }
+ check_publication_add_relation(targetrel->relation, target_cols);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -188,6 +230,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ values[Anum_pg_publication_rel_prattrs - 1] =
+ PointerGetDatum(strlist_to_textarray(target_cols));
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -196,7 +240,15 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
heap_freetuple(tup);
ObjectAddressSet(myself, PublicationRelRelationId, prrelid);
+ foreach(lc, target_cols)
+ {
+ int attnum;
+ attnum = get_attnum(relid, lfirst(lc));
+ /* Add dependency on the column */
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attnum);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 179a0ef982..3c71bfb1f2 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -413,7 +413,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
pubrel = palloc(sizeof(PublicationRelInfo));
pubrel->relation = oldrel;
-
+ /* This is not needed to delete a table */
+ pubrel->columns = NIL;
delrels = lappend(delrels, pubrel);
}
}
@@ -548,6 +549,8 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = NIL;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
@@ -581,8 +584,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
+
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = NIL;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index e308de170e..857129a371 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4945,6 +4945,7 @@ _copyPublicationTable(const PublicationTable *from)
PublicationTable *newnode = makeNode(PublicationTable);
COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 99440b40be..b0f37b2ceb 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -3118,6 +3118,7 @@ static bool
_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 6a0f46505c..e5cd7ea74f 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9637,10 +9637,11 @@ publication_table_list:
{ $$ = lappend($1, $3); }
;
-publication_table: relation_expr
+publication_table: relation_expr opt_column_list
{
PublicationTable *n = makeNode(PublicationTable);
n->relation = $1;
+ n->columns = $2;
$$ = (Node *) n;
}
;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 9f5bf4b639..f9e5179860 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,9 +29,9 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *att_map);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary, Bitmapset *att_map);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +398,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *att_map)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +410,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_map);
}
/*
@@ -442,7 +442,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary, Bitmapset *att_map)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -463,11 +463,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, att_map);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_map);
}
/*
@@ -536,7 +536,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -651,7 +651,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *att_map)
{
char *relname;
@@ -673,7 +673,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, att_map);
}
/*
@@ -749,20 +749,37 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary,
+ Bitmapset *att_map)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
bool isnull[MaxTupleAttributeNumber];
int i;
uint16 nliveatts = 0;
+ Bitmapset *idattrs = NULL;
+ bool replidentfull;
+ Form_pg_attribute att;
desc = RelationGetDescr(rel);
+ replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ if (!replidentfull)
+ idattrs = RelationGetIdentityKeyBitmap(rel);
+
for (i = 0; i < desc->natts; i++)
{
+ att = TupleDescAttr(desc, i);
if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
continue;
+ /*
+ * Do not increment count of attributes if not a part of column filters
+ * except for replica identity columns or if replica identity is full.
+ */
+ if (att_map != NULL && !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, att_map)
+ && !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, idattrs)
+ && !replidentfull)
+ continue;
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -800,6 +817,16 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
continue;
}
+ /*
+ * Do not send attribute data if it is not a part of column filters,
+ * except if it is a part of REPLICA IDENTITY or REPLICA IDENTITY is
+ * full, send the data.
+ */
+ if (att_map != NULL && !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, att_map)
+ && !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber, idattrs)
+ && !replidentfull)
+ continue;
+
typtup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(att->atttypid));
if (!HeapTupleIsValid(typtup))
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
@@ -904,7 +931,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *att_map)
{
TupleDesc desc;
int i;
@@ -914,20 +941,34 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
desc = RelationGetDescr(rel);
- /* send number of live attributes */
- for (i = 0; i < desc->natts; i++)
- {
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
- continue;
- nliveatts++;
- }
- pq_sendint16(out, nliveatts);
-
/* fetch bitmap of REPLICATION IDENTITY attributes */
replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
if (!replidentfull)
idattrs = RelationGetIdentityKeyBitmap(rel);
+ /* send number of live attributes */
+ for (i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ /* REPLICA IDENTITY FULL means all columns are sent as part of key. */
+ if (replidentfull || bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs))
+ {
+ nliveatts++;
+ continue;
+ }
+ /* Skip sending if not a part of column filter */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map))
+ continue;
+ nliveatts++;
+ }
+ pq_sendint16(out, nliveatts);
+
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -937,6 +978,13 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ /* Exlude filtered columns, REPLICA IDENTITY COLUMNS CAN'T BE EXCLUDED */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map) && !bms_is_member(att->attnum
+ - FirstLowInvalidHeapAttributeNumber, idattrs)
+ && !replidentfull)
+ continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index f07983a43c..9bd834914b 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -111,6 +111,7 @@
#include "replication/origin.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -695,19 +696,27 @@ fetch_remote_table_info(char *nspname, char *relname,
LogicalRepRelation *lrel)
{
WalRcvExecResult *res;
+ WalRcvExecResult *res_pub;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
+ TupleTableSlot *slot_pub;
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid pubRow[] = {TEXTARRAYOID};
bool isnull;
- int natt;
+ int natt,i;
+ Datum *elems;
+ int nelems;
+ List *pub_columns = NIL;
+ ListCell *lc;
+ bool am_partition = false;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -737,6 +746,7 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetChar(slot_getattr(slot, 4, &isnull));
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
@@ -774,11 +784,79 @@ fetch_remote_table_info(char *nspname, char *relname,
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+
+ /*
+ * Now, fetch the values of publications' column filters
+ * For a partition, use pg_inherit to find the parent,
+ * as the pg_publication_rel contains only the topmost parent
+ * table entry in case the table is partitioned.
+ * Run a recursive query to iterate through all the parents
+ * of the partition and retreive the record for the parent
+ * that exists in pg_publication_rel.
+ */
+ resetStringInfo(&cmd);
+ if (!am_partition)
+ appendStringInfo(&cmd, "SELECT prattrs from pg_publication_rel"
+ " WHERE prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd, "WITH RECURSIVE t(inhparent) AS ( SELECT inhparent from pg_inherits where inhrelid = %u"
+ " UNION SELECT pg.inhparent from pg_inherits pg, t where inhrelid = t.inhparent)"
+ " SELECT prattrs from pg_publication_rel WHERE prrelid IN (SELECT inhparent from t)", lrel->remoteid);
+
+ res_pub = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(pubRow), pubRow);
+
+ if (res_pub->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch published columns info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, res_pub->err)));
+ slot_pub = MakeSingleTupleTableSlot(res_pub->tupledesc, &TTSOpsMinimalTuple);
+
+ while (tuplestore_gettupleslot(res_pub->tuplestore, true, false, slot_pub))
+ {
+ deconstruct_array(DatumGetArrayTypePCopy(slot_getattr(slot_pub, 1, &isnull)),
+ TEXTOID, -1, false, 'i',
+ &elems, NULL, &nelems);
+ for (i = 0; i < nelems; i++)
+ pub_columns = lappend(pub_columns, TextDatumGetCString(elems[i]));
+ ExecClearTuple(slot_pub);
+ }
+ ExecDropSingleTupleTableSlot(slot_pub);
+ walrcv_clear_result(res_pub);
+
+ /*
+ * Store the column names only if they are contained in column filter
+ * LogicalRepRelation will only contain attributes corresponding
+ * to those specficied in column filters.
+ */
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
+ char * rel_colname =
TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ bool found = false;
Assert(!isnull);
+ if (pub_columns != NIL)
+ {
+ foreach(lc, pub_columns)
+ {
+ char *pub_colname = lfirst(lc);
+ if(!strcmp(pub_colname, rel_colname))
+ {
+ found = true;
+ lrel->attnames[natt] = rel_colname;
+ break;
+ }
+ }
+ }
+ else
+ {
+ found = true;
+ lrel->attnames[natt] = rel_colname;
+ }
+ if (!found)
+ continue;
+
lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
@@ -829,8 +907,17 @@ copy_table(Relation rel)
/* Start copy on the publisher. */
initStringInfo(&cmd);
if (lrel.relkind == RELKIND_RELATION)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 14d737fd93..bbaf8dcd5e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,16 +15,19 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -81,7 +84,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *att_map);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -130,6 +134,7 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+ Bitmapset *att_map;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -570,11 +575,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
}
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->att_map);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->att_map);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -587,7 +592,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *att_map)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -609,14 +615,24 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
-
+ /*
+ * Do not send type information if attribute is
+ * not present in column filter.
+ * XXX Allow sending type information for REPLICA
+ * IDENTITY COLUMNS with user created type.
+ * even when they are not mentioned in column filters.
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map))
+ continue;
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, att_map);
OutputPluginWrite(ctx, false);
}
@@ -693,7 +709,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->att_map);
OutputPluginWrite(ctx, true);
break;
}
@@ -722,7 +738,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->att_map);
OutputPluginWrite(ctx, true);
break;
}
@@ -1119,6 +1135,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
bool found;
+ Oid ancestor_id;
MemoryContext oldctx;
Assert(RelationSyncCache != NULL);
@@ -1139,8 +1156,8 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
- entry->map = NULL; /* will be set by maybe_send_schema() if
- * needed */
+ entry->att_map = NULL;
+ entry->map = NULL; /* will be set by maybe_send_schema() if needed */
}
/* Validate the entry */
@@ -1171,6 +1188,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
if (pub->alltables)
{
@@ -1181,7 +1199,6 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (!publish)
{
- bool ancestor_published = false;
/*
* For a partition, check if any of the ancestors are
@@ -1206,6 +1223,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
pub->oid))
{
ancestor_published = true;
+ ancestor_id = ancestor;
if (pub->pubviaroot)
publish_as_relid = ancestor;
}
@@ -1224,15 +1242,41 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ int nelems, i;
+ bool isnull;
+ Datum *elems;
+ HeapTuple pub_rel_tuple;
+ Datum pub_rel_cols;
+ List *columns = NIL;
+
+ if (ancestor_published)
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP, ObjectIdGetDatum(ancestor_id),
+ ObjectIdGetDatum(pub->oid));
+ else
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP, ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP, pub_rel_tuple, Anum_pg_publication_rel_prattrs, &isnull);
+ if (!isnull)
+ {
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ deconstruct_array(DatumGetArrayTypePCopy(pub_rel_cols),
+ TEXTOID, -1, false, 'i',
+ &elems, NULL, &nelems);
+ for (i = 0; i < nelems; i++)
+ columns = lappend(columns, TextDatumGetCString(elems[i]));
+ entry->att_map = get_table_columnset(publish_as_relid, columns, entry->att_map);
+ MemoryContextSwitchTo(oldctx);
+ }
+ ReleaseSysCache(pub_rel_tuple);
+ }
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
}
- if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
- break;
}
list_free(pubids);
@@ -1328,6 +1372,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->att_map);
+ entry->att_map = NULL;
if (entry->map)
{
/*
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 561266aa3e..809413938f 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -86,6 +86,7 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index b5d5504cbb..d1d4eec2c0 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN
+ text prattrs[1]; /* Variable length field starts here */
+#endif
} FormData_pg_publication_rel;
/* ----------------
@@ -40,6 +43,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
*/
typedef FormData_pg_publication_rel *Form_pg_publication_rel;
+DECLARE_TOAST(pg_publication_rel, 8895, 8896);
DECLARE_UNIQUE_INDEX_PKEY(pg_publication_rel_oid_index, 6112, PublicationRelObjectIndexId, on pg_publication_rel using btree(oid oid_ops));
DECLARE_UNIQUE_INDEX(pg_publication_rel_prrelid_prpubid_index, 6113, PublicationRelPrrelidPrpubidIndexId, on pg_publication_rel using btree(prrelid oid_ops, prpubid oid_ops));
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 743e5aa4f3..4c0c5afacd 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3628,6 +3628,7 @@ typedef struct PublicationTable
{
NodeTag type;
RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
typedef struct CreatePublicationStmt
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 83741dcf42..709b4be916 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -207,11 +207,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *att_map);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *att_map);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -228,7 +228,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *att_map);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index b4faa1c123..b4c49fa32f 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -680,5 +680,6 @@ RelationGetSmgr(Relation rel)
/* routines in utils/cache/relcache.c */
extern void RelationIncrementReferenceCount(Relation rel);
extern void RelationDecrementReferenceCount(Relation rel);
+extern Bitmapset* get_table_columnset(Oid relid, List *columns, Bitmapset *att_map);
#endif /* REL_H */
--
2.20.1
The code in get_rel_sync_entry() changes current memory context to
CacheMemoryContext, then does a bunch of memory-leaking things. This is
not good, because that memory context has to be very carefully managed
to avoid permanent memory leaks. I suppose you added that because you
need something -- probably entry->att_map -- to survive memory context
resets, but if so then you need to change to CacheMemoryContext only
when that memory is allocated, not other chunks of memory. I suspect
you can fix this by moving the MemoryContextSwitchTo() to just before
calling get_table_columnset; then all the leaky thinkgs are done in
whatever the original memory context is, which is fine.
(However, you also need to make sure that ->att_map is carefully freed
at the right time. It looks like this already happens in
rel_sync_cache_relation_cb, but is rel_sync_cache_publication_cb
correct? And in get_rel_sync_entry() itself, what if the entry already
has att_map -- should it be freed prior to allocating another one?)
By the way, I notice that your patch doesn't add documentation changes,
which are of course necessary.
/me is left wondering about PGOutputData->publication_names memory
handling ...
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
On 9/6/21 7:51 PM, Alvaro Herrera wrote:
On 2021-Sep-06, Rahila Syed wrote:
... ugh. Since CASCADE is already defined to be a
potentially-data-loss operation, then that may be acceptable
behavior. For sure the default RESTRICT behavior shouldn't do it,
though.That makes sense to me.
However, the default (RESTRICT) behaviour of DROP TABLE allows
removing the table from the publication. I have implemented the
removal of table from publication on drop column (RESTRICT) on the
same lines.But dropping the table is quite a different action from dropping a
column, isn't it? If you drop a table, it seems perfectly reasonable
that it has to be removed from the publication -- essentially, when the
user drops a table, she is saying "I don't care about this table
anymore". However, if you drop just one column, that doesn't
necessarily mean that the user wants to stop publishing the whole table.
Removing the table from the publication in ALTER TABLE DROP COLUMN seems
like an overreaction. (Except perhaps in the special case were the
column being dropped is the only one that was being published.)So let's discuss what should happen. If you drop a column, and the
column is filtered out, then it seems to me that the publication should
continue to have the table, and it should continue to filter out the
other columns that were being filtered out, regardless of CASCADE/RESTRICT.
However, if the column is *included* in the publication, and you drop
it, ISTM there are two cases:1. If it's DROP CASCADE, then the list of columns to replicate should
continue to have all columns it previously had, so just remove the
column that is being dropped.2. If it's DROP RESTRICT, then an error should be raised so that the
user can make a concious decision to remove the column from the filter
before dropping the column.
FWIW I think this is a sensible behavior.
I don't quite see why dropping a column should remove the table from
publication (assuming there are some columns still replicated).
Of course, it may break the subscriber (e.g. when there was NOT NULL
constraint on that column), but DROP RESTRICT (which I assume is the
default mode) prevents that. And if DROP CASCADE is specified, I think
it's reasonable to require the user to fix the fallout.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Here are some v5 review comments for your consideration:
------
1. src/backend/access/common/relation.c
@@ -215,3 +217,22 @@ relation_close(Relation relation, LOCKMODE lockmode)
if (lockmode != NoLock)
UnlockRelationId(&relid, lockmode);
}
+
+/*
+ * Return a bitmapset of attributes given the list of column names
+ */
+Bitmapset*
+get_table_columnset(Oid relid, List *columns, Bitmapset *att_map)
+{
IIUC that 3rd parameter (att_map) is always passed as NULL to
get_table_columnset function because you are constructing this
Bitmapset from scratch. Maybe I am mistaken, but if not then what is
the purpose of that att_map parameter?
------
2. src/backend/catalog/pg_publication.c
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot add relation \"%s\" to publication",
+ RelationGetRelationName(targetrel)),
+ errdetail("Column filter must include REPLICA IDENTITY columns")));
Is ERRCODE_INVALID_COLUMN_REFERENCE a more appropriate errcode to use here?
------
3. src/backend/catalog/pg_publication.c
+ else
+ {
+ Bitmapset *filtermap = NULL;
+ idattrs = RelationGetIndexAttrBitmap(targetrel,
INDEX_ATTR_BITMAP_IDENTITY_KEY);
The RelationGetIndexAttrBitmap function comment says "should be
bms_free'd when not needed anymore" but it seems the patch code is not
freeing idattrs when finished using it.
------
Kind Regards,
Peter Smith.
Fujitsu Australia
On Mon, Sep 6, 2021 at 9:25 PM Rahila Syed <rahilasyed90@gmail.com> wrote:
On Mon, Sep 6, 2021 at 8:53 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
Did you give any thoughts to my earlier suggestion related to syntax [1]?
[1] - /messages/by-id/CAA4eK1J9b_0_PMnJ2jq9E55bcbmTKdUmy6jPnkf1Zwy2jxah_g@mail.gmail.com
For future support to replicate all columns except (x,y,z), I think some optional keywords like
COLUMNS NOT IN can be inserted between table name and (*columns_list*) as follows.
ALTER PUBLICATION ADD TABLE tab_name [COLUMNS NOT IN] (x,y,z)
I think this should be possible as a future addition to proposed syntax in the patch.
Please let me know your opinion.
Right, I don't want you to implement that feature as part of this
patch but how about using COLUMNS or similar keyword in column filter
like ALTER PUBLICATION ADD TABLE tab_name COLUMNS (c1, c2, ...)? This
can make it easier to extend in the future.
--
With Regards,
Amit Kapila.
On Mon, Sep 6, 2021 at 11:21 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-06, Rahila Syed wrote:
... ugh. Since CASCADE is already defined to be a
potentially-data-loss operation, then that may be acceptable
behavior. For sure the default RESTRICT behavior shouldn't do it,
though.That makes sense to me.
However, the default (RESTRICT) behaviour of DROP TABLE allows
removing the table from the publication. I have implemented the
removal of table from publication on drop column (RESTRICT) on the
same lines.But dropping the table is quite a different action from dropping a
column, isn't it? If you drop a table, it seems perfectly reasonable
that it has to be removed from the publication -- essentially, when the
user drops a table, she is saying "I don't care about this table
anymore". However, if you drop just one column, that doesn't
necessarily mean that the user wants to stop publishing the whole table.
Removing the table from the publication in ALTER TABLE DROP COLUMN seems
like an overreaction. (Except perhaps in the special case were the
column being dropped is the only one that was being published.)So let's discuss what should happen. If you drop a column, and the
column is filtered out, then it seems to me that the publication should
continue to have the table, and it should continue to filter out the
other columns that were being filtered out, regardless of CASCADE/RESTRICT.
Yeah, for this case we don't need to do anything and I am not sure if
the patch is dropping tables in this case?
However, if the column is *included* in the publication, and you drop
it, ISTM there are two cases:1. If it's DROP CASCADE, then the list of columns to replicate should
continue to have all columns it previously had, so just remove the
column that is being dropped.
Note that for a somewhat similar case in the index (where the index
has an expression) we drop the index if one of the columns used in the
index expression is dropped, so we might want to just remove the
entire filter here instead of just removing the particular column or
remove the entire table from publication as Rahila is proposing.
I think removing just a particular column can break the replication
for Updates and Deletes if the removed column is part of replica
identity. If the entire filter is removed then also the entire
replication can break, so, I think Rahila's proposal is worth
considering.
2. If it's DROP RESTRICT, then an error should be raised so that the
user can make a concious decision to remove the column from the filter
before dropping the column.
I think one can argue for a similar case for index. If we are allowing
the index to be dropped even with RESTRICT then why not column filter?
Did you give any thoughts to my earlier suggestion related to syntax [1]?
[1] /messages/by-id/CAA4eK1J9b_0_PMnJ2jq9E55bcbmTKdUmy6jPnkf1Zwy2jxah_g@mail.gmail.com
This is a great followup idea, after the current feature is committed.
As mentioned in my response to Rahila, I was just thinking of using an
optional keyword Column for column filter so that we can extend it
later.
--
With Regards,
Amit Kapila.
On Tue, Sep 7, 2021 at 11:06 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Sep 6, 2021 at 11:21 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:On 2021-Sep-06, Rahila Syed wrote:
... ugh. Since CASCADE is already defined to be a
potentially-data-loss operation, then that may be acceptable
behavior. For sure the default RESTRICT behavior shouldn't do it,
though.That makes sense to me.
However, the default (RESTRICT) behaviour of DROP TABLE allows
removing the table from the publication. I have implemented the
removal of table from publication on drop column (RESTRICT) on the
same lines.But dropping the table is quite a different action from dropping a
column, isn't it? If you drop a table, it seems perfectly reasonable
that it has to be removed from the publication -- essentially, when the
user drops a table, she is saying "I don't care about this table
anymore". However, if you drop just one column, that doesn't
necessarily mean that the user wants to stop publishing the whole table.
Removing the table from the publication in ALTER TABLE DROP COLUMN seems
like an overreaction. (Except perhaps in the special case were the
column being dropped is the only one that was being published.)So let's discuss what should happen. If you drop a column, and the
column is filtered out, then it seems to me that the publication should
continue to have the table, and it should continue to filter out the
other columns that were being filtered out, regardless ofCASCADE/RESTRICT.
Yeah, for this case we don't need to do anything and I am not sure if
the patch is dropping tables in this case?However, if the column is *included* in the publication, and you drop
it, ISTM there are two cases:1. If it's DROP CASCADE, then the list of columns to replicate should
continue to have all columns it previously had, so just remove the
column that is being dropped.Note that for a somewhat similar case in the index (where the index
has an expression) we drop the index if one of the columns used in the
index expression is dropped, so we might want to just remove the
entire filter here instead of just removing the particular column or
remove the entire table from publication as Rahila is proposing.I think removing just a particular column can break the replication
for Updates and Deletes if the removed column is part of replica
identity.
But how this is specific to this patch, I think the behavior should be the
same as what is there now, I mean now also we can drop the columns which
are part of replica identity right.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Tue, Sep 7, 2021 at 11:26 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Sep 7, 2021 at 11:06 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Sep 6, 2021 at 11:21 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-06, Rahila Syed wrote:
... ugh. Since CASCADE is already defined to be a
potentially-data-loss operation, then that may be acceptable
behavior. For sure the default RESTRICT behavior shouldn't do it,
though.That makes sense to me.
However, the default (RESTRICT) behaviour of DROP TABLE allows
removing the table from the publication. I have implemented the
removal of table from publication on drop column (RESTRICT) on the
same lines.But dropping the table is quite a different action from dropping a
column, isn't it? If you drop a table, it seems perfectly reasonable
that it has to be removed from the publication -- essentially, when the
user drops a table, she is saying "I don't care about this table
anymore". However, if you drop just one column, that doesn't
necessarily mean that the user wants to stop publishing the whole table.
Removing the table from the publication in ALTER TABLE DROP COLUMN seems
like an overreaction. (Except perhaps in the special case were the
column being dropped is the only one that was being published.)So let's discuss what should happen. If you drop a column, and the
column is filtered out, then it seems to me that the publication should
continue to have the table, and it should continue to filter out the
other columns that were being filtered out, regardless of CASCADE/RESTRICT.Yeah, for this case we don't need to do anything and I am not sure if
the patch is dropping tables in this case?However, if the column is *included* in the publication, and you drop
it, ISTM there are two cases:1. If it's DROP CASCADE, then the list of columns to replicate should
continue to have all columns it previously had, so just remove the
column that is being dropped.Note that for a somewhat similar case in the index (where the index
has an expression) we drop the index if one of the columns used in the
index expression is dropped, so we might want to just remove the
entire filter here instead of just removing the particular column or
remove the entire table from publication as Rahila is proposing.I think removing just a particular column can break the replication
for Updates and Deletes if the removed column is part of replica
identity.But how this is specific to this patch, I think the behavior should be the same as what is there now, I mean now also we can drop the columns which are part of replica identity right.
Sure, but we drop replica identity and corresponding index as well.
The patch ensures that replica identity columns must be part of the
column filter and now that restriction won't hold anymore. I think if
we want to retain that restriction then it is better to either remove
the entire filter or remove the entire table. Anyway, the main point
was that if we can remove the index/replica identity, it seems like
there should be the same treatment for column filter.
Another related point that occurred to me is that if the user changes
replica identity then probably we should ensure that the column
filters for the table still holds the creteria or maybe we need to
remove the filter in that case as well. I am not sure if the patch is
already doing something about it and if not then isn't it better to do
something about it?
--
With Regards,
Amit Kapila.
On Mon, Sep 6, 2021, at 2:51 PM, Alvaro Herrera wrote:
I pushed the clerical part of this -- namely the addition of
PublicationTable node and PublicationRelInfo struct. I attach the part
of your v4 patch that I didn't include. It contains a couple of small
corrections, but I didn't do anything invasive (such as pgindent)
because that would perhaps cause you too much merge pain.
While updating the row filter patch [1]/messages/by-id/0c2464d4-65f4-4d91-aeb2-c5584c1350f5@www.fastmail.com (because it also uses these
structures), I noticed that you use PublicationRelInfo as a type name instead
of PublicationRelationInfo. I choose the latter because there is already a data
structure named PublicationRelInfo (pg_dump.h). It is a client-side data
structure but I doesn't seem a good practice to duplicate data structure names
over the same code base.
[1]: /messages/by-id/0c2464d4-65f4-4d91-aeb2-c5584c1350f5@www.fastmail.com
--
Euler Taveira
EDB https://www.enterprisedb.com/
On Mon, Sep 6, 2021 at 11:21 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
I pushed the clerical part of this -- namely the addition of
PublicationTable node and PublicationRelInfo struct.
One point to note here is that we are developing a generic grammar for
publications where not only tables but other objects like schema,
sequences, etc. can be specified, see [1]/messages/by-id/877603.1629120678@sss.pgh.pa.us. So, there is some overlap
in the grammar modifications being made by this patch and the work
being done in that other thread. As both the patches are being
developed at the same time, it might be better to be in sync,
otherwise, some of the work needs to be changed. I can see that in the
patch [2]postgresql.org/message-id/CALDaNm0OudeDeFN7bSWPro0hgKx%3D1zPgcNFWnvU_G6w3mDPX0Q%40mail.gmail.com (v28-0002-Added-schema-level-support-for-publication) being
developed there the changes made by the above commit needs to be
changed again to represent a generic object for publication. It is
possible that we can do it some other way but I think it would be
better to coordinate the work in both threads. The other approach is
to continue independently and the later patch can adapt to the earlier
one which is fine too but it might be more work for the later one.
[1]: /messages/by-id/877603.1629120678@sss.pgh.pa.us
[2]: postgresql.org/message-id/CALDaNm0OudeDeFN7bSWPro0hgKx%3D1zPgcNFWnvU_G6w3mDPX0Q%40mail.gmail.com
--
With Regards,
Amit Kapila.
On 2021-Sep-15, Amit Kapila wrote:
On Mon, Sep 6, 2021 at 11:21 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
I pushed the clerical part of this -- namely the addition of
PublicationTable node and PublicationRelInfo struct.One point to note here is that we are developing a generic grammar for
publications where not only tables but other objects like schema,
sequences, etc. can be specified, see [1]. So, there is some overlap
in the grammar modifications being made by this patch and the work
being done in that other thread.
Oh rats. I was not aware of that thread, or indeed of the fact that
adding multiple object types to publications was being considered.
I do see that 0002 there contains gram.y changes, but AFAICS those
changes don't allow specifying a column list for a table, so there are
some changes needed in that patch for that either way.
I agree that it's better to move forward in unison.
I noticed that 0002 in that other patch uses a void * pointer in
PublicationObjSpec that "could be either RangeVar or String", which
strikes me as a really bad idea. (Already discussed in some other
thread recently, maybe this one or the row filtering one.)
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
On Wed, Sep 15, 2021 at 5:20 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-15, Amit Kapila wrote:
On Mon, Sep 6, 2021 at 11:21 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
I pushed the clerical part of this -- namely the addition of
PublicationTable node and PublicationRelInfo struct.One point to note here is that we are developing a generic grammar for
publications where not only tables but other objects like schema,
sequences, etc. can be specified, see [1]. So, there is some overlap
in the grammar modifications being made by this patch and the work
being done in that other thread.Oh rats. I was not aware of that thread, or indeed of the fact that
adding multiple object types to publications was being considered.I do see that 0002 there contains gram.y changes, but AFAICS those
changes don't allow specifying a column list for a table, so there are
some changes needed in that patch for that either way.I agree that it's better to move forward in unison.
I noticed that 0002 in that other patch uses a void * pointer in
PublicationObjSpec that "could be either RangeVar or String", which
strikes me as a really bad idea. (Already discussed in some other
thread recently, maybe this one or the row filtering one.)
I have extracted the parser code and attached it here, so that it will
be easy to go through. We wanted to support the following syntax as in
[1]: /messages/by-id/877603.1629120678@sss.pgh.pa.us
CREATE PUBLICATION pub1 FOR
TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2,
SEQUENCE seq1,seq2, ALL SEQUENCES IN SCHEMA s3,s4;
Columns can be added to PublicationObjSpec data structure. The patch
Generic_object_type_parser_002_table_schema_publication.patch has the
changes that were used to handle the parsing. Schema and Relation both
are different objects, schema is of string type and relation is of
RangeVar type. While parsing, schema name is parsed in string format
and relation is parsed and converted to rangevar type, these objects
will be then handled accordingly during post processing. That is the
reason it used void * type which could hold both RangeVar and String.
Thoughts?
[1]: /messages/by-id/877603.1629120678@sss.pgh.pa.us
Regards,
Vignesh
Attachments:
Generic_object_type_parser_001_table_publication.patchtext/x-patch; charset=US-ASCII; name=Generic_object_type_parser_001_table_publication.patchDownload
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index d6fddd6efe..2a2fe03c13 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -141,14 +141,14 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
* Insert new publication / relation mapping.
*/
ObjectAddress
-publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
+publication_add_relation(Oid pubid, Relation targetrel,
bool if_not_exists)
{
Relation rel;
HeapTuple tup;
Datum values[Natts_pg_publication_rel];
bool nulls[Natts_pg_publication_rel];
- Oid relid = RelationGetRelid(targetrel->relation);
+ Oid relid = RelationGetRelid(targetrel);
Oid prrelid;
Publication *pub = GetPublication(pubid);
ObjectAddress myself,
@@ -172,10 +172,10 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_OBJECT),
errmsg("relation \"%s\" is already member of publication \"%s\"",
- RelationGetRelationName(targetrel->relation), pub->name)));
+ RelationGetRelationName(targetrel), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ check_publication_add_relation(targetrel);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -209,7 +209,7 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
table_close(rel, RowExclusiveLock);
/* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcache(targetrel->relation);
+ CacheInvalidateRelcache(targetrel);
return myself;
}
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 30929da1f5..f5fd463c15 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -34,6 +34,7 @@
#include "commands/publicationcmds.h"
#include "funcapi.h"
#include "miscadmin.h"
+#include "nodes/makefuncs.h"
#include "utils/acl.h"
#include "utils/array.h"
#include "utils/builtins.h"
@@ -138,6 +139,34 @@ parse_publication_options(ParseState *pstate,
}
}
+/*
+ * Convert the PublicationObjSpecType list into rangevar list.
+ */
+static void
+ObjectsInPublicationToOids(List *pubobjspec_list, ParseState *pstate,
+ List **rels)
+{
+ ListCell *cell;
+ PublicationObjSpec *pubobj;
+
+ if (!pubobjspec_list)
+ return;
+
+ foreach(cell, pubobjspec_list)
+ {
+ Node *node;
+
+ pubobj = (PublicationObjSpec *) lfirst(cell);
+ node = (Node *) pubobj->object;
+
+ if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLE)
+ {
+ if (IsA(node, RangeVar))
+ *rels = lappend(*rels, (RangeVar *) node);
+ }
+ }
+}
+
/*
* Create new publication.
*/
@@ -155,6 +184,7 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
bool publish_via_partition_root_given;
bool publish_via_partition_root;
AclResult aclresult;
+ List *relations = NIL;
/* must have CREATE privilege on database */
aclresult = pg_database_aclcheck(MyDatabaseId, GetUserId(), ACL_CREATE);
@@ -224,13 +254,14 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
/* Make the changes visible. */
CommandCounterIncrement();
- if (stmt->tables)
+ ObjectsInPublicationToOids(stmt->pubobjects, pstate, &relations);
+ if (relations != NIL)
{
List *rels;
- Assert(list_length(stmt->tables) > 0);
+ Assert(list_length(relations) > 0);
- rels = OpenTableList(stmt->tables);
+ rels = OpenTableList(relations);
PublicationAddTables(puboid, rels, true, NULL);
CloseTableList(rels);
}
@@ -360,7 +391,7 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
*/
static void
AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
- HeapTuple tup)
+ HeapTuple tup, List *tables)
{
List *rels = NIL;
Form_pg_publication pubform = (Form_pg_publication) GETSTRUCT(tup);
@@ -374,13 +405,13 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
NameStr(pubform->pubname)),
errdetail("Tables cannot be added to or dropped from FOR ALL TABLES publications.")));
- Assert(list_length(stmt->tables) > 0);
+ Assert(list_length(tables) > 0);
- rels = OpenTableList(stmt->tables);
+ rels = OpenTableList(tables);
- if (stmt->tableAction == DEFELEM_ADD)
+ if (stmt->action == DEFELEM_ADD)
PublicationAddTables(pubid, rels, false, stmt);
- else if (stmt->tableAction == DEFELEM_DROP)
+ else if (stmt->action == DEFELEM_DROP)
PublicationDropTables(pubid, rels, false);
else /* DEFELEM_SET */
{
@@ -398,10 +429,9 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
foreach(newlc, rels)
{
- PublicationRelInfo *newpubrel;
+ Relation newrel = (Relation) lfirst(newlc);
- newpubrel = (PublicationRelInfo *) lfirst(newlc);
- if (RelationGetRelid(newpubrel->relation) == oldrelid)
+ if (RelationGetRelid(newrel) == oldrelid)
{
found = true;
break;
@@ -410,16 +440,10 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
/* Not yet in the list, open it and add to the list */
if (!found)
{
- Relation oldrel;
- PublicationRelInfo *pubrel;
-
- /* Wrap relation into PublicationRelInfo */
- oldrel = table_open(oldrelid, ShareUpdateExclusiveLock);
+ Relation oldrel = table_open(oldrelid,
+ ShareUpdateExclusiveLock);
- pubrel = palloc(sizeof(PublicationRelInfo));
- pubrel->relation = oldrel;
-
- delrels = lappend(delrels, pubrel);
+ delrels = lappend(delrels, oldrel);
}
}
@@ -450,6 +474,7 @@ AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt)
Relation rel;
HeapTuple tup;
Form_pg_publication pubform;
+ List *relations = NIL;
rel = table_open(PublicationRelationId, RowExclusiveLock);
@@ -469,10 +494,12 @@ AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt)
aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_PUBLICATION,
stmt->pubname);
+ ObjectsInPublicationToOids(stmt->pubobjects, pstate, &relations);
+
if (stmt->options)
AlterPublicationOptions(pstate, stmt, rel, tup);
else
- AlterPublicationTables(stmt, rel, tup);
+ AlterPublicationTables(stmt, rel, tup, relations);
/* Cleanup. */
heap_freetuple(tup);
@@ -540,7 +567,7 @@ RemovePublicationById(Oid pubid)
/*
* Open relations specified by a PublicationTable list.
- * In the returned list of PublicationRelInfo, tables are locked
+ * In the returned list of Relation, tables are locked
* in ShareUpdateExclusiveLock mode in order to add them to a publication.
*/
static List *
@@ -555,16 +582,15 @@ OpenTableList(List *tables)
*/
foreach(lc, tables)
{
- PublicationTable *t = lfirst_node(PublicationTable, lc);
- bool recurse = t->relation->inh;
+ RangeVar *rv = lfirst_node(RangeVar, lc);
+ bool recurse = rv->inh;
Relation rel;
Oid myrelid;
- PublicationRelInfo *pub_rel;
/* Allow query cancel in case this takes a long time */
CHECK_FOR_INTERRUPTS();
- rel = table_openrv(t->relation, ShareUpdateExclusiveLock);
+ rel = table_openrv(rv, ShareUpdateExclusiveLock);
myrelid = RelationGetRelid(rel);
/*
@@ -580,9 +606,7 @@ OpenTableList(List *tables)
continue;
}
- pub_rel = palloc(sizeof(PublicationRelInfo));
- pub_rel->relation = rel;
- rels = lappend(rels, pub_rel);
+ rels = lappend(rels, rel);
relids = lappend_oid(relids, myrelid);
/*
@@ -615,9 +639,7 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
- pub_rel = palloc(sizeof(PublicationRelInfo));
- pub_rel->relation = rel;
- rels = lappend(rels, pub_rel);
+ rels = lappend(rels, rel);
relids = lappend_oid(relids, childrelid);
}
}
@@ -638,10 +660,9 @@ CloseTableList(List *rels)
foreach(lc, rels)
{
- PublicationRelInfo *pub_rel;
+ Relation rel = (Relation) lfirst(lc);
- pub_rel = (PublicationRelInfo *) lfirst(lc);
- table_close(pub_rel->relation, NoLock);
+ table_close(rel, NoLock);
}
}
@@ -658,8 +679,7 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
foreach(lc, rels)
{
- PublicationRelInfo *pub_rel = (PublicationRelInfo *) lfirst(lc);
- Relation rel = pub_rel->relation;
+ Relation rel = (Relation) lfirst(lc);
ObjectAddress obj;
/* Must be owner of the table or superuser. */
@@ -667,7 +687,7 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(rel->rd_rel->relkind),
RelationGetRelationName(rel));
- obj = publication_add_relation(pubid, pub_rel, if_not_exists);
+ obj = publication_add_relation(pubid, rel, if_not_exists);
if (stmt)
{
EventTriggerCollectSimpleCommand(obj, InvalidObjectAddress,
@@ -691,8 +711,7 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
foreach(lc, rels)
{
- PublicationRelInfo *pubrel = (PublicationRelInfo *) lfirst(lc);
- Relation rel = pubrel->relation;
+ Relation rel = (Relation) lfirst(lc);
Oid relid = RelationGetRelid(rel);
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 228387eaee..ade93023b8 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4817,7 +4817,7 @@ _copyCreatePublicationStmt(const CreatePublicationStmt *from)
COPY_STRING_FIELD(pubname);
COPY_NODE_FIELD(options);
- COPY_NODE_FIELD(tables);
+ COPY_NODE_FIELD(pubobjects);
COPY_SCALAR_FIELD(for_all_tables);
return newnode;
@@ -4830,9 +4830,9 @@ _copyAlterPublicationStmt(const AlterPublicationStmt *from)
COPY_STRING_FIELD(pubname);
COPY_NODE_FIELD(options);
- COPY_NODE_FIELD(tables);
+ COPY_NODE_FIELD(pubobjects);
COPY_SCALAR_FIELD(for_all_tables);
- COPY_SCALAR_FIELD(tableAction);
+ COPY_SCALAR_FIELD(action);
return newnode;
}
@@ -4958,12 +4958,14 @@ _copyForeignKeyCacheInfo(const ForeignKeyCacheInfo *from)
return newnode;
}
-static PublicationTable *
-_copyPublicationTable(const PublicationTable *from)
+static PublicationObjSpec *
+_copyPublicationObject(const PublicationObjSpec *from)
{
- PublicationTable *newnode = makeNode(PublicationTable);
+ PublicationObjSpec *newnode = makeNode(PublicationObjSpec);
- COPY_NODE_FIELD(relation);
+ COPY_SCALAR_FIELD(pubobjtype);
+ COPY_NODE_FIELD(object);
+ COPY_LOCATION_FIELD(location);
return newnode;
}
@@ -5887,8 +5889,8 @@ copyObjectImpl(const void *from)
case T_PartitionCmd:
retval = _copyPartitionCmd(from);
break;
- case T_PublicationTable:
- retval = _copyPublicationTable(from);
+ case T_PublicationObjSpec:
+ retval = _copyPublicationObject(from);
break;
/*
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 800f588b5c..06917598f4 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2302,7 +2302,7 @@ _equalCreatePublicationStmt(const CreatePublicationStmt *a,
{
COMPARE_STRING_FIELD(pubname);
COMPARE_NODE_FIELD(options);
- COMPARE_NODE_FIELD(tables);
+ COMPARE_NODE_FIELD(pubobjects);
COMPARE_SCALAR_FIELD(for_all_tables);
return true;
@@ -2314,9 +2314,9 @@ _equalAlterPublicationStmt(const AlterPublicationStmt *a,
{
COMPARE_STRING_FIELD(pubname);
COMPARE_NODE_FIELD(options);
- COMPARE_NODE_FIELD(tables);
+ COMPARE_NODE_FIELD(pubobjects);
COMPARE_SCALAR_FIELD(for_all_tables);
- COMPARE_SCALAR_FIELD(tableAction);
+ COMPARE_SCALAR_FIELD(action);
return true;
}
@@ -3134,9 +3134,9 @@ _equalBitString(const BitString *a, const BitString *b)
}
static bool
-_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
+_equalPublicationObject(const PublicationObjSpec *a, const PublicationObjSpec *b)
{
- COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(object);
return true;
}
@@ -3894,8 +3894,8 @@ equal(const void *a, const void *b)
case T_PartitionCmd:
retval = _equalPartitionCmd(a, b);
break;
- case T_PublicationTable:
- retval = _equalPublicationTable(a, b);
+ case T_PublicationObjSpec:
+ retval = _equalPublicationObject(a, b);
break;
default:
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index e3068a374e..d3d7683511 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -256,6 +256,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
PartitionSpec *partspec;
PartitionBoundSpec *partboundspec;
RoleSpec *rolespec;
+ PublicationObjSpec *publicationobjectspec;
struct SelectLimit *selectlimit;
SetQuantifier setquantifier;
struct GroupClause *groupclause;
@@ -425,14 +426,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
transform_element_list transform_type_list
TriggerTransitions TriggerReferencing
vacuum_relation_list opt_vacuum_relation_list
- drop_option_list publication_table_list
+ drop_option_list pub_obj_list
%type <node> opt_routine_body
%type <groupclause> group_clause
%type <list> group_by_list
%type <node> group_by_item empty_grouping_set rollup_clause cube_clause
%type <node> grouping_sets_clause
-%type <node> opt_publication_for_tables publication_for_tables publication_table
%type <list> opt_fdw_options fdw_options
%type <defelt> fdw_option
@@ -554,6 +554,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <node> var_value zone_value
%type <rolespec> auth_ident RoleSpec opt_granted_by
+%type <publicationobjectspec> PublicationObjSpec
%type <keyword> unreserved_keyword type_func_name_keyword
%type <keyword> col_name_keyword reserved_keyword
%type <keyword> bare_label_keyword
@@ -9591,69 +9592,75 @@ AlterOwnerStmt: ALTER AGGREGATE aggregate_with_argtypes OWNER TO RoleSpec
/*****************************************************************************
*
- * CREATE PUBLICATION name [ FOR TABLE ] [ WITH options ]
+ * CREATE PUBLICATION name [WITH options]
+ *
+ * CREATE PUBLICATION FOR ALL TABLES [WITH options]
+ *
+ * CREATE PUBLICATION FOR pub_obj [, pub_obj2] [WITH options]
+ *
+ * pub_obj is one of:
+ *
+ * TABLE table [, table2]
+ * ALL TABLES IN SCHEMA schema [, schema2]
*
*****************************************************************************/
CreatePublicationStmt:
- CREATE PUBLICATION name opt_publication_for_tables opt_definition
+ CREATE PUBLICATION name opt_definition
{
CreatePublicationStmt *n = makeNode(CreatePublicationStmt);
n->pubname = $3;
- n->options = $5;
- if ($4 != NULL)
- {
- /* FOR TABLE */
- if (IsA($4, List))
- n->tables = (List *)$4;
- /* FOR ALL TABLES */
- else
- n->for_all_tables = true;
- }
+ n->options = $4;
$$ = (Node *)n;
}
- ;
-
-opt_publication_for_tables:
- publication_for_tables { $$ = $1; }
- | /* EMPTY */ { $$ = NULL; }
- ;
-
-publication_for_tables:
- FOR TABLE publication_table_list
+ | CREATE PUBLICATION name FOR ALL TABLES opt_definition
{
- $$ = (Node *) $3;
+ CreatePublicationStmt *n = makeNode(CreatePublicationStmt);
+ n->pubname = $3;
+ n->options = $7;
+ n->for_all_tables = true;
+ $$ = (Node *)n;
}
- | FOR ALL TABLES
+ | CREATE PUBLICATION name FOR TABLE pub_obj_list opt_definition
{
- $$ = (Node *) makeInteger(true);
+ CreatePublicationStmt *n = makeNode(CreatePublicationStmt);
+ n->pubname = $3;
+ n->options = $7;
+ n->pubobjects = (List *)$6;
+ $$ = (Node *)n;
}
;
-publication_table_list:
- publication_table
- { $$ = list_make1($1); }
- | publication_table_list ',' publication_table
- { $$ = lappend($1, $3); }
+/* FOR TABLE and FOR ALL TABLES IN SCHEMA specifications */
+PublicationObjSpec: relation_expr
+ {
+ PublicationObjSpec *n = makeNode(PublicationObjSpec);
+ n->object = (Node*)$1;
+ n->pubobjtype = PUBLICATIONOBJ_TABLE;
+ $$ = n;
+ }
;
-publication_table: relation_expr
- {
- PublicationTable *n = makeNode(PublicationTable);
- n->relation = $1;
- $$ = (Node *) n;
- }
+pub_obj_list: PublicationObjSpec
+ { $$ = list_make1($1); }
+ | pub_obj_list ',' PublicationObjSpec
+ { $$ = lappend($1, $3); }
;
/*****************************************************************************
*
* ALTER PUBLICATION name SET ( options )
*
- * ALTER PUBLICATION name ADD TABLE table [, table2]
+ * ALTER PUBLICATION name ADD pub_obj [, pub_obj ...]
+ *
+ * ALTER PUBLICATION name DROP pub_obj [, pub_obj ...]
+ *
+ * ALTER PUBLICATION name SET pub_obj [, pub_obj ...]
*
- * ALTER PUBLICATION name DROP TABLE table [, table2]
+ * pub_obj is one of:
*
- * ALTER PUBLICATION name SET TABLE table [, table2]
+ * TABLE table_name [, table_name ...]
+ * ALL TABLES IN SCHEMA schema_name [, schema_name ...]
*
*****************************************************************************/
@@ -9665,28 +9672,28 @@ AlterPublicationStmt:
n->options = $5;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name ADD_P TABLE publication_table_list
+ | ALTER PUBLICATION name ADD_P TABLE pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
- n->tables = $6;
- n->tableAction = DEFELEM_ADD;
+ n->pubobjects = $6;
+ n->action = DEFELEM_ADD;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name SET TABLE publication_table_list
+ | ALTER PUBLICATION name SET TABLE pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
- n->tables = $6;
- n->tableAction = DEFELEM_SET;
+ n->pubobjects = $6;
+ n->action = DEFELEM_SET;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name DROP TABLE publication_table_list
+ | ALTER PUBLICATION name DROP TABLE pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
- n->tables = $6;
- n->tableAction = DEFELEM_DROP;
+ n->pubobjects = $6;
+ n->action = DEFELEM_DROP;
$$ = (Node *)n;
}
;
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 561266aa3e..f332bad4d4 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -83,11 +83,6 @@ typedef struct Publication
PublicationActions pubactions;
} Publication;
-typedef struct PublicationRelInfo
-{
- Relation relation;
-} PublicationRelInfo;
-
extern Publication *GetPublication(Oid pubid);
extern Publication *GetPublicationByName(const char *pubname, bool missing_ok);
extern List *GetRelationPublications(Oid relid);
@@ -113,7 +108,7 @@ extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern bool is_publishable_relation(Relation rel);
-extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
+extern ObjectAddress publication_add_relation(Oid pubid, Relation targetrel,
bool if_not_exists);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index b3ee4194d3..40aadb37d9 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -487,7 +487,7 @@ typedef enum NodeTag
T_PartitionRangeDatum,
T_PartitionCmd,
T_VacuumRelation,
- T_PublicationTable,
+ T_PublicationObjSpec,
/*
* TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 45e4f2a16e..3e88965316 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -353,6 +353,26 @@ typedef struct RoleSpec
int location; /* token location, or -1 if unknown */
} RoleSpec;
+/*
+ * Publication object type
+ */
+typedef enum PublicationObjSpecType
+{
+ PUBLICATIONOBJ_TABLE, /* Table type */
+ PUBLICATIONOBJ_REL_IN_SCHEMA, /* Relations in schema type */
+ PUBLICATIONOBJ_UNKNOWN /* Unknown type */
+} PublicationObjSpecType;
+
+typedef struct PublicationObjSpec
+{
+ NodeTag type;
+ PublicationObjSpecType pubobjtype; /* type of this publication object */
+ Node *object; /* publication object could be:
+ * RangeVar - table object
+ * String - tablename or schemaname */
+ int location; /* token location, or -1 if unknown */
+} PublicationObjSpec;
+
/*
* FuncCall - a function or aggregate invocation
*
@@ -3636,18 +3656,12 @@ typedef struct AlterTSConfigurationStmt
bool missing_ok; /* for DROP - skip error if missing? */
} AlterTSConfigurationStmt;
-typedef struct PublicationTable
-{
- NodeTag type;
- RangeVar *relation; /* relation to be published */
-} PublicationTable;
-
typedef struct CreatePublicationStmt
{
NodeTag type;
char *pubname; /* Name of the publication */
List *options; /* List of DefElem nodes */
- List *tables; /* Optional list of tables to add */
+ List *pubobjects; /* Optional list of publication objects */
bool for_all_tables; /* Special publication for all tables in db */
} CreatePublicationStmt;
@@ -3659,10 +3673,11 @@ typedef struct AlterPublicationStmt
/* parameters used for ALTER PUBLICATION ... WITH */
List *options; /* List of DefElem nodes */
- /* parameters used for ALTER PUBLICATION ... ADD/DROP TABLE */
- List *tables; /* List of tables to add/drop */
+ /* ALTER PUBLICATION ... ADD/DROP TABLE/ALL TABLES IN SCHEMA parameters */
+ List *pubobjects; /* Optional list of publication objects */
bool for_all_tables; /* Special publication for all tables in db */
- DefElemAction tableAction; /* What action to perform with the tables */
+ DefElemAction action; /* What action to perform with the
+ * tables/schemas */
} AlterPublicationStmt;
typedef struct CreateSubscriptionStmt
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 423780652f..8a1b97836e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2045,8 +2045,9 @@ PsqlSettings
Publication
PublicationActions
PublicationInfo
+PublicationObjSpec
+PublicationObjSpecType
PublicationPartOpt
-PublicationRelInfo
PublicationTable
PullFilter
PullFilterOps
Generic_object_type_parser_002_table_schema_publication.patchtext/x-patch; charset=US-ASCII; name=Generic_object_type_parser_002_table_schema_publication.patchDownload
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index d6fddd6efe..2a2fe03c13 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -141,14 +141,14 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
* Insert new publication / relation mapping.
*/
ObjectAddress
-publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
+publication_add_relation(Oid pubid, Relation targetrel,
bool if_not_exists)
{
Relation rel;
HeapTuple tup;
Datum values[Natts_pg_publication_rel];
bool nulls[Natts_pg_publication_rel];
- Oid relid = RelationGetRelid(targetrel->relation);
+ Oid relid = RelationGetRelid(targetrel);
Oid prrelid;
Publication *pub = GetPublication(pubid);
ObjectAddress myself,
@@ -172,10 +172,10 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_OBJECT),
errmsg("relation \"%s\" is already member of publication \"%s\"",
- RelationGetRelationName(targetrel->relation), pub->name)));
+ RelationGetRelationName(targetrel), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ check_publication_add_relation(targetrel);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -209,7 +209,7 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
table_close(rel, RowExclusiveLock);
/* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcache(targetrel->relation);
+ CacheInvalidateRelcache(targetrel);
return myself;
}
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 30929da1f5..4e4e02ba70 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -34,6 +34,7 @@
#include "commands/publicationcmds.h"
#include "funcapi.h"
#include "miscadmin.h"
+#include "nodes/makefuncs.h"
#include "utils/acl.h"
#include "utils/array.h"
#include "utils/builtins.h"
@@ -138,6 +139,85 @@ parse_publication_options(ParseState *pstate,
}
}
+/*
+ * Convert the PublicationObjSpecType list into schema oid list and rangevar
+ * list.
+ */
+static void
+ObjectsInPublicationToOids(List *pubobjspec_list, ParseState *pstate,
+ List **rels, List **schemas)
+{
+ ListCell *cell;
+ PublicationObjSpec *pubobj;
+ PublicationObjSpecType prevobjtype = PUBLICATIONOBJ_UNKNOWN;
+
+ if (!pubobjspec_list)
+ return;
+
+ pubobj = (PublicationObjSpec *) linitial(pubobjspec_list);
+ if (pubobj->pubobjtype == PUBLICATIONOBJ_UNKNOWN)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("FOR TABLE/FOR ALL TABLES IN SCHEMA should be specified before the table/schema name(s)"),
+ parser_errposition(pstate, pubobj->location));
+
+ foreach(cell, pubobjspec_list)
+ {
+ Node *node;
+
+ pubobj = (PublicationObjSpec *) lfirst(cell);
+ node = (Node *) pubobj->object;
+
+ if (pubobj->pubobjtype == PUBLICATIONOBJ_UNKNOWN)
+ pubobj->pubobjtype = prevobjtype;
+ else
+ prevobjtype = pubobj->pubobjtype;
+
+ if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLE)
+ {
+ if (IsA(node, RangeVar))
+ *rels = lappend(*rels, (RangeVar *) node);
+ else if (IsA(node, String))
+ {
+ RangeVar *rel = makeRangeVar(NULL, strVal(node),
+ pubobj->location);
+ *rels = lappend(*rels, rel);
+ }
+ }
+ else if (pubobj->pubobjtype == PUBLICATIONOBJ_REL_IN_SCHEMA)
+ {
+ Oid schemaid;
+ char *schemaname;
+
+ if (!IsA(node, String))
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("invalid schema name at or near"),
+ parser_errposition(pstate, pubobj->location));
+
+ schemaname = strVal(node);
+ if (strcmp(schemaname, "CURRENT_SCHEMA") == 0)
+ {
+ List *search_path;
+
+ search_path = fetch_search_path(false);
+ if (search_path == NIL) /* nothing valid in search_path? */
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_SCHEMA),
+ errmsg("no schema has been selected for CURRENT_SCHEMA"));
+
+ schemaid = linitial_oid(search_path);
+ list_free(search_path);
+ }
+ else
+ schemaid = get_namespace_oid(schemaname, false);
+
+ /* Filter out duplicates if user specifies "sch1, sch1" */
+ *schemas = list_append_unique_oid(*schemas, schemaid);
+ }
+ }
+}
+
/*
* Create new publication.
*/
@@ -155,6 +235,8 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
bool publish_via_partition_root_given;
bool publish_via_partition_root;
AclResult aclresult;
+ List *relations = NIL;
+ List *schemaidlist = NIL;
/* must have CREATE privilege on database */
aclresult = pg_database_aclcheck(MyDatabaseId, GetUserId(), ACL_CREATE);
@@ -224,13 +306,15 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
/* Make the changes visible. */
CommandCounterIncrement();
- if (stmt->tables)
+ ObjectsInPublicationToOids(stmt->pubobjects, pstate, &relations,
+ &schemaidlist);
+ if (relations != NIL)
{
List *rels;
- Assert(list_length(stmt->tables) > 0);
+ Assert(list_length(relations) > 0);
- rels = OpenTableList(stmt->tables);
+ rels = OpenTableList(relations);
PublicationAddTables(puboid, rels, true, NULL);
CloseTableList(rels);
}
@@ -360,7 +444,7 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
*/
static void
AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
- HeapTuple tup)
+ HeapTuple tup, List *tables, List *schemaidlist)
{
List *rels = NIL;
Form_pg_publication pubform = (Form_pg_publication) GETSTRUCT(tup);
@@ -374,13 +458,13 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
NameStr(pubform->pubname)),
errdetail("Tables cannot be added to or dropped from FOR ALL TABLES publications.")));
- Assert(list_length(stmt->tables) > 0);
+ Assert(list_length(tables) > 0);
- rels = OpenTableList(stmt->tables);
+ rels = OpenTableList(tables);
- if (stmt->tableAction == DEFELEM_ADD)
+ if (stmt->action == DEFELEM_ADD)
PublicationAddTables(pubid, rels, false, stmt);
- else if (stmt->tableAction == DEFELEM_DROP)
+ else if (stmt->action == DEFELEM_DROP)
PublicationDropTables(pubid, rels, false);
else /* DEFELEM_SET */
{
@@ -398,10 +482,9 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
foreach(newlc, rels)
{
- PublicationRelInfo *newpubrel;
+ Relation newrel = (Relation) lfirst(newlc);
- newpubrel = (PublicationRelInfo *) lfirst(newlc);
- if (RelationGetRelid(newpubrel->relation) == oldrelid)
+ if (RelationGetRelid(newrel) == oldrelid)
{
found = true;
break;
@@ -410,16 +493,10 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
/* Not yet in the list, open it and add to the list */
if (!found)
{
- Relation oldrel;
- PublicationRelInfo *pubrel;
-
- /* Wrap relation into PublicationRelInfo */
- oldrel = table_open(oldrelid, ShareUpdateExclusiveLock);
+ Relation oldrel = table_open(oldrelid,
+ ShareUpdateExclusiveLock);
- pubrel = palloc(sizeof(PublicationRelInfo));
- pubrel->relation = oldrel;
-
- delrels = lappend(delrels, pubrel);
+ delrels = lappend(delrels, oldrel);
}
}
@@ -450,6 +527,8 @@ AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt)
Relation rel;
HeapTuple tup;
Form_pg_publication pubform;
+ List *relations = NIL;
+ List *schemaidlist = NIL;
rel = table_open(PublicationRelationId, RowExclusiveLock);
@@ -469,10 +548,16 @@ AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt)
aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_PUBLICATION,
stmt->pubname);
+ ObjectsInPublicationToOids(stmt->pubobjects, pstate, &relations,
+ &schemaidlist);
+
if (stmt->options)
AlterPublicationOptions(pstate, stmt, rel, tup);
else
- AlterPublicationTables(stmt, rel, tup);
+ {
+ if (relations)
+ AlterPublicationTables(stmt, rel, tup, relations, schemaidlist);
+ }
/* Cleanup. */
heap_freetuple(tup);
@@ -540,7 +625,7 @@ RemovePublicationById(Oid pubid)
/*
* Open relations specified by a PublicationTable list.
- * In the returned list of PublicationRelInfo, tables are locked
+ * In the returned list of Relation, tables are locked
* in ShareUpdateExclusiveLock mode in order to add them to a publication.
*/
static List *
@@ -555,16 +640,15 @@ OpenTableList(List *tables)
*/
foreach(lc, tables)
{
- PublicationTable *t = lfirst_node(PublicationTable, lc);
- bool recurse = t->relation->inh;
+ RangeVar *rv = lfirst_node(RangeVar, lc);
+ bool recurse = rv->inh;
Relation rel;
Oid myrelid;
- PublicationRelInfo *pub_rel;
/* Allow query cancel in case this takes a long time */
CHECK_FOR_INTERRUPTS();
- rel = table_openrv(t->relation, ShareUpdateExclusiveLock);
+ rel = table_openrv(rv, ShareUpdateExclusiveLock);
myrelid = RelationGetRelid(rel);
/*
@@ -580,9 +664,7 @@ OpenTableList(List *tables)
continue;
}
- pub_rel = palloc(sizeof(PublicationRelInfo));
- pub_rel->relation = rel;
- rels = lappend(rels, pub_rel);
+ rels = lappend(rels, rel);
relids = lappend_oid(relids, myrelid);
/*
@@ -615,9 +697,7 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
- pub_rel = palloc(sizeof(PublicationRelInfo));
- pub_rel->relation = rel;
- rels = lappend(rels, pub_rel);
+ rels = lappend(rels, rel);
relids = lappend_oid(relids, childrelid);
}
}
@@ -638,10 +718,9 @@ CloseTableList(List *rels)
foreach(lc, rels)
{
- PublicationRelInfo *pub_rel;
+ Relation rel = (Relation) lfirst(lc);
- pub_rel = (PublicationRelInfo *) lfirst(lc);
- table_close(pub_rel->relation, NoLock);
+ table_close(rel, NoLock);
}
}
@@ -658,8 +737,7 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
foreach(lc, rels)
{
- PublicationRelInfo *pub_rel = (PublicationRelInfo *) lfirst(lc);
- Relation rel = pub_rel->relation;
+ Relation rel = (Relation) lfirst(lc);
ObjectAddress obj;
/* Must be owner of the table or superuser. */
@@ -667,7 +745,7 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(rel->rd_rel->relkind),
RelationGetRelationName(rel));
- obj = publication_add_relation(pubid, pub_rel, if_not_exists);
+ obj = publication_add_relation(pubid, rel, if_not_exists);
if (stmt)
{
EventTriggerCollectSimpleCommand(obj, InvalidObjectAddress,
@@ -691,8 +769,7 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
foreach(lc, rels)
{
- PublicationRelInfo *pubrel = (PublicationRelInfo *) lfirst(lc);
- Relation rel = pubrel->relation;
+ Relation rel = (Relation) lfirst(lc);
Oid relid = RelationGetRelid(rel);
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 228387eaee..ade93023b8 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4817,7 +4817,7 @@ _copyCreatePublicationStmt(const CreatePublicationStmt *from)
COPY_STRING_FIELD(pubname);
COPY_NODE_FIELD(options);
- COPY_NODE_FIELD(tables);
+ COPY_NODE_FIELD(pubobjects);
COPY_SCALAR_FIELD(for_all_tables);
return newnode;
@@ -4830,9 +4830,9 @@ _copyAlterPublicationStmt(const AlterPublicationStmt *from)
COPY_STRING_FIELD(pubname);
COPY_NODE_FIELD(options);
- COPY_NODE_FIELD(tables);
+ COPY_NODE_FIELD(pubobjects);
COPY_SCALAR_FIELD(for_all_tables);
- COPY_SCALAR_FIELD(tableAction);
+ COPY_SCALAR_FIELD(action);
return newnode;
}
@@ -4958,12 +4958,14 @@ _copyForeignKeyCacheInfo(const ForeignKeyCacheInfo *from)
return newnode;
}
-static PublicationTable *
-_copyPublicationTable(const PublicationTable *from)
+static PublicationObjSpec *
+_copyPublicationObject(const PublicationObjSpec *from)
{
- PublicationTable *newnode = makeNode(PublicationTable);
+ PublicationObjSpec *newnode = makeNode(PublicationObjSpec);
- COPY_NODE_FIELD(relation);
+ COPY_SCALAR_FIELD(pubobjtype);
+ COPY_NODE_FIELD(object);
+ COPY_LOCATION_FIELD(location);
return newnode;
}
@@ -5887,8 +5889,8 @@ copyObjectImpl(const void *from)
case T_PartitionCmd:
retval = _copyPartitionCmd(from);
break;
- case T_PublicationTable:
- retval = _copyPublicationTable(from);
+ case T_PublicationObjSpec:
+ retval = _copyPublicationObject(from);
break;
/*
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 800f588b5c..d384af2db7 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2302,7 +2302,7 @@ _equalCreatePublicationStmt(const CreatePublicationStmt *a,
{
COMPARE_STRING_FIELD(pubname);
COMPARE_NODE_FIELD(options);
- COMPARE_NODE_FIELD(tables);
+ COMPARE_NODE_FIELD(pubobjects);
COMPARE_SCALAR_FIELD(for_all_tables);
return true;
@@ -2314,9 +2314,9 @@ _equalAlterPublicationStmt(const AlterPublicationStmt *a,
{
COMPARE_STRING_FIELD(pubname);
COMPARE_NODE_FIELD(options);
- COMPARE_NODE_FIELD(tables);
+ COMPARE_NODE_FIELD(pubobjects);
COMPARE_SCALAR_FIELD(for_all_tables);
- COMPARE_SCALAR_FIELD(tableAction);
+ COMPARE_SCALAR_FIELD(action);
return true;
}
@@ -3134,12 +3134,12 @@ _equalBitString(const BitString *a, const BitString *b)
}
static bool
-_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
+_equalPublicationObject(const PublicationObjSpec *a, const PublicationObjSpec *b)
{
- COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(object);
return true;
-}
+}
/*
* equal
@@ -3894,8 +3894,8 @@ equal(const void *a, const void *b)
case T_PartitionCmd:
retval = _equalPartitionCmd(a, b);
break;
- case T_PublicationTable:
- retval = _equalPublicationTable(a, b);
+ case T_PublicationObjSpec:
+ retval = _equalPublicationObject(a, b);
break;
default:
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index e3068a374e..f7d7cab596 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -195,6 +195,9 @@ static Node *makeXmlExpr(XmlExprOp op, char *name, List *named_args,
static List *mergeTableFuncParameters(List *func_args, List *columns);
static TypeName *TableFuncTypeName(List *columns);
static RangeVar *makeRangeVarFromAnyName(List *names, int position, core_yyscan_t yyscanner);
+static RangeVar *makeRangeVarFromQualifiedName(char *name, List *rels,
+ int location,
+ core_yyscan_t yyscanner);
static void SplitColQualList(List *qualList,
List **constraintList, CollateClause **collClause,
core_yyscan_t yyscanner);
@@ -256,6 +259,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
PartitionSpec *partspec;
PartitionBoundSpec *partboundspec;
RoleSpec *rolespec;
+ PublicationObjSpec *publicationobjectspec;
struct SelectLimit *selectlimit;
SetQuantifier setquantifier;
struct GroupClause *groupclause;
@@ -425,14 +429,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
transform_element_list transform_type_list
TriggerTransitions TriggerReferencing
vacuum_relation_list opt_vacuum_relation_list
- drop_option_list publication_table_list
+ drop_option_list pub_obj_list
%type <node> opt_routine_body
%type <groupclause> group_clause
%type <list> group_by_list
%type <node> group_by_item empty_grouping_set rollup_clause cube_clause
%type <node> grouping_sets_clause
-%type <node> opt_publication_for_tables publication_for_tables publication_table
%type <list> opt_fdw_options fdw_options
%type <defelt> fdw_option
@@ -517,6 +520,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <node> table_ref
%type <jexpr> joined_table
%type <range> relation_expr
+%type <range> extended_relation_expr
%type <range> relation_expr_opt_alias
%type <node> tablesample_clause opt_repeatable_clause
%type <target> target_el set_target insert_column_item
@@ -554,6 +558,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <node> var_value zone_value
%type <rolespec> auth_ident RoleSpec opt_granted_by
+%type <publicationobjectspec> PublicationObjSpec
+%type <publicationobjectspec> pubobj_expr
+%type <node> pubobj_name
%type <keyword> unreserved_keyword type_func_name_keyword
%type <keyword> col_name_keyword reserved_keyword
%type <keyword> bare_label_keyword
@@ -9591,69 +9598,117 @@ AlterOwnerStmt: ALTER AGGREGATE aggregate_with_argtypes OWNER TO RoleSpec
/*****************************************************************************
*
- * CREATE PUBLICATION name [ FOR TABLE ] [ WITH options ]
+ * CREATE PUBLICATION name [WITH options]
+ *
+ * CREATE PUBLICATION FOR ALL TABLES [WITH options]
+ *
+ * CREATE PUBLICATION FOR pub_obj [, pub_obj2] [WITH options]
+ *
+ * pub_obj is one of:
+ *
+ * TABLE table [, table2]
+ * ALL TABLES IN SCHEMA schema [, schema2]
*
*****************************************************************************/
CreatePublicationStmt:
- CREATE PUBLICATION name opt_publication_for_tables opt_definition
+ CREATE PUBLICATION name opt_definition
{
CreatePublicationStmt *n = makeNode(CreatePublicationStmt);
n->pubname = $3;
- n->options = $5;
- if ($4 != NULL)
- {
- /* FOR TABLE */
- if (IsA($4, List))
- n->tables = (List *)$4;
- /* FOR ALL TABLES */
- else
- n->for_all_tables = true;
- }
+ n->options = $4;
+ $$ = (Node *)n;
+ }
+ | CREATE PUBLICATION name FOR ALL TABLES opt_definition
+ {
+ CreatePublicationStmt *n = makeNode(CreatePublicationStmt);
+ n->pubname = $3;
+ n->options = $7;
+ n->for_all_tables = true;
+ $$ = (Node *)n;
+ }
+ | CREATE PUBLICATION name FOR pub_obj_list opt_definition
+ {
+ CreatePublicationStmt *n = makeNode(CreatePublicationStmt);
+ n->pubname = $3;
+ n->options = $6;
+ n->pubobjects = (List *)$5;
$$ = (Node *)n;
}
;
-opt_publication_for_tables:
- publication_for_tables { $$ = $1; }
- | /* EMPTY */ { $$ = NULL; }
+pubobj_expr:
+ pubobj_name
+ {
+ /* inheritance query, implicitly */
+ $$ = makeNode(PublicationObjSpec);
+ $$->object = $1;
+ }
+ | extended_relation_expr
+ {
+ $$ = makeNode(PublicationObjSpec);
+ $$->object = $1;
+ }
+ | CURRENT_SCHEMA
+ {
+ $$ = makeNode(PublicationObjSpec);
+ $$->object = makeString("CURRENT_SCHEMA");
+ }
;
-publication_for_tables:
- FOR TABLE publication_table_list
+/* This can be either a schema or relation name. */
+pubobj_name:
+ ColId
{
- $$ = (Node *) $3;
+ $$ = (Node *) makeString($1);
}
- | FOR ALL TABLES
+ | ColId indirection
{
- $$ = (Node *) makeInteger(true);
+ $$ = (Node *) makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
}
;
-publication_table_list:
- publication_table
- { $$ = list_make1($1); }
- | publication_table_list ',' publication_table
- { $$ = lappend($1, $3); }
+/* FOR TABLE and FOR ALL TABLES IN SCHEMA specifications */
+PublicationObjSpec: TABLE pubobj_expr
+ {
+ $$ = $2;
+ $$->pubobjtype = PUBLICATIONOBJ_TABLE;
+ $$->location = @1;
+ }
+ | ALL TABLES IN_P SCHEMA pubobj_expr
+ {
+ $$ = $5;
+ $$->pubobjtype = PUBLICATIONOBJ_REL_IN_SCHEMA;
+ $$->location = @1;
+ }
+ | pubobj_expr
+ {
+ $$ = $1;
+ $$->pubobjtype = PUBLICATIONOBJ_UNKNOWN;
+ $$->location = @1;
+ }
;
-publication_table: relation_expr
- {
- PublicationTable *n = makeNode(PublicationTable);
- n->relation = $1;
- $$ = (Node *) n;
- }
+pub_obj_list: PublicationObjSpec
+ { $$ = list_make1($1); }
+ | pub_obj_list ',' PublicationObjSpec
+ { $$ = lappend($1, $3); }
;
/*****************************************************************************
*
* ALTER PUBLICATION name SET ( options )
*
- * ALTER PUBLICATION name ADD TABLE table [, table2]
+ * ALTER PUBLICATION name ADD pub_obj [, pub_obj ...]
*
- * ALTER PUBLICATION name DROP TABLE table [, table2]
+ * ALTER PUBLICATION name DROP pub_obj [, pub_obj ...]
*
- * ALTER PUBLICATION name SET TABLE table [, table2]
+ * ALTER PUBLICATION name SET pub_obj [, pub_obj ...]
+ *
+ * pub_obj is one of:
+ *
+ * TABLE table_name [, table_name ...]
+ * ALL TABLES IN SCHEMA schema_name [, schema_name ...]
*
*****************************************************************************/
@@ -9665,28 +9720,28 @@ AlterPublicationStmt:
n->options = $5;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name ADD_P TABLE publication_table_list
+ | ALTER PUBLICATION name ADD_P pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
- n->tables = $6;
- n->tableAction = DEFELEM_ADD;
+ n->pubobjects = $5;
+ n->action = DEFELEM_ADD;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name SET TABLE publication_table_list
+ | ALTER PUBLICATION name SET pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
- n->tables = $6;
- n->tableAction = DEFELEM_SET;
+ n->pubobjects = $5;
+ n->action = DEFELEM_SET;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name DROP TABLE publication_table_list
+ | ALTER PUBLICATION name DROP pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
- n->tables = $6;
- n->tableAction = DEFELEM_DROP;
+ n->pubobjects = $5;
+ n->action = DEFELEM_DROP;
$$ = (Node *)n;
}
;
@@ -12430,7 +12485,14 @@ relation_expr:
$$->inh = true;
$$->alias = NULL;
}
- | qualified_name '*'
+ | extended_relation_expr
+ {
+ $$ = $1;
+ }
+ ;
+
+extended_relation_expr:
+ qualified_name '*'
{
/* inheritance query, explicitly */
$$ = $1;
@@ -12453,7 +12515,6 @@ relation_expr:
}
;
-
relation_expr_list:
relation_expr { $$ = list_make1($1); }
| relation_expr_list ',' relation_expr { $$ = lappend($1, $3); }
@@ -15104,28 +15165,7 @@ qualified_name:
}
| ColId indirection
{
- check_qualified_name($2, yyscanner);
- $$ = makeRangeVar(NULL, NULL, @1);
- switch (list_length($2))
- {
- case 1:
- $$->catalogname = NULL;
- $$->schemaname = $1;
- $$->relname = strVal(linitial($2));
- break;
- case 2:
- $$->catalogname = $1;
- $$->schemaname = strVal(linitial($2));
- $$->relname = strVal(lsecond($2));
- break;
- default:
- ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("improper qualified name (too many dotted names): %s",
- NameListToString(lcons(makeString($1), $2))),
- parser_errposition(@1)));
- break;
- }
+ $$ = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
}
;
@@ -17045,6 +17085,41 @@ TableFuncTypeName(List *columns)
return result;
}
+/*
+ * Convert a relation_name with name and namelist to a RangeVar using
+ * makeRangeVar.
+ */
+static RangeVar *
+makeRangeVarFromQualifiedName(char *name, List *namelist, int location,
+ core_yyscan_t yyscanner)
+{
+ RangeVar *r = makeRangeVar(NULL, NULL, location);
+
+ check_qualified_name(namelist, yyscanner);
+ switch (list_length(namelist))
+ {
+ case 1:
+ r->catalogname = NULL;
+ r->schemaname = name;
+ r->relname = strVal(linitial(namelist));
+ break;
+ case 2:
+ r->catalogname = name;
+ r->schemaname = strVal(linitial(namelist));
+ r->relname = strVal(lsecond(namelist));
+ break;
+ default:
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("improper qualified name (too many dotted names): %s",
+ NameListToString(lcons(makeString(name), namelist))),
+ parser_errposition(location));
+ break;
+ }
+
+ return r;
+}
+
/*
* Convert a list of (dotted) names to a RangeVar (like
* makeRangeVarFromNameList, but with position support). The
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 561266aa3e..f332bad4d4 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -83,11 +83,6 @@ typedef struct Publication
PublicationActions pubactions;
} Publication;
-typedef struct PublicationRelInfo
-{
- Relation relation;
-} PublicationRelInfo;
-
extern Publication *GetPublication(Oid pubid);
extern Publication *GetPublicationByName(const char *pubname, bool missing_ok);
extern List *GetRelationPublications(Oid relid);
@@ -113,7 +108,7 @@ extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern bool is_publishable_relation(Relation rel);
-extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
+extern ObjectAddress publication_add_relation(Oid pubid, Relation targetrel,
bool if_not_exists);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index b3ee4194d3..40aadb37d9 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -487,7 +487,7 @@ typedef enum NodeTag
T_PartitionRangeDatum,
T_PartitionCmd,
T_VacuumRelation,
- T_PublicationTable,
+ T_PublicationObjSpec,
/*
* TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 45e4f2a16e..6f1239017e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -353,6 +353,26 @@ typedef struct RoleSpec
int location; /* token location, or -1 if unknown */
} RoleSpec;
+/*
+ * Publication object type
+ */
+typedef enum PublicationObjSpecType
+{
+ PUBLICATIONOBJ_TABLE, /* Table type */
+ PUBLICATIONOBJ_REL_IN_SCHEMA, /* Relations in schema type */
+ PUBLICATIONOBJ_UNKNOWN /* Unknown type */
+} PublicationObjSpecType;
+
+typedef struct PublicationObjSpec
+{
+ NodeTag type;
+ PublicationObjSpecType pubobjtype; /* type of this publication object */
+ void *object; /* publication object could be:
+ * RangeVar - table object
+ * String - tablename or schemaname */
+ int location; /* token location, or -1 if unknown */
+} PublicationObjSpec;
+
/*
* FuncCall - a function or aggregate invocation
*
@@ -3636,18 +3656,12 @@ typedef struct AlterTSConfigurationStmt
bool missing_ok; /* for DROP - skip error if missing? */
} AlterTSConfigurationStmt;
-typedef struct PublicationTable
-{
- NodeTag type;
- RangeVar *relation; /* relation to be published */
-} PublicationTable;
-
typedef struct CreatePublicationStmt
{
NodeTag type;
char *pubname; /* Name of the publication */
List *options; /* List of DefElem nodes */
- List *tables; /* Optional list of tables to add */
+ List *pubobjects; /* Optional list of publication objects */
bool for_all_tables; /* Special publication for all tables in db */
} CreatePublicationStmt;
@@ -3659,10 +3673,11 @@ typedef struct AlterPublicationStmt
/* parameters used for ALTER PUBLICATION ... WITH */
List *options; /* List of DefElem nodes */
- /* parameters used for ALTER PUBLICATION ... ADD/DROP TABLE */
- List *tables; /* List of tables to add/drop */
+ /* ALTER PUBLICATION ... ADD/DROP TABLE/ALL TABLES IN SCHEMA parameters */
+ List *pubobjects; /* Optional list of publication objects */
bool for_all_tables; /* Special publication for all tables in db */
- DefElemAction tableAction; /* What action to perform with the tables */
+ DefElemAction action; /* What action to perform with the
+ * tables/schemas */
} AlterPublicationStmt;
typedef struct CreateSubscriptionStmt
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 423780652f..8a1b97836e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2045,8 +2045,9 @@ PsqlSettings
Publication
PublicationActions
PublicationInfo
+PublicationObjSpec
+PublicationObjSpecType
PublicationPartOpt
-PublicationRelInfo
PublicationTable
PullFilter
PullFilterOps
On 2021-Sep-15, vignesh C wrote:
I have extracted the parser code and attached it here, so that it will
be easy to go through. We wanted to support the following syntax as in
[1]:
CREATE PUBLICATION pub1 FOR
TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2,
SEQUENCE seq1,seq2, ALL SEQUENCES IN SCHEMA s3,s4;
Oh, thanks, it looks like this can be useful. We can get the common
grammar done and then rebase all the other patches (I was also just told
about support for sequences in [1]/messages/by-id/3d6df331-5532-6848-eb45-344b265e0238@enterprisedb.com) on top.
[1]: /messages/by-id/3d6df331-5532-6848-eb45-344b265e0238@enterprisedb.com
Columns can be added to PublicationObjSpec data structure.
Right. (As a List of String, I imagine.)
The patch
Generic_object_type_parser_002_table_schema_publication.patch has the
changes that were used to handle the parsing. Schema and Relation both
are different objects, schema is of string type and relation is of
RangeVar type. While parsing, schema name is parsed in string format
and relation is parsed and converted to rangevar type, these objects
will be then handled accordingly during post processing.
Yeah, I think it'd be cleaner if the node type has two members, something like
this
typedef struct PublicationObjSpec
{
NodeTag type;
PublicationObjSpecType pubobjtype; /* type of this publication object */
RangeVar *rv; /* if a table */
String *objname; /* if a schema */
int location; /* token location, or -1 if unknown */
} PublicationObjSpec;
and only one of them is set, the other is NULL, depending on the object type.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
On Wed, Sep 15, 2021, at 9:19 AM, vignesh C wrote:
I have extracted the parser code and attached it here, so that it will
be easy to go through. We wanted to support the following syntax as in
[1]:
CREATE PUBLICATION pub1 FOR
TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2,
SEQUENCE seq1,seq2, ALL SEQUENCES IN SCHEMA s3,s4;
I don't like this syntax. It seems too much syntax for the same purpose in a
single command. If you look at GRANT command whose ALL TABLES IN SCHEMA syntax
was extracted, you can use ON TABLE or ON ALL TABLES IN SCHEMA; you cannot use
both. This proposal allows duplicate objects (of course, you can ignore it but
the current code prevent duplicates -- see publication_add_relation).
IMO you should mimic the GRANT grammar and have multiple commands for row
filtering, column filtering, and ALL FOO IN SCHEMA. The filtering patches only
use the FOR TABLE syntax. The later won't have filtering syntax. Having said
that the grammar should be:
CREATE PUBLICATION name
[ FOR TABLE [ ONLY ] table_name [ * ] [ (column_name [, ...] ) ] [ WHERE (expression) ] [, ...]
| FOR ALL TABLES
| FOR ALL TABLES IN SCHEMA schema_name, [, ...]
| FOR ALL SEQUENCES IN SCHEMA schema_name, [, ...] ]
[ WITH ( publication_parameter [= value] [, ... ] ) ]
ALTER PUBLICATION name ADD TABLE [ ONLY ] table_name [ * ] [ (column_name [, ...] ) ] [ WHERE (expression) ]
ALTER PUBLICATION name ADD ALL TABLES IN SCHEMA schema_name, [, ...]
ALTER PUBLICATION name ADD ALL SEQUENCES IN SCHEMA schema_name, [, ...]
ALTER PUBLICATION name SET TABLE [ ONLY ] table_name [ * ] [ (column_name [, ...] ) ] [ WHERE (expression) ]
ALTER PUBLICATION name SET ALL TABLES IN SCHEMA schema_name, [, ...]
ALTER PUBLICATION name SET ALL SEQUENCES IN SCHEMA schema_name, [, ...]
ALTER PUBLICATION name DROP TABLE [ ONLY ] table_name [ * ]
ALTER PUBLICATION name DROP ALL TABLES IN SCHEMA schema_name, [, ...]
ALTER PUBLICATION name DROP ALL SEQUENCES IN SCHEMA schema_name, [, ...]
Opinions?
--
Euler Taveira
EDB https://www.enterprisedb.com/
On Wednesday, September 15, 2021 8:19 PM vignesh C <vignesh21@gmail.com> wrote:
I have extracted the parser code and attached it here, so that it will be easy to
go through. We wanted to support the following syntax as in
[1]:
CREATE PUBLICATION pub1 FOR
TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2, SEQUENCE seq1,seq2, ALL
SEQUENCES IN SCHEMA s3,s4;
I am +1 for this syntax.
This syntax is more flexible than adding or dropping different type objects in
separate commands. User can either use one single command to add serval different
objects or use serval commands to add each type objects.
Best regards,
Hou zj
On Tue, Sep 7, 2021 at 3:51 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
...
I pushed the clerical part of this -- namely the addition of
PublicationTable node and PublicationRelInfo struct. I attach the part
of your v4 patch that I didn't include. It contains a couple of small
corrections, but I didn't do anything invasive (such as pgindent)
because that would perhaps cause you too much merge pain.
I noticed that the latest v5 no longer includes the TAP test which was
in the v4 patch.
(src/test/subscription/t/021_column_filter.pl)
Was that omission deliberate?
------
Kind Regards,
Peter Smith.
Fujitsu Australia.
On Wed, Sep 15, 2021 at 6:06 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-15, vignesh C wrote:
The patch
Generic_object_type_parser_002_table_schema_publication.patch has the
changes that were used to handle the parsing. Schema and Relation both
are different objects, schema is of string type and relation is of
RangeVar type. While parsing, schema name is parsed in string format
and relation is parsed and converted to rangevar type, these objects
will be then handled accordingly during post processing.Yeah, I think it'd be cleaner if the node type has two members, something like
thistypedef struct PublicationObjSpec
{
NodeTag type;
PublicationObjSpecType pubobjtype; /* type of this publication object */
RangeVar *rv; /* if a table */
String *objname; /* if a schema */
int location; /* token location, or -1 if unknown */
} PublicationObjSpec;and only one of them is set, the other is NULL, depending on the object type.
I think the problem here is that with the proposed grammar we won't be
always able to distinguish names at the gram.y stage. Some post
parsing analysis is required to attribute the right type to name as is
done in the patch. The same seems to be indicated by Tom in his email
as well where he has proposed this syntax [1]/messages/by-id/877603.1629120678@sss.pgh.pa.us. Also, something similar
is done for privilege_target (GRANT syntax) where we have a list of
objects but here the story is slightly more advanced because we are
planning to allow specifying multiple objects in one command. One
might think that we can identify each type of objects lists separately
but that gives grammar conflicts as it is not able to identify whether
the comma ',' is used for the same type object or for the next type.
Due to which we need to come up with a generic object for names to
which we attribute the right type in post parse analysis. Now, I think
instead of void *, it might be better to use Node * for generic
objects unless we have some problem.
[1]: /messages/by-id/877603.1629120678@sss.pgh.pa.us
--
With Regards,
Amit Kapila.
On Wed, Sep 15, 2021 at 8:19 PM Euler Taveira <euler@eulerto.com> wrote:
On Wed, Sep 15, 2021, at 9:19 AM, vignesh C wrote:
I have extracted the parser code and attached it here, so that it will
be easy to go through. We wanted to support the following syntax as in
[1]:
CREATE PUBLICATION pub1 FOR
TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2,
SEQUENCE seq1,seq2, ALL SEQUENCES IN SCHEMA s3,s4;I don't like this syntax. It seems too much syntax for the same purpose in a
single command. If you look at GRANT command whose ALL TABLES IN SCHEMA syntax
was extracted, you can use ON TABLE or ON ALL TABLES IN SCHEMA; you cannot use
both. This proposal allows duplicate objects (of course, you can ignore it but
the current code prevent duplicates -- see publication_add_relation).IMO you should mimic the GRANT grammar and have multiple commands for row
filtering, column filtering, and ALL FOO IN SCHEMA. The filtering patches only
use the FOR TABLE syntax. The later won't have filtering syntax.
Sure, but we don't prevent if the user uses only FOR TABLE variant.
OTOH, it is better to provide flexibility to allow multiple objects in
one command unless that is not feasible. It saves the effort of users
in many cases. In short, +1 for the syntax where multiple objects can
be allowed.
--
With Regards,
Amit Kapila.
On Thu, Sep 16, 2021 at 8:45 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Sep 15, 2021 at 6:06 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-15, vignesh C wrote:
The patch
Generic_object_type_parser_002_table_schema_publication.patch has the
changes that were used to handle the parsing. Schema and Relation both
are different objects, schema is of string type and relation is of
RangeVar type. While parsing, schema name is parsed in string format
and relation is parsed and converted to rangevar type, these objects
will be then handled accordingly during post processing.Yeah, I think it'd be cleaner if the node type has two members, something like
thistypedef struct PublicationObjSpec
{
NodeTag type;
PublicationObjSpecType pubobjtype; /* type of this publication object */
RangeVar *rv; /* if a table */
String *objname; /* if a schema */
int location; /* token location, or -1 if unknown */
} PublicationObjSpec;and only one of them is set, the other is NULL, depending on the object type.
I think the problem here is that with the proposed grammar we won't be
always able to distinguish names at the gram.y stage.
This is the issue that Amit was talking about:
gram.y: error: shift/reduce conflicts: 2 found, 0 expected
gram.y: warning: shift/reduce conflict on token ',' [-Wcounterexamples]
First example: CREATE PUBLICATION name FOR TABLE relation_expr_list
• ',' relation_expr ',' PublicationObjSpec opt_definition $end
Shift derivation
$accept
↳ parse_toplevel
$end
↳ stmtmulti
↳ toplevel_stmt
↳ stmt
↳ CreatePublicationStmt
↳ CREATE PUBLICATION name FOR pub_obj_list
opt_definition
↳ PublicationObjSpec
',' PublicationObjSpec
↳ TABLE relation_expr_list
↳
relation_expr_list • ',' relation_expr
Second example: CREATE PUBLICATION name FOR TABLE relation_expr_list
• ',' PublicationObjSpec opt_definition $end
Reduce derivation
$accept
↳ parse_toplevel
$end
↳ stmtmulti
↳ toplevel_stmt
↳ stmt
↳ CreatePublicationStmt
↳ CREATE PUBLICATION name FOR pub_obj_list
opt_definition
↳ pub_obj_list
',' PublicationObjSpec
↳ PublicationObjSpec
↳ TABLE relation_expr_list •
Here it is not able to distinguish if ',' is used for the next table
name or the next object.
I was able to reproduce this issue with the attached patch.
Regards,
Vignesh
Attachments:
Generic_object_type_parser_issue.patchtext/x-patch; charset=US-ASCII; name=Generic_object_type_parser_issue.patchDownload
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index d6fddd6efe..2a2fe03c13 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -141,14 +141,14 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
* Insert new publication / relation mapping.
*/
ObjectAddress
-publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
+publication_add_relation(Oid pubid, Relation targetrel,
bool if_not_exists)
{
Relation rel;
HeapTuple tup;
Datum values[Natts_pg_publication_rel];
bool nulls[Natts_pg_publication_rel];
- Oid relid = RelationGetRelid(targetrel->relation);
+ Oid relid = RelationGetRelid(targetrel);
Oid prrelid;
Publication *pub = GetPublication(pubid);
ObjectAddress myself,
@@ -172,10 +172,10 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_OBJECT),
errmsg("relation \"%s\" is already member of publication \"%s\"",
- RelationGetRelationName(targetrel->relation), pub->name)));
+ RelationGetRelationName(targetrel), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ check_publication_add_relation(targetrel);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -209,7 +209,7 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
table_close(rel, RowExclusiveLock);
/* Invalidate relcache so that publication info is rebuilt. */
- CacheInvalidateRelcache(targetrel->relation);
+ CacheInvalidateRelcache(targetrel);
return myself;
}
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 30929da1f5..4e4e02ba70 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -34,6 +34,7 @@
#include "commands/publicationcmds.h"
#include "funcapi.h"
#include "miscadmin.h"
+#include "nodes/makefuncs.h"
#include "utils/acl.h"
#include "utils/array.h"
#include "utils/builtins.h"
@@ -138,6 +139,85 @@ parse_publication_options(ParseState *pstate,
}
}
+/*
+ * Convert the PublicationObjSpecType list into schema oid list and rangevar
+ * list.
+ */
+static void
+ObjectsInPublicationToOids(List *pubobjspec_list, ParseState *pstate,
+ List **rels, List **schemas)
+{
+ ListCell *cell;
+ PublicationObjSpec *pubobj;
+ PublicationObjSpecType prevobjtype = PUBLICATIONOBJ_UNKNOWN;
+
+ if (!pubobjspec_list)
+ return;
+
+ pubobj = (PublicationObjSpec *) linitial(pubobjspec_list);
+ if (pubobj->pubobjtype == PUBLICATIONOBJ_UNKNOWN)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("FOR TABLE/FOR ALL TABLES IN SCHEMA should be specified before the table/schema name(s)"),
+ parser_errposition(pstate, pubobj->location));
+
+ foreach(cell, pubobjspec_list)
+ {
+ Node *node;
+
+ pubobj = (PublicationObjSpec *) lfirst(cell);
+ node = (Node *) pubobj->object;
+
+ if (pubobj->pubobjtype == PUBLICATIONOBJ_UNKNOWN)
+ pubobj->pubobjtype = prevobjtype;
+ else
+ prevobjtype = pubobj->pubobjtype;
+
+ if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLE)
+ {
+ if (IsA(node, RangeVar))
+ *rels = lappend(*rels, (RangeVar *) node);
+ else if (IsA(node, String))
+ {
+ RangeVar *rel = makeRangeVar(NULL, strVal(node),
+ pubobj->location);
+ *rels = lappend(*rels, rel);
+ }
+ }
+ else if (pubobj->pubobjtype == PUBLICATIONOBJ_REL_IN_SCHEMA)
+ {
+ Oid schemaid;
+ char *schemaname;
+
+ if (!IsA(node, String))
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("invalid schema name at or near"),
+ parser_errposition(pstate, pubobj->location));
+
+ schemaname = strVal(node);
+ if (strcmp(schemaname, "CURRENT_SCHEMA") == 0)
+ {
+ List *search_path;
+
+ search_path = fetch_search_path(false);
+ if (search_path == NIL) /* nothing valid in search_path? */
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_SCHEMA),
+ errmsg("no schema has been selected for CURRENT_SCHEMA"));
+
+ schemaid = linitial_oid(search_path);
+ list_free(search_path);
+ }
+ else
+ schemaid = get_namespace_oid(schemaname, false);
+
+ /* Filter out duplicates if user specifies "sch1, sch1" */
+ *schemas = list_append_unique_oid(*schemas, schemaid);
+ }
+ }
+}
+
/*
* Create new publication.
*/
@@ -155,6 +235,8 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
bool publish_via_partition_root_given;
bool publish_via_partition_root;
AclResult aclresult;
+ List *relations = NIL;
+ List *schemaidlist = NIL;
/* must have CREATE privilege on database */
aclresult = pg_database_aclcheck(MyDatabaseId, GetUserId(), ACL_CREATE);
@@ -224,13 +306,15 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
/* Make the changes visible. */
CommandCounterIncrement();
- if (stmt->tables)
+ ObjectsInPublicationToOids(stmt->pubobjects, pstate, &relations,
+ &schemaidlist);
+ if (relations != NIL)
{
List *rels;
- Assert(list_length(stmt->tables) > 0);
+ Assert(list_length(relations) > 0);
- rels = OpenTableList(stmt->tables);
+ rels = OpenTableList(relations);
PublicationAddTables(puboid, rels, true, NULL);
CloseTableList(rels);
}
@@ -360,7 +444,7 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
*/
static void
AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
- HeapTuple tup)
+ HeapTuple tup, List *tables, List *schemaidlist)
{
List *rels = NIL;
Form_pg_publication pubform = (Form_pg_publication) GETSTRUCT(tup);
@@ -374,13 +458,13 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
NameStr(pubform->pubname)),
errdetail("Tables cannot be added to or dropped from FOR ALL TABLES publications.")));
- Assert(list_length(stmt->tables) > 0);
+ Assert(list_length(tables) > 0);
- rels = OpenTableList(stmt->tables);
+ rels = OpenTableList(tables);
- if (stmt->tableAction == DEFELEM_ADD)
+ if (stmt->action == DEFELEM_ADD)
PublicationAddTables(pubid, rels, false, stmt);
- else if (stmt->tableAction == DEFELEM_DROP)
+ else if (stmt->action == DEFELEM_DROP)
PublicationDropTables(pubid, rels, false);
else /* DEFELEM_SET */
{
@@ -398,10 +482,9 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
foreach(newlc, rels)
{
- PublicationRelInfo *newpubrel;
+ Relation newrel = (Relation) lfirst(newlc);
- newpubrel = (PublicationRelInfo *) lfirst(newlc);
- if (RelationGetRelid(newpubrel->relation) == oldrelid)
+ if (RelationGetRelid(newrel) == oldrelid)
{
found = true;
break;
@@ -410,16 +493,10 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
/* Not yet in the list, open it and add to the list */
if (!found)
{
- Relation oldrel;
- PublicationRelInfo *pubrel;
-
- /* Wrap relation into PublicationRelInfo */
- oldrel = table_open(oldrelid, ShareUpdateExclusiveLock);
+ Relation oldrel = table_open(oldrelid,
+ ShareUpdateExclusiveLock);
- pubrel = palloc(sizeof(PublicationRelInfo));
- pubrel->relation = oldrel;
-
- delrels = lappend(delrels, pubrel);
+ delrels = lappend(delrels, oldrel);
}
}
@@ -450,6 +527,8 @@ AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt)
Relation rel;
HeapTuple tup;
Form_pg_publication pubform;
+ List *relations = NIL;
+ List *schemaidlist = NIL;
rel = table_open(PublicationRelationId, RowExclusiveLock);
@@ -469,10 +548,16 @@ AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt)
aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_PUBLICATION,
stmt->pubname);
+ ObjectsInPublicationToOids(stmt->pubobjects, pstate, &relations,
+ &schemaidlist);
+
if (stmt->options)
AlterPublicationOptions(pstate, stmt, rel, tup);
else
- AlterPublicationTables(stmt, rel, tup);
+ {
+ if (relations)
+ AlterPublicationTables(stmt, rel, tup, relations, schemaidlist);
+ }
/* Cleanup. */
heap_freetuple(tup);
@@ -540,7 +625,7 @@ RemovePublicationById(Oid pubid)
/*
* Open relations specified by a PublicationTable list.
- * In the returned list of PublicationRelInfo, tables are locked
+ * In the returned list of Relation, tables are locked
* in ShareUpdateExclusiveLock mode in order to add them to a publication.
*/
static List *
@@ -555,16 +640,15 @@ OpenTableList(List *tables)
*/
foreach(lc, tables)
{
- PublicationTable *t = lfirst_node(PublicationTable, lc);
- bool recurse = t->relation->inh;
+ RangeVar *rv = lfirst_node(RangeVar, lc);
+ bool recurse = rv->inh;
Relation rel;
Oid myrelid;
- PublicationRelInfo *pub_rel;
/* Allow query cancel in case this takes a long time */
CHECK_FOR_INTERRUPTS();
- rel = table_openrv(t->relation, ShareUpdateExclusiveLock);
+ rel = table_openrv(rv, ShareUpdateExclusiveLock);
myrelid = RelationGetRelid(rel);
/*
@@ -580,9 +664,7 @@ OpenTableList(List *tables)
continue;
}
- pub_rel = palloc(sizeof(PublicationRelInfo));
- pub_rel->relation = rel;
- rels = lappend(rels, pub_rel);
+ rels = lappend(rels, rel);
relids = lappend_oid(relids, myrelid);
/*
@@ -615,9 +697,7 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
- pub_rel = palloc(sizeof(PublicationRelInfo));
- pub_rel->relation = rel;
- rels = lappend(rels, pub_rel);
+ rels = lappend(rels, rel);
relids = lappend_oid(relids, childrelid);
}
}
@@ -638,10 +718,9 @@ CloseTableList(List *rels)
foreach(lc, rels)
{
- PublicationRelInfo *pub_rel;
+ Relation rel = (Relation) lfirst(lc);
- pub_rel = (PublicationRelInfo *) lfirst(lc);
- table_close(pub_rel->relation, NoLock);
+ table_close(rel, NoLock);
}
}
@@ -658,8 +737,7 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
foreach(lc, rels)
{
- PublicationRelInfo *pub_rel = (PublicationRelInfo *) lfirst(lc);
- Relation rel = pub_rel->relation;
+ Relation rel = (Relation) lfirst(lc);
ObjectAddress obj;
/* Must be owner of the table or superuser. */
@@ -667,7 +745,7 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(rel->rd_rel->relkind),
RelationGetRelationName(rel));
- obj = publication_add_relation(pubid, pub_rel, if_not_exists);
+ obj = publication_add_relation(pubid, rel, if_not_exists);
if (stmt)
{
EventTriggerCollectSimpleCommand(obj, InvalidObjectAddress,
@@ -691,8 +769,7 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
foreach(lc, rels)
{
- PublicationRelInfo *pubrel = (PublicationRelInfo *) lfirst(lc);
- Relation rel = pubrel->relation;
+ Relation rel = (Relation) lfirst(lc);
Oid relid = RelationGetRelid(rel);
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 228387eaee..ade93023b8 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4817,7 +4817,7 @@ _copyCreatePublicationStmt(const CreatePublicationStmt *from)
COPY_STRING_FIELD(pubname);
COPY_NODE_FIELD(options);
- COPY_NODE_FIELD(tables);
+ COPY_NODE_FIELD(pubobjects);
COPY_SCALAR_FIELD(for_all_tables);
return newnode;
@@ -4830,9 +4830,9 @@ _copyAlterPublicationStmt(const AlterPublicationStmt *from)
COPY_STRING_FIELD(pubname);
COPY_NODE_FIELD(options);
- COPY_NODE_FIELD(tables);
+ COPY_NODE_FIELD(pubobjects);
COPY_SCALAR_FIELD(for_all_tables);
- COPY_SCALAR_FIELD(tableAction);
+ COPY_SCALAR_FIELD(action);
return newnode;
}
@@ -4958,12 +4958,14 @@ _copyForeignKeyCacheInfo(const ForeignKeyCacheInfo *from)
return newnode;
}
-static PublicationTable *
-_copyPublicationTable(const PublicationTable *from)
+static PublicationObjSpec *
+_copyPublicationObject(const PublicationObjSpec *from)
{
- PublicationTable *newnode = makeNode(PublicationTable);
+ PublicationObjSpec *newnode = makeNode(PublicationObjSpec);
- COPY_NODE_FIELD(relation);
+ COPY_SCALAR_FIELD(pubobjtype);
+ COPY_NODE_FIELD(object);
+ COPY_LOCATION_FIELD(location);
return newnode;
}
@@ -5887,8 +5889,8 @@ copyObjectImpl(const void *from)
case T_PartitionCmd:
retval = _copyPartitionCmd(from);
break;
- case T_PublicationTable:
- retval = _copyPublicationTable(from);
+ case T_PublicationObjSpec:
+ retval = _copyPublicationObject(from);
break;
/*
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 800f588b5c..d384af2db7 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2302,7 +2302,7 @@ _equalCreatePublicationStmt(const CreatePublicationStmt *a,
{
COMPARE_STRING_FIELD(pubname);
COMPARE_NODE_FIELD(options);
- COMPARE_NODE_FIELD(tables);
+ COMPARE_NODE_FIELD(pubobjects);
COMPARE_SCALAR_FIELD(for_all_tables);
return true;
@@ -2314,9 +2314,9 @@ _equalAlterPublicationStmt(const AlterPublicationStmt *a,
{
COMPARE_STRING_FIELD(pubname);
COMPARE_NODE_FIELD(options);
- COMPARE_NODE_FIELD(tables);
+ COMPARE_NODE_FIELD(pubobjects);
COMPARE_SCALAR_FIELD(for_all_tables);
- COMPARE_SCALAR_FIELD(tableAction);
+ COMPARE_SCALAR_FIELD(action);
return true;
}
@@ -3134,12 +3134,12 @@ _equalBitString(const BitString *a, const BitString *b)
}
static bool
-_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
+_equalPublicationObject(const PublicationObjSpec *a, const PublicationObjSpec *b)
{
- COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(object);
return true;
-}
+}
/*
* equal
@@ -3894,8 +3894,8 @@ equal(const void *a, const void *b)
case T_PartitionCmd:
retval = _equalPartitionCmd(a, b);
break;
- case T_PublicationTable:
- retval = _equalPublicationTable(a, b);
+ case T_PublicationObjSpec:
+ retval = _equalPublicationObject(a, b);
break;
default:
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index e3068a374e..c50bb570ea 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -256,6 +256,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
PartitionSpec *partspec;
PartitionBoundSpec *partboundspec;
RoleSpec *rolespec;
+ PublicationObjSpec *publicationobjectspec;
struct SelectLimit *selectlimit;
SetQuantifier setquantifier;
struct GroupClause *groupclause;
@@ -425,14 +426,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
transform_element_list transform_type_list
TriggerTransitions TriggerReferencing
vacuum_relation_list opt_vacuum_relation_list
- drop_option_list publication_table_list
+ drop_option_list pub_obj_list pubobj_expr_list
%type <node> opt_routine_body
%type <groupclause> group_clause
%type <list> group_by_list
%type <node> group_by_item empty_grouping_set rollup_clause cube_clause
%type <node> grouping_sets_clause
-%type <node> opt_publication_for_tables publication_for_tables publication_table
%type <list> opt_fdw_options fdw_options
%type <defelt> fdw_option
@@ -554,6 +554,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <node> var_value zone_value
%type <rolespec> auth_ident RoleSpec opt_granted_by
+%type <publicationobjectspec> PublicationObjSpec
+%type <publicationobjectspec> pubobj_expr
+%type <node> pubobj_name
%type <keyword> unreserved_keyword type_func_name_keyword
%type <keyword> col_name_keyword reserved_keyword
%type <keyword> bare_label_keyword
@@ -9591,69 +9594,107 @@ AlterOwnerStmt: ALTER AGGREGATE aggregate_with_argtypes OWNER TO RoleSpec
/*****************************************************************************
*
- * CREATE PUBLICATION name [ FOR TABLE ] [ WITH options ]
+ * CREATE PUBLICATION name [WITH options]
+ *
+ * CREATE PUBLICATION FOR ALL TABLES [WITH options]
+ *
+ * CREATE PUBLICATION FOR pub_obj [, pub_obj2] [WITH options]
+ *
+ * pub_obj is one of:
+ *
+ * TABLE table [, table2]
+ * ALL TABLES IN SCHEMA schema [, schema2]
*
*****************************************************************************/
CreatePublicationStmt:
- CREATE PUBLICATION name opt_publication_for_tables opt_definition
+ CREATE PUBLICATION name opt_definition
{
CreatePublicationStmt *n = makeNode(CreatePublicationStmt);
n->pubname = $3;
- n->options = $5;
- if ($4 != NULL)
- {
- /* FOR TABLE */
- if (IsA($4, List))
- n->tables = (List *)$4;
- /* FOR ALL TABLES */
- else
- n->for_all_tables = true;
- }
+ n->options = $4;
+ $$ = (Node *)n;
+ }
+ | CREATE PUBLICATION name FOR ALL TABLES opt_definition
+ {
+ CreatePublicationStmt *n = makeNode(CreatePublicationStmt);
+ n->pubname = $3;
+ n->options = $7;
+ n->for_all_tables = true;
+ $$ = (Node *)n;
+ }
+ | CREATE PUBLICATION name FOR pub_obj_list opt_definition
+ {
+ CreatePublicationStmt *n = makeNode(CreatePublicationStmt);
+ n->pubname = $3;
+ n->options = $6;
+ n->pubobjects = (List *)$5;
$$ = (Node *)n;
}
;
-opt_publication_for_tables:
- publication_for_tables { $$ = $1; }
- | /* EMPTY */ { $$ = NULL; }
+pubobj_expr:
+ pubobj_name
+ {
+ /* inheritance query, implicitly */
+ $$ = makeNode(PublicationObjSpec);
+ $$->object = $1;
+ }
;
-publication_for_tables:
- FOR TABLE publication_table_list
+/* This can be either a schema or relation name. */
+pubobj_name:
+ ColId
{
- $$ = (Node *) $3;
+ $$ = (Node *) makeString($1);
}
- | FOR ALL TABLES
+ | ColId indirection
{
- $$ = (Node *) makeInteger(true);
+ $$ = (Node *) makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
}
;
-publication_table_list:
- publication_table
- { $$ = list_make1($1); }
- | publication_table_list ',' publication_table
- { $$ = lappend($1, $3); }
+/* FOR TABLE and FOR ALL TABLES IN SCHEMA specifications */
+PublicationObjSpec: TABLE relation_expr_list
+ {
+ $$ = $2;
+ $$->pubobjtype = PUBLICATIONOBJ_TABLE;
+ $$->location = @1;
+ }
+ | ALL TABLES IN_P SCHEMA pubobj_expr_list
+ {
+ $$ = $5;
+ $$->pubobjtype = PUBLICATIONOBJ_REL_IN_SCHEMA;
+ $$->location = @1;
+ }
;
-publication_table: relation_expr
- {
- PublicationTable *n = makeNode(PublicationTable);
- n->relation = $1;
- $$ = (Node *) n;
- }
+pubobj_expr_list: pubobj_expr
+ { $$ = list_make1($1); }
+ | pubobj_expr_list ',' pubobj_expr
+ { $$ = lappend($1, $3); }
+ ;
+
+pub_obj_list: PublicationObjSpec
+ { $$ = list_make1($1); }
+ | pub_obj_list ',' PublicationObjSpec
+ { $$ = lappend($1, $3); }
;
/*****************************************************************************
*
* ALTER PUBLICATION name SET ( options )
*
- * ALTER PUBLICATION name ADD TABLE table [, table2]
+ * ALTER PUBLICATION name ADD pub_obj [, pub_obj ...]
+ *
+ * ALTER PUBLICATION name DROP pub_obj [, pub_obj ...]
+ *
+ * ALTER PUBLICATION name SET pub_obj [, pub_obj ...]
*
- * ALTER PUBLICATION name DROP TABLE table [, table2]
+ * pub_obj is one of:
*
- * ALTER PUBLICATION name SET TABLE table [, table2]
+ * TABLE table_name [, table_name ...]
+ * ALL TABLES IN SCHEMA schema_name [, schema_name ...]
*
*****************************************************************************/
@@ -9665,28 +9706,28 @@ AlterPublicationStmt:
n->options = $5;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name ADD_P TABLE publication_table_list
+ | ALTER PUBLICATION name ADD_P pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
- n->tables = $6;
- n->tableAction = DEFELEM_ADD;
+ n->pubobjects = $5;
+ n->action = DEFELEM_ADD;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name SET TABLE publication_table_list
+ | ALTER PUBLICATION name SET pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
- n->tables = $6;
- n->tableAction = DEFELEM_SET;
+ n->pubobjects = $5;
+ n->action = DEFELEM_SET;
$$ = (Node *)n;
}
- | ALTER PUBLICATION name DROP TABLE publication_table_list
+ | ALTER PUBLICATION name DROP pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
- n->tables = $6;
- n->tableAction = DEFELEM_DROP;
+ n->pubobjects = $5;
+ n->action = DEFELEM_DROP;
$$ = (Node *)n;
}
;
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 561266aa3e..f332bad4d4 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -83,11 +83,6 @@ typedef struct Publication
PublicationActions pubactions;
} Publication;
-typedef struct PublicationRelInfo
-{
- Relation relation;
-} PublicationRelInfo;
-
extern Publication *GetPublication(Oid pubid);
extern Publication *GetPublicationByName(const char *pubname, bool missing_ok);
extern List *GetRelationPublications(Oid relid);
@@ -113,7 +108,7 @@ extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern bool is_publishable_relation(Relation rel);
-extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
+extern ObjectAddress publication_add_relation(Oid pubid, Relation targetrel,
bool if_not_exists);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index e0057daa06..d34b4ac8e5 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -487,7 +487,7 @@ typedef enum NodeTag
T_PartitionRangeDatum,
T_PartitionCmd,
T_VacuumRelation,
- T_PublicationTable,
+ T_PublicationObjSpec,
/*
* TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3138877553..20eeb12022 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -353,6 +353,26 @@ typedef struct RoleSpec
int location; /* token location, or -1 if unknown */
} RoleSpec;
+/*
+ * Publication object type
+ */
+typedef enum PublicationObjSpecType
+{
+ PUBLICATIONOBJ_TABLE, /* Table type */
+ PUBLICATIONOBJ_REL_IN_SCHEMA, /* Relations in schema type */
+ PUBLICATIONOBJ_UNKNOWN /* Unknown type */
+} PublicationObjSpecType;
+
+typedef struct PublicationObjSpec
+{
+ NodeTag type;
+ PublicationObjSpecType pubobjtype; /* type of this publication object */
+ void *object; /* publication object could be:
+ * RangeVar - table object
+ * String - tablename or schemaname */
+ int location; /* token location, or -1 if unknown */
+} PublicationObjSpec;
+
/*
* FuncCall - a function or aggregate invocation
*
@@ -3636,18 +3656,12 @@ typedef struct AlterTSConfigurationStmt
bool missing_ok; /* for DROP - skip error if missing? */
} AlterTSConfigurationStmt;
-typedef struct PublicationTable
-{
- NodeTag type;
- RangeVar *relation; /* relation to be published */
-} PublicationTable;
-
typedef struct CreatePublicationStmt
{
NodeTag type;
char *pubname; /* Name of the publication */
List *options; /* List of DefElem nodes */
- List *tables; /* Optional list of tables to add */
+ List *pubobjects; /* Optional list of publication objects */
bool for_all_tables; /* Special publication for all tables in db */
} CreatePublicationStmt;
@@ -3659,10 +3673,11 @@ typedef struct AlterPublicationStmt
/* parameters used for ALTER PUBLICATION ... WITH */
List *options; /* List of DefElem nodes */
- /* parameters used for ALTER PUBLICATION ... ADD/DROP TABLE */
- List *tables; /* List of tables to add/drop */
+ /* ALTER PUBLICATION ... ADD/DROP TABLE/ALL TABLES IN SCHEMA parameters */
+ List *pubobjects; /* Optional list of publication objects */
bool for_all_tables; /* Special publication for all tables in db */
- DefElemAction tableAction; /* What action to perform with the tables */
+ DefElemAction action; /* What action to perform with the
+ * tables/schemas */
} AlterPublicationStmt;
typedef struct CreateSubscriptionStmt
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 423780652f..8a1b97836e 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2045,8 +2045,9 @@ PsqlSettings
Publication
PublicationActions
PublicationInfo
+PublicationObjSpec
+PublicationObjSpecType
PublicationPartOpt
-PublicationRelInfo
PublicationTable
PullFilter
PullFilterOps
On 2021-Sep-16, Amit Kapila wrote:
I think the problem here is that with the proposed grammar we won't be
always able to distinguish names at the gram.y stage. Some post
parsing analysis is required to attribute the right type to name as is
done in the patch.
Doesn't it work to stuff them all into RangeVars? Then you don't need
to make the node type a monstrosity, just bail out in parse analysis if
an object spec has more elements in the RV than the object type allows.
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
On Thu, Sep 16, 2021 at 6:14 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-16, Amit Kapila wrote:
I think the problem here is that with the proposed grammar we won't be
always able to distinguish names at the gram.y stage. Some post
parsing analysis is required to attribute the right type to name as is
done in the patch.Doesn't it work to stuff them all into RangeVars? Then you don't need
to make the node type a monstrosity, just bail out in parse analysis if
an object spec has more elements in the RV than the object type allows.
So, are you suggesting that we store even schema names corresponding
to FOR ALL TABLES IN SCHEMA s1 [, ...] grammar in RangeVars in some
way (say store schema name in relname or schemaname field of RangeVar)
at gram.y stage and then later extract it from RangeVar? If so, why do
you think it would be better than the current proposed way?
--
With Regards,
Amit Kapila.
On 2021-Sep-16, vignesh C wrote:
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index e3068a374e..c50bb570ea 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y
Yeah, on a quick glance this looks all wrong. Your PublicationObjSpec
production should return a node with tag PublicationObjSpec, and
pubobj_expr should not exist at all -- that stuff is just making it all
more confusing.
I think it'd be something like this:
PublicationObjSpec:
ALL TABLES
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES;
$$->location = @1;
}
| TABLE qualified_name
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubobj = $2;
$$->location = @1;
}
| ALL TABLES IN_P SCHEMA name
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES_IN_SCHEMA;
$$->pubobj = makeRangeVar( ... $5 ... );
$$->location = @1;
}
| qualified_name
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubobj = $1;
$$->location = @1;
};
You need a single object name under TABLE, not a list -- this was Tom's
point about needing post-processing to determine how to assign a type to
a object that's what I named PUBLICATIONOBJ_CONTINUATION here.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Puedes vivir sólo una vez, pero si lo haces bien, una vez es suficiente"
On 2021-Sep-16, Alvaro Herrera wrote:
Actually, something like this might be better:
PublicationObjSpec:
| TABLE qualified_name
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubrvobj = $2;
$$->location = @1;
}
| ALL TABLES IN_P SCHEMA name
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES_IN_SCHEMA;
$$->pubplainobj = $5;
$$->location = @1;
}
So you don't have to cram the schema name in a RangeVar, which would
indeed be quite awkward. (I'm sure you can come up with better names
for the struct members there ...)
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Porque francamente, si para saber manejarse a uno mismo hubiera que
rendir examen... ¿Quién es el machito que tendría carnet?" (Mafalda)
On Thu, Sep 16, 2021 at 7:20 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-16, vignesh C wrote:
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index e3068a374e..c50bb570ea 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.yYeah, on a quick glance this looks all wrong. Your PublicationObjSpec
production should return a node with tag PublicationObjSpec, and
pubobj_expr should not exist at all -- that stuff is just making it all
more confusing.I think it'd be something like this:
PublicationObjSpec:
ALL TABLES
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES;
$$->location = @1;
}
| TABLE qualified_name
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubobj = $2;
$$->location = @1;
}
| ALL TABLES IN_P SCHEMA name
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES_IN_SCHEMA;
$$->pubobj = makeRangeVar( ... $5 ... );
$$->location = @1;
}
| qualified_name
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubobj = $1;
$$->location = @1;
};You need a single object name under TABLE, not a list -- this was Tom's
point about needing post-processing to determine how to assign a type to
a object that's what I named PUBLICATIONOBJ_CONTINUATION here.
In the above, we will not be able to use qualified_name, as
qualified_name will not support the following syntaxes:
create publication pub1 for table t1 *;
create publication pub1 for table ONLY t1 *;
create publication pub1 for table ONLY (t1);
To solve this problem we can change qualified_name to relation_expr
but the problem with doing that is that the user will be able to
provide the following syntaxes:
create publication pub1 for all tables in schema sch1 *;
create publication pub1 for all tables in schema ONLY sch1 *;
create publication pub1 for all tables in schema ONLY (sch1);
To handle this we will need some special flag which will differentiate
these and throw errors at post processing time. We need to define an
expression similar to relation_expr say pub_expr which handles all
variants of qualified_name and then use a special flag so that we can
throw an error if somebody uses the above type of syntax for schema
names. And then if we have to distinguish between schema name and
relation name variant, then we need few other things.
We proposed the below solution which handles all these problems and
also used Node type which need not store schemaname in RangeVar type:
pubobj_expr:
pubobj_name
{
/* inheritance query, implicitly */
$$ = makeNode(PublicationObjSpec);
$$->object = $1;
}
| extended_relation_expr
{
$$ = makeNode(PublicationObjSpec);
$$->object = (Node *)$1;
}
| CURRENT_SCHEMA
{
$$ = makeNode(PublicationObjSpec);
$$->object = (Node
*)makeString("CURRENT_SCHEMA");
}
;
/* This can be either a schema or relation name. */
pubobj_name:
ColId
{
$$ = (Node *) makeString($1);
}
| ColId indirection
{
$$ = (Node *)
makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
}
;
/* FOR TABLE and FOR ALL TABLES IN SCHEMA specifications */
PublicationObjSpec: TABLE pubobj_expr
{
$$ = $2;
$$->pubobjtype =
PUBLICATIONOBJ_TABLE;
$$->location = @1;
}
| ALL TABLES IN_P SCHEMA pubobj_expr
{
$$ = $5;
$$->pubobjtype =
PUBLICATIONOBJ_REL_IN_SCHEMA;
$$->location = @1;
}
| pubobj_expr
{
$$ = $1;
$$->pubobjtype =
PUBLICATIONOBJ_UNKNOWN;
$$->location = @1;
}
;
The same has been proposed in the recent version of patch [1]/messages/by-id/CALDaNm0OudeDeFN7bSWPro0hgKx=1zPgcNFWnvU_G6w3mDPX0Q@mail.gmail.com Thoughts?.
[1]: /messages/by-id/CALDaNm0OudeDeFN7bSWPro0hgKx=1zPgcNFWnvU_G6w3mDPX0Q@mail.gmail.com Thoughts?
Thoughts?
Regards,
Vignesh
On Thurs, Sep 16, 2021 10:37 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-16, Alvaro Herrera wrote:
Actually, something like this might be better:
PublicationObjSpec:
| TABLE qualified_name
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubrvobj = $2;
$$->location = @1;
}
| ALL TABLES IN_P SCHEMA name
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES_IN_SCHEMA;
$$->pubplainobj = $5;
$$->location = @1;
}So you don't have to cram the schema name in a RangeVar, which would indeed
be quite awkward. (I'm sure you can come up with better names for the struct
members there ...)>
Did you mean something like the following ?
-----
PublicationObjSpec:
TABLE qualified_name {...}
| ALL TABLES IN_P SCHEMA name {...}
;
pub_obj_list:
PublicationObjSpec
| pub_obj_list ',' PublicationObjSpec
-----
If so, I think it only supports syntaxes like "TABLE a, TABLE b, TABLE c" while
we cannnot use "TABLE a,b,c". To support multiple objects, we need a bare name
in PublicationObjSpec.
Or Did you mean something like this ?
-----
PublicationObjSpec:
TABLE qualified_name {...}
| ALL TABLES IN_P SCHEMA name {...}
| qualified_name {...}
;
-----
I think this doesn't support relation expression like "table */ONLY table/ONLY
(table)" as memtioned by Vignesh [1]/messages/by-id/CALDaNm06=LDytYyY+xcAQd8UK_YpJ3zMo4P5V8KBArw6MoDWDg@mail.gmail.com.
Thoughts ?
[1]: /messages/by-id/CALDaNm06=LDytYyY+xcAQd8UK_YpJ3zMo4P5V8KBArw6MoDWDg@mail.gmail.com
Best regards,
Hou zj
On Fri, Sep 17, 2021 at 9:36 AM vignesh C <vignesh21@gmail.com> wrote:
On Thu, Sep 16, 2021 at 7:20 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-16, vignesh C wrote:
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index e3068a374e..c50bb570ea 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.yYeah, on a quick glance this looks all wrong. Your PublicationObjSpec
production should return a node with tag PublicationObjSpec, and
pubobj_expr should not exist at all -- that stuff is just making it all
more confusing.I think it'd be something like this:
PublicationObjSpec:
ALL TABLES
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES;
$$->location = @1;
}
| TABLE qualified_name
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubobj = $2;
$$->location = @1;
}
| ALL TABLES IN_P SCHEMA name
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_ALL_TABLES_IN_SCHEMA;
$$->pubobj = makeRangeVar( ... $5 ... );
$$->location = @1;
}
| qualified_name
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubobj = $1;
$$->location = @1;
};You need a single object name under TABLE, not a list -- this was Tom's
point about needing post-processing to determine how to assign a type to
a object that's what I named PUBLICATIONOBJ_CONTINUATION here.In the above, we will not be able to use qualified_name, as
qualified_name will not support the following syntaxes:
create publication pub1 for table t1 *;
create publication pub1 for table ONLY t1 *;
create publication pub1 for table ONLY (t1);To solve this problem we can change qualified_name to relation_expr
but the problem with doing that is that the user will be able to
provide the following syntaxes:
create publication pub1 for all tables in schema sch1 *;
create publication pub1 for all tables in schema ONLY sch1 *;
create publication pub1 for all tables in schema ONLY (sch1);To handle this we will need some special flag which will differentiate
these and throw errors at post processing time. We need to define an
expression similar to relation_expr say pub_expr which handles all
variants of qualified_name and then use a special flag so that we can
throw an error if somebody uses the above type of syntax for schema
names. And then if we have to distinguish between schema name and
relation name variant, then we need few other things.We proposed the below solution which handles all these problems and
also used Node type which need not store schemaname in RangeVar type:
Alvaro, do you have any thoughts on these proposed grammar changes?
--
With Regards,
Amit Kapila.
Hi,
I wanted to do a review of this patch, but I'm a bit confused about
which patch(es) to review. There's the v5 patch, and then these two
patches - which seem to be somewhat duplicate, though.
Can anyone explain what's the "current" patch version, or perhaps tell
me which of the patches to combine?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Fri, Sep 24, 2021 at 12:45 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
Hi,
I wanted to do a review of this patch, but I'm a bit confused about
which patch(es) to review. There's the v5 patch, and then these two
patches - which seem to be somewhat duplicate, though.Can anyone explain what's the "current" patch version, or perhaps tell
me which of the patches to combine?
I think v5 won't work atop a common grammar patch. There need some
adjustments in v5. I think it would be good if we can first get the
common grammar patch reviewed/committed and then build this on top of
it. The common grammar and the corresponding implementation are being
accomplished in the Schema support patch, the latest version of which
is at [1]/messages/by-id/OS3PR01MB571844A87B6A83B7C10F9D6B94A39@OS3PR01MB5718.jpnprd01.prod.outlook.com. Now, Vignesh seems to have extracted just the grammar
portion of that work in his patch
Generic_object_type_parser_002_table_schema_publication [2]/messages/by-id/CALDaNm1YoxJCs=uiyPM=tFDDc2qn0ja01nb2TCPqrjZH2jR0sQ@mail.gmail.com (there are
some changes after that but not anything fundamentally different till
now) then he seems to have prepared a patch
(Generic_object_type_parser_001_table_publication [2]/messages/by-id/CALDaNm1YoxJCs=uiyPM=tFDDc2qn0ja01nb2TCPqrjZH2jR0sQ@mail.gmail.com) on similar
lines only for tables.
[1]: /messages/by-id/OS3PR01MB571844A87B6A83B7C10F9D6B94A39@OS3PR01MB5718.jpnprd01.prod.outlook.com
[2]: /messages/by-id/CALDaNm1YoxJCs=uiyPM=tFDDc2qn0ja01nb2TCPqrjZH2jR0sQ@mail.gmail.com
--
With Regards,
Amit Kapila.
On Fri, Sep 24, 2021 at 8:40 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Sep 24, 2021 at 12:45 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Hi,
I wanted to do a review of this patch, but I'm a bit confused about
which patch(es) to review. There's the v5 patch, and then these two
patches - which seem to be somewhat duplicate, though.Can anyone explain what's the "current" patch version, or perhaps tell
me which of the patches to combine?I think v5 won't work atop a common grammar patch. There need some
adjustments in v5. I think it would be good if we can first get the
common grammar patch reviewed/committed and then build this on top of
it. The common grammar and the corresponding implementation are being
accomplished in the Schema support patch, the latest version of which
is at [1].
I have posted an updated patch with the fixes at [1]/messages/by-id/CALDaNm1R-xbQvz4LU5OXu3KKwbWOz3uDcT_YjRU6V0R5FZDYDg@mail.gmail.com, please review
the updated patch.
[1]: /messages/by-id/CALDaNm1R-xbQvz4LU5OXu3KKwbWOz3uDcT_YjRU6V0R5FZDYDg@mail.gmail.com
Regards,
Vignesh
On 2021-Sep-23, Amit Kapila wrote:
Alvaro, do you have any thoughts on these proposed grammar changes?
Yeah, I think pubobj_name remains a problem in that you don't know its
return type -- could be a String or a RangeVar, and the user of that
production can't distinguish. So you're still (unnecessarily, IMV)
stashing an object of undetermined type into ->object.
I think you should get rid of both pubobj_name and pubobj_expr and do
somethine like this:
/* FOR TABLE and FOR ALL TABLES IN SCHEMA specifications */
PublicationObjSpec: TABLE ColId
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->rangevar = makeRangeVarFromQualifiedName($1, NULL, @1, yyscanner);
}
| TABLE ColId indirection
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
}
| ALL TABLES IN_P SCHEMA ColId
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_REL_IN_SCHEMA;
$$->name = $4;
}
| ALL TABLES IN_P SCHEMA CURRENT_SCHEMA /* XXX should this be "IN_P CURRENT_SCHEMA"? */
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
$$->name = $4;
}
| ColId
{
$$ = makeNode(PublicationObjSpec);
$$->name = $1;
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
}
| ColId indirection
{
$$ = makeNode(PublicationObjSpec);
$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
}
| CURRENT_SCHEMA
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
}
;
so in AlterPublicationStmt you would have stanzas like
| ALTER PUBLICATION name ADD_P pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
n->pubobjects = preprocess_pubobj_list($5);
n->action = DEFELEM_ADD;
$$ = (Node *)n;
}
where preprocess_pubobj_list (defined right after processCASbits and
somewhat mimicking it and SplitColQualList) takes all
PUBLICATIONOBJ_CONTINUATION and turns them into either
PUBLICATIONOBJ_TABLE entries or PUBLICATIONOBJ_REL_IN_SCHEMA entries,
depending on what the previous entry was. (And of course if there is no
previous entry, raise an error immediately). Note that node
PublicationObjSpec now has two fields, one for RangeVar and another for
a plain name, and tables always use the second one, except when they are
continuations, but of course those continuations that use name are
turned into rangevars in the preprocess step. I think that would make
the code in ObjectsInPublicationToOids less messy.
(I don't think using the string "CURRENT_SCHEMA" is a great solution.
Did you try having a schema named CURRENT_SCHEMA?)
I verified that bison is happy with the grammar I proposed; I also
verified that you can add opt_column_list to the stanzas for tables, and
it remains happy.
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
Y una voz del caos me habló y me dijo
"Sonríe y sé feliz, podría ser peor".
Y sonreí. Y fui feliz.
Y fue peor.
On 9/24/21 7:05 AM, vignesh C wrote:
On Fri, Sep 24, 2021 at 8:40 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Sep 24, 2021 at 12:45 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Hi,
I wanted to do a review of this patch, but I'm a bit confused about
which patch(es) to review. There's the v5 patch, and then these two
patches - which seem to be somewhat duplicate, though.Can anyone explain what's the "current" patch version, or perhaps tell
me which of the patches to combine?I think v5 won't work atop a common grammar patch. There need some
adjustments in v5. I think it would be good if we can first get the
common grammar patch reviewed/committed and then build this on top of
it. The common grammar and the corresponding implementation are being
accomplished in the Schema support patch, the latest version of which
is at [1].I have posted an updated patch with the fixes at [1], please review
the updated patch.
[1] - /messages/by-id/CALDaNm1R-xbQvz4LU5OXu3KKwbWOz3uDcT_YjRU6V0R5FZDYDg@mail.gmail.com
But that's not the column filtering patch, right? Why would this patch
depend on "schema level support", but maybe the consensus is there's
some common part that we need to get in first?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 2021-Sep-24, Tomas Vondra wrote:
But that's not the column filtering patch, right? Why would this patch
depend on "schema level support", but maybe the consensus is there's some
common part that we need to get in first?
Yes, the grammar needs to be common. I posted a proposed grammar in
/messages/by-id/202109241325.eag5g6mpvoup@alvherre.pgsql
(this thread) which should serve both. I forgot to test the addition of
a WHERE clause for row filtering, though, and I didn't think to look at
adding SEQUENCE support either.
(I'm not sure what's going to be the proposal regarding FOR ALL TABLES
IN SCHEMA for sequences. Are we going to have "FOR ALL SEQUENCES IN
SCHEMA" and "FOR ALL TABLES AND SEQUENCES IN SCHEMA"?)
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
Thou shalt study thy libraries and strive not to reinvent them without
cause, that thy code may be short and readable and thy days pleasant
and productive. (7th Commandment for C Programmers)
On 9/25/21 12:24 AM, Alvaro Herrera wrote:
On 2021-Sep-24, Tomas Vondra wrote:
But that's not the column filtering patch, right? Why would this patch
depend on "schema level support", but maybe the consensus is there's some
common part that we need to get in first?Yes, the grammar needs to be common. I posted a proposed grammar in
/messages/by-id/202109241325.eag5g6mpvoup@alvherre.pgsql
(this thread) which should serve both. I forgot to test the addition of
a WHERE clause for row filtering, though, and I didn't think to look at
adding SEQUENCE support either.
Fine with me, but I still don't know which version of the column
filtering patch should I look at ... maybe there's none up to date, at
the moment?
(I'm not sure what's going to be the proposal regarding FOR ALL TABLES
IN SCHEMA for sequences. Are we going to have "FOR ALL SEQUENCES IN
SCHEMA" and "FOR ALL TABLES AND SEQUENCES IN SCHEMA"?)
Should be "FOR ABSOLUTELY EVERYTHING IN SCHEMA" of course ;-)
On a more serious note, a comma-separated list of objects seems like the
best / most flexible choice, i.e. "FOR TABLES, SEQUENCES IN SCHEMA"?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 2021-Sep-25, Tomas Vondra wrote:
On 9/25/21 12:24 AM, Alvaro Herrera wrote:
On 2021-Sep-24, Tomas Vondra wrote:
But that's not the column filtering patch, right? Why would this patch
depend on "schema level support", but maybe the consensus is there's some
common part that we need to get in first?Yes, the grammar needs to be common. I posted a proposed grammar in
/messages/by-id/202109241325.eag5g6mpvoup@alvherre.pgsql
(this thread) which should serve both. I forgot to test the addition of
a WHERE clause for row filtering, though, and I didn't think to look at
adding SEQUENCE support either.Fine with me, but I still don't know which version of the column filtering
patch should I look at ... maybe there's none up to date, at the moment?
I don't think there is one. I think the latest is what I posted in
/messages/by-id/202109061751.3qz5xpugwx6w@alvherre.pgsql (At least I
don't see any reply from Rahila with attachments after that), but that
wasn't addressing a bunch of review comments that had been made; and I
suspect that Amit K has already committed a few conflicting patches
after that.
(I'm not sure what's going to be the proposal regarding FOR ALL TABLES
IN SCHEMA for sequences. Are we going to have "FOR ALL SEQUENCES IN
SCHEMA" and "FOR ALL TABLES AND SEQUENCES IN SCHEMA"?)Should be "FOR ABSOLUTELY EVERYTHING IN SCHEMA" of course ;-)
hahah ...
On a more serious note, a comma-separated list of objects seems like the
best / most flexible choice, i.e. "FOR TABLES, SEQUENCES IN SCHEMA"?
Hmm, not sure if bison is going to like that. Maybe it's OK if
SEQUENCES is a fully reserved word? But nothing beats experimentation!
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
Thou shalt check the array bounds of all strings (indeed, all arrays), for
surely where thou typest "foo" someone someday shall type
"supercalifragilisticexpialidocious" (5th Commandment for C programmers)
From Fri, Sep 24, 2021 9:25 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-23, Amit Kapila wrote:
Alvaro, do you have any thoughts on these proposed grammar changes?
Yeah, I think pubobj_name remains a problem in that you don't know its return
type -- could be a String or a RangeVar, and the user of that production can't
distinguish. So you're still (unnecessarily, IMV) stashing an object of
undetermined type into ->object.I think you should get rid of both pubobj_name and pubobj_expr and do
somethine like this:
PublicationObjSpec: TABLE ColId
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->rangevar = makeRangeVarFromQualifiedName($1, NULL, @1, yyscanner);
}
| TABLE ColId indirection
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
}
Hi,
IIRC, the above grammar doesn't support extended relation expression (like:
"tablename * ", "ONLY tablename", "ONLY '( tablename )") which is part of rule
relation_expr. I think we should add these too. And if we move forward with the
design you proposed, we should do something like the following:
/* FOR TABLE and FOR ALL TABLES IN SCHEMA specifications */
PublicationObjSpec:
TABLE relation_expr
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->rangevar = $2;
}
| ALL TABLES IN_P SCHEMA ColId
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_REL_IN_SCHEMA;
$$->name = $5;
}
| ALL TABLES IN_P SCHEMA CURRENT_SCHEMA
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
$$->name = $5;
}
| extended_relation_expr /* grammar like tablename * , ONLY tablename, ONLY ( tablename )*/
{
$$ = makeNode(PublicationObjSpec);
$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
}
| ColId
{
$$ = makeNode(PublicationObjSpec);
$$->name = $1;
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
}
| ColId indirection
{
$$ = makeNode(PublicationObjSpec);
$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
}
Best regards,
Hou zj
On Fri, Sep 24, 2021 at 6:55 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-23, Amit Kapila wrote:
Alvaro, do you have any thoughts on these proposed grammar changes?
Yeah, I think pubobj_name remains a problem in that you don't know its
return type -- could be a String or a RangeVar, and the user of that
production can't distinguish. So you're still (unnecessarily, IMV)
stashing an object of undetermined type into ->object.I think you should get rid of both pubobj_name and pubobj_expr and do
somethine like this:/* FOR TABLE and FOR ALL TABLES IN SCHEMA specifications */
PublicationObjSpec: TABLE ColId
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->rangevar = makeRangeVarFromQualifiedName($1, NULL, @1, yyscanner);
}
| TABLE ColId indirection
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
}
| ALL TABLES IN_P SCHEMA ColId
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_REL_IN_SCHEMA;
$$->name = $4;
}
| ALL TABLES IN_P SCHEMA CURRENT_SCHEMA /* XXX should this be "IN_P CURRENT_SCHEMA"? */
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
$$->name = $4;
}
| ColId
{
$$ = makeNode(PublicationObjSpec);
$$->name = $1;
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
}
| ColId indirection
{
$$ = makeNode(PublicationObjSpec);
$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
}
| CURRENT_SCHEMA
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
}
;
Apart from the issue that Hou San pointed, I found one issue with
introduction of PUBLICATIONOBJ_CURRSCHEMA, I was not able to
differentiate if it is table or schema in the following cases:
CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA CURRENT_SCHEMA;
CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA sch1, CURRENT_SCHEMA;
CREATE PUBLICATION pub1 FOR table t1, CURRENT_SCHEMA;
The differentiation is required to differentiate and add a schema or a table.
I felt it was better to use PUBLICATIONOBJ_CONTINUATION in case of
CURRENT_SCHEMA in multiple object cases like:
PublicationObjSpec: TABLE relation_expr
{
$$ =
makeNode(PublicationObjSpec);
$$->pubobjtype =
PUBLICATIONOBJ_TABLE;
$$->rangevar = $2;
}
| ALL TABLES IN_P SCHEMA ColId
{
$$ =
makeNode(PublicationObjSpec);
$$->pubobjtype =
PUBLICATIONOBJ_REL_IN_SCHEMA;
$$->name = $5;
$$->location = @5;
}
| ALL TABLES IN_P SCHEMA CURRENT_SCHEMA /* XXX
should this be "IN_P CURRENT_SCHEMA"? */
{
$$ =
makeNode(PublicationObjSpec);
$$->pubobjtype =
PUBLICATIONOBJ_REL_IN_SCHEMA;
$$->name = "CURRENT_SCHEMA";
$$->location = @5;
}
| ColId
{
$$ =
makeNode(PublicationObjSpec);
$$->name = $1;
$$->pubobjtype =
PUBLICATIONOBJ_CONTINUATION;
$$->location = @1;
}
| ColId indirection
{
$$ =
makeNode(PublicationObjSpec);
$$->rangevar =
makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
$$->pubobjtype =
PUBLICATIONOBJ_CONTINUATION;
$$->location = @1;
}
| CURRENT_SCHEMA
{
$$ =
makeNode(PublicationObjSpec);
$$->pubobjtype =
PUBLICATIONOBJ_CONTINUATION;
$$->name = "CURRENT_SCHEMA";
$$->location = @1;
}
| extended_relation_expr /* grammar
like tablename * , ONLY tablename, ONLY ( tablename )*/
{
$$ =
makeNode(PublicationObjSpec);
/*$$->rangevar =
makeRangeVarFromQualifiedName($1, $2, @1, yyscanner); */
$$->rangevar = $1;
$$->pubobjtype =
PUBLICATIONOBJ_CONTINUATION;
}
;
I'm ok with your suggestion along with the above proposed changes. I
felt the changes proposed at [1]/messages/by-id/CALDaNm1R-xbQvz4LU5OXu3KKwbWOz3uDcT_YjRU6V0R5FZDYDg@mail.gmail.com were also fine. Let's change it to
whichever is better, easily extendable and can handle the Column
filtering project, ALL TABLES IN SCHEMA, ALL SEQUENCES IN SCHEMA
projects, and other projects in the future. Based on that we can check
in the parser changes independently and then the remaining series of
the patches can be rebased on top of it accordingly. Thoughts?
so in AlterPublicationStmt you would have stanzas like
| ALTER PUBLICATION name ADD_P pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
n->pubobjects = preprocess_pubobj_list($5);
n->action = DEFELEM_ADD;
$$ = (Node *)n;
}where preprocess_pubobj_list (defined right after processCASbits and
somewhat mimicking it and SplitColQualList) takes all
PUBLICATIONOBJ_CONTINUATION and turns them into either
PUBLICATIONOBJ_TABLE entries or PUBLICATIONOBJ_REL_IN_SCHEMA entries,
depending on what the previous entry was. (And of course if there is no
previous entry, raise an error immediately). Note that node
PublicationObjSpec now has two fields, one for RangeVar and another for
a plain name, and tables always use the second one, except when they are
continuations, but of course those continuations that use name are
turned into rangevars in the preprocess step. I think that would make
the code in ObjectsInPublicationToOids less messy.
I agree with this. I will make the changes for this in the next version.
(I don't think using the string "CURRENT_SCHEMA" is a great solution.
Did you try having a schema named CURRENT_SCHEMA?)
Here CURRENT_SCHEMA is not used for the schema name, it will be
replaced with the name of the schema that is first in the search path.
[1]: /messages/by-id/CALDaNm1R-xbQvz4LU5OXu3KKwbWOz3uDcT_YjRU6V0R5FZDYDg@mail.gmail.com
Regards,
Vignesh
On Sat, Sep 25, 2021 at 1:15 PM vignesh C <vignesh21@gmail.com> wrote:
On Fri, Sep 24, 2021 at 6:55 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-23, Amit Kapila wrote:
Alvaro, do you have any thoughts on these proposed grammar changes?
Yeah, I think pubobj_name remains a problem in that you don't know its
return type -- could be a String or a RangeVar, and the user of that
production can't distinguish. So you're still (unnecessarily, IMV)
stashing an object of undetermined type into ->object.I think you should get rid of both pubobj_name and pubobj_expr and do
somethine like this:/* FOR TABLE and FOR ALL TABLES IN SCHEMA specifications */
PublicationObjSpec: TABLE ColId
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->rangevar = makeRangeVarFromQualifiedName($1, NULL, @1, yyscanner);
}
| TABLE ColId indirection
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
}
| ALL TABLES IN_P SCHEMA ColId
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_REL_IN_SCHEMA;
$$->name = $4;
}
| ALL TABLES IN_P SCHEMA CURRENT_SCHEMA /* XXX should this be "IN_P CURRENT_SCHEMA"? */
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
$$->name = $4;
}
| ColId
{
$$ = makeNode(PublicationObjSpec);
$$->name = $1;
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
}
| ColId indirection
{
$$ = makeNode(PublicationObjSpec);
$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
}
| CURRENT_SCHEMA
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
}
;Apart from the issue that Hou San pointed, I found one issue with
introduction of PUBLICATIONOBJ_CURRSCHEMA, I was not able to
differentiate if it is table or schema in the following cases:
CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA CURRENT_SCHEMA;
CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA sch1, CURRENT_SCHEMA;
CREATE PUBLICATION pub1 FOR table t1, CURRENT_SCHEMA;
The differentiation is required to differentiate and add a schema or a table.
I am not sure what makes you say that we can't distinguish the above
cases when there is already a separate rule for CURRENT_SCHEMA? I
think you can distinguish by tracking the previous objects as we are
already doing in the patch. But one thing that is not clear to me is
is the reason to introduce a new type PUBLICATIONOBJ_CURRSCHEMA when
we use PUBLICATIONOBJ_REL_IN_SCHEMA and PUBLICATIONOBJ_CONTINUATION to
distinguish all cases of CURRENT_SCHEMA. Alvaro might have something
in mind for this which is not apparent and that might have caused
confusion to you as well?
--
With Regards,
Amit Kapila.
On Mon, Sep 27, 2021 at 4:41 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Sat, Sep 25, 2021 at 1:15 PM vignesh C <vignesh21@gmail.com> wrote:
On Fri, Sep 24, 2021 at 6:55 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-23, Amit Kapila wrote:
Alvaro, do you have any thoughts on these proposed grammar changes?
Yeah, I think pubobj_name remains a problem in that you don't know its
return type -- could be a String or a RangeVar, and the user of that
production can't distinguish. So you're still (unnecessarily, IMV)
stashing an object of undetermined type into ->object.I think you should get rid of both pubobj_name and pubobj_expr and do
somethine like this:/* FOR TABLE and FOR ALL TABLES IN SCHEMA specifications */
PublicationObjSpec: TABLE ColId
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->rangevar = makeRangeVarFromQualifiedName($1, NULL, @1, yyscanner);
}
| TABLE ColId indirection
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
}
| ALL TABLES IN_P SCHEMA ColId
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_REL_IN_SCHEMA;
$$->name = $4;
}
| ALL TABLES IN_P SCHEMA CURRENT_SCHEMA /* XXX should this be "IN_P CURRENT_SCHEMA"? */
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
$$->name = $4;
}
| ColId
{
$$ = makeNode(PublicationObjSpec);
$$->name = $1;
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
}
| ColId indirection
{
$$ = makeNode(PublicationObjSpec);
$$->rangevar = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
}
| CURRENT_SCHEMA
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CURRSCHEMA;
}
;Apart from the issue that Hou San pointed, I found one issue with
introduction of PUBLICATIONOBJ_CURRSCHEMA, I was not able to
differentiate if it is table or schema in the following cases:
CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA CURRENT_SCHEMA;
CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA sch1, CURRENT_SCHEMA;
CREATE PUBLICATION pub1 FOR table t1, CURRENT_SCHEMA;
The differentiation is required to differentiate and add a schema or a table.I am not sure what makes you say that we can't distinguish the above
cases when there is already a separate rule for CURRENT_SCHEMA? I
think you can distinguish by tracking the previous objects as we are
already doing in the patch. But one thing that is not clear to me is
is the reason to introduce a new type PUBLICATIONOBJ_CURRSCHEMA when
we use PUBLICATIONOBJ_REL_IN_SCHEMA and PUBLICATIONOBJ_CONTINUATION to
distinguish all cases of CURRENT_SCHEMA. Alvaro might have something
in mind for this which is not apparent and that might have caused
confusion to you as well?
It is difficult to identify this case:
1) create publication pub1 for all tables in schema CURRENT_SCHEMA;
2) create publication pub1 for CURRENT_SCHEMA;
Here case 1 should succeed and case 2 should throw error:
Since the object type will be set to PUBLICATIONOBJ_CURRSCHEMA in both
cases, we cannot differentiate between them:
1) ALL TABLES IN_P SCHEMA CURRENT_SCHEMA /* XXX should this be "IN_P
CURRENT_SCHEMA"? */
{
$$ =
makeNode(PublicationObjSpec);
$$->pubobjtype =
PUBLICATIONOBJ_CURRSCHEMA;
$$->name = $4;
}
2) CURRENT_SCHEMA
{
$$ =
makeNode(PublicationObjSpec);
$$->pubobjtype =
PUBLICATIONOBJ_CURRSCHEMA;
}
I felt it will work, if we set object type to
PUBLICATIONOBJ_CONTINUATION in 2nd case(CURRENT_SCHEMA) and setting
object type to PUBLICATIONOBJ_REL_IN_SCHEMA or
PUBLICATIONOBJ_CURRSCHEMA in 1st case( ALL TABLES IN_P SCHEMA
CURRENT_SCHEMA).
Thoughts?
Regards,
Vignesh
On 2021-Sep-27, Amit Kapila wrote:
I am not sure what makes you say that we can't distinguish the above
cases when there is already a separate rule for CURRENT_SCHEMA? I
think you can distinguish by tracking the previous objects as we are
already doing in the patch. But one thing that is not clear to me is
is the reason to introduce a new type PUBLICATIONOBJ_CURRSCHEMA when
we use PUBLICATIONOBJ_REL_IN_SCHEMA and PUBLICATIONOBJ_CONTINUATION to
distinguish all cases of CURRENT_SCHEMA. Alvaro might have something
in mind for this which is not apparent and that might have caused
confusion to you as well?
My issue is what happens if you have a schema that is named
CURRENT_SCHEMA. In the normal case where you do ALL TABLES IN SCHEMA
"CURRENT_SCHEMA" you would end up with a String containing
"CURRENT_SCHEMA", so how do you distinguish that from ALL TABLES IN
SCHEMA CURRENT_SCHEMA, which does not refer to the schema named
"CURRENT_SCHEMA" but in Vignesh's proposal also uses a String containing
"CURRENT_SCHEMA"?
Now you could say "but who would be stupid enough to do that??!", but it
seems easier to dodge the problem entirely. AFAICS our grammar never
uses String "CURRENT_SCHEMA" to represent CURRENT_SCHEMA, but rather
some special enum value.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
Hi,
I don't think there is one. I think the latest is what I posted in
/messages/by-id/202109061751.3qz5xpugwx6w@alvherre.pgsql (At least I
don't see any reply from Rahila with attachments after that), but that
wasn't addressing a bunch of review comments that had been made; and I
suspect that Amit K has already committed a few conflicting patches
after that.Yes, the v5 version of the patch attached by Alvaro is the latest one.
IIUC, the review comments that are yet to be addressed apart from the
ongoing grammar
discussion, are as follows:
1. Behaviour on dropping a column from the table, that is a part of column
filter.
In the latest patch, the entire table is dropped from publication on
dropping a column
that is a part of the column filter. However, there is preference for
another approach
to drop just the column from the filter on DROP column CASCADE(continue to
filter
the other columns), and an error for DROP RESTRICT.
2. Instead of WITH RECURSIVE query to find the topmost parent of the
partition
in fetch_remote_table_info, use pg_partition_tree and pg_partition_root.
3. Report of memory leakage in get_rel_sync_entry().
4. Missing documentation
5. Latest comments(last two messages) by Peter Smith.
Thank you,
Rahila Syed
On Mon, Sep 27, 2021 at 5:53 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-27, Amit Kapila wrote:
I am not sure what makes you say that we can't distinguish the above
cases when there is already a separate rule for CURRENT_SCHEMA? I
think you can distinguish by tracking the previous objects as we are
already doing in the patch. But one thing that is not clear to me is
is the reason to introduce a new type PUBLICATIONOBJ_CURRSCHEMA when
we use PUBLICATIONOBJ_REL_IN_SCHEMA and PUBLICATIONOBJ_CONTINUATION to
distinguish all cases of CURRENT_SCHEMA. Alvaro might have something
in mind for this which is not apparent and that might have caused
confusion to you as well?My issue is what happens if you have a schema that is named
CURRENT_SCHEMA. In the normal case where you do ALL TABLES IN SCHEMA
"CURRENT_SCHEMA" you would end up with a String containing
"CURRENT_SCHEMA", so how do you distinguish that from ALL TABLES IN
SCHEMA CURRENT_SCHEMA, which does not refer to the schema named
"CURRENT_SCHEMA" but in Vignesh's proposal also uses a String containing
"CURRENT_SCHEMA"?Now you could say "but who would be stupid enough to do that??!",
But it is not allowed to create schema or table with the name
CURRENT_SCHEMA, so not sure if we need to do anything special for it.
However, if we want to handle it as a separate enum then the handling
would be something like:
| ALL TABLES IN_P SCHEMA CURRENT_SCHEMA
{
$$ =
makeNode(PublicationObjSpec);
$$->pubobjtype =
PUBLICATIONOBJ_CURRSCHEMA;
}
...
...
| CURRENT_SCHEMA
{
$$ =
makeNode(PublicationObjSpec);
$$->pubobjtype =
PUBLICATIONOBJ_CONTINUATION;
}
;
Now, during post-processing, the PUBLICATIONOBJ_CONTINUATION will be
distinguished as CURRENT_SCHEMA because both rangeVar and name will be
NULL. Do you have other ideas to deal with it? Vignesh has already
point in his email [1]/messages/by-id/CALDaNm06shp+ALwC2s-dV-S4k2o6bcmXnXGX4ETkoXxKHQfjfA@mail.gmail.com why we can't keep pubobjtype as
PUBLICATIONOBJ_CURRSCHEMA in the second case, so I used
PUBLICATIONOBJ_CONTINUATION.
[1]: /messages/by-id/CALDaNm06shp+ALwC2s-dV-S4k2o6bcmXnXGX4ETkoXxKHQfjfA@mail.gmail.com
--
With Regards,
Amit Kapila.
On Mon, Sep 27, 2021 at 6:41 PM Rahila Syed <rahilasyed90@gmail.com> wrote:
I don't think there is one. I think the latest is what I posted in
/messages/by-id/202109061751.3qz5xpugwx6w@alvherre.pgsql (At least I
don't see any reply from Rahila with attachments after that), but that
wasn't addressing a bunch of review comments that had been made; and I
suspect that Amit K has already committed a few conflicting patches
after that.Yes, the v5 version of the patch attached by Alvaro is the latest one.
IIUC, the review comments that are yet to be addressed apart from the ongoing grammar
discussion, are as follows:1. Behaviour on dropping a column from the table, that is a part of column filter.
In the latest patch, the entire table is dropped from publication on dropping a column
that is a part of the column filter. However, there is preference for another approach
to drop just the column from the filter on DROP column CASCADE(continue to filter
the other columns), and an error for DROP RESTRICT.
I am not sure if we can do this as pointed by me in one of the
previous emails [1]/messages/by-id/CAA4eK1KCGF43pfLv8+mixcTMs=Nkd6YdWL53LhiT1DvnuTg01g@mail.gmail.com. I think additionally, you might want to take some
action if the replica identity is changed as requested in the same
email [1]/messages/by-id/CAA4eK1KCGF43pfLv8+mixcTMs=Nkd6YdWL53LhiT1DvnuTg01g@mail.gmail.com.
[1]: /messages/by-id/CAA4eK1KCGF43pfLv8+mixcTMs=Nkd6YdWL53LhiT1DvnuTg01g@mail.gmail.com
--
With Regards,
Amit Kapila.
On 2021-Sep-28, Amit Kapila wrote:
But it is not allowed to create schema or table with the name
CURRENT_SCHEMA, so not sure if we need to do anything special for it.
Oh? You certainly can.
alvherre=# create schema "CURRENT_SCHEMA";
CREATE SCHEMA
alvherre=# \dn
Listado de esquemas
Nombre | Dueño
----------------+-------------------
CURRENT_SCHEMA | alvherre
public | pg_database_owner
temp | alvherre
(3 filas)
alvherre=# create table "CURRENT_SCHEMA"."CURRENT_SCHEMA" ("bother amit for a while" int);
CREATE TABLE
alvherre=# \d "CURRENT_SCHEMA".*
Tabla «CURRENT_SCHEMA.CURRENT_SCHEMA»
Columna | Tipo | Ordenamiento | Nulable | Por omisión
-------------------------+---------+--------------+---------+-------------
bother amit for a while | integer | | |
Now, during post-processing, the PUBLICATIONOBJ_CONTINUATION will be
distinguished as CURRENT_SCHEMA because both rangeVar and name will be
NULL. Do you have other ideas to deal with it?
That sounds plausible. There's no need for a name-free object of any other
kind AFAICS, so there should be no conflict. If we ever do find a
conflict, we can add another struct member to disambiguate.
Thanks
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
"Doing what he did amounts to sticking his fingers under the hood of the
implementation; if he gets his fingers burnt, it's his problem." (Tom Lane)
On Wed, Sep 29, 2021 at 6:49 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-28, Amit Kapila wrote:
But it is not allowed to create schema or table with the name
CURRENT_SCHEMA, so not sure if we need to do anything special for it.Oh? You certainly can.
alvherre=# create schema "CURRENT_SCHEMA";
CREATE SCHEMA
alvherre=# \dn
Listado de esquemas
Nombre | Dueño
----------------+-------------------
CURRENT_SCHEMA | alvherre
public | pg_database_owner
temp | alvherre
(3 filas)alvherre=# create table "CURRENT_SCHEMA"."CURRENT_SCHEMA" ("bother amit for a while" int);
CREATE TABLE
alvherre=# \d "CURRENT_SCHEMA".*
Tabla «CURRENT_SCHEMA.CURRENT_SCHEMA»
Columna | Tipo | Ordenamiento | Nulable | Por omisión
-------------------------+---------+--------------+---------+-------------
bother amit for a while | integer | | |
oops, I was trying without quotes.
Now, during post-processing, the PUBLICATIONOBJ_CONTINUATION will be
distinguished as CURRENT_SCHEMA because both rangeVar and name will be
NULL. Do you have other ideas to deal with it?That sounds plausible. There's no need for a name-free object of any other
kind AFAICS, so there should be no conflict. If we ever do find a
conflict, we can add another struct member to disambiguate.
Okay, thanks. I feel now we are in agreement on the grammar rules.
--
With Regards,
Amit Kapila.
Hi
I took the latest posted patch, rebased on current sources, fixed the
conflicts, and pgindented. No further changes. Here's the result. All
tests are passing for me. Some review comments that were posted have
not been addressed yet; I'll look into that soon.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Java is clearly an example of money oriented programming" (A. Stepanov)
Attachments:
v6-0001-Add-column-filtering-to-logical-replication.patchtext/x-diff; charset=utf-8Download
From bb01d00a9ee8f19bc1d9c36bde6cd0ca178c859c Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 6 Sep 2021 10:34:29 -0300
Subject: [PATCH v6] Add column filtering to logical replication
Add capability to specifiy column names while linking
the table to a publication, at the time of CREATE or ALTER
publication. This will allow replicating only the specified
columns. Other columns, if any, on the subscriber will be populated
locally or NULL will be inserted if no value is supplied for the column
by the upstream during INSERT.
This facilitates replication to a table on subscriber
containing only the subscribed/filtered columns.
If no filter is specified, all the columns are replicated.
REPLICA IDENTITY columns are always replicated.
Thus, prohibit adding relation to publication, if column filters
do not contain REPLICA IDENTITY.
Add a tap test for the same in src/test/subscription.
Author: Rahila Syed <rahilasyed90@gmail.com>
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
src/backend/access/common/relation.c | 22 +++++
src/backend/catalog/pg_publication.c | 58 ++++++++++-
src/backend/commands/publicationcmds.c | 8 +-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 30 +++++-
src/backend/replication/logical/proto.c | 103 ++++++++++++++++----
src/backend/replication/logical/tablesync.c | 101 +++++++++++++++++--
src/backend/replication/pgoutput/pgoutput.c | 77 ++++++++++++---
src/include/catalog/pg_publication.h | 1 +
src/include/catalog/pg_publication_rel.h | 4 +
src/include/nodes/parsenodes.h | 1 +
src/include/replication/logicalproto.h | 6 +-
src/include/utils/rel.h | 1 +
14 files changed, 365 insertions(+), 49 deletions(-)
diff --git a/src/backend/access/common/relation.c b/src/backend/access/common/relation.c
index 632d13c1ea..05d6fcba26 100644
--- a/src/backend/access/common/relation.c
+++ b/src/backend/access/common/relation.c
@@ -21,12 +21,14 @@
#include "postgres.h"
#include "access/relation.h"
+#include "access/sysattr.h"
#include "access/xact.h"
#include "catalog/namespace.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/lmgr.h"
#include "utils/inval.h"
+#include "utils/lsyscache.h"
#include "utils/syscache.h"
@@ -215,3 +217,23 @@ relation_close(Relation relation, LOCKMODE lockmode)
if (lockmode != NoLock)
UnlockRelationId(&relid, lockmode);
}
+
+/*
+ * Return a bitmapset of attributes given the list of column names
+ */
+Bitmapset *
+get_table_columnset(Oid relid, List *columns, Bitmapset *att_map)
+{
+ ListCell *cell;
+
+ foreach(cell, columns)
+ {
+ const char *attname = lfirst(cell);
+ int attnum = get_attnum(relid, attname);
+
+ if (!bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, att_map))
+ att_map = bms_add_member(att_map,
+ attnum - FirstLowInvalidHeapAttributeNumber);
+ }
+ return att_map;
+}
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 63579b2f82..de5f3266cd 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -50,8 +50,12 @@
* error if not.
*/
static void
-check_publication_add_relation(Relation targetrel)
+check_publication_add_relation(Relation targetrel, List *targetcols)
{
+ bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ Oid relid = RelationGetRelid(targetrel);
+ Bitmapset *idattrs;
+
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
@@ -82,6 +86,36 @@ check_publication_add_relation(Relation targetrel)
errmsg("cannot add relation \"%s\" to publication",
RelationGetRelationName(targetrel)),
errdetail("This operation is not supported for unlogged tables.")));
+
+ /*
+ * Cannot specify column filter when REPLICA IDENTITY IS FULL or if column
+ * filter does not contain REPLICA IDENITY columns
+ */
+ if (targetcols != NIL)
+ {
+ if (replidentfull)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot add relation \"%s\" to publication",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter with REPLICA IDENTITY FULL")));
+ else
+ {
+ Bitmapset *filtermap = NULL;
+
+ idattrs = RelationGetIndexAttrBitmap(targetrel, INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ filtermap = get_table_columnset(relid, targetcols, filtermap);
+ if (!bms_is_subset(idattrs, filtermap))
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot add relation \"%s\" to publication",
+ RelationGetRelationName(targetrel)),
+ errdetail("Column filter must include REPLICA IDENTITY columns")));
+ }
+ }
+ }
}
/*
@@ -270,6 +304,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectAddress myself,
referenced;
List *relids = NIL;
+ ListCell *lc;
+ List *target_cols = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -292,7 +328,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ foreach(lc, targetrel->columns)
+ {
+ char *colname;
+
+ colname = strVal(lfirst(lc));
+ target_cols = lappend(target_cols, colname);
+ }
+ check_publication_add_relation(targetrel->relation, target_cols);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -305,6 +348,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ values[Anum_pg_publication_rel_prattrs - 1] =
+ PointerGetDatum(strlist_to_textarray(target_cols));
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -313,7 +358,16 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
heap_freetuple(tup);
ObjectAddressSet(myself, PublicationRelRelationId, prrelid);
+ foreach(lc, target_cols)
+ {
+ int attnum;
+ attnum = get_attnum(relid, lfirst(lc));
+
+ /* Add dependency on the column */
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attnum);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 7d4a0e95f6..2b06deed6b 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -561,7 +561,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
pubrel = palloc(sizeof(PublicationRelInfo));
pubrel->relation = oldrel;
-
+ /* This is not needed to delete a table */
+ pubrel->columns = NIL;
delrels = lappend(delrels, pubrel);
}
}
@@ -932,6 +933,8 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = NIL;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
@@ -965,8 +968,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
+
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = NIL;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 297b6ee715..081b432d8f 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4832,6 +4832,7 @@ _copyPublicationTable(const PublicationTable *from)
PublicationTable *newnode = makeNode(PublicationTable);
COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f537d3eb96..f871e61f5f 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2312,6 +2312,7 @@ static bool
_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 86ce33bd97..695efd784d 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9654,12 +9654,13 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr
+ TABLE relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
+ $$->pubtable->columns = $3;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9674,28 +9675,37 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLE_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId
+ | ColId opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- $$->name = $1;
+ if ($2)
+ {
+ $$->pubtable = makeNode(PublicationTable);
+ $$->pubtable->relation = makeRangeVar(NULL, $1, @1);
+ $$->pubtable->columns = $2;
+ }
+ else
+ $$->name = $1;
$$->location = @1;
}
- | ColId indirection
+ | ColId indirection opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
+ $$->pubtable->columns = $3;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr
+ | extended_relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
+ $$->pubtable->columns = $2;
}
| CURRENT_SCHEMA
{
@@ -17356,6 +17366,16 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
else if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLE_IN_SCHEMA ||
pubobj->pubobjtype == PUBLICATIONOBJ_TABLE_IN_CUR_SCHEMA)
{
+ /*
+ * This can happen if a column list is specified in a continuation
+ * for a schema entry; reject it.
+ */
+ if (pubobj->pubtable)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schemas"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 9f5bf4b639..15d8192238 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,9 +29,9 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *att_map);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary, Bitmapset *att_map);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +398,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *att_map)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +410,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_map);
}
/*
@@ -442,7 +442,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary, Bitmapset *att_map)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -463,11 +463,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, att_map);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_map);
}
/*
@@ -536,7 +536,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -651,7 +651,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *att_map)
{
char *relname;
@@ -673,7 +673,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, att_map);
}
/*
@@ -749,20 +749,42 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary,
+ Bitmapset *att_map)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
bool isnull[MaxTupleAttributeNumber];
int i;
uint16 nliveatts = 0;
+ Bitmapset *idattrs = NULL;
+ bool replidentfull;
+ Form_pg_attribute att;
desc = RelationGetDescr(rel);
+ replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ if (!replidentfull)
+ idattrs = RelationGetIdentityKeyBitmap(rel);
+
for (i = 0; i < desc->natts; i++)
{
+ att = TupleDescAttr(desc, i);
if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
continue;
+
+ /*
+ * Do not increment count of attributes if not a part of column
+ * filters except for replica identity columns or if replica identity
+ * is full.
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map) &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs) &&
+ !replidentfull)
+ continue;
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -800,6 +822,19 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
continue;
}
+ /*
+ * Do not send attribute data if it is not a part of column filters,
+ * except if it is a part of REPLICA IDENTITY or REPLICA IDENTITY is
+ * full, send the data.
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map) &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs) &&
+ !replidentfull)
+ continue;
+
typtup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(att->atttypid));
if (!HeapTupleIsValid(typtup))
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
@@ -904,7 +939,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *att_map)
{
TupleDesc desc;
int i;
@@ -914,20 +949,35 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
desc = RelationGetDescr(rel);
- /* send number of live attributes */
- for (i = 0; i < desc->natts; i++)
- {
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
- continue;
- nliveatts++;
- }
- pq_sendint16(out, nliveatts);
-
/* fetch bitmap of REPLICATION IDENTITY attributes */
replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
if (!replidentfull)
idattrs = RelationGetIdentityKeyBitmap(rel);
+ /* send number of live attributes */
+ for (i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ /* REPLICA IDENTITY FULL means all columns are sent as part of key. */
+ if (replidentfull ||
+ bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs))
+ {
+ nliveatts++;
+ continue;
+ }
+ /* Skip sending if not a part of column filter */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map))
+ continue;
+ nliveatts++;
+ }
+ pq_sendint16(out, nliveatts);
+
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -937,6 +987,17 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ /*
+ * Exclude filtered columns, but REPLICA IDENTITY columns can't be
+ * excluded
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map) &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs)
+ && !replidentfull)
+ continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index f07983a43c..953304b200 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -111,6 +111,7 @@
#include "replication/origin.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -695,19 +696,28 @@ fetch_remote_table_info(char *nspname, char *relname,
LogicalRepRelation *lrel)
{
WalRcvExecResult *res;
+ WalRcvExecResult *res_pub;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
+ TupleTableSlot *slot_pub;
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid pubRow[] = {TEXTARRAYOID};
bool isnull;
- int natt;
+ int natt,
+ i;
+ Datum *elems;
+ int nelems;
+ List *pub_columns = NIL;
+ ListCell *lc;
+ bool am_partition = false;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -737,6 +747,7 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetChar(slot_getattr(slot, 4, &isnull));
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
@@ -774,11 +785,80 @@ fetch_remote_table_info(char *nspname, char *relname,
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+
+ /*
+ * Now, fetch the values of publications' column filters For a partition,
+ * use pg_inherit to find the parent, as the pg_publication_rel contains
+ * only the topmost parent table entry in case the table is partitioned.
+ * Run a recursive query to iterate through all the parents of the
+ * partition and retreive the record for the parent that exists in
+ * pg_publication_rel.
+ */
+ resetStringInfo(&cmd);
+ if (!am_partition)
+ appendStringInfo(&cmd, "SELECT prattrs from pg_publication_rel"
+ " WHERE prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd, "WITH RECURSIVE t(inhparent) AS ( SELECT inhparent from pg_inherits where inhrelid = %u"
+ " UNION SELECT pg.inhparent from pg_inherits pg, t where inhrelid = t.inhparent)"
+ " SELECT prattrs from pg_publication_rel WHERE prrelid IN (SELECT inhparent from t)", lrel->remoteid);
+
+ res_pub = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(pubRow), pubRow);
+
+ if (res_pub->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch published columns info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, res_pub->err)));
+ slot_pub = MakeSingleTupleTableSlot(res_pub->tupledesc, &TTSOpsMinimalTuple);
+
+ while (tuplestore_gettupleslot(res_pub->tuplestore, true, false, slot_pub))
+ {
+ deconstruct_array(DatumGetArrayTypePCopy(slot_getattr(slot_pub, 1, &isnull)),
+ TEXTOID, -1, false, 'i',
+ &elems, NULL, &nelems);
+ for (i = 0; i < nelems; i++)
+ pub_columns = lappend(pub_columns, TextDatumGetCString(elems[i]));
+ ExecClearTuple(slot_pub);
+ }
+ ExecDropSingleTupleTableSlot(slot_pub);
+ walrcv_clear_result(res_pub);
+
+ /*
+ * Store the column names only if they are contained in column filter
+ * LogicalRepRelation will only contain attributes corresponding to those
+ * specficied in column filters.
+ */
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ bool found = false;
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 1, &isnull));
Assert(!isnull);
+ if (pub_columns != NIL)
+ {
+ foreach(lc, pub_columns)
+ {
+ char *pub_colname = lfirst(lc);
+
+ if (!strcmp(pub_colname, rel_colname))
+ {
+ found = true;
+ lrel->attnames[natt] = rel_colname;
+ break;
+ }
+ }
+ }
+ else
+ {
+ found = true;
+ lrel->attnames[natt] = rel_colname;
+ }
+ if (!found)
+ continue;
+
lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
@@ -829,8 +909,17 @@ copy_table(Relation rel)
/* Start copy on the publisher. */
initStringInfo(&cmd);
if (lrel.relkind == RELKIND_RELATION)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 6f6a203dea..e33cb29cb3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,16 +15,19 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -81,7 +84,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *att_map);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -130,6 +134,7 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+ Bitmapset *att_map;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -570,11 +575,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
}
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->att_map);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->att_map);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -587,7 +592,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *att_map)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -610,13 +616,23 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /*
+ * Do not send type information if attribute is not present in column
+ * filter. XXX Allow sending type information for REPLICA IDENTITY
+ * COLUMNS with user created type. even when they are not mentioned in
+ * column filters.
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map))
+ continue;
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, att_map);
OutputPluginWrite(ctx, false);
}
@@ -693,7 +709,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->att_map);
OutputPluginWrite(ctx, true);
break;
}
@@ -722,7 +738,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->att_map);
OutputPluginWrite(ctx, true);
break;
}
@@ -1122,6 +1138,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
bool found;
+ Oid ancestor_id;
MemoryContext oldctx;
Assert(RelationSyncCache != NULL);
@@ -1142,6 +1159,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
+ entry->att_map = NULL;
entry->map = NULL; /* will be set by maybe_send_schema() if
* needed */
}
@@ -1182,6 +1200,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
if (pub->alltables)
{
@@ -1192,7 +1211,6 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (!publish)
{
- bool ancestor_published = false;
/*
* For a partition, check if any of the ancestors are
@@ -1219,6 +1237,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
pub->oid))
{
ancestor_published = true;
+ ancestor_id = ancestor;
if (pub->pubviaroot)
publish_as_relid = ancestor;
}
@@ -1239,15 +1258,49 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ int nelems,
+ i;
+ bool isnull;
+ Datum *elems;
+ HeapTuple pub_rel_tuple;
+ Datum pub_rel_cols;
+ List *columns = NIL;
+
+ pub_rel_tuple =
+ SearchSysCache2(PUBLICATIONRELMAP,
+ ancestor_published ? ObjectIdGetDatum(ancestor_id) :
+ ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP,
+ pub_rel_tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (!isnull)
+ {
+ /* FIXME it's a bad idea to use CacheMemoryContext
+ * directly here. Must produce the map first in a
+ * tmp context, then copy to CacheMemoryContext
+ */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ deconstruct_array(DatumGetArrayTypePCopy(pub_rel_cols),
+ TEXTOID, -1, false, 'i',
+ &elems, NULL, &nelems);
+ for (i = 0; i < nelems; i++)
+ columns = lappend(columns, TextDatumGetCString(elems[i]));
+ entry->att_map = get_table_columnset(publish_as_relid, columns, entry->att_map);
+ MemoryContextSwitchTo(oldctx);
+ }
+ ReleaseSysCache(pub_rel_tuple);
+ }
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
}
- if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
- break;
}
list_free(pubids);
@@ -1343,6 +1396,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->att_map);
+ entry->att_map = NULL;
if (entry->map)
{
/*
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 1ae439e6f3..d16e0218dc 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -86,6 +86,7 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index b5d5504cbb..11695e8478 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN
+ text prattrs[1]; /* Variable length field starts here */
+#endif
} FormData_pg_publication_rel;
/* ----------------
@@ -40,6 +43,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
*/
typedef FormData_pg_publication_rel *Form_pg_publication_rel;
+DECLARE_TOAST(pg_publication_rel, 8895, 8896);
DECLARE_UNIQUE_INDEX_PKEY(pg_publication_rel_oid_index, 6112, PublicationRelObjectIndexId, on pg_publication_rel using btree(oid oid_ops));
DECLARE_UNIQUE_INDEX(pg_publication_rel_prrelid_prpubid_index, 6113, PublicationRelPrrelidPrpubidIndexId, on pg_publication_rel using btree(prrelid oid_ops, prpubid oid_ops));
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 067138e6b5..f19e9508ba 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3641,6 +3641,7 @@ typedef struct PublicationTable
{
NodeTag type;
RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 83741dcf42..709b4be916 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -207,11 +207,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *att_map);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *att_map);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -228,7 +228,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *att_map);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 31281279cf..1a4eb490db 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -681,5 +681,6 @@ RelationGetSmgr(Relation rel)
/* routines in utils/cache/relcache.c */
extern void RelationIncrementReferenceCount(Relation rel);
extern void RelationDecrementReferenceCount(Relation rel);
+extern Bitmapset *get_table_columnset(Oid relid, List *columns, Bitmapset *att_map);
#endif /* REL_H */
--
2.30.2
Oh, I just noticed that for some reason the test file was lost in the
rebase, so those tests I thought I was running ... I wasn't. And of
course if I put it back, it fails.
More later.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
"Crear es tan difícil como ser libre" (Elsa Triolet)
On 2021-Dec-01, Alvaro Herrera wrote:
Hi
I took the latest posted patch, rebased on current sources, fixed the
conflicts, and pgindented. No further changes. Here's the result. All
tests are passing for me. Some review comments that were posted have
not been addressed yet; I'll look into that soon.
In v7 I have reinstated the test file and fixed the silly problem that
caused it to fail (probably a mistake of mine while rebasing).
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
Maybe there's lots of data loss but the records of data loss are also lost.
(Lincoln Yeoh)
Attachments:
v7-0001-Add-column-filtering-to-logical-replication.patchtext/x-diff; charset=utf-8Download
From 1fadd74c39967bca2807f5f7ad8f894ed7c4ad50 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 6 Sep 2021 10:34:29 -0300
Subject: [PATCH v7] Add column filtering to logical replication
Add capability to specifiy column names while linking
the table to a publication, at the time of CREATE or ALTER
publication. This will allow replicating only the specified
columns. Other columns, if any, on the subscriber will be populated
locally or NULL will be inserted if no value is supplied for the column
by the upstream during INSERT.
This facilitates replication to a table on subscriber
containing only the subscribed/filtered columns.
If no filter is specified, all the columns are replicated.
REPLICA IDENTITY columns are always replicated.
Thus, prohibit adding relation to publication, if column filters
do not contain REPLICA IDENTITY.
Add a tap test for the same in src/test/subscription.
Author: Rahila Syed <rahilasyed90@gmail.com>
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
src/backend/access/common/relation.c | 22 +++
src/backend/catalog/pg_publication.c | 58 +++++++-
src/backend/commands/publicationcmds.c | 8 +-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 36 ++++-
src/backend/replication/logical/proto.c | 103 ++++++++++---
src/backend/replication/logical/tablesync.c | 101 ++++++++++++-
src/backend/replication/pgoutput/pgoutput.c | 77 ++++++++--
src/include/catalog/pg_publication.h | 1 +
src/include/catalog/pg_publication_rel.h | 4 +
src/include/nodes/parsenodes.h | 1 +
src/include/replication/logicalproto.h | 6 +-
src/include/utils/rel.h | 1 +
src/test/subscription/t/021_column_filter.pl | 143 +++++++++++++++++++
15 files changed, 512 insertions(+), 51 deletions(-)
create mode 100644 src/test/subscription/t/021_column_filter.pl
diff --git a/src/backend/access/common/relation.c b/src/backend/access/common/relation.c
index 632d13c1ea..05d6fcba26 100644
--- a/src/backend/access/common/relation.c
+++ b/src/backend/access/common/relation.c
@@ -21,12 +21,14 @@
#include "postgres.h"
#include "access/relation.h"
+#include "access/sysattr.h"
#include "access/xact.h"
#include "catalog/namespace.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/lmgr.h"
#include "utils/inval.h"
+#include "utils/lsyscache.h"
#include "utils/syscache.h"
@@ -215,3 +217,23 @@ relation_close(Relation relation, LOCKMODE lockmode)
if (lockmode != NoLock)
UnlockRelationId(&relid, lockmode);
}
+
+/*
+ * Return a bitmapset of attributes given the list of column names
+ */
+Bitmapset *
+get_table_columnset(Oid relid, List *columns, Bitmapset *att_map)
+{
+ ListCell *cell;
+
+ foreach(cell, columns)
+ {
+ const char *attname = lfirst(cell);
+ int attnum = get_attnum(relid, attname);
+
+ if (!bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, att_map))
+ att_map = bms_add_member(att_map,
+ attnum - FirstLowInvalidHeapAttributeNumber);
+ }
+ return att_map;
+}
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 63579b2f82..de5f3266cd 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -50,8 +50,12 @@
* error if not.
*/
static void
-check_publication_add_relation(Relation targetrel)
+check_publication_add_relation(Relation targetrel, List *targetcols)
{
+ bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ Oid relid = RelationGetRelid(targetrel);
+ Bitmapset *idattrs;
+
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
@@ -82,6 +86,36 @@ check_publication_add_relation(Relation targetrel)
errmsg("cannot add relation \"%s\" to publication",
RelationGetRelationName(targetrel)),
errdetail("This operation is not supported for unlogged tables.")));
+
+ /*
+ * Cannot specify column filter when REPLICA IDENTITY IS FULL or if column
+ * filter does not contain REPLICA IDENITY columns
+ */
+ if (targetcols != NIL)
+ {
+ if (replidentfull)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot add relation \"%s\" to publication",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter with REPLICA IDENTITY FULL")));
+ else
+ {
+ Bitmapset *filtermap = NULL;
+
+ idattrs = RelationGetIndexAttrBitmap(targetrel, INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ filtermap = get_table_columnset(relid, targetcols, filtermap);
+ if (!bms_is_subset(idattrs, filtermap))
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot add relation \"%s\" to publication",
+ RelationGetRelationName(targetrel)),
+ errdetail("Column filter must include REPLICA IDENTITY columns")));
+ }
+ }
+ }
}
/*
@@ -270,6 +304,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectAddress myself,
referenced;
List *relids = NIL;
+ ListCell *lc;
+ List *target_cols = NIL;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -292,7 +328,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ foreach(lc, targetrel->columns)
+ {
+ char *colname;
+
+ colname = strVal(lfirst(lc));
+ target_cols = lappend(target_cols, colname);
+ }
+ check_publication_add_relation(targetrel->relation, target_cols);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -305,6 +348,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ values[Anum_pg_publication_rel_prattrs - 1] =
+ PointerGetDatum(strlist_to_textarray(target_cols));
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -313,7 +358,16 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
heap_freetuple(tup);
ObjectAddressSet(myself, PublicationRelRelationId, prrelid);
+ foreach(lc, target_cols)
+ {
+ int attnum;
+ attnum = get_attnum(relid, lfirst(lc));
+
+ /* Add dependency on the column */
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attnum);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 7d4a0e95f6..128ee40d41 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -561,7 +561,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
pubrel = palloc(sizeof(PublicationRelInfo));
pubrel->relation = oldrel;
-
+ /* This is not needed to delete a table */
+ pubrel->columns = NIL;
delrels = lappend(delrels, pubrel);
}
}
@@ -932,6 +933,8 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
@@ -965,8 +968,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
+
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 297b6ee715..081b432d8f 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4832,6 +4832,7 @@ _copyPublicationTable(const PublicationTable *from)
PublicationTable *newnode = makeNode(PublicationTable);
COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f537d3eb96..f871e61f5f 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2312,6 +2312,7 @@ static bool
_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 86ce33bd97..fba968bc8b 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9654,12 +9654,13 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr
+ TABLE relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
+ $$->pubtable->columns = $3;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9674,28 +9675,38 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLE_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId
+ | ColId opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- $$->name = $1;
+ if ($2 != NULL)
+ {
+ $$->pubtable = makeNode(PublicationTable);
+ $$->pubtable->relation = makeRangeVar(NULL, $1, @1);
+ $$->pubtable->columns = $2;
+ $$->name = NULL;
+ }
+ else
+ $$->name = $1;
$$->location = @1;
}
- | ColId indirection
+ | ColId indirection opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
+ $$->pubtable->columns = $3;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr
+ | extended_relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
+ $$->pubtable->columns = $2;
}
| CURRENT_SCHEMA
{
@@ -17347,8 +17358,9 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
{
/* convert it to PublicationTable */
PublicationTable *pubtable = makeNode(PublicationTable);
- pubtable->relation = makeRangeVar(NULL, pubobj->name,
- pubobj->location);
+
+ pubtable->relation =
+ makeRangeVar(NULL, pubobj->name, pubobj->location);
pubobj->pubtable = pubtable;
pubobj->name = NULL;
}
@@ -17356,6 +17368,16 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
else if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLE_IN_SCHEMA ||
pubobj->pubobjtype == PUBLICATIONOBJ_TABLE_IN_CUR_SCHEMA)
{
+ /*
+ * This can happen if a column list is specified in a continuation
+ * for a schema entry; reject it.
+ */
+ if (pubobj->pubtable)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schemas"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 9f5bf4b639..15d8192238 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,9 +29,9 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *att_map);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary, Bitmapset *att_map);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +398,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *att_map)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +410,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_map);
}
/*
@@ -442,7 +442,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary, Bitmapset *att_map)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -463,11 +463,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, att_map);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_map);
}
/*
@@ -536,7 +536,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -651,7 +651,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *att_map)
{
char *relname;
@@ -673,7 +673,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, att_map);
}
/*
@@ -749,20 +749,42 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary,
+ Bitmapset *att_map)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
bool isnull[MaxTupleAttributeNumber];
int i;
uint16 nliveatts = 0;
+ Bitmapset *idattrs = NULL;
+ bool replidentfull;
+ Form_pg_attribute att;
desc = RelationGetDescr(rel);
+ replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ if (!replidentfull)
+ idattrs = RelationGetIdentityKeyBitmap(rel);
+
for (i = 0; i < desc->natts; i++)
{
+ att = TupleDescAttr(desc, i);
if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
continue;
+
+ /*
+ * Do not increment count of attributes if not a part of column
+ * filters except for replica identity columns or if replica identity
+ * is full.
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map) &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs) &&
+ !replidentfull)
+ continue;
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -800,6 +822,19 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
continue;
}
+ /*
+ * Do not send attribute data if it is not a part of column filters,
+ * except if it is a part of REPLICA IDENTITY or REPLICA IDENTITY is
+ * full, send the data.
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map) &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs) &&
+ !replidentfull)
+ continue;
+
typtup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(att->atttypid));
if (!HeapTupleIsValid(typtup))
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
@@ -904,7 +939,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *att_map)
{
TupleDesc desc;
int i;
@@ -914,20 +949,35 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
desc = RelationGetDescr(rel);
- /* send number of live attributes */
- for (i = 0; i < desc->natts; i++)
- {
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
- continue;
- nliveatts++;
- }
- pq_sendint16(out, nliveatts);
-
/* fetch bitmap of REPLICATION IDENTITY attributes */
replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
if (!replidentfull)
idattrs = RelationGetIdentityKeyBitmap(rel);
+ /* send number of live attributes */
+ for (i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ /* REPLICA IDENTITY FULL means all columns are sent as part of key. */
+ if (replidentfull ||
+ bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs))
+ {
+ nliveatts++;
+ continue;
+ }
+ /* Skip sending if not a part of column filter */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map))
+ continue;
+ nliveatts++;
+ }
+ pq_sendint16(out, nliveatts);
+
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -937,6 +987,17 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ /*
+ * Exclude filtered columns, but REPLICA IDENTITY columns can't be
+ * excluded
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map) &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs)
+ && !replidentfull)
+ continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index f07983a43c..953304b200 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -111,6 +111,7 @@
#include "replication/origin.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -695,19 +696,28 @@ fetch_remote_table_info(char *nspname, char *relname,
LogicalRepRelation *lrel)
{
WalRcvExecResult *res;
+ WalRcvExecResult *res_pub;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
+ TupleTableSlot *slot_pub;
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid pubRow[] = {TEXTARRAYOID};
bool isnull;
- int natt;
+ int natt,
+ i;
+ Datum *elems;
+ int nelems;
+ List *pub_columns = NIL;
+ ListCell *lc;
+ bool am_partition = false;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -737,6 +747,7 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetChar(slot_getattr(slot, 4, &isnull));
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
@@ -774,11 +785,80 @@ fetch_remote_table_info(char *nspname, char *relname,
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+
+ /*
+ * Now, fetch the values of publications' column filters For a partition,
+ * use pg_inherit to find the parent, as the pg_publication_rel contains
+ * only the topmost parent table entry in case the table is partitioned.
+ * Run a recursive query to iterate through all the parents of the
+ * partition and retreive the record for the parent that exists in
+ * pg_publication_rel.
+ */
+ resetStringInfo(&cmd);
+ if (!am_partition)
+ appendStringInfo(&cmd, "SELECT prattrs from pg_publication_rel"
+ " WHERE prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd, "WITH RECURSIVE t(inhparent) AS ( SELECT inhparent from pg_inherits where inhrelid = %u"
+ " UNION SELECT pg.inhparent from pg_inherits pg, t where inhrelid = t.inhparent)"
+ " SELECT prattrs from pg_publication_rel WHERE prrelid IN (SELECT inhparent from t)", lrel->remoteid);
+
+ res_pub = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(pubRow), pubRow);
+
+ if (res_pub->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch published columns info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, res_pub->err)));
+ slot_pub = MakeSingleTupleTableSlot(res_pub->tupledesc, &TTSOpsMinimalTuple);
+
+ while (tuplestore_gettupleslot(res_pub->tuplestore, true, false, slot_pub))
+ {
+ deconstruct_array(DatumGetArrayTypePCopy(slot_getattr(slot_pub, 1, &isnull)),
+ TEXTOID, -1, false, 'i',
+ &elems, NULL, &nelems);
+ for (i = 0; i < nelems; i++)
+ pub_columns = lappend(pub_columns, TextDatumGetCString(elems[i]));
+ ExecClearTuple(slot_pub);
+ }
+ ExecDropSingleTupleTableSlot(slot_pub);
+ walrcv_clear_result(res_pub);
+
+ /*
+ * Store the column names only if they are contained in column filter
+ * LogicalRepRelation will only contain attributes corresponding to those
+ * specficied in column filters.
+ */
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ bool found = false;
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 1, &isnull));
Assert(!isnull);
+ if (pub_columns != NIL)
+ {
+ foreach(lc, pub_columns)
+ {
+ char *pub_colname = lfirst(lc);
+
+ if (!strcmp(pub_colname, rel_colname))
+ {
+ found = true;
+ lrel->attnames[natt] = rel_colname;
+ break;
+ }
+ }
+ }
+ else
+ {
+ found = true;
+ lrel->attnames[natt] = rel_colname;
+ }
+ if (!found)
+ continue;
+
lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
@@ -829,8 +909,17 @@ copy_table(Relation rel)
/* Start copy on the publisher. */
initStringInfo(&cmd);
if (lrel.relkind == RELKIND_RELATION)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 6f6a203dea..e33cb29cb3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,16 +15,19 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -81,7 +84,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *att_map);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -130,6 +134,7 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+ Bitmapset *att_map;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -570,11 +575,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
}
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->att_map);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->att_map);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -587,7 +592,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *att_map)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -610,13 +616,23 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /*
+ * Do not send type information if attribute is not present in column
+ * filter. XXX Allow sending type information for REPLICA IDENTITY
+ * COLUMNS with user created type. even when they are not mentioned in
+ * column filters.
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map))
+ continue;
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, att_map);
OutputPluginWrite(ctx, false);
}
@@ -693,7 +709,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->att_map);
OutputPluginWrite(ctx, true);
break;
}
@@ -722,7 +738,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->att_map);
OutputPluginWrite(ctx, true);
break;
}
@@ -1122,6 +1138,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
bool found;
+ Oid ancestor_id;
MemoryContext oldctx;
Assert(RelationSyncCache != NULL);
@@ -1142,6 +1159,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
+ entry->att_map = NULL;
entry->map = NULL; /* will be set by maybe_send_schema() if
* needed */
}
@@ -1182,6 +1200,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
if (pub->alltables)
{
@@ -1192,7 +1211,6 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (!publish)
{
- bool ancestor_published = false;
/*
* For a partition, check if any of the ancestors are
@@ -1219,6 +1237,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
pub->oid))
{
ancestor_published = true;
+ ancestor_id = ancestor;
if (pub->pubviaroot)
publish_as_relid = ancestor;
}
@@ -1239,15 +1258,49 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ int nelems,
+ i;
+ bool isnull;
+ Datum *elems;
+ HeapTuple pub_rel_tuple;
+ Datum pub_rel_cols;
+ List *columns = NIL;
+
+ pub_rel_tuple =
+ SearchSysCache2(PUBLICATIONRELMAP,
+ ancestor_published ? ObjectIdGetDatum(ancestor_id) :
+ ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP,
+ pub_rel_tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (!isnull)
+ {
+ /* FIXME it's a bad idea to use CacheMemoryContext
+ * directly here. Must produce the map first in a
+ * tmp context, then copy to CacheMemoryContext
+ */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ deconstruct_array(DatumGetArrayTypePCopy(pub_rel_cols),
+ TEXTOID, -1, false, 'i',
+ &elems, NULL, &nelems);
+ for (i = 0; i < nelems; i++)
+ columns = lappend(columns, TextDatumGetCString(elems[i]));
+ entry->att_map = get_table_columnset(publish_as_relid, columns, entry->att_map);
+ MemoryContextSwitchTo(oldctx);
+ }
+ ReleaseSysCache(pub_rel_tuple);
+ }
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
}
- if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
- break;
}
list_free(pubids);
@@ -1343,6 +1396,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->att_map);
+ entry->att_map = NULL;
if (entry->map)
{
/*
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 1ae439e6f3..d16e0218dc 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -86,6 +86,7 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index b5d5504cbb..11695e8478 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN
+ text prattrs[1]; /* Variable length field starts here */
+#endif
} FormData_pg_publication_rel;
/* ----------------
@@ -40,6 +43,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
*/
typedef FormData_pg_publication_rel *Form_pg_publication_rel;
+DECLARE_TOAST(pg_publication_rel, 8895, 8896);
DECLARE_UNIQUE_INDEX_PKEY(pg_publication_rel_oid_index, 6112, PublicationRelObjectIndexId, on pg_publication_rel using btree(oid oid_ops));
DECLARE_UNIQUE_INDEX(pg_publication_rel_prrelid_prpubid_index, 6113, PublicationRelPrrelidPrpubidIndexId, on pg_publication_rel using btree(prrelid oid_ops, prpubid oid_ops));
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 067138e6b5..f19e9508ba 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3641,6 +3641,7 @@ typedef struct PublicationTable
{
NodeTag type;
RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 83741dcf42..709b4be916 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -207,11 +207,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *att_map);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *att_map);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -228,7 +228,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *att_map);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 31281279cf..1a4eb490db 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -681,5 +681,6 @@ RelationGetSmgr(Relation rel)
/* routines in utils/cache/relcache.c */
extern void RelationIncrementReferenceCount(Relation rel);
extern void RelationDecrementReferenceCount(Relation rel);
+extern Bitmapset *get_table_columnset(Oid relid, List *columns, Bitmapset *att_map);
#endif /* REL_H */
diff --git a/src/test/subscription/t/021_column_filter.pl b/src/test/subscription/t/021_column_filter.pl
new file mode 100644
index 0000000000..7b52d4593d
--- /dev/null
+++ b/src/test/subscription/t/021_column_filter.pl
@@ -0,0 +1,143 @@
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test TRUNCATE
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 9;
+
+# setup
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int)");
+# Test with weird column names
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, B varchar, \"c'\" int)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+#Test replication with multi-level partition
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, \"c'\" int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+#Test create publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub1 FOR TABLE tab1(a, \"B\"), tab3(\"a'\",\"c'\"), test_part(a,b)");
+
+my $result = $node_publisher->safe_psql('postgres',
+ "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
+is($result, qq(tab1|{a,B}
+tab3|{a',c'}
+test_part|{a,b}), 'publication relation updated');
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+);
+#Initial sync
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab1 VALUES (1,2,3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 VALUES (1,2,3)");
+#Test for replication of partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (1,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (2,'bcd', '2021-07-03 11:12:13')");
+#Test for replication of multi-level partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (4,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (5,'bcd', '2021-07-03 11:12:13')");
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'insert on column c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab3");
+is($result, qq(1|3), 'insert on column b is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part");
+is($result, qq(1|abc\n2|bcd\n4|abc\n5|bcd), 'insert on all columns is replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab1 SET c = 5 where a = 1");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'update on column c is not replicated');
+
+#Test alter publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION"
+);
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 VALUES (1,'abc',3)");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2");
+is($result, qq(1|abc), 'insert on column c is not replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET c = 5 where a = 1");
+is($result, qq(1|abc), 'update on column c is not replicated');
+
+#Test error conditions
+my ($psql_rc, $psql_out, $psql_err) = $node_publisher->psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE test_part(b)");
+like($psql_err, qr/Column filter must include REPLICA IDENTITY columns/, 'Error when column filter does not contain REPLICA IDENTITY');
+
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE test_part DROP COLUMN b");
+$result = $node_publisher->safe_psql('postgres',
+ "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
+is($result, qq(tab1|{a,B}
+tab2|{a,b}
+tab3|{a',c'}), 'publication relation test_part removed');
--
2.30.2
On 2021-Sep-16, Peter Smith wrote:
I noticed that the latest v5 no longer includes the TAP test which was
in the v4 patch.(src/test/subscription/t/021_column_filter.pl)
Was that omission deliberate?
Somehow I not only failed to notice the omission, but also your email
where you told us about it. I have since posted a version of the patch
that again includes it.
Thanks!
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"No renuncies a nada. No te aferres a nada."
On Fri, Dec 3, 2021 at 12:45 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Sep-16, Peter Smith wrote:
I noticed that the latest v5 no longer includes the TAP test which was
in the v4 patch.(src/test/subscription/t/021_column_filter.pl)
Was that omission deliberate?
Somehow I not only failed to notice the omission, but also your email
where you told us about it. I have since posted a version of the patch
that again includes it.
Thanks for the patch, Few comments:
I had a look at the patch, I felt the following should be handled:
1) Dump changes to include the column filters while adding table to
publication in dumpPublicationTable
2) Documentation changes for column filtering in create_publication.sgml
3) describe publication changes to support \dRp command in describePublications
4) I felt we need not allow specifying columns in case of "alter
publication drop table" as currently dropping column filter is not
allowed.
5) We should check if the column specified is present in the table,
currently we are able to specify non existent column for column
filtering
+ foreach(lc, targetrel->columns)
+ {
+ char *colname;
+
+ colname = strVal(lfirst(lc));
+ target_cols = lappend(target_cols, colname);
+ }
+ check_publication_add_relation(targetrel->relation, target_cols);
Regards,
Vignesh
On 02.12.21 15:23, Alvaro Herrera wrote:
I took the latest posted patch, rebased on current sources, fixed the
conflicts, and pgindented. No further changes. Here's the result. All
tests are passing for me. Some review comments that were posted have
not been addressed yet; I'll look into that soon.In v7 I have reinstated the test file and fixed the silly problem that
caused it to fail (probably a mistake of mine while rebasing).
I looked through this a bit. You had said that you are still going to
integrate past review comments, so I didn't look to deeply before you
get to that.
Attached are a few fixup patches that you could integrate.
There was no documentation, so I wrote a bit (patch 0001). It only
touches the CREATE PUBLICATION and ALTER PUBLICATION pages at the
moment. There was no mention in the Logical Replication chapter that
warranted updating. Perhaps we should revisit that chapter at the end
of the release cycle.
DDL tests should be done in src/test/regress/sql/publication.sql rather
than through TAP tests, to keep it simpler. I have added a few that I
came up with (patch 0002). Note the FIXME marker that it does not
recognize if the listed columns don't exist. I removed a now redundant
test from the TAP test file. The other error condition test in the TAP
test file ('publication relation test_part removed') I didn't
understand: test_part was added with columns (a, b), so why would
dropping column b remove the whole entry? Maybe I missed something, or
this could be explained better.
I was curious what happens when you have different publications with
different column lists, so I wrote a test for that (patch 0003). It
turns out it works, so there is nothing to do, but perhaps the test is
useful to keep.
The test file 021_column_filter.pl should be renamed to an unused number
(would be 027 currently). Also, it contains references to "TRUNCATE",
where it was presumably copied from.
On the implementation side, I think the added catalog column
pg_publication_rel.prattrs should be an int2 array, not a text array.
That would also fix the above problem. If you have to look up the
columns at DDL time, then you will notice when they don't exist.
Finally, I suggest not naming this feature "column filter". I think
this name arose because of the analogy with the "row filter" feature
also being developed. But a filter is normally a dynamic data-driven
action, which this is not. Golden Gate calls it in their documentation
"Selecting Columns", or we could just call it "column list".
Attachments:
0001-Add-some-documentation.patchtext/plain; charset=UTF-8; name=0001-Add-some-documentation.patchDownload
From cc9082fe6f98e5d9d992377761d06b8aadf3b27f Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Fri, 10 Dec 2021 13:19:52 +0100
Subject: [PATCH 1/3] Add some documentation
---
doc/src/sgml/ref/alter_publication.sgml | 4 +++-
doc/src/sgml/ref/create_publication.sgml | 11 ++++++++++-
2 files changed, 13 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index bb4ef5e5e2..c86055b93c 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -30,7 +30,7 @@
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -110,6 +110,8 @@ <title>Parameters</title>
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index d805e8e77a..73a23cbb02 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -78,6 +78,15 @@ <title>Parameters</title>
publication, so they are never explicitly added to the publication.
</para>
+ <para>
+ When a column list is specified, only the listed columns are replicated;
+ any other columns are ignored for the purpose of replication through
+ this publication. If no column list is specified, all columns of the
+ table are replicated through this publication, including any columns
+ added later. If a column list is specified, it must include the replica
+ identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
--
2.34.1
0002-Add-regression-tests.patchtext/plain; charset=UTF-8; name=0002-Add-regression-tests.patchDownload
From dadc41b139c352811f5956892c02135905263347 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Fri, 10 Dec 2021 13:40:50 +0100
Subject: [PATCH 2/3] Add regression tests
---
src/test/regress/expected/publication.out | 16 +++++++++++++++-
src/test/regress/sql/publication.sql | 12 +++++++++++-
src/test/subscription/t/021_column_filter.pl | 6 +-----
3 files changed, 27 insertions(+), 7 deletions(-)
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 5ac2d666a2..c6017c0514 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -165,7 +165,21 @@ Publications:
regress_publication_user | t | t | t | f | f | f
(1 row)
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (x, y, z); -- error
+ERROR: cannot add relation "testpub_tbl5" to publication
+DETAIL: Column filter must include REPLICA IDENTITY columns
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error FIXME
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ERROR: relation "testpub_tbl5" is already member of publication "testpub_fortable"
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ERROR: relation "testpub_tbl5" is already member of publication "testpub_fortable"
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ERROR: cannot add relation "testpub_tbl6" to publication
+DETAIL: Cannot have column filter with REPLICA IDENTITY FULL
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
CREATE TABLE testpub_tbl3a (b text) INHERITS (testpub_tbl3);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 56dd358554..b2fd793c61 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -89,7 +89,17 @@ CREATE PUBLICATION testpub_for_tbl_schema FOR ALL TABLES IN SCHEMA pub_test, TAB
\d+ testpub_tbl2
\dRp+ testpub_foralltables
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (x, y, z); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error FIXME
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
diff --git a/src/test/subscription/t/021_column_filter.pl b/src/test/subscription/t/021_column_filter.pl
index 7b52d4593d..02e05420f1 100644
--- a/src/test/subscription/t/021_column_filter.pl
+++ b/src/test/subscription/t/021_column_filter.pl
@@ -5,7 +5,7 @@
use warnings;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
-use Test::More tests => 9;
+use Test::More tests => 8;
# setup
@@ -130,10 +130,6 @@
is($result, qq(1|abc), 'update on column c is not replicated');
#Test error conditions
-my ($psql_rc, $psql_out, $psql_err) = $node_publisher->psql('postgres',
- "CREATE PUBLICATION pub2 FOR TABLE test_part(b)");
-like($psql_err, qr/Column filter must include REPLICA IDENTITY columns/, 'Error when column filter does not contain REPLICA IDENTITY');
-
$node_publisher->safe_psql('postgres',
"ALTER TABLE test_part DROP COLUMN b");
$result = $node_publisher->safe_psql('postgres',
--
2.34.1
0003-Test-case-for-overlapping-column-lists.patchtext/plain; charset=UTF-8; name=0003-Test-case-for-overlapping-column-lists.patchDownload
From 8f8307000fcd047f3a11ddc4366e6fd386908296 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Fri, 10 Dec 2021 13:51:38 +0100
Subject: [PATCH 3/3] Test case for overlapping column lists
---
src/test/subscription/t/021_column_filter.pl | 17 ++++++++++++++++-
1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/src/test/subscription/t/021_column_filter.pl b/src/test/subscription/t/021_column_filter.pl
index 02e05420f1..3814ad2367 100644
--- a/src/test/subscription/t/021_column_filter.pl
+++ b/src/test/subscription/t/021_column_filter.pl
@@ -5,7 +5,7 @@
use warnings;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
-use Test::More tests => 8;
+use Test::More tests => 9;
# setup
@@ -137,3 +137,18 @@
is($result, qq(tab1|{a,B}
tab2|{a,b}
tab3|{a',c'}), 'publication relation test_part removed');
+
+
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab4 (a int PRIMARY KEY, b int, c int, d int)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab4 (a int PRIMARY KEY, b int, d int)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub2 FOR TABLE tab4 (a, b)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub3 FOR TABLE tab4 (a, d)");
+$node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2, pub3");
+$node_publisher->wait_for_catchup('sub2');
+$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (1, 11, 111, 1111)");
+$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (2, 22, 222, 2222)");
+$node_publisher->wait_for_catchup('sub2');
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab4;"),
+ qq(1|11|1111
+2|22|2222),
+ 'overlapping publications with overlapping column lists');
--
2.34.1
On 2021-Dec-10, Peter Eisentraut wrote:
I looked through this a bit. You had said that you are still going to
integrate past review comments, so I didn't look to deeply before you get to
that.
Thanks for doing this! As it happens I've spent the last couple of days
working on some of these details.
There was no documentation, so I wrote a bit (patch 0001). It only touches
the CREATE PUBLICATION and ALTER PUBLICATION pages at the moment. There was
no mention in the Logical Replication chapter that warranted updating.
Perhaps we should revisit that chapter at the end of the release cycle.
Thanks. I hadn't looked at the docs yet, so I'll definitely take this.
DDL tests should be done in src/test/regress/sql/publication.sql rather than
through TAP tests, to keep it simpler.
Yeah, I noticed this too but hadn't done it yet.
Note the FIXME marker that it does not recognize if the
listed columns don't exist.
I had fixed this already, so I suppose it should be okay.
I removed a now redundant test from the TAP
test file. The other error condition test in the TAP test file
('publication relation test_part removed') I didn't understand: test_part
was added with columns (a, b), so why would dropping column b remove the
whole entry? Maybe I missed something, or this could be explained better.
There was some discussion about it earlier in the thread and I was also
against this proposed behavior.
I was curious what happens when you have different publications with
different column lists, so I wrote a test for that (patch 0003). It turns
out it works, so there is nothing to do, but perhaps the test is useful to
keep.
Great, thanks. Yes, I think it will be.
On the implementation side, I think the added catalog column
pg_publication_rel.prattrs should be an int2 array, not a text array.
I already rewrote it to use a int2vector column in pg_publication_rel.
This interacted badly with the previous behavior on dropping columns,
which I have to revisit, but otherwise it seems much better.
(Particularly since we don't need to care about quoting names and such.)
Finally, I suggest not naming this feature "column filter". I think this
name arose because of the analogy with the "row filter" feature also being
developed. But a filter is normally a dynamic data-driven action, which
this is not. Golden Gate calls it in their documentation "Selecting
Columns", or we could just call it "column list".
Hmm, I hadn't thought of renaming the feature, but I have to admit that
I was confused because of the name, so I agree with choosing some other
name.
I'll integrate your changes and post the whole thing later.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
Si no sabes adonde vas, es muy probable que acabes en otra parte.
On 2021-Sep-02, Alvaro Herrera wrote:
On 2021-Sep-02, Rahila Syed wrote:
After thinking about this, I think it is best to remove the entire table
from publication,
if a column specified in the column filter is dropped from the table.Hmm, I think it would be cleanest to give responsibility to the user: if
the column to be dropped is in the filter, then raise an error, aborting
the drop. Then it is up to them to figure out what to do.
I thought about this some more and realized that our earlier conclusions
were wrong or at least inconvenient. I think that the best behavior if
you drop a column from a table is to remove the column from the
publication column list, and do nothing else.
Consider the case where you add a table to a publication without a
column filter, and later drop the column. You don't get an error that
the relation is part of a publication; simply, the subscribers of that
publication will no longer receive that column.
Similarly for this case: if you add a table to a publication with a
column list, and later drop a column in that list, then you shouldn't
get an error either. Simply the subscribers of that publication should
receive one column less.
Should be fairly quick to implement ... on it now.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"The problem with the facetime model is not just that it's demoralizing, but
that the people pretending to work interrupt the ones actually working."
(Paul Graham)
On 2021-Dec-10, Alvaro Herrera wrote:
I thought about this some more and realized that our earlier conclusions
were wrong or at least inconvenient. I think that the best behavior if
you drop a column from a table is to remove the column from the
publication column list, and do nothing else.
Should be fairly quick to implement ... on it now.
Actually it's not so easy to implement.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"No hay ausente sin culpa ni presente sin disculpa" (Prov. francés)
On 2021-Dec-10, Alvaro Herrera wrote:
Actually it's not so easy to implement.
So I needed to add "sub object id" support for pg_publication_rel
objects in pg_depend / dependency.c. What I have now is partial (the
describe routines need patched) but it's sufficient to show what's
needed. In essence, we now set these depend entries with column
numbers, so that they can be dropped independently; when the drop comes,
the existing pg_publication_rel row is modified to cover the remaining
columns. As far as I can tell, it works correctly.
There is one policy decision to make: what if ALTER TABLE drops the last
remaining column in the publication? I opted to raise a specific error
in this case, though we could just the same opt to drop the relation
from the publication. Are there opinions on this?
This version incorporates the fixups Peter submitted, plus some other
fixes of my own. Notably, as Peter also mentioned, I changed
pg_publication_rel.prattrs to store int2vector rather than an array of
column names. This makes for better behavior if columns are renamed and
things like that, and also we don't need to be so cautious about
quoting. It does mean we need a slightly more complicated query in a
couple of spots, but that should be okay.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
"Always assume the user will do much worse than the stupidest thing
you can imagine." (Julien PUYDT)
Attachments:
column-filtering-8.patchtext/x-diff; charset=utf-8Download
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index fe9c714257..a88d12e8ae 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -1472,7 +1472,7 @@ doDeletion(const ObjectAddress *object, int flags)
break;
case OCLASS_PUBLICATION_REL:
- RemovePublicationRelById(object->objectId);
+ RemovePublicationRelById(object->objectId, object->objectSubId);
break;
case OCLASS_PUBLICATION:
@@ -2754,8 +2754,12 @@ free_object_addresses(ObjectAddresses *addrs)
ObjectClass
getObjectClass(const ObjectAddress *object)
{
- /* only pg_class entries can have nonzero objectSubId */
+ /*
+ * only pg_class and pg_publication_rel entries can have nonzero
+ * objectSubId
+ */
if (object->classId != RelationRelationId &&
+ object->classId != PublicationRelRelationId &&
object->objectSubId != 0)
elog(ERROR, "invalid non-zero objectSubId for object class %u",
object->classId);
diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c
index 2bae3fbb17..5eed248dcb 100644
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -4019,6 +4019,7 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok)
/* translator: first %s is, e.g., "table %s" */
appendStringInfo(&buffer, _("publication of %s in publication %s"),
rel.data, pubname);
+ /* FIXME add objectSubId support */
pfree(rel.data);
ReleaseSysCache(tup);
break;
@@ -5853,9 +5854,16 @@ getObjectIdentityParts(const ObjectAddress *object,
getRelationIdentity(&buffer, prform->prrelid, objname, false);
appendStringInfo(&buffer, " in publication %s", pubname);
+ if (object->objectSubId) /* FIXME maybe get_attname */
+ appendStringInfo(&buffer, " column %d", object->objectSubId);
if (objargs)
+ {
*objargs = list_make1(pubname);
+ if (object->objectSubId)
+ *objargs = lappend(*objargs,
+ psprintf("%d", object->objectSubId));
+ }
ReleaseSysCache(tup);
break;
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 07bcdc463a..462c8efe70 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -658,6 +658,56 @@ isObjectPinned(const ObjectAddress *object)
* Various special-purpose lookups and manipulations of pg_depend.
*/
+/*
+ * Find all objects of the given class that reference the specified object,
+ * and add them to the given ObjectAddresses.
+ */
+void
+findAndAddAddresses(ObjectAddresses *addrs, Oid classId,
+ Oid refclassId, Oid refobjectId, int32 refobjsubId)
+{
+ Relation depRel;
+ ScanKeyData key[3];
+ SysScanDesc scan;
+ HeapTuple tup;
+
+ depRel = table_open(DependRelationId, AccessShareLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_refclassid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(refclassId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_refobjid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(refobjectId));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_refobjsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum(refobjsubId));
+
+ scan = systable_beginscan(depRel, DependReferenceIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+ ObjectAddress object;
+
+ if (depform->classid != classId)
+ continue;
+
+ ObjectAddressSubSet(object, depform->classid, depform->objid,
+ depform->refobjsubid);
+
+ add_exact_object_address(&object, addrs);
+ }
+
+ systable_endscan(scan);
+
+ table_close(depRel, AccessShareLock);
+}
+
/*
* Find the extension containing the specified object, if any
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index a5229aea51..b99b3748c9 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -328,6 +328,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
{
table_close(rel, RowExclusiveLock);
+ /* FIXME need to handle the case of different column list */
+
if (if_not_exists)
return InvalidObjectAddress;
@@ -395,12 +397,6 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectAddressSet(myself, PublicationRelRelationId, prrelid);
- while ((attnum = bms_first_member(attmap)) >= 0)
- {
- /* Add dependency on the column */
- ObjectAddressSubSet(referenced, RelationRelationId, relid, attnum);
- recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
- }
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -409,6 +405,21 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectAddressSet(referenced, RelationRelationId, relid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ /*
+ * If there's an explicit column list, make one dependency entry for each
+ * column. Note that the referencing side of the dependency is also
+ * specific to one column, so that it can be dropped separately if the
+ * column is dropped.
+ */
+ while ((attnum = bms_first_member(attmap)) >= 0)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid,
+ attnum + FirstLowInvalidHeapAttributeNumber);
+ myself.objectSubId = attnum + FirstLowInvalidHeapAttributeNumber;
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL);
+ }
+ myself.objectSubId = 0; /* need to undo this bit */
+
/* Close the table. */
table_close(rel, RowExclusiveLock);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 753df44613..0078c5986c 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -561,7 +561,6 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
pubrel = palloc(sizeof(PublicationRelInfo));
pubrel->relation = oldrel;
- /* This is not needed to delete a table */
pubrel->columns = NIL;
delrels = lappend(delrels, pubrel);
}
@@ -758,10 +757,11 @@ AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt)
}
/*
- * Remove relation from publication by mapping OID.
+ * Remove relation from publication by mapping OID, or publication status
+ * of one column of that relation in the publication if an attnum is given.
*/
void
-RemovePublicationRelById(Oid proid)
+RemovePublicationRelById(Oid proid, int32 attnum)
{
Relation rel;
HeapTuple tup;
@@ -791,7 +791,81 @@ RemovePublicationRelById(Oid proid)
InvalidatePublicationRels(relids);
- CatalogTupleDelete(rel, &tup->t_self);
+ /*
+ * If no column is given, simply delete the relation from the publication.
+ *
+ * If a column is given, what we do instead is to remove that column from
+ * the column list. The relation remains in the publication, with the
+ * other columns. However, dropping the last column is disallowed.
+ */
+ if (attnum == 0)
+ {
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ else
+ {
+ Datum adatum;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ int16 *newelems;
+ int2vector *newvec;
+ Datum values[Natts_pg_publication_rel];
+ bool nulls[Natts_pg_publication_rel];
+ bool replace[Natts_pg_publication_rel];
+ HeapTuple newtup;
+ int i,
+ j;
+ bool isnull;
+
+ /* Obtain the original column list */
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP,
+ tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull) /* shouldn't happen */
+ elog(ERROR, "can't drop column from publication without a column list");
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* Construct a list excluding the given column */
+ newelems = palloc(sizeof(int16) * nelems - 1);
+ for (i = 0, j = 0; i < nelems - 1; i++)
+ {
+ if (elems[i] == attnum)
+ continue;
+ newelems[j++] = elems[i];
+ }
+
+ /*
+ * If this is the last column used in the publication, disallow the
+ * command. We could alternatively just drop the relation from the
+ * publication.
+ */
+ if (j == 0)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot drop the last column in publication \"%s\"",
+ get_publication_name(pubrel->prpubid, false)),
+ errhint("Remove table \"%s\" from the publication first.",
+ get_rel_name(pubrel->prrelid)));
+ }
+
+ /* Build the updated tuple */
+ MemSet(values, 0, sizeof(values));
+ MemSet(nulls, false, sizeof(nulls));
+ MemSet(replace, false, sizeof(replace));
+ newvec = buildint2vector(newelems, j);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(newvec);
+ replace[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ /* Execute the update */
+ newtup = heap_modify_tuple(tup, RelationGetDescr(rel),
+ values, nulls, replace);
+ CatalogTupleUpdate(rel, &tup->t_self, newtup);
+ }
ReleaseSysCache(tup);
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index c821271306..705bddc773 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -40,8 +40,9 @@
#include "catalog/pg_inherits.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
-#include "catalog/pg_tablespace.h"
+#include "catalog/pg_publication_rel.h"
#include "catalog/pg_statistic_ext.h"
+#include "catalog/pg_tablespace.h"
#include "catalog/pg_trigger.h"
#include "catalog/pg_type.h"
#include "catalog/storage.h"
@@ -8415,6 +8416,13 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * If the column is part of a replication column list, arrange to get that
+ * removed too.
+ */
+ findAndAddAddresses(addrs, PublicationRelRelationId,
+ RelationRelationId, RelationGetRelid(rel), attnum);
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index dfb5d0430c..f9f9ecd0c0 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -621,6 +621,8 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
* filter. XXX Allow sending type information for REPLICA IDENTITY
* COLUMNS with user created type. even when they are not mentioned in
* column filters.
+ *
+ * FIXME -- this code seems not verified by tests.
*/
if (att_map != NULL &&
!bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 2f412ca3db..84ee807e0b 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1648,6 +1648,8 @@ psql_completion(const char *text, int start, int end)
/* ALTER PUBLICATION <name> ADD */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
+ else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD", "TABLE"))
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL);
/* ALTER PUBLICATION <name> DROP */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "DROP"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h
index 3eca295ff4..76d421e09e 100644
--- a/src/include/catalog/dependency.h
+++ b/src/include/catalog/dependency.h
@@ -214,6 +214,9 @@ extern long changeDependenciesOf(Oid classId, Oid oldObjectId,
extern long changeDependenciesOn(Oid refClassId, Oid oldRefObjectId,
Oid newRefObjectId);
+extern void findAndAddAddresses(ObjectAddresses *addrs, Oid classId,
+ Oid refclassId, Oid refobjectId, int32 refobjsubId);
+
extern Oid getExtensionOfObject(Oid classId, Oid objectId);
extern List *getAutoExtensionsOfObject(Oid classId, Oid objectId);
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index 4ba68c70ee..23f037df7f 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -25,7 +25,7 @@
extern ObjectAddress CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt);
extern void AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt);
extern void RemovePublicationById(Oid pubid);
-extern void RemovePublicationRelById(Oid proid);
+extern void RemovePublicationRelById(Oid proid, int32 attnum);
extern void RemovePublicationSchemaById(Oid psoid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 93ef6e21eb..aef2f905a1 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -167,18 +167,32 @@ Publications:
CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (x, y, z); -- error
-ERROR: cannot add relation "testpub_tbl5" to publication
-DETAIL: Column filter must include REPLICA IDENTITY columns
-ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error FIXME
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ERROR: column "x" of relation "testpub_tbl5" does not exist
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
-ERROR: relation "testpub_tbl5" is already member of publication "testpub_fortable"
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
-ERROR: relation "testpub_tbl5" is already member of publication "testpub_fortable"
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+\dRp+ testpub_fortable
+ Publication testpub_fortable
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | t | t | t | f
+Tables:
+ "public.testpub_tbl5" (a)
+Tables from schemas:
+ "pub_test"
+
+ALTER TABLE testpub_tbl5 DROP COLUMN a;
+ERROR: cannot drop the last column in publication "testpub_fortable"
+HINT: Remove table "testpub_tbl5" from the publication first.
CREATE TABLE testpub_tbl6 (a int, b text, c text);
ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
-ERROR: cannot add relation "testpub_tbl6" to publication
-DETAIL: Cannot have column filter with REPLICA IDENTITY FULL
+ERROR: invalid column list for publishing relation "testpub_tbl6"
+DETAIL: Cannot have column filter on relations with REPLICA IDENTITY FULL.
DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index eb0e71ea62..18b87803c0 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -91,9 +91,12 @@ SELECT pubname, puballtables FROM pg_publication WHERE pubname = 'testpub_forall
CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (x, y, z); -- error
-ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error FIXME
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+\dRp+ testpub_fortable
+ALTER TABLE testpub_tbl5 DROP COLUMN a;
CREATE TABLE testpub_tbl6 (a int, b text, c text);
ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
diff --git a/src/test/subscription/t/021_column_filter.pl b/src/test/subscription/t/021_column_filter.pl
index 27d0537621..354e6ac363 100644
--- a/src/test/subscription/t/021_column_filter.pl
+++ b/src/test/subscription/t/021_column_filter.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
-use Test::More tests => 8;
+use Test::More tests => 10;
# setup
@@ -129,14 +129,23 @@ $node_publisher->safe_psql('postgres',
"UPDATE tab2 SET c = 5 where a = 1");
is($result, qq(1|abc), 'update on column c is not replicated');
-# Test error conditions
+# Test behavior when a column is dropped
$node_publisher->safe_psql('postgres',
"ALTER TABLE test_part DROP COLUMN b");
$result = $node_publisher->safe_psql('postgres',
- "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
-is($result, qq(tab1|1 2
+ "select prrelid::regclass, prattrs from pg_publication_rel pb;");
+is($result,
+ q(tab1|1 2
+tab3|1 3
tab2|1 2
-tab3|1 3), 'publication relation test_part removed');
+test_part|1), 'column test_part.b removed');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (3, '2021-12-13 12:13:14')");
+$node_publisher->wait_for_catchup('sub1');
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part WHERE a = 3");
+is($result, "3|", 'only column a is replicated');
$node_publisher->safe_psql('postgres', "CREATE TABLE tab4 (a int PRIMARY KEY, b int, c int, d int)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab4 (a int PRIMARY KEY, b int, d int)");
Hmm, I messed up the patch file I sent. Here's the complete patch.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Doing what he did amounts to sticking his fingers under the hood of the
implementation; if he gets his fingers burnt, it's his problem." (Tom Lane)
Attachments:
column-filtering-9.patchtext/x-diff; charset=utf-8Download
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index bb4ef5e5e2..c86055b93c 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -30,7 +30,7 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -110,6 +110,8 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index d805e8e77a..73a23cbb02 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -78,6 +78,15 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
publication, so they are never explicitly added to the publication.
</para>
+ <para>
+ When a column list is specified, only the listed columns are replicated;
+ any other columns are ignored for the purpose of replication through
+ this publication. If no column list is specified, all columns of the
+ table are replicated through this publication, including any columns
+ added later. If a column list is specified, it must include the replica
+ identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index fe9c714257..a88d12e8ae 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -1472,7 +1472,7 @@ doDeletion(const ObjectAddress *object, int flags)
break;
case OCLASS_PUBLICATION_REL:
- RemovePublicationRelById(object->objectId);
+ RemovePublicationRelById(object->objectId, object->objectSubId);
break;
case OCLASS_PUBLICATION:
@@ -2754,8 +2754,12 @@ free_object_addresses(ObjectAddresses *addrs)
ObjectClass
getObjectClass(const ObjectAddress *object)
{
- /* only pg_class entries can have nonzero objectSubId */
+ /*
+ * only pg_class and pg_publication_rel entries can have nonzero
+ * objectSubId
+ */
if (object->classId != RelationRelationId &&
+ object->classId != PublicationRelRelationId &&
object->objectSubId != 0)
elog(ERROR, "invalid non-zero objectSubId for object class %u",
object->classId);
diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c
index 2bae3fbb17..5eed248dcb 100644
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -4019,6 +4019,7 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok)
/* translator: first %s is, e.g., "table %s" */
appendStringInfo(&buffer, _("publication of %s in publication %s"),
rel.data, pubname);
+ /* FIXME add objectSubId support */
pfree(rel.data);
ReleaseSysCache(tup);
break;
@@ -5853,9 +5854,16 @@ getObjectIdentityParts(const ObjectAddress *object,
getRelationIdentity(&buffer, prform->prrelid, objname, false);
appendStringInfo(&buffer, " in publication %s", pubname);
+ if (object->objectSubId) /* FIXME maybe get_attname */
+ appendStringInfo(&buffer, " column %d", object->objectSubId);
if (objargs)
+ {
*objargs = list_make1(pubname);
+ if (object->objectSubId)
+ *objargs = lappend(*objargs,
+ psprintf("%d", object->objectSubId));
+ }
ReleaseSysCache(tup);
break;
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 5f37bf6d10..dfcb450e61 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -658,6 +658,56 @@ isObjectPinned(const ObjectAddress *object)
* Various special-purpose lookups and manipulations of pg_depend.
*/
+/*
+ * Find all objects of the given class that reference the specified object,
+ * and add them to the given ObjectAddresses.
+ */
+void
+findAndAddAddresses(ObjectAddresses *addrs, Oid classId,
+ Oid refclassId, Oid refobjectId, int32 refobjsubId)
+{
+ Relation depRel;
+ ScanKeyData key[3];
+ SysScanDesc scan;
+ HeapTuple tup;
+
+ depRel = table_open(DependRelationId, AccessShareLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_refclassid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(refclassId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_refobjid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(refobjectId));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_refobjsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum(refobjsubId));
+
+ scan = systable_beginscan(depRel, DependReferenceIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+ ObjectAddress object;
+
+ if (depform->classid != classId)
+ continue;
+
+ ObjectAddressSubSet(object, depform->classid, depform->objid,
+ depform->refobjsubid);
+
+ add_exact_object_address(&object, addrs);
+ }
+
+ systable_endscan(scan);
+
+ table_close(depRel, AccessShareLock);
+}
+
/*
* Find the extension containing the specified object, if any
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 62f10bcbd2..ae58adc8e5 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -46,12 +46,18 @@
#include "utils/syscache.h"
/*
- * Check if relation can be in given publication and throws appropriate
- * error if not.
+ * Check if relation can be in given publication and that the column
+ * filter is sensible, and throws appropriate error if not.
+ *
+ * targetcols is the bitmapset of column specified as column filter
+ * (shifted by FirstLowInvalidHeapAttributeNumber), or NULL if no column
+ * filter was specified.
*/
static void
-check_publication_add_relation(Relation targetrel)
+check_publication_add_relation(Relation targetrel, Bitmapset *columns)
{
+ bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
@@ -82,6 +88,40 @@ check_publication_add_relation(Relation targetrel)
errmsg("cannot add relation \"%s\" to publication",
RelationGetRelationName(targetrel)),
errdetail("This operation is not supported for unlogged tables.")));
+
+ /*
+ * Enforce that the column filter can only leave out columns that aren't
+ * forced to be sent.
+ *
+ * No column can be excluded if REPLICA IDENTITY is FULL (since all the
+ * columns need to be sent regardless); and in other cases, the columns in
+ * the REPLICA IDENTITY cannot be left out.
+ */
+ if (columns != NULL)
+ {
+ if (replidentfull)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter on relations with REPLICA IDENTITY FULL."));
+ else
+ {
+ Bitmapset *idattrs;
+
+ idattrs = RelationGetIndexAttrBitmap(targetrel,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+ if (!bms_is_subset(idattrs, columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
+
+ if (idattrs)
+ pfree(idattrs);
+ }
+ }
}
/*
@@ -289,9 +329,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
Oid relid = RelationGetRelid(targetrel->relation);
Oid prrelid;
Publication *pub = GetPublication(pubid);
+ Bitmapset *attmap = NULL;
+ AttrNumber *attarray;
+ int natts = 0;
+ int attnum;
ObjectAddress myself,
referenced;
List *relids = NIL;
+ ListCell *lc;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -305,6 +350,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
{
table_close(rel, RowExclusiveLock);
+ /* FIXME need to handle the case of different column list */
+
if (if_not_exists)
return InvalidObjectAddress;
@@ -314,7 +361,34 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ attarray = palloc(sizeof(AttrNumber) * list_length(targetrel->columns));
+ foreach(lc, targetrel->columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(relid, colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel->relation)));
+ if (attnum < 0)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, attmap))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("column \"%s\" specified twice in publication column list",
+ colname));
+
+ attmap = bms_add_member(attmap, attnum - FirstLowInvalidHeapAttributeNumber);
+ attarray[natts++] = attnum;
+ }
+
+ check_publication_add_relation(targetrel->relation, attmap);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -327,6 +401,15 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ if (targetrel->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -344,6 +427,21 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectAddressSet(referenced, RelationRelationId, relid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ /*
+ * If there's an explicit column list, make one dependency entry for each
+ * column. Note that the referencing side of the dependency is also
+ * specific to one column, so that it can be dropped separately if the
+ * column is dropped.
+ */
+ while ((attnum = bms_first_member(attmap)) >= 0)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid,
+ attnum + FirstLowInvalidHeapAttributeNumber);
+ myself.objectSubId = attnum + FirstLowInvalidHeapAttributeNumber;
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL);
+ }
+ myself.objectSubId = 0; /* need to undo this bit */
+
/* Close the table. */
table_close(rel, RowExclusiveLock);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 404bb5d0c8..a070914bdd 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -561,7 +561,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
pubrel = palloc(sizeof(PublicationRelInfo));
pubrel->relation = oldrel;
-
+ pubrel->columns = NIL;
delrels = lappend(delrels, pubrel);
}
}
@@ -757,10 +757,11 @@ AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt)
}
/*
- * Remove relation from publication by mapping OID.
+ * Remove relation from publication by mapping OID, or publication status
+ * of one column of that relation in the publication if an attnum is given.
*/
void
-RemovePublicationRelById(Oid proid)
+RemovePublicationRelById(Oid proid, int32 attnum)
{
Relation rel;
HeapTuple tup;
@@ -790,7 +791,81 @@ RemovePublicationRelById(Oid proid)
InvalidatePublicationRels(relids);
- CatalogTupleDelete(rel, &tup->t_self);
+ /*
+ * If no column is given, simply delete the relation from the publication.
+ *
+ * If a column is given, what we do instead is to remove that column from
+ * the column list. The relation remains in the publication, with the
+ * other columns. However, dropping the last column is disallowed.
+ */
+ if (attnum == 0)
+ {
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ else
+ {
+ Datum adatum;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ int16 *newelems;
+ int2vector *newvec;
+ Datum values[Natts_pg_publication_rel];
+ bool nulls[Natts_pg_publication_rel];
+ bool replace[Natts_pg_publication_rel];
+ HeapTuple newtup;
+ int i,
+ j;
+ bool isnull;
+
+ /* Obtain the original column list */
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP,
+ tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull) /* shouldn't happen */
+ elog(ERROR, "can't drop column from publication without a column list");
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* Construct a list excluding the given column */
+ newelems = palloc(sizeof(int16) * nelems - 1);
+ for (i = 0, j = 0; i < nelems - 1; i++)
+ {
+ if (elems[i] == attnum)
+ continue;
+ newelems[j++] = elems[i];
+ }
+
+ /*
+ * If this is the last column used in the publication, disallow the
+ * command. We could alternatively just drop the relation from the
+ * publication.
+ */
+ if (j == 0)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot drop the last column in publication \"%s\"",
+ get_publication_name(pubrel->prpubid, false)),
+ errhint("Remove table \"%s\" from the publication first.",
+ get_rel_name(pubrel->prrelid)));
+ }
+
+ /* Build the updated tuple */
+ MemSet(values, 0, sizeof(values));
+ MemSet(nulls, false, sizeof(nulls));
+ MemSet(replace, false, sizeof(replace));
+ newvec = buildint2vector(newelems, j);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(newvec);
+ replace[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ /* Execute the update */
+ newtup = heap_modify_tuple(tup, RelationGetDescr(rel),
+ values, nulls, replace);
+ CatalogTupleUpdate(rel, &tup->t_self, newtup);
+ }
ReleaseSysCache(tup);
@@ -932,6 +1007,8 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
@@ -965,8 +1042,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
+
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
@@ -1074,6 +1154,12 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list may not be specified for relation \"%s\" in ALTER PUBLICATION ... SET/DROP command",
+ RelationGetRelationName(pubrel->relation)));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 47b29001d5..7207dcf9c0 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -40,8 +40,9 @@
#include "catalog/pg_inherits.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
-#include "catalog/pg_tablespace.h"
+#include "catalog/pg_publication_rel.h"
#include "catalog/pg_statistic_ext.h"
+#include "catalog/pg_tablespace.h"
#include "catalog/pg_trigger.h"
#include "catalog/pg_type.h"
#include "catalog/storage.h"
@@ -8420,6 +8421,13 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * If the column is part of a replication column list, arrange to get that
+ * removed too.
+ */
+ findAndAddAddresses(addrs, PublicationRelRelationId,
+ RelationRelationId, RelationGetRelid(rel), attnum);
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index df0b747883..0ff4c1ceac 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4833,6 +4833,7 @@ _copyPublicationTable(const PublicationTable *from)
PublicationTable *newnode = makeNode(PublicationTable);
COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index cb7ddd463c..d786a688ac 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2312,6 +2312,7 @@ static bool
_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 3d4dd43e47..4dad6fedfb 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9742,12 +9742,13 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr
+ TABLE relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
+ $$->pubtable->columns = $3;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9762,28 +9763,38 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLE_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId
+ | ColId opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- $$->name = $1;
+ if ($2 != NULL)
+ {
+ $$->pubtable = makeNode(PublicationTable);
+ $$->pubtable->relation = makeRangeVar(NULL, $1, @1);
+ $$->pubtable->columns = $2;
+ $$->name = NULL;
+ }
+ else
+ $$->name = $1;
$$->location = @1;
}
- | ColId indirection
+ | ColId indirection opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
+ $$->pubtable->columns = $3;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr
+ | extended_relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
+ $$->pubtable->columns = $2;
}
| CURRENT_SCHEMA
{
@@ -17435,8 +17446,9 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
{
/* convert it to PublicationTable */
PublicationTable *pubtable = makeNode(PublicationTable);
- pubtable->relation = makeRangeVar(NULL, pubobj->name,
- pubobj->location);
+
+ pubtable->relation =
+ makeRangeVar(NULL, pubobj->name, pubobj->location);
pubobj->pubtable = pubtable;
pubobj->name = NULL;
}
@@ -17444,6 +17456,16 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
else if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLE_IN_SCHEMA ||
pubobj->pubobjtype == PUBLICATIONOBJ_TABLE_IN_CUR_SCHEMA)
{
+ /*
+ * This can happen if a column list is specified in a continuation
+ * for a schema entry; reject it.
+ */
+ if (pubobj->pubtable)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schemas"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 9f5bf4b639..15d8192238 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,9 +29,9 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *att_map);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary, Bitmapset *att_map);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +398,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *att_map)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +410,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_map);
}
/*
@@ -442,7 +442,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary, Bitmapset *att_map)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -463,11 +463,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, att_map);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_map);
}
/*
@@ -536,7 +536,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -651,7 +651,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *att_map)
{
char *relname;
@@ -673,7 +673,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, att_map);
}
/*
@@ -749,20 +749,42 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary,
+ Bitmapset *att_map)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
bool isnull[MaxTupleAttributeNumber];
int i;
uint16 nliveatts = 0;
+ Bitmapset *idattrs = NULL;
+ bool replidentfull;
+ Form_pg_attribute att;
desc = RelationGetDescr(rel);
+ replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ if (!replidentfull)
+ idattrs = RelationGetIdentityKeyBitmap(rel);
+
for (i = 0; i < desc->natts; i++)
{
+ att = TupleDescAttr(desc, i);
if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
continue;
+
+ /*
+ * Do not increment count of attributes if not a part of column
+ * filters except for replica identity columns or if replica identity
+ * is full.
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map) &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs) &&
+ !replidentfull)
+ continue;
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -800,6 +822,19 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
continue;
}
+ /*
+ * Do not send attribute data if it is not a part of column filters,
+ * except if it is a part of REPLICA IDENTITY or REPLICA IDENTITY is
+ * full, send the data.
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map) &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs) &&
+ !replidentfull)
+ continue;
+
typtup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(att->atttypid));
if (!HeapTupleIsValid(typtup))
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
@@ -904,7 +939,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *att_map)
{
TupleDesc desc;
int i;
@@ -914,20 +949,35 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
desc = RelationGetDescr(rel);
- /* send number of live attributes */
- for (i = 0; i < desc->natts; i++)
- {
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
- continue;
- nliveatts++;
- }
- pq_sendint16(out, nliveatts);
-
/* fetch bitmap of REPLICATION IDENTITY attributes */
replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
if (!replidentfull)
idattrs = RelationGetIdentityKeyBitmap(rel);
+ /* send number of live attributes */
+ for (i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ /* REPLICA IDENTITY FULL means all columns are sent as part of key. */
+ if (replidentfull ||
+ bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs))
+ {
+ nliveatts++;
+ continue;
+ }
+ /* Skip sending if not a part of column filter */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map))
+ continue;
+ nliveatts++;
+ }
+ pq_sendint16(out, nliveatts);
+
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -937,6 +987,17 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ /*
+ * Exclude filtered columns, but REPLICA IDENTITY columns can't be
+ * excluded
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map) &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs)
+ && !replidentfull)
+ continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index f07983a43c..15902faf56 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -111,6 +111,7 @@
#include "replication/origin.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -695,19 +696,25 @@ fetch_remote_table_info(char *nspname, char *relname,
LogicalRepRelation *lrel)
{
WalRcvExecResult *res;
+ WalRcvExecResult *res_pub;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
+ TupleTableSlot *slot_pub;
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid pubRow[] = {TEXTARRAYOID};
bool isnull;
int natt;
+ List *pub_columns = NIL;
+ ListCell *lc;
+ bool am_partition = false;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -737,6 +744,7 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetChar(slot_getattr(slot, 4, &isnull));
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
@@ -774,11 +782,101 @@ fetch_remote_table_info(char *nspname, char *relname,
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+
+ /*
+ * Now, fetch the values of publications' column filters.
+ *
+ * For a partition, use pg_inherit to find the parent, as the
+ * pg_publication_rel contains only the topmost parent table entry in case
+ * the table is partitioned. Run a recursive query to iterate through all
+ * the parents of the partition and retreive the record for the parent
+ * that exists in pg_publication_rel.
+ */
+ resetStringInfo(&cmd);
+ appendStringInfoString(&cmd,
+ "SELECT CASE WHEN prattrs IS NOT NULL THEN\n"
+ " ARRAY(SELECT attname\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(prattrs::int[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END AS columns\n"
+ "FROM pg_catalog.pg_publication_rel\n");
+ if (!am_partition)
+ appendStringInfo(&cmd, "WHERE prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "WHERE prrelid IN (SELECT relid \n"
+ "FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);
+
+ res_pub = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(pubRow), pubRow);
+
+ if (res_pub->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch published columns info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, res_pub->err)));
+ slot_pub = MakeSingleTupleTableSlot(res_pub->tupledesc, &TTSOpsMinimalTuple);
+
+ while (tuplestore_gettupleslot(res_pub->tuplestore, true, false, slot_pub))
+ {
+ Datum adatum;
+ Datum *elems;
+ bool *nulls;
+ int nelems;
+
+ adatum = slot_getattr(slot_pub, 1, &isnull);
+ if (isnull) /* shouldn't happen */
+ elog(ERROR, "unexpected null value in publication column filter");
+ deconstruct_array(DatumGetArrayTypeP(adatum),
+ TEXTOID, -1, false, TYPALIGN_INT,
+ &elems, &nulls, &nelems);
+ for (int i = 0; i < nelems; i++)
+ {
+ if (nulls[i]) /* shouldn't happen */
+ elog(ERROR, "unexpected null value in publication column filter");
+ pub_columns = lappend(pub_columns, TextDatumGetCString(elems[i]));
+ }
+ ExecClearTuple(slot_pub);
+ }
+ ExecDropSingleTupleTableSlot(slot_pub);
+ walrcv_clear_result(res_pub);
+
+ /*
+ * Store the column names only if they are contained in column filter
+ * LogicalRepRelation will only contain attributes corresponding to those
+ * specficied in column filters.
+ */
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ bool found = false;
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 1, &isnull));
Assert(!isnull);
+ if (pub_columns != NIL)
+ {
+ foreach(lc, pub_columns)
+ {
+ char *pub_colname = lfirst(lc);
+
+ if (!strcmp(pub_colname, rel_colname))
+ {
+ found = true;
+ lrel->attnames[natt] = rel_colname;
+ break;
+ }
+ }
+ }
+ else
+ {
+ found = true;
+ lrel->attnames[natt] = rel_colname;
+ }
+ if (!found)
+ continue;
+
lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
@@ -829,8 +927,17 @@ copy_table(Relation rel)
/* Start copy on the publisher. */
initStringInfo(&cmd);
if (lrel.relkind == RELKIND_RELATION)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 6f6a203dea..f9f9ecd0c0 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,16 +15,19 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -81,7 +84,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *att_map);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -130,6 +134,7 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+ Bitmapset *att_map;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -570,11 +575,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
}
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->att_map);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->att_map);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -587,7 +592,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *att_map)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -610,13 +616,25 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /*
+ * Do not send type information if attribute is not present in column
+ * filter. XXX Allow sending type information for REPLICA IDENTITY
+ * COLUMNS with user created type. even when they are not mentioned in
+ * column filters.
+ *
+ * FIXME -- this code seems not verified by tests.
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map))
+ continue;
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, att_map);
OutputPluginWrite(ctx, false);
}
@@ -693,7 +711,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->att_map);
OutputPluginWrite(ctx, true);
break;
}
@@ -722,7 +740,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->att_map);
OutputPluginWrite(ctx, true);
break;
}
@@ -1122,6 +1140,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
bool found;
+ Oid ancestor_id;
MemoryContext oldctx;
Assert(RelationSyncCache != NULL);
@@ -1142,6 +1161,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
+ entry->att_map = NULL;
entry->map = NULL; /* will be set by maybe_send_schema() if
* needed */
}
@@ -1182,6 +1202,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
if (pub->alltables)
{
@@ -1192,8 +1213,6 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (!publish)
{
- bool ancestor_published = false;
-
/*
* For a partition, check if any of the ancestors are
* published. If so, note down the topmost ancestor that is
@@ -1219,6 +1238,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
pub->oid))
{
ancestor_published = true;
+ ancestor_id = ancestor;
if (pub->pubviaroot)
publish_as_relid = ancestor;
}
@@ -1239,15 +1259,47 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ Oid relid;
+ HeapTuple pub_rel_tuple;
+
+ relid = ancestor_published ? ancestor_id : publish_as_relid;
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ Datum pub_rel_cols;
+ bool isnull;
+
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP,
+ pub_rel_tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(pub_rel_cols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
+ entry->att_map = bms_add_member(entry->att_map,
+ elems[i] - FirstLowInvalidHeapAttributeNumber);
+ MemoryContextSwitchTo(oldctx);
+ }
+ ReleaseSysCache(pub_rel_tuple);
+ }
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
}
-
- if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
- break;
}
list_free(pubids);
@@ -1343,6 +1395,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->att_map);
+ entry->att_map = NULL;
if (entry->map)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 10a86f9810..0c438481dc 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4265,6 +4265,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_oid;
int i_prpubid;
int i_prrelid;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4276,8 +4277,13 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
/* Collect all publication membership info. */
appendPQExpBufferStr(query,
- "SELECT tableoid, oid, prpubid, prrelid "
- "FROM pg_catalog.pg_publication_rel");
+ "SELECT tableoid, oid, prpubid, prrelid");
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBufferStr(query, ", prattrs");
+ else
+ appendPQExpBufferStr(query, ", NULL as prattrs");
+ appendPQExpBufferStr(query,
+ " FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
ntups = PQntuples(res);
@@ -4286,6 +4292,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_oid = PQfnumber(res, "oid");
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4327,6 +4334,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
pubrinfo[j].publication = pubinfo;
pubrinfo[j].pubtable = tbinfo;
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[i].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4391,10 +4420,12 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s;\n",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+ appendPQExpBufferStr(query, ";\n");
/*
* There is no point in creating a drop query as the drop is done by table
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 6dccb4be4e..50a5b885f6 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -633,6 +633,7 @@ typedef struct _PublicationRelInfo
DumpableObject dobj;
PublicationInfo *publication;
TableInfo *pubtable;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 72d8547628..46fa616406 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6302,7 +6302,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -6319,10 +6319,14 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (!as_schema) /* as table */
+ {
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
- else
+ if (!PQgetisnull(res, i, 2))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 2));
+ }
+ else /* as schema */
printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
@@ -6450,8 +6454,20 @@ describePublications(const char *pattern)
{
/* Get the tables for the specified publication */
printfPQExpBuffer(&buf,
- "SELECT n.nspname, c.relname\n"
- "FROM pg_catalog.pg_class c,\n"
+ "SELECT n.nspname, c.relname, \n");
+ if (pset.sversion >= 150000)
+ appendPQExpBufferStr(&buf,
+ " CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string"
+ "(ARRAY(SELECT attname\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::int[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END AS columns");
+ else
+ appendPQExpBufferStr(&buf, "NULL as columns");
+ appendPQExpBuffer(&buf,
+ "\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
" pg_catalog.pg_publication_rel pr\n"
"WHERE c.relnamespace = n.oid\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 2f412ca3db..84ee807e0b 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1648,6 +1648,8 @@ psql_completion(const char *text, int start, int end)
/* ALTER PUBLICATION <name> ADD */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
+ else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD", "TABLE"))
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL);
/* ALTER PUBLICATION <name> DROP */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "DROP"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h
index 3eca295ff4..76d421e09e 100644
--- a/src/include/catalog/dependency.h
+++ b/src/include/catalog/dependency.h
@@ -214,6 +214,9 @@ extern long changeDependenciesOf(Oid classId, Oid oldObjectId,
extern long changeDependenciesOn(Oid refClassId, Oid oldRefObjectId,
Oid newRefObjectId);
+extern void findAndAddAddresses(ObjectAddresses *addrs, Oid classId,
+ Oid refclassId, Oid refobjectId, int32 refobjsubId);
+
extern Oid getExtensionOfObject(Oid classId, Oid objectId);
extern List *getAutoExtensionsOfObject(Oid classId, Oid objectId);
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 902f2f2f0d..f5ae2065e9 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -86,6 +86,7 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index b5d5504cbb..7ad285faae 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN
+ int2vector prattrs; /* Variable length field starts here */
+#endif
} FormData_pg_publication_rel;
/* ----------------
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index 4ba68c70ee..23f037df7f 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -25,7 +25,7 @@
extern ObjectAddress CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt);
extern void AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt);
extern void RemovePublicationById(Oid pubid);
-extern void RemovePublicationRelById(Oid proid);
+extern void RemovePublicationRelById(Oid proid, int32 attnum);
extern void RemovePublicationSchemaById(Oid psoid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 4c5a8a39bf..02b547d044 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3642,6 +3642,7 @@ typedef struct PublicationTable
{
NodeTag type;
RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 83741dcf42..709b4be916 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -207,11 +207,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *att_map);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *att_map);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -228,7 +228,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *att_map);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 5ac2d666a2..84afe0ebef 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -165,7 +165,35 @@ Publications:
regress_publication_user | t | t | t | f | f | f
(1 row)
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (x, y, z); -- error
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+\dRp+ testpub_fortable
+ Publication testpub_fortable
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | t | t | t | f
+Tables:
+ "public.testpub_tbl5" (a)
+Tables from schemas:
+ "pub_test"
+
+ALTER TABLE testpub_tbl5 DROP COLUMN a;
+ERROR: cannot drop the last column in publication "testpub_fortable"
+HINT: Remove table "testpub_tbl5" from the publication first.
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl6"
+DETAIL: Cannot have column filter on relations with REPLICA IDENTITY FULL.
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
CREATE TABLE testpub_tbl3a (b text) INHERITS (testpub_tbl3);
@@ -669,6 +697,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schemas
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 56dd358554..200158ba69 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -89,7 +89,20 @@ SELECT pubname, puballtables FROM pg_publication WHERE pubname = 'testpub_forall
\d+ testpub_tbl2
\dRp+ testpub_foralltables
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (x, y, z); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+\dRp+ testpub_fortable
+ALTER TABLE testpub_tbl5 DROP COLUMN a;
+
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
@@ -362,6 +375,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/021_column_filter.pl b/src/test/subscription/t/021_column_filter.pl
new file mode 100644
index 0000000000..354e6ac363
--- /dev/null
+++ b/src/test/subscription/t/021_column_filter.pl
@@ -0,0 +1,162 @@
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test TRUNCATE
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
+
+# setup
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int)");
+# Test with weird column names
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, B varchar, \"c'\" int)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+#Test replication with multi-level partition
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, \"c'\" int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+#Test create publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub1 FOR TABLE tab1(a, \"B\"), tab3(\"a'\",\"c'\"), test_part(a,b)");
+
+my $result = $node_publisher->safe_psql('postgres',
+ "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
+is($result, qq(tab1|1 2
+tab3|1 3
+test_part|1 2), 'publication relation updated');
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+);
+#Initial sync
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab1 VALUES (1,2,3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 VALUES (1,2,3)");
+#Test for replication of partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (1,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (2,'bcd', '2021-07-03 11:12:13')");
+#Test for replication of multi-level partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (4,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (5,'bcd', '2021-07-03 11:12:13')");
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'insert on column c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab3");
+is($result, qq(1|3), 'insert on column b is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part");
+is($result, qq(1|abc\n2|bcd\n4|abc\n5|bcd), 'insert on all columns is replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab1 SET c = 5 where a = 1");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'update on column c is not replicated');
+
+#Test alter publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION"
+);
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 VALUES (1,'abc',3)");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2");
+is($result, qq(1|abc), 'insert on column c is not replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET c = 5 where a = 1");
+is($result, qq(1|abc), 'update on column c is not replicated');
+
+# Test behavior when a column is dropped
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE test_part DROP COLUMN b");
+$result = $node_publisher->safe_psql('postgres',
+ "select prrelid::regclass, prattrs from pg_publication_rel pb;");
+is($result,
+ q(tab1|1 2
+tab3|1 3
+tab2|1 2
+test_part|1), 'column test_part.b removed');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (3, '2021-12-13 12:13:14')");
+$node_publisher->wait_for_catchup('sub1');
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part WHERE a = 3");
+is($result, "3|", 'only column a is replicated');
+
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab4 (a int PRIMARY KEY, b int, c int, d int)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab4 (a int PRIMARY KEY, b int, d int)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub2 FOR TABLE tab4 (a, b)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub3 FOR TABLE tab4 (a, d)");
+$node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2, pub3");
+$node_publisher->wait_for_catchup('sub2');
+$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (1, 11, 111, 1111)");
+$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (2, 22, 222, 2222)");
+$node_publisher->wait_for_catchup('sub2');
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab4;"),
+ qq(1|11|1111
+2|22|2222),
+ 'overlapping publications with overlapping column lists');
On 2021-Dec-13, Alvaro Herrera wrote:
Hmm, I messed up the patch file I sent. Here's the complete patch.
Actually, this requires even a bit more mess than this to be really
complete if we want to be strict about it. The reason is that, with the
patch I just posted, we're creating a new type of representable object
that will need to have some way of making it through pg_identify_object,
pg_get_object_address, pg_identify_object_as_address. This is only
visible as one tries to patch object_address.sql (auditability of DDL
operations being the goal).
I think this means we need a new OBJECT_PUBLICATION_REL_COLUMN value in
the ObjectType (paralelling OBJECT_COLUMN), and no new ObjectClass
value. Looking now to confirm.
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
"El que vive para el futuro es un iluso, y el que vive para el pasado,
un imbécil" (Luis Adler, "Los tripulantes de la noche")
On 2021-Dec-13, Alvaro Herrera wrote:
I think this means we need a new OBJECT_PUBLICATION_REL_COLUMN value in
the ObjectType (paralelling OBJECT_COLUMN), and no new ObjectClass
value. Looking now to confirm.
After working on this a little bit more, I realized that this is a bad
idea overall. It causes lots of complications and it's just not worth
it. So I'm back at my original thought that we need to throw an ERROR
at ALTER TABLE .. DROP COLUMN time if the column is part of a
replication column filter, and suggest the user to remove the column
from the filter first and reattempt the DROP COLUMN.
This means that we need to support changing the column list of a table
in a publication. I'm looking at implementing some form of ALTER
PUBLICATION for that.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Find a bug in a program, and fix it, and the program will work today.
Show the program how to find and fix a bug, and the program
will work forever" (Oliver Silfridge)
On 12/14/21 17:43, Alvaro Herrera wrote:
On 2021-Dec-13, Alvaro Herrera wrote:
I think this means we need a new OBJECT_PUBLICATION_REL_COLUMN value in
the ObjectType (paralelling OBJECT_COLUMN), and no new ObjectClass
value. Looking now to confirm.After working on this a little bit more, I realized that this is a bad
idea overall. It causes lots of complications and it's just not worth
it. So I'm back at my original thought that we need to throw an ERROR
at ALTER TABLE .. DROP COLUMN time if the column is part of a
replication column filter, and suggest the user to remove the column
from the filter first and reattempt the DROP COLUMN.This means that we need to support changing the column list of a table
in a publication. I'm looking at implementing some form of ALTER
PUBLICATION for that.
Yeah. I think it's not clear if this should behave more like an index or
a view. When an indexed column gets dropped we simply drop the index.
But if you drop a column referenced by a view, we fail with an error. I
think we should handle this more like a view, because publications are
externally visible objects too (while indexes are pretty much just an
implementation detail).
But why would it be easier not to add new object type? We still need to
check there is no publication referencing the column - either you do
that automatically through a dependency, or you do that by custom code.
Using a dependency seems better to me, but I don't know what are the
complications you mentioned.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 2021-Dec-14, Tomas Vondra wrote:
Yeah. I think it's not clear if this should behave more like an index or a
view. When an indexed column gets dropped we simply drop the index. But if
you drop a column referenced by a view, we fail with an error. I think we
should handle this more like a view, because publications are externally
visible objects too (while indexes are pretty much just an implementation
detail).
I agree -- I think it's more like a view than like an index. (The
original proposal was that if you dropped a column that was part of the
column list of a relation in a publication, the entire relation is
dropped from the view, but that doesn't seem very friendly behavior --
you break the replication stream immediately if you do that, and the
only way to fix it is to send a fresh copy of the remaining subset of
columns.)
But why would it be easier not to add new object type? We still need to
check there is no publication referencing the column - either you do that
automatically through a dependency, or you do that by custom code. Using a
dependency seems better to me, but I don't know what are the complications
you mentioned.
The problem is that we need a way to represent the object "column of a
table in a publication". I found myself adding a lot of additional code
to support OBJECT_PUBLICATION_REL_COLUMN and that seemed like too much.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
On 12/14/21 20:35, Alvaro Herrera wrote:
On 2021-Dec-14, Tomas Vondra wrote:
Yeah. I think it's not clear if this should behave more like an index or a
view. When an indexed column gets dropped we simply drop the index. But if
you drop a column referenced by a view, we fail with an error. I think we
should handle this more like a view, because publications are externally
visible objects too (while indexes are pretty much just an implementation
detail).I agree -- I think it's more like a view than like an index. (The
original proposal was that if you dropped a column that was part of the
column list of a relation in a publication, the entire relation is
dropped from the view, but that doesn't seem very friendly behavior --
you break the replication stream immediately if you do that, and the
only way to fix it is to send a fresh copy of the remaining subset of
columns.)
Right, that's my reasoning too.
But why would it be easier not to add new object type? We still need to
check there is no publication referencing the column - either you do that
automatically through a dependency, or you do that by custom code. Using a
dependency seems better to me, but I don't know what are the complications
you mentioned.The problem is that we need a way to represent the object "column of a
table in a publication". I found myself adding a lot of additional code
to support OBJECT_PUBLICATION_REL_COLUMN and that seemed like too much.
My experience with dependencies is pretty limited, but can't we simply
make a dependency between the whole publication and the column?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Hi,
I went through the v9 patch, and I have a couple comments / questions.
Apologies if some of this was already discussed earlier, it's hard to
cross-check in such a long thread. Most of the comments are in 0002 to
make it easier to locate, and it also makes proposed code changes
clearer I think.
1) check_publication_add_relation - the "else" branch is not really
needed, because the "if (replidentfull)" always errors-out
2) publication_add_relation has a FIXME about handling cases with
different column list
So what's the right behavior for ADD TABLE with different column list?
I'd say we should allow that, and that it should be mostly the same
thing as adding/removing columns to the list incrementally, i.e. we
should replace the column lists. We could also prohibit such changes,
but that seems like a really annoying limitation, forcing people to
remove/add the relation.
I added some comments to the attmap translation block, and replaced <0
check with AttrNumberIsForUserDefinedAttr.
But I wonder if we could get rid of the offset, considering we're
dealing with just user-defined attributes. That'd make the code clearer,
but it would break if we're comparing it to other bitmaps with offsets.
But I don't think we do.
3) I doubt "att_map" is the right name, though. AFAICS it's just a list
of columns for the relation, not a map, right? So maybe attr_list?
4) AlterPublication talks about "publication status" for a column, but
do we actually track that? Or what does that mean?
5) PublicationDropTables does a check
if (pubrel->columns)
ereport(ERROR,
errcode(ERRCODE_SYNTAX_ERROR),
Shouldn't this be prevented by the grammar, really? Also, it should be
in regression tests.
6) Another thing that should be in the test is partitioned table with
attribute mapping and column list, to see how map and attr_map interact.
7) There's a couple places doing this
if (att_map != NULL &&
!bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
att_map) &&
!bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
idattrs) &&
!replidentfull)
which is really hard to understand (even if we get rid of the offset),
so maybe let's move that to a function with sensible name. Also, some
places don't check indattrs - seems a bit suspicious.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
0001-v9.patchtext/x-patch; charset=UTF-8; name=0001-v9.patchDownload
From fb5ce02d36b46f92ab01c9a823cc4e315cfcb73c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Tue, 14 Dec 2021 20:53:28 +0100
Subject: [PATCH 1/2] v9
---
doc/src/sgml/ref/alter_publication.sgml | 4 +-
doc/src/sgml/ref/create_publication.sgml | 11 +-
src/backend/catalog/dependency.c | 8 +-
src/backend/catalog/objectaddress.c | 8 +
src/backend/catalog/pg_depend.c | 50 ++++++
src/backend/catalog/pg_publication.c | 106 +++++++++++-
src/backend/commands/publicationcmds.c | 94 ++++++++++-
src/backend/commands/tablecmds.c | 10 +-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 36 ++++-
src/backend/replication/logical/proto.c | 97 ++++++++---
src/backend/replication/logical/tablesync.c | 117 +++++++++++++-
src/backend/replication/pgoutput/pgoutput.c | 80 +++++++--
src/bin/pg_dump/pg_dump.c | 41 ++++-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 26 ++-
src/bin/psql/tab-complete.c | 2 +
src/include/catalog/dependency.h | 3 +
src/include/catalog/pg_publication.h | 1 +
src/include/catalog/pg_publication_rel.h | 3 +
src/include/commands/publicationcmds.h | 2 +-
src/include/nodes/parsenodes.h | 1 +
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 39 ++++-
src/test/regress/sql/publication.sql | 19 ++-
src/test/subscription/t/021_column_filter.pl | 162 +++++++++++++++++++
27 files changed, 857 insertions(+), 72 deletions(-)
create mode 100644 src/test/subscription/t/021_column_filter.pl
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index bb4ef5e5e2..c86055b93c 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -30,7 +30,7 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -110,6 +110,8 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index d805e8e77a..73a23cbb02 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -78,6 +78,15 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
publication, so they are never explicitly added to the publication.
</para>
+ <para>
+ When a column list is specified, only the listed columns are replicated;
+ any other columns are ignored for the purpose of replication through
+ this publication. If no column list is specified, all columns of the
+ table are replicated through this publication, including any columns
+ added later. If a column list is specified, it must include the replica
+ identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index fe9c714257..a88d12e8ae 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -1472,7 +1472,7 @@ doDeletion(const ObjectAddress *object, int flags)
break;
case OCLASS_PUBLICATION_REL:
- RemovePublicationRelById(object->objectId);
+ RemovePublicationRelById(object->objectId, object->objectSubId);
break;
case OCLASS_PUBLICATION:
@@ -2754,8 +2754,12 @@ free_object_addresses(ObjectAddresses *addrs)
ObjectClass
getObjectClass(const ObjectAddress *object)
{
- /* only pg_class entries can have nonzero objectSubId */
+ /*
+ * only pg_class and pg_publication_rel entries can have nonzero
+ * objectSubId
+ */
if (object->classId != RelationRelationId &&
+ object->classId != PublicationRelRelationId &&
object->objectSubId != 0)
elog(ERROR, "invalid non-zero objectSubId for object class %u",
object->classId);
diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c
index 2bae3fbb17..5eed248dcb 100644
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -4019,6 +4019,7 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok)
/* translator: first %s is, e.g., "table %s" */
appendStringInfo(&buffer, _("publication of %s in publication %s"),
rel.data, pubname);
+ /* FIXME add objectSubId support */
pfree(rel.data);
ReleaseSysCache(tup);
break;
@@ -5853,9 +5854,16 @@ getObjectIdentityParts(const ObjectAddress *object,
getRelationIdentity(&buffer, prform->prrelid, objname, false);
appendStringInfo(&buffer, " in publication %s", pubname);
+ if (object->objectSubId) /* FIXME maybe get_attname */
+ appendStringInfo(&buffer, " column %d", object->objectSubId);
if (objargs)
+ {
*objargs = list_make1(pubname);
+ if (object->objectSubId)
+ *objargs = lappend(*objargs,
+ psprintf("%d", object->objectSubId));
+ }
ReleaseSysCache(tup);
break;
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 5f37bf6d10..dfcb450e61 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -658,6 +658,56 @@ isObjectPinned(const ObjectAddress *object)
* Various special-purpose lookups and manipulations of pg_depend.
*/
+/*
+ * Find all objects of the given class that reference the specified object,
+ * and add them to the given ObjectAddresses.
+ */
+void
+findAndAddAddresses(ObjectAddresses *addrs, Oid classId,
+ Oid refclassId, Oid refobjectId, int32 refobjsubId)
+{
+ Relation depRel;
+ ScanKeyData key[3];
+ SysScanDesc scan;
+ HeapTuple tup;
+
+ depRel = table_open(DependRelationId, AccessShareLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_refclassid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(refclassId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_refobjid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(refobjectId));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_refobjsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum(refobjsubId));
+
+ scan = systable_beginscan(depRel, DependReferenceIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+ ObjectAddress object;
+
+ if (depform->classid != classId)
+ continue;
+
+ ObjectAddressSubSet(object, depform->classid, depform->objid,
+ depform->refobjsubid);
+
+ add_exact_object_address(&object, addrs);
+ }
+
+ systable_endscan(scan);
+
+ table_close(depRel, AccessShareLock);
+}
+
/*
* Find the extension containing the specified object, if any
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 62f10bcbd2..ae58adc8e5 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -46,12 +46,18 @@
#include "utils/syscache.h"
/*
- * Check if relation can be in given publication and throws appropriate
- * error if not.
+ * Check if relation can be in given publication and that the column
+ * filter is sensible, and throws appropriate error if not.
+ *
+ * targetcols is the bitmapset of column specified as column filter
+ * (shifted by FirstLowInvalidHeapAttributeNumber), or NULL if no column
+ * filter was specified.
*/
static void
-check_publication_add_relation(Relation targetrel)
+check_publication_add_relation(Relation targetrel, Bitmapset *columns)
{
+ bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
@@ -82,6 +88,40 @@ check_publication_add_relation(Relation targetrel)
errmsg("cannot add relation \"%s\" to publication",
RelationGetRelationName(targetrel)),
errdetail("This operation is not supported for unlogged tables.")));
+
+ /*
+ * Enforce that the column filter can only leave out columns that aren't
+ * forced to be sent.
+ *
+ * No column can be excluded if REPLICA IDENTITY is FULL (since all the
+ * columns need to be sent regardless); and in other cases, the columns in
+ * the REPLICA IDENTITY cannot be left out.
+ */
+ if (columns != NULL)
+ {
+ if (replidentfull)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter on relations with REPLICA IDENTITY FULL."));
+ else
+ {
+ Bitmapset *idattrs;
+
+ idattrs = RelationGetIndexAttrBitmap(targetrel,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+ if (!bms_is_subset(idattrs, columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
+
+ if (idattrs)
+ pfree(idattrs);
+ }
+ }
}
/*
@@ -289,9 +329,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
Oid relid = RelationGetRelid(targetrel->relation);
Oid prrelid;
Publication *pub = GetPublication(pubid);
+ Bitmapset *attmap = NULL;
+ AttrNumber *attarray;
+ int natts = 0;
+ int attnum;
ObjectAddress myself,
referenced;
List *relids = NIL;
+ ListCell *lc;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -305,6 +350,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
{
table_close(rel, RowExclusiveLock);
+ /* FIXME need to handle the case of different column list */
+
if (if_not_exists)
return InvalidObjectAddress;
@@ -314,7 +361,34 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ attarray = palloc(sizeof(AttrNumber) * list_length(targetrel->columns));
+ foreach(lc, targetrel->columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(relid, colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel->relation)));
+ if (attnum < 0)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, attmap))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("column \"%s\" specified twice in publication column list",
+ colname));
+
+ attmap = bms_add_member(attmap, attnum - FirstLowInvalidHeapAttributeNumber);
+ attarray[natts++] = attnum;
+ }
+
+ check_publication_add_relation(targetrel->relation, attmap);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -327,6 +401,15 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ if (targetrel->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -344,6 +427,21 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectAddressSet(referenced, RelationRelationId, relid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ /*
+ * If there's an explicit column list, make one dependency entry for each
+ * column. Note that the referencing side of the dependency is also
+ * specific to one column, so that it can be dropped separately if the
+ * column is dropped.
+ */
+ while ((attnum = bms_first_member(attmap)) >= 0)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid,
+ attnum + FirstLowInvalidHeapAttributeNumber);
+ myself.objectSubId = attnum + FirstLowInvalidHeapAttributeNumber;
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL);
+ }
+ myself.objectSubId = 0; /* need to undo this bit */
+
/* Close the table. */
table_close(rel, RowExclusiveLock);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 404bb5d0c8..a070914bdd 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -561,7 +561,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
pubrel = palloc(sizeof(PublicationRelInfo));
pubrel->relation = oldrel;
-
+ pubrel->columns = NIL;
delrels = lappend(delrels, pubrel);
}
}
@@ -757,10 +757,11 @@ AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt)
}
/*
- * Remove relation from publication by mapping OID.
+ * Remove relation from publication by mapping OID, or publication status
+ * of one column of that relation in the publication if an attnum is given.
*/
void
-RemovePublicationRelById(Oid proid)
+RemovePublicationRelById(Oid proid, int32 attnum)
{
Relation rel;
HeapTuple tup;
@@ -790,7 +791,81 @@ RemovePublicationRelById(Oid proid)
InvalidatePublicationRels(relids);
- CatalogTupleDelete(rel, &tup->t_self);
+ /*
+ * If no column is given, simply delete the relation from the publication.
+ *
+ * If a column is given, what we do instead is to remove that column from
+ * the column list. The relation remains in the publication, with the
+ * other columns. However, dropping the last column is disallowed.
+ */
+ if (attnum == 0)
+ {
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ else
+ {
+ Datum adatum;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ int16 *newelems;
+ int2vector *newvec;
+ Datum values[Natts_pg_publication_rel];
+ bool nulls[Natts_pg_publication_rel];
+ bool replace[Natts_pg_publication_rel];
+ HeapTuple newtup;
+ int i,
+ j;
+ bool isnull;
+
+ /* Obtain the original column list */
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP,
+ tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull) /* shouldn't happen */
+ elog(ERROR, "can't drop column from publication without a column list");
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* Construct a list excluding the given column */
+ newelems = palloc(sizeof(int16) * nelems - 1);
+ for (i = 0, j = 0; i < nelems - 1; i++)
+ {
+ if (elems[i] == attnum)
+ continue;
+ newelems[j++] = elems[i];
+ }
+
+ /*
+ * If this is the last column used in the publication, disallow the
+ * command. We could alternatively just drop the relation from the
+ * publication.
+ */
+ if (j == 0)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot drop the last column in publication \"%s\"",
+ get_publication_name(pubrel->prpubid, false)),
+ errhint("Remove table \"%s\" from the publication first.",
+ get_rel_name(pubrel->prrelid)));
+ }
+
+ /* Build the updated tuple */
+ MemSet(values, 0, sizeof(values));
+ MemSet(nulls, false, sizeof(nulls));
+ MemSet(replace, false, sizeof(replace));
+ newvec = buildint2vector(newelems, j);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(newvec);
+ replace[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ /* Execute the update */
+ newtup = heap_modify_tuple(tup, RelationGetDescr(rel),
+ values, nulls, replace);
+ CatalogTupleUpdate(rel, &tup->t_self, newtup);
+ }
ReleaseSysCache(tup);
@@ -932,6 +1007,8 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
@@ -965,8 +1042,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
+
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
@@ -1074,6 +1154,12 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list may not be specified for relation \"%s\" in ALTER PUBLICATION ... SET/DROP command",
+ RelationGetRelationName(pubrel->relation)));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 47b29001d5..7207dcf9c0 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -40,8 +40,9 @@
#include "catalog/pg_inherits.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
-#include "catalog/pg_tablespace.h"
+#include "catalog/pg_publication_rel.h"
#include "catalog/pg_statistic_ext.h"
+#include "catalog/pg_tablespace.h"
#include "catalog/pg_trigger.h"
#include "catalog/pg_type.h"
#include "catalog/storage.h"
@@ -8420,6 +8421,13 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * If the column is part of a replication column list, arrange to get that
+ * removed too.
+ */
+ findAndAddAddresses(addrs, PublicationRelRelationId,
+ RelationRelationId, RelationGetRelid(rel), attnum);
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index df0b747883..0ff4c1ceac 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4833,6 +4833,7 @@ _copyPublicationTable(const PublicationTable *from)
PublicationTable *newnode = makeNode(PublicationTable);
COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index cb7ddd463c..d786a688ac 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2312,6 +2312,7 @@ static bool
_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 3d4dd43e47..4dad6fedfb 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9742,12 +9742,13 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr
+ TABLE relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
+ $$->pubtable->columns = $3;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9762,28 +9763,38 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLE_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId
+ | ColId opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- $$->name = $1;
+ if ($2 != NULL)
+ {
+ $$->pubtable = makeNode(PublicationTable);
+ $$->pubtable->relation = makeRangeVar(NULL, $1, @1);
+ $$->pubtable->columns = $2;
+ $$->name = NULL;
+ }
+ else
+ $$->name = $1;
$$->location = @1;
}
- | ColId indirection
+ | ColId indirection opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
+ $$->pubtable->columns = $3;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr
+ | extended_relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
+ $$->pubtable->columns = $2;
}
| CURRENT_SCHEMA
{
@@ -17435,8 +17446,9 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
{
/* convert it to PublicationTable */
PublicationTable *pubtable = makeNode(PublicationTable);
- pubtable->relation = makeRangeVar(NULL, pubobj->name,
- pubobj->location);
+
+ pubtable->relation =
+ makeRangeVar(NULL, pubobj->name, pubobj->location);
pubobj->pubtable = pubtable;
pubobj->name = NULL;
}
@@ -17444,6 +17456,16 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
else if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLE_IN_SCHEMA ||
pubobj->pubobjtype == PUBLICATIONOBJ_TABLE_IN_CUR_SCHEMA)
{
+ /*
+ * This can happen if a column list is specified in a continuation
+ * for a schema entry; reject it.
+ */
+ if (pubobj->pubtable)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schemas"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 9f5bf4b639..15d8192238 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,9 +29,9 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *att_map);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary, Bitmapset *att_map);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +398,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *att_map)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +410,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_map);
}
/*
@@ -442,7 +442,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary, Bitmapset *att_map)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -463,11 +463,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, att_map);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, att_map);
}
/*
@@ -536,7 +536,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -651,7 +651,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *att_map)
{
char *relname;
@@ -673,7 +673,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, att_map);
}
/*
@@ -749,20 +749,42 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary,
+ Bitmapset *att_map)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
bool isnull[MaxTupleAttributeNumber];
int i;
uint16 nliveatts = 0;
+ Bitmapset *idattrs = NULL;
+ bool replidentfull;
+ Form_pg_attribute att;
desc = RelationGetDescr(rel);
+ replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ if (!replidentfull)
+ idattrs = RelationGetIdentityKeyBitmap(rel);
+
for (i = 0; i < desc->natts; i++)
{
+ att = TupleDescAttr(desc, i);
if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
continue;
+
+ /*
+ * Do not increment count of attributes if not a part of column
+ * filters except for replica identity columns or if replica identity
+ * is full.
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map) &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs) &&
+ !replidentfull)
+ continue;
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -800,6 +822,19 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
continue;
}
+ /*
+ * Do not send attribute data if it is not a part of column filters,
+ * except if it is a part of REPLICA IDENTITY or REPLICA IDENTITY is
+ * full, send the data.
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map) &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs) &&
+ !replidentfull)
+ continue;
+
typtup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(att->atttypid));
if (!HeapTupleIsValid(typtup))
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
@@ -904,7 +939,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *att_map)
{
TupleDesc desc;
int i;
@@ -914,20 +949,35 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
desc = RelationGetDescr(rel);
+ /* fetch bitmap of REPLICATION IDENTITY attributes */
+ replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ if (!replidentfull)
+ idattrs = RelationGetIdentityKeyBitmap(rel);
+
/* send number of live attributes */
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ /* REPLICA IDENTITY FULL means all columns are sent as part of key. */
+ if (replidentfull ||
+ bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs))
+ {
+ nliveatts++;
+ continue;
+ }
+ /* Skip sending if not a part of column filter */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map))
continue;
nliveatts++;
}
pq_sendint16(out, nliveatts);
- /* fetch bitmap of REPLICATION IDENTITY attributes */
- replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
- if (!replidentfull)
- idattrs = RelationGetIdentityKeyBitmap(rel);
-
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -937,6 +987,17 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ /*
+ * Exclude filtered columns, but REPLICA IDENTITY columns can't be
+ * excluded
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map) &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ idattrs)
+ && !replidentfull)
+ continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index f07983a43c..15902faf56 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -111,6 +111,7 @@
#include "replication/origin.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -695,19 +696,25 @@ fetch_remote_table_info(char *nspname, char *relname,
LogicalRepRelation *lrel)
{
WalRcvExecResult *res;
+ WalRcvExecResult *res_pub;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
+ TupleTableSlot *slot_pub;
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid pubRow[] = {TEXTARRAYOID};
bool isnull;
int natt;
+ List *pub_columns = NIL;
+ ListCell *lc;
+ bool am_partition = false;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -737,6 +744,7 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetChar(slot_getattr(slot, 4, &isnull));
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
@@ -774,11 +782,101 @@ fetch_remote_table_info(char *nspname, char *relname,
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+
+ /*
+ * Now, fetch the values of publications' column filters.
+ *
+ * For a partition, use pg_inherit to find the parent, as the
+ * pg_publication_rel contains only the topmost parent table entry in case
+ * the table is partitioned. Run a recursive query to iterate through all
+ * the parents of the partition and retreive the record for the parent
+ * that exists in pg_publication_rel.
+ */
+ resetStringInfo(&cmd);
+ appendStringInfoString(&cmd,
+ "SELECT CASE WHEN prattrs IS NOT NULL THEN\n"
+ " ARRAY(SELECT attname\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(prattrs::int[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END AS columns\n"
+ "FROM pg_catalog.pg_publication_rel\n");
+ if (!am_partition)
+ appendStringInfo(&cmd, "WHERE prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "WHERE prrelid IN (SELECT relid \n"
+ "FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);
+
+ res_pub = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(pubRow), pubRow);
+
+ if (res_pub->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch published columns info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, res_pub->err)));
+ slot_pub = MakeSingleTupleTableSlot(res_pub->tupledesc, &TTSOpsMinimalTuple);
+
+ while (tuplestore_gettupleslot(res_pub->tuplestore, true, false, slot_pub))
+ {
+ Datum adatum;
+ Datum *elems;
+ bool *nulls;
+ int nelems;
+
+ adatum = slot_getattr(slot_pub, 1, &isnull);
+ if (isnull) /* shouldn't happen */
+ elog(ERROR, "unexpected null value in publication column filter");
+ deconstruct_array(DatumGetArrayTypeP(adatum),
+ TEXTOID, -1, false, TYPALIGN_INT,
+ &elems, &nulls, &nelems);
+ for (int i = 0; i < nelems; i++)
+ {
+ if (nulls[i]) /* shouldn't happen */
+ elog(ERROR, "unexpected null value in publication column filter");
+ pub_columns = lappend(pub_columns, TextDatumGetCString(elems[i]));
+ }
+ ExecClearTuple(slot_pub);
+ }
+ ExecDropSingleTupleTableSlot(slot_pub);
+ walrcv_clear_result(res_pub);
+
+ /*
+ * Store the column names only if they are contained in column filter
+ * LogicalRepRelation will only contain attributes corresponding to those
+ * specficied in column filters.
+ */
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ bool found = false;
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 1, &isnull));
Assert(!isnull);
+ if (pub_columns != NIL)
+ {
+ foreach(lc, pub_columns)
+ {
+ char *pub_colname = lfirst(lc);
+
+ if (!strcmp(pub_colname, rel_colname))
+ {
+ found = true;
+ lrel->attnames[natt] = rel_colname;
+ break;
+ }
+ }
+ }
+ else
+ {
+ found = true;
+ lrel->attnames[natt] = rel_colname;
+ }
+ if (!found)
+ continue;
+
lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
@@ -829,8 +927,17 @@ copy_table(Relation rel)
/* Start copy on the publisher. */
initStringInfo(&cmd);
if (lrel.relkind == RELKIND_RELATION)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 6f6a203dea..f9f9ecd0c0 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,16 +15,19 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -81,7 +84,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *att_map);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -130,6 +134,7 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+ Bitmapset *att_map;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -570,11 +575,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
}
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->att_map);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->att_map);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -587,7 +592,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *att_map)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -610,13 +616,25 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /*
+ * Do not send type information if attribute is not present in column
+ * filter. XXX Allow sending type information for REPLICA IDENTITY
+ * COLUMNS with user created type. even when they are not mentioned in
+ * column filters.
+ *
+ * FIXME -- this code seems not verified by tests.
+ */
+ if (att_map != NULL &&
+ !bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
+ att_map))
+ continue;
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, att_map);
OutputPluginWrite(ctx, false);
}
@@ -693,7 +711,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->att_map);
OutputPluginWrite(ctx, true);
break;
}
@@ -722,7 +740,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->att_map);
OutputPluginWrite(ctx, true);
break;
}
@@ -1122,6 +1140,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
bool found;
+ Oid ancestor_id;
MemoryContext oldctx;
Assert(RelationSyncCache != NULL);
@@ -1142,6 +1161,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
+ entry->att_map = NULL;
entry->map = NULL; /* will be set by maybe_send_schema() if
* needed */
}
@@ -1182,6 +1202,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
if (pub->alltables)
{
@@ -1192,8 +1213,6 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (!publish)
{
- bool ancestor_published = false;
-
/*
* For a partition, check if any of the ancestors are
* published. If so, note down the topmost ancestor that is
@@ -1219,6 +1238,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
pub->oid))
{
ancestor_published = true;
+ ancestor_id = ancestor;
if (pub->pubviaroot)
publish_as_relid = ancestor;
}
@@ -1239,15 +1259,47 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ Oid relid;
+ HeapTuple pub_rel_tuple;
+
+ relid = ancestor_published ? ancestor_id : publish_as_relid;
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ Datum pub_rel_cols;
+ bool isnull;
+
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP,
+ pub_rel_tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(pub_rel_cols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
+ entry->att_map = bms_add_member(entry->att_map,
+ elems[i] - FirstLowInvalidHeapAttributeNumber);
+ MemoryContextSwitchTo(oldctx);
+ }
+ ReleaseSysCache(pub_rel_tuple);
+ }
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
}
-
- if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
- break;
}
list_free(pubids);
@@ -1343,6 +1395,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->att_map);
+ entry->att_map = NULL;
if (entry->map)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 10a86f9810..0c438481dc 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4265,6 +4265,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_oid;
int i_prpubid;
int i_prrelid;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4276,8 +4277,13 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
/* Collect all publication membership info. */
appendPQExpBufferStr(query,
- "SELECT tableoid, oid, prpubid, prrelid "
- "FROM pg_catalog.pg_publication_rel");
+ "SELECT tableoid, oid, prpubid, prrelid");
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBufferStr(query, ", prattrs");
+ else
+ appendPQExpBufferStr(query, ", NULL as prattrs");
+ appendPQExpBufferStr(query,
+ " FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
ntups = PQntuples(res);
@@ -4286,6 +4292,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_oid = PQfnumber(res, "oid");
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4327,6 +4334,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
pubrinfo[j].publication = pubinfo;
pubrinfo[j].pubtable = tbinfo;
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[i].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4391,10 +4420,12 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s;\n",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+ appendPQExpBufferStr(query, ";\n");
/*
* There is no point in creating a drop query as the drop is done by table
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 6dccb4be4e..50a5b885f6 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -633,6 +633,7 @@ typedef struct _PublicationRelInfo
DumpableObject dobj;
PublicationInfo *publication;
TableInfo *pubtable;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 72d8547628..46fa616406 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6302,7 +6302,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -6319,10 +6319,14 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (!as_schema) /* as table */
+ {
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
- else
+ if (!PQgetisnull(res, i, 2))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 2));
+ }
+ else /* as schema */
printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
@@ -6450,8 +6454,20 @@ describePublications(const char *pattern)
{
/* Get the tables for the specified publication */
printfPQExpBuffer(&buf,
- "SELECT n.nspname, c.relname\n"
- "FROM pg_catalog.pg_class c,\n"
+ "SELECT n.nspname, c.relname, \n");
+ if (pset.sversion >= 150000)
+ appendPQExpBufferStr(&buf,
+ " CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string"
+ "(ARRAY(SELECT attname\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::int[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END AS columns");
+ else
+ appendPQExpBufferStr(&buf, "NULL as columns");
+ appendPQExpBuffer(&buf,
+ "\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
" pg_catalog.pg_publication_rel pr\n"
"WHERE c.relnamespace = n.oid\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 2f412ca3db..84ee807e0b 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1648,6 +1648,8 @@ psql_completion(const char *text, int start, int end)
/* ALTER PUBLICATION <name> ADD */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
+ else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD", "TABLE"))
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL);
/* ALTER PUBLICATION <name> DROP */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "DROP"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h
index 3eca295ff4..76d421e09e 100644
--- a/src/include/catalog/dependency.h
+++ b/src/include/catalog/dependency.h
@@ -214,6 +214,9 @@ extern long changeDependenciesOf(Oid classId, Oid oldObjectId,
extern long changeDependenciesOn(Oid refClassId, Oid oldRefObjectId,
Oid newRefObjectId);
+extern void findAndAddAddresses(ObjectAddresses *addrs, Oid classId,
+ Oid refclassId, Oid refobjectId, int32 refobjsubId);
+
extern Oid getExtensionOfObject(Oid classId, Oid objectId);
extern List *getAutoExtensionsOfObject(Oid classId, Oid objectId);
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 902f2f2f0d..f5ae2065e9 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -86,6 +86,7 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index b5d5504cbb..7ad285faae 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN
+ int2vector prattrs; /* Variable length field starts here */
+#endif
} FormData_pg_publication_rel;
/* ----------------
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index 4ba68c70ee..23f037df7f 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -25,7 +25,7 @@
extern ObjectAddress CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt);
extern void AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt);
extern void RemovePublicationById(Oid pubid);
-extern void RemovePublicationRelById(Oid proid);
+extern void RemovePublicationRelById(Oid proid, int32 attnum);
extern void RemovePublicationSchemaById(Oid psoid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 4c5a8a39bf..02b547d044 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3642,6 +3642,7 @@ typedef struct PublicationTable
{
NodeTag type;
RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 83741dcf42..709b4be916 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -207,11 +207,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *att_map);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *att_map);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -228,7 +228,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *att_map);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 5ac2d666a2..84afe0ebef 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -165,7 +165,35 @@ Publications:
regress_publication_user | t | t | t | f | f | f
(1 row)
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (x, y, z); -- error
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+\dRp+ testpub_fortable
+ Publication testpub_fortable
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | t | t | t | f
+Tables:
+ "public.testpub_tbl5" (a)
+Tables from schemas:
+ "pub_test"
+
+ALTER TABLE testpub_tbl5 DROP COLUMN a;
+ERROR: cannot drop the last column in publication "testpub_fortable"
+HINT: Remove table "testpub_tbl5" from the publication first.
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl6"
+DETAIL: Cannot have column filter on relations with REPLICA IDENTITY FULL.
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
CREATE TABLE testpub_tbl3a (b text) INHERITS (testpub_tbl3);
@@ -669,6 +697,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schemas
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 56dd358554..200158ba69 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -89,7 +89,20 @@ SELECT pubname, puballtables FROM pg_publication WHERE pubname = 'testpub_forall
\d+ testpub_tbl2
\dRp+ testpub_foralltables
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (x, y, z); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+\dRp+ testpub_fortable
+ALTER TABLE testpub_tbl5 DROP COLUMN a;
+
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
@@ -362,6 +375,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/021_column_filter.pl b/src/test/subscription/t/021_column_filter.pl
new file mode 100644
index 0000000000..354e6ac363
--- /dev/null
+++ b/src/test/subscription/t/021_column_filter.pl
@@ -0,0 +1,162 @@
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test TRUNCATE
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 10;
+
+# setup
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int)");
+# Test with weird column names
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, B varchar, \"c'\" int)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+#Test replication with multi-level partition
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, \"c'\" int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+#Test create publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub1 FOR TABLE tab1(a, \"B\"), tab3(\"a'\",\"c'\"), test_part(a,b)");
+
+my $result = $node_publisher->safe_psql('postgres',
+ "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
+is($result, qq(tab1|1 2
+tab3|1 3
+test_part|1 2), 'publication relation updated');
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+);
+#Initial sync
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab1 VALUES (1,2,3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 VALUES (1,2,3)");
+#Test for replication of partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (1,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (2,'bcd', '2021-07-03 11:12:13')");
+#Test for replication of multi-level partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (4,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (5,'bcd', '2021-07-03 11:12:13')");
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'insert on column c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab3");
+is($result, qq(1|3), 'insert on column b is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part");
+is($result, qq(1|abc\n2|bcd\n4|abc\n5|bcd), 'insert on all columns is replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab1 SET c = 5 where a = 1");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'update on column c is not replicated');
+
+#Test alter publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION"
+);
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 VALUES (1,'abc',3)");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2");
+is($result, qq(1|abc), 'insert on column c is not replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET c = 5 where a = 1");
+is($result, qq(1|abc), 'update on column c is not replicated');
+
+# Test behavior when a column is dropped
+$node_publisher->safe_psql('postgres',
+ "ALTER TABLE test_part DROP COLUMN b");
+$result = $node_publisher->safe_psql('postgres',
+ "select prrelid::regclass, prattrs from pg_publication_rel pb;");
+is($result,
+ q(tab1|1 2
+tab3|1 3
+tab2|1 2
+test_part|1), 'column test_part.b removed');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (3, '2021-12-13 12:13:14')");
+$node_publisher->wait_for_catchup('sub1');
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part WHERE a = 3");
+is($result, "3|", 'only column a is replicated');
+
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab4 (a int PRIMARY KEY, b int, c int, d int)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab4 (a int PRIMARY KEY, b int, d int)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub2 FOR TABLE tab4 (a, b)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub3 FOR TABLE tab4 (a, d)");
+$node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2, pub3");
+$node_publisher->wait_for_catchup('sub2');
+$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (1, 11, 111, 1111)");
+$node_publisher->safe_psql('postgres', "INSERT INTO tab4 VALUES (2, 22, 222, 2222)");
+$node_publisher->wait_for_catchup('sub2');
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab4;"),
+ qq(1|11|1111
+2|22|2222),
+ 'overlapping publications with overlapping column lists');
--
2.31.1
0002-review.patchtext/x-patch; charset=UTF-8; name=0002-review.patchDownload
From a1cb64a17b35dd75daf22a9f59da176ed109612a Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Tue, 14 Dec 2021 22:11:10 +0100
Subject: [PATCH 2/2] review
---
src/backend/catalog/pg_publication.c | 60 +++++++++++++++------
src/backend/commands/publicationcmds.c | 12 +++++
src/backend/commands/tablecmds.c | 2 +-
src/backend/replication/logical/proto.c | 8 +++
src/backend/replication/pgoutput/pgoutput.c | 8 +++
5 files changed, 72 insertions(+), 18 deletions(-)
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index ae58adc8e5..7a478b7072 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -99,28 +99,27 @@ check_publication_add_relation(Relation targetrel, Bitmapset *columns)
*/
if (columns != NULL)
{
+ Bitmapset *idattrs;
+
if (replidentfull)
ereport(ERROR,
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("invalid column list for publishing relation \"%s\"",
RelationGetRelationName(targetrel)),
errdetail("Cannot have column filter on relations with REPLICA IDENTITY FULL."));
- else
- {
- Bitmapset *idattrs;
-
- idattrs = RelationGetIndexAttrBitmap(targetrel,
- INDEX_ATTR_BITMAP_IDENTITY_KEY);
- if (!bms_is_subset(idattrs, columns))
- ereport(ERROR,
- errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
- errmsg("invalid column list for publishing relation \"%s\"",
- RelationGetRelationName(targetrel)),
- errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
-
- if (idattrs)
- pfree(idattrs);
- }
+
+ /* XXX The else was unnecessary, because the "if" always errors-out */
+ idattrs = RelationGetIndexAttrBitmap(targetrel,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+ if (!bms_is_subset(idattrs, columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
+
+ if (idattrs)
+ pfree(idattrs);
}
}
@@ -352,6 +351,15 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
/* FIXME need to handle the case of different column list */
+ /*
+ * XXX So what's the right behavior for ADD TABLE with different column
+ * list? I'd say we should allow that, and that it should be mostly the
+ * same thing as adding/removing columns to the list incrementally, i.e.
+ * we should replace the column lists. We could also prohibit, but that
+ * seems like a really annoying limitation, forcing people to remove/add
+ * the relation.
+ */
+
if (if_not_exists)
return InvalidObjectAddress;
@@ -361,6 +369,10 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
RelationGetRelationName(targetrel->relation), pub->name)));
}
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
attarray = palloc(sizeof(AttrNumber) * list_length(targetrel->columns));
foreach(lc, targetrel->columns)
{
@@ -372,12 +384,23 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
errcode(ERRCODE_UNDEFINED_COLUMN),
errmsg("column \"%s\" of relation \"%s\" does not exist",
colname, RelationGetRelationName(targetrel->relation)));
- if (attnum < 0)
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
ereport(ERROR,
errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
errmsg("cannot reference system column \"%s\" in publication column list",
colname));
+ /*
+ * XXX The offset seems kinda pointless, because at this point we're not
+ * dealing with system attributes, so all attnums are > 0. Are we comparing
+ * it to arbitrary attnums later? This would simplify adding the deps later
+ * because we're offsetting it back.
+ *
+ * XXX I wouldn't say "twice" though. It can be specified multiple times,
+ * it's just that we discover it during the second occurrence. I'd say
+ * "duplicate column name in publication column list" instead.
+ */
if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, attmap))
ereport(ERROR,
errcode(ERRCODE_DUPLICATE_OBJECT),
@@ -417,6 +440,9 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
CatalogTupleInsert(rel, tup);
heap_freetuple(tup);
+ /* XXX not sure if worth an explicit free, but if we free the tuple ... */
+ pfree(attarray);
+
ObjectAddressSet(myself, PublicationRelRelationId, prrelid);
/* Add dependency on the publication */
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index a070914bdd..38efc5ad59 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -759,6 +759,8 @@ AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt)
/*
* Remove relation from publication by mapping OID, or publication status
* of one column of that relation in the publication if an attnum is given.
+ *
+ * XXX Whats "publication status"?
*/
void
RemovePublicationRelById(Oid proid, int32 attnum)
@@ -825,6 +827,9 @@ RemovePublicationRelById(Oid proid, int32 attnum)
&isnull);
if (isnull) /* shouldn't happen */
elog(ERROR, "can't drop column from publication without a column list");
+
+ /* XXX We're reading the array in multiple places, I suggest to move it
+ * to a separate function, maybe? */
arr = DatumGetArrayTypeP(adatum);
nelems = ARR_DIMS(arr)[0];
elems = (int16 *) ARR_DATA_PTR(arr);
@@ -833,6 +838,7 @@ RemovePublicationRelById(Oid proid, int32 attnum)
newelems = palloc(sizeof(int16) * nelems - 1);
for (i = 0, j = 0; i < nelems - 1; i++)
{
+ /* XXX Can it happen that we never find a match? Seems like an error. */
if (elems[i] == attnum)
continue;
newelems[j++] = elems[i];
@@ -842,6 +848,10 @@ RemovePublicationRelById(Oid proid, int32 attnum)
* If this is the last column used in the publication, disallow the
* command. We could alternatively just drop the relation from the
* publication.
+ *
+ * XXX Alternatively we could switch to "replicate all column", but
+ * that seems counterintuitive. However, is there a way to switch
+ * between these two modes, somehow?
*/
if (j == 0)
{
@@ -1154,6 +1164,8 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ /* XXX Shouldn't this be prevented by the grammar, ideally? Can it actually
+ * happen? It does not seem to be tested in the regression tests. */
if (pubrel->columns)
ereport(ERROR,
errcode(ERRCODE_SYNTAX_ERROR),
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 7207dcf9c0..f7c21158c0 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -8422,7 +8422,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
/*
- * If the column is part of a replication column list, arrange to get that
+ * If the column is part of a publication column list, arrange to get that
* removed too.
*/
findAndAddAddresses(addrs, PublicationRelRelationId,
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 15d8192238..885acd6c9e 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -777,6 +777,13 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
* Do not increment count of attributes if not a part of column
* filters except for replica identity columns or if replica identity
* is full.
+ *
+ * XXX This exact check is done in multiple places, so maybe let's move it
+ * to a separate function and call it. Also, att_map is for user attrs only
+ * so maybe we can get rid of the offsets?
+ *
+ * XXX Why do we need to check both att_map and idattrs? Aren't we enforcing
+ * that indattrs is always a subset of att_map?
*/
if (att_map != NULL &&
!bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
@@ -970,6 +977,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *att_map)
continue;
}
/* Skip sending if not a part of column filter */
+ /* XXX How come this is not checking idattrs and replindentfull? */
if (att_map != NULL &&
!bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
att_map))
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index f9f9ecd0c0..551c4d5315 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -134,6 +134,14 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+
+ /* XXX Needs a comment explaining what att_map does, how it's different
+ * from map. The naming seems really confusing and it's a bit unclear how
+ * these two bits interact (i.e. partitioning + column list).
+ *
+ * XXX In fact, is it really mapping anything? It seems more like a simple
+ * list of attnums, translated from list of names.
+ */
Bitmapset *att_map;
} RelationSyncEntry;
--
2.31.1
On Tues, Dec 14, 2021 1:48 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
Hmm, I messed up the patch file I sent. Here's the complete patch.
Hi,
I have a minor question about the replica identity check of this patch.
+check_publication_add_relation(Relation targetrel, Bitmapset *columns)
...
+ idattrs = RelationGetIndexAttrBitmap(targetrel,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+ if (!bms_is_subset(idattrs, columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
+
The patch ensures all columns of RT are in column list when CREATE/ALTER
publication, but it seems doesn't prevent user from changing the replica
identity or dropping the index used in replica identity. Do we also need to
check those cases ?
Best regards,
Hou zj
On 2021-Dec-16, houzj.fnst@fujitsu.com wrote:
The patch ensures all columns of RT are in column list when CREATE/ALTER
publication, but it seems doesn't prevent user from changing the replica
identity or dropping the index used in replica identity. Do we also need to
check those cases ?
Yes, we do. As it happens, I spent a couple of hours yesterday writing
code for that, at least partially. I haven't yet checked what happens
with cases like REPLICA NOTHING, or REPLICA INDEX <xyz> and then
dropping that index.
My initial ideas were a bit wrong BTW: I thought we should check the
combination of column lists in all publications (a bitwise-OR of column
bitmaps, so to speak). But conceptually that's wrong: we need to check
the column list of each publication individually instead. Otherwise, if
you wanted to hide a column from some publication but that column was
part of the replica identity, there'd be no way to identify the tuple in
the replica. (Or, if the pgouput code disobeys the column list and
sends the replica identity even if it's not in the column list, then
you'd be potentially publishing data that you wanted to hide.)
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
On 2021-Dec-14, Tomas Vondra wrote:
7) There's a couple places doing this
if (att_map != NULL &&
!bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
att_map) &&
!bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
idattrs) &&
!replidentfull)which is really hard to understand (even if we get rid of the offset), so
maybe let's move that to a function with sensible name. Also, some places
don't check indattrs - seems a bit suspicious.
It is indeed pretty hard to read ... but I think this is completely
unnecessary. Any column that is part of the identity should have been
included in the column filter, so there is no need to check for the
identity attributes separately. Testing just for the columns in the
filter ought to be sufficient; and the cases "if att_map NULL" and "is
replica identity FULL" are also equivalent, because in the case of FULL,
you're disallowed from setting a column list. So this whole thing can
be reduced to just this:
if (att_map != NULL && !bms_is_member(att->attnum, att_map))
continue; /* that is, don't send this attribute */
so I don't think this merits a separate function.
[ says he, after already trying to write said function ]
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Before you were born your parents weren't as boring as they are now. They
got that way paying your bills, cleaning up your room and listening to you
tell them how idealistic you are." -- Charles J. Sykes' advice to teenagers
On Wed, Dec 15, 2021 at 1:05 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Dec-14, Tomas Vondra wrote:
Yeah. I think it's not clear if this should behave more like an index or a
view. When an indexed column gets dropped we simply drop the index. But if
you drop a column referenced by a view, we fail with an error. I think we
should handle this more like a view, because publications are externally
visible objects too (while indexes are pretty much just an implementation
detail).I agree -- I think it's more like a view than like an index. (The
original proposal was that if you dropped a column that was part of the
column list of a relation in a publication, the entire relation is
dropped from the view,
I think in the above sentence, you mean to say "dropped from the
publication". So, IIUC, you are proposing that if one drops a column
that was part of the column list of a relation in a publication, an
error will be raised. Also, if the user specifies CASCADE in Alter
Table ... Drop Column, then we drop the relation from publication. Is
that right? BTW, this is somewhat on the lines of what row_filter
patch is also doing where if the user drops the column that was part
of row_filter for a relation in publication, we give an error and if
the user tries to drop the column with CASCADE then the relation is
removed from the publication.
--
With Regards,
Amit Kapila.
On 17.12.21 05:47, Amit Kapila wrote:
I think in the above sentence, you mean to say "dropped from the
publication". So, IIUC, you are proposing that if one drops a column
that was part of the column list of a relation in a publication, an
error will be raised. Also, if the user specifies CASCADE in Alter
Table ... Drop Column, then we drop the relation from publication. Is
that right? BTW, this is somewhat on the lines of what row_filter
patch is also doing where if the user drops the column that was part
of row_filter for a relation in publication, we give an error and if
the user tries to drop the column with CASCADE then the relation is
removed from the publication.
That looks correct. Consider how triggers behave: Dropping a column
that a trigger uses (either in UPDATE OF or a WHEN condition) errors
with RESTRICT and drops the trigger with CASCADE.
On Friday, December 17, 2021 1:55 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Dec-16, houzj.fnst@fujitsu.com wrote:
The patch ensures all columns of RT are in column list when
CREATE/ALTER publication, but it seems doesn't prevent user from
changing the replica identity or dropping the index used in replica
identity. Do we also need to check those cases ?Yes, we do. As it happens, I spent a couple of hours yesterday writing code for
that, at least partially. I haven't yet checked what happens with cases like
REPLICA NOTHING, or REPLICA INDEX <xyz> and then dropping that index.My initial ideas were a bit wrong BTW: I thought we should check the
combination of column lists in all publications (a bitwise-OR of column bitmaps,
so to speak). But conceptually that's wrong: we need to check the column list
of each publication individually instead. Otherwise, if you wanted to hide a
column from some publication but that column was part of the replica identity,
there'd be no way to identify the tuple in the replica. (Or, if the pgouput code
disobeys the column list and sends the replica identity even if it's not in the
column list, then you'd be potentially publishing data that you wanted to hide.)
Thanks for the explanation.
Apart from ALTER REPLICA IDENTITY and DROP INDEX, I think there could be
some other cases we need to handle for the replica identity check:
1)
When adding a partitioned table with column list to the publication, I think we
need to check the RI of all its leaf partition. Because the RI on the partition
is the one actually takes effect.
2)
ALTER TABLE ADD PRIMARY KEY;
ALTER TABLE DROP CONSTRAINT "PRIMAEY KEY";
If the replica identity is default, it will use the primary key. we might also
need to prevent user from adding or removing primary key in this case.
Based on the above cases, the RI check seems could bring considerable amount of
code. So, how about we follow what we already did in CheckCmdReplicaIdentity(),
we can put the check for RI in that function, so that we can cover all the
cases and reduce the code change. And if we are worried about the cost of do
the check for UPDATE and DELETE every time, we can also save the result in the
relcache. It's safe because every operation change the RI will invalidate the
relcache. We are using this approach in row filter patch to make sure all
columns in row filter expression are part of RI.
Best regards,
Hou zj
Hi,
Thank you for updating the patch. The regression tests and tap tests pass
with v9 patch.
After working on this a little bit more, I realized that this is a bad
idea overall. It causes lots of complications and it's just not worth
it. So I'm back at my original thought that we need to throw an ERROR
at ALTER TABLE .. DROP COLUMN time if the column is part of a
replication column filter, and suggest the user to remove the column
from the filter first and reattempt the DROP COLUMN.This means that we need to support changing the column list of a table
in a publication. I'm looking at implementing some form of ALTER
PUBLICATION for that.
I think right now the patch contains support only for ALTER PUBLICATION..
ADD TABLE with column filters.
In order to achieve changing the column lists of a published table, I think
we can extend the
ALTER TABLE ..SET TABLE syntax to support specification of column list.
So this whole thing can
be reduced to just this:
if (att_map != NULL && !bms_is_member(att->attnum, att_map))
continue; /* that is, don't send this attribute */
I agree the condition can be shortened now. The long if condition was
included because initially the feature
allowed specifying filters without replica identity columns(sent those
columns internally without user
having to specify).
900 + * the table is partitioned. Run a recursive query to iterate
through all
901 + * the parents of the partition and retreive the record for
the parent
902 + * that exists in pg_publication_rel.
903 + */
The above comment in fetch_remote_table_info() can be changed as the
recursive query
is no longer used.
Thank you,
Rahila Syed
On 2021-Dec-17, Rahila Syed wrote:
This means that we need to support changing the column list of a
table in a publication. I'm looking at implementing some form of
ALTER PUBLICATION for that.I think right now the patch contains support only for ALTER
PUBLICATION.. ADD TABLE with column filters. In order to achieve
changing the column lists of a published table, I think we can extend
the ALTER TABLE ..SET TABLE syntax to support specification of column
list.
Yeah, that's what I was thinking too.
So this whole thing can be reduced to just this:
if (att_map != NULL && !bms_is_member(att->attnum, att_map))
continue; /* that is, don't send this attribute */I agree the condition can be shortened now. The long if condition was
included because initially the feature allowed specifying filters
without replica identity columns(sent those columns internally without
user having to specify).
Ah, true, I had forgotten that. Thanks.
900 + * the table is partitioned. Run a recursive query to iterate through all
901 + * the parents of the partition and retreive the record for the parent
902 + * that exists in pg_publication_rel.
903 + */The above comment in fetch_remote_table_info() can be changed as the
recursive query is no longer used.
Oh, of course.
I'll finish some loose ends and submit a v10, but it's still not final.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
"Right now the sectors on the hard disk run clockwise, but I heard a rumor that
you can squeeze 0.2% more throughput by running them counterclockwise.
It's worth the effort. Recommended." (Gerry Pourwelle)
So I've been thinking about this as a "security" item (you can see my
comments to that effect sprinkled all over this thread), in the sense
that if a publication "hides" some column, then the replica just won't
get access to it. But in reality that's mistaken: the filtering that
this patch implements is done based on the queries that *the replica*
executes at its own volition; if the replica decides to ignore the list
of columns, it'll be able to get all columns. All it takes is an
uncooperative replica in order for the lot of data to be exposed anyway.
If the server has a *separate* security mechanism to hide the columns
(per-column privs), it is that feature that will protect the data, not
the logical-replication-feature to filter out columns.
This led me to realize that the replica-side code in tablesync.c is
totally oblivious to what's the publication through which a table is
being received from in the replica. So we're not aware of a replica
being exposed only a subset of columns through some specific
publication; and a lot more hacking is needed than this patch does, in
order to be aware of which publications are being used.
I'm going to have a deeper look at this whole thing.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
On 12/17/21 22:07, Alvaro Herrera wrote:
So I've been thinking about this as a "security" item (you can see my
comments to that effect sprinkled all over this thread), in the sense
that if a publication "hides" some column, then the replica just won't
get access to it. But in reality that's mistaken: the filtering that
this patch implements is done based on the queries that *the replica*
executes at its own volition; if the replica decides to ignore the list
of columns, it'll be able to get all columns. All it takes is an
uncooperative replica in order for the lot of data to be exposed anyway.
Interesting, I haven't really looked at this as a security feature. And
in my experience if something is not carefully designed to be secure
from the get go, it's really hard to add that bit later ...
You say it's the replica making the decisions, but my mental model is
it's the publisher decoding the data for a given list of publications
(which indeed is specified by the subscriber). But the subscriber can't
tweak the definition of publications, right? Or what do you mean by
queries executed by the replica? What are the gap?
If the server has a *separate* security mechanism to hide the columns
(per-column privs), it is that feature that will protect the data, not
the logical-replication-feature to filter out columns.
Right. Although I haven't thought about how logical decoding interacts
with column privileges. I don't think logical decoding actually checks
column privileges - I certainly don't recall any ACL checks in
src/backend/replication ...
AFAIK we only really check privileges during initial sync (when creating
the slot and copying data), but then we keep replicating data even if
the privilege gets revoked for the table/column. In principle the
replication role is pretty close to superuser.
This led me to realize that the replica-side code in tablesync.c is
totally oblivious to what's the publication through which a table is
being received from in the replica. So we're not aware of a replica
being exposed only a subset of columns through some specific
publication; and a lot more hacking is needed than this patch does, in
order to be aware of which publications are being used.I'm going to have a deeper look at this whole thing.
Does that mean we currently sync all the columns in the initial sync,
and only start filtering columns later while decoding transactions?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 2021-Dec-17, Tomas Vondra wrote:
On 12/17/21 22:07, Alvaro Herrera wrote:
So I've been thinking about this as a "security" item (you can see my
comments to that effect sprinkled all over this thread), in the sense
that if a publication "hides" some column, then the replica just won't
get access to it. But in reality that's mistaken: the filtering that
this patch implements is done based on the queries that *the replica*
executes at its own volition; if the replica decides to ignore the list
of columns, it'll be able to get all columns. All it takes is an
uncooperative replica in order for the lot of data to be exposed anyway.Interesting, I haven't really looked at this as a security feature. And in
my experience if something is not carefully designed to be secure from the
get go, it's really hard to add that bit later ...
I guess the way to really harden replication is to use the GRANT system
at the publisher's side to restrict access for the replication user.
This would provide actual security. So you're right that I seem to be
barking at the wrong tree ... maybe I need to give a careful look at
the documentation for logical replication to understand what is being
offered, and to make sure that we explicitly indicate that limiting the
column list does not provide any actual security.
You say it's the replica making the decisions, but my mental model is it's
the publisher decoding the data for a given list of publications (which
indeed is specified by the subscriber). But the subscriber can't tweak the
definition of publications, right? Or what do you mean by queries executed
by the replica? What are the gap?
I am thinking in somebody modifying the code that the replica runs, so
that it ignores the column list that the publication has been configured
to provide; instead of querying only those columns, it would query all
columns.
If the server has a *separate* security mechanism to hide the columns
(per-column privs), it is that feature that will protect the data, not
the logical-replication-feature to filter out columns.Right. Although I haven't thought about how logical decoding interacts with
column privileges. I don't think logical decoding actually checks column
privileges - I certainly don't recall any ACL checks in
src/backend/replication ...
Well, in practice if you're confronted with a replica that's controlled
by a malicious user that can tweak its behavior, then replica-side
privilege checking won't do anything useful.
This led me to realize that the replica-side code in tablesync.c is
totally oblivious to what's the publication through which a table is
being received from in the replica. So we're not aware of a replica
being exposed only a subset of columns through some specific
publication; and a lot more hacking is needed than this patch does, in
order to be aware of which publications are being used.
Does that mean we currently sync all the columns in the initial sync, and
only start filtering columns later while decoding transactions?
No, it does filter the list of columns in the initial sync. But the
current implementation is bogus, because it obtains the list of *all*
publications in which the table is published, not just the ones that the
subscription is configured to get data from. And the sync code doesn't
receive the list of publications. We need more thorough patching of the
sync code to close that hole.
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
On 12/18/21 02:34, Alvaro Herrera wrote:
On 2021-Dec-17, Tomas Vondra wrote:
On 12/17/21 22:07, Alvaro Herrera wrote:
So I've been thinking about this as a "security" item (you can see my
comments to that effect sprinkled all over this thread), in the sense
that if a publication "hides" some column, then the replica just won't
get access to it. But in reality that's mistaken: the filtering that
this patch implements is done based on the queries that *the replica*
executes at its own volition; if the replica decides to ignore the list
of columns, it'll be able to get all columns. All it takes is an
uncooperative replica in order for the lot of data to be exposed anyway.Interesting, I haven't really looked at this as a security feature. And in
my experience if something is not carefully designed to be secure from the
get go, it's really hard to add that bit later ...I guess the way to really harden replication is to use the GRANT system
at the publisher's side to restrict access for the replication user.
This would provide actual security. So you're right that I seem to be
barking at the wrong tree ... maybe I need to give a careful look at
the documentation for logical replication to understand what is being
offered, and to make sure that we explicitly indicate that limiting the
column list does not provide any actual security.You say it's the replica making the decisions, but my mental model is it's
the publisher decoding the data for a given list of publications (which
indeed is specified by the subscriber). But the subscriber can't tweak the
definition of publications, right? Or what do you mean by queries executed
by the replica? What are the gap?I am thinking in somebody modifying the code that the replica runs, so
that it ignores the column list that the publication has been configured
to provide; instead of querying only those columns, it would query all
columns.If the server has a *separate* security mechanism to hide the columns
(per-column privs), it is that feature that will protect the data, not
the logical-replication-feature to filter out columns.Right. Although I haven't thought about how logical decoding interacts with
column privileges. I don't think logical decoding actually checks column
privileges - I certainly don't recall any ACL checks in
src/backend/replication ...Well, in practice if you're confronted with a replica that's controlled
by a malicious user that can tweak its behavior, then replica-side
privilege checking won't do anything useful.
I don't follow. Surely the decoding happens on the primary node, right?
Which is where the ACL checks would happen, using the role the
replication connection is opened with.
This led me to realize that the replica-side code in tablesync.c is
totally oblivious to what's the publication through which a table is
being received from in the replica. So we're not aware of a replica
being exposed only a subset of columns through some specific
publication; and a lot more hacking is needed than this patch does, in
order to be aware of which publications are being used.Does that mean we currently sync all the columns in the initial sync, and
only start filtering columns later while decoding transactions?No, it does filter the list of columns in the initial sync. But the
current implementation is bogus, because it obtains the list of *all*
publications in which the table is published, not just the ones that the
subscription is configured to get data from. And the sync code doesn't
receive the list of publications. We need more thorough patching of the
sync code to close that hole.
Ah, got it. Thanks for the explanation. Yeah, that makes no sense.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Sat, Dec 18, 2021 at 7:04 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Dec-17, Tomas Vondra wrote:
On 12/17/21 22:07, Alvaro Herrera wrote:
So I've been thinking about this as a "security" item (you can see my
comments to that effect sprinkled all over this thread), in the sense
that if a publication "hides" some column, then the replica just won't
get access to it. But in reality that's mistaken: the filtering that
this patch implements is done based on the queries that *the replica*
executes at its own volition; if the replica decides to ignore the list
of columns, it'll be able to get all columns. All it takes is an
uncooperative replica in order for the lot of data to be exposed anyway.Interesting, I haven't really looked at this as a security feature. And in
my experience if something is not carefully designed to be secure from the
get go, it's really hard to add that bit later ...I guess the way to really harden replication is to use the GRANT system
at the publisher's side to restrict access for the replication user.
This would provide actual security. So you're right that I seem to be
barking at the wrong tree ... maybe I need to give a careful look at
the documentation for logical replication to understand what is being
offered, and to make sure that we explicitly indicate that limiting the
column list does not provide any actual security.
IIRC, the use cases as mentioned by other databases (like Oracle) are
(a) this helps when the target table doesn't have the same set of
columns or (b) when the columns contain some sensitive information
like personal identification number, etc. I think there could be a
side benefit in this which comes from the fact that the lesser data
will flow across the network which could lead to faster replication
especially when the user filters large column data.
--
With Regards,
Amit Kapila.
On Fri, Dec 17, 2021 at 3:16 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
On Friday, December 17, 2021 1:55 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Dec-16, houzj.fnst@fujitsu.com wrote:
The patch ensures all columns of RT are in column list when
CREATE/ALTER publication, but it seems doesn't prevent user from
changing the replica identity or dropping the index used in replica
identity. Do we also need to check those cases ?Yes, we do. As it happens, I spent a couple of hours yesterday writing code for
that, at least partially. I haven't yet checked what happens with cases like
REPLICA NOTHING, or REPLICA INDEX <xyz> and then dropping that index.My initial ideas were a bit wrong BTW: I thought we should check the
combination of column lists in all publications (a bitwise-OR of column bitmaps,
so to speak). But conceptually that's wrong: we need to check the column list
of each publication individually instead. Otherwise, if you wanted to hide a
column from some publication but that column was part of the replica identity,
there'd be no way to identify the tuple in the replica. (Or, if the pgouput code
disobeys the column list and sends the replica identity even if it's not in the
column list, then you'd be potentially publishing data that you wanted to hide.)Thanks for the explanation.
Apart from ALTER REPLICA IDENTITY and DROP INDEX, I think there could be
some other cases we need to handle for the replica identity check:1)
When adding a partitioned table with column list to the publication, I think we
need to check the RI of all its leaf partition. Because the RI on the partition
is the one actually takes effect.2)
ALTER TABLE ADD PRIMARY KEY;
ALTER TABLE DROP CONSTRAINT "PRIMAEY KEY";If the replica identity is default, it will use the primary key. we might also
need to prevent user from adding or removing primary key in this case.Based on the above cases, the RI check seems could bring considerable amount of
code. So, how about we follow what we already did in CheckCmdReplicaIdentity(),
we can put the check for RI in that function, so that we can cover all the
cases and reduce the code change. And if we are worried about the cost of do
the check for UPDATE and DELETE every time, we can also save the result in the
relcache. It's safe because every operation change the RI will invalidate the
relcache. We are using this approach in row filter patch to make sure all
columns in row filter expression are part of RI.
Another point related to RI is that this patch seems to restrict
specifying the RI columns in the column filter list irrespective of
publish action. Do we need to have such a restriction if the
publication publishes 'insert' or 'truncate'?
--
With Regards,
Amit Kapila.
On 2021-Dec-18, Tomas Vondra wrote:
On 12/18/21 02:34, Alvaro Herrera wrote:
On 2021-Dec-17, Tomas Vondra wrote:
If the server has a *separate* security mechanism to hide the
columns (per-column privs), it is that feature that will protect
the data, not the logical-replication-feature to filter out
columns.Right. Although I haven't thought about how logical decoding
interacts with column privileges. I don't think logical decoding
actually checks column privileges - I certainly don't recall any
ACL checks in src/backend/replication ...Well, in practice if you're confronted with a replica that's
controlled by a malicious user that can tweak its behavior, then
replica-side privilege checking won't do anything useful.I don't follow. Surely the decoding happens on the primary node,
right? Which is where the ACL checks would happen, using the role the
replication connection is opened with.
I think you do follow. Yes, the decoding happens on the primary node,
and the security checks should occur in the primary node, because to do
otherwise is folly(*). Which means that column filtering, being a
replica-side feature, is *not* a security feature. I was mistaken about
it, is all. If you want security, you need to use column-level
privileges, as you say.
(*) The checks *must* occur in the primary side, because the primary
does not control the code that runs in the replica side. The primary
must treat the replica as running potentially hostile code. Trying to
defend against that is not practical.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
On 17.12.21 22:07, Alvaro Herrera wrote:
So I've been thinking about this as a "security" item (you can see my
comments to that effect sprinkled all over this thread), in the sense
that if a publication "hides" some column, then the replica just won't
get access to it. But in reality that's mistaken: the filtering that
this patch implements is done based on the queries that *the replica*
executes at its own volition; if the replica decides to ignore the list
of columns, it'll be able to get all columns. All it takes is an
uncooperative replica in order for the lot of data to be exposed anyway.
During normal replication, the publisher should only send the columns
that are configured to be part of the publication. So I don't see a
problem there.
During the initial table sync, the subscriber indeed can construct any
COPY command. We could maybe replace this with a more customized COPY
command variant, like COPY table OF publication TO STDOUT.
But right now the subscriber is sort of assumed to have access to
everything on the publisher anyway, so I doubt that this is the only
problem. But it's worth considering.
Determining that an array has a NULL element seems convoluted. I ended
up with this query, where comparing the result of array_positions() with
an empty array does that. If anybody knows of a simpler way, or any
situations in which this fails, I'm all ears.
with published_cols as (
select case when
pg_catalog.array_positions(pg_catalog.array_agg(unnest), null) <> '{}' then null else
pg_catalog.array_agg(distinct unnest order by unnest) end AS attrs
from pg_catalog.pg_publication p join
pg_catalog.pg_publication_rel pr on (p.oid = pr.prpubid) left join
unnest(prattrs) on (true)
where prrelid = 38168 and p.pubname in ('pub1', 'pub2')
)
SELECT a.attname,
a.atttypid,
a.attnum = ANY(i.indkey)
FROM pg_catalog.pg_attribute a
LEFT JOIN pg_catalog.pg_index i
ON (i.indexrelid = pg_get_replica_identity_index(38168)),
published_cols
WHERE a.attnum > 0::pg_catalog.int2
AND NOT a.attisdropped and a.attgenerated = ''
AND a.attrelid = 38168
AND (published_cols.attrs IS NULL OR attnum = ANY(published_cols.attrs))
ORDER BY a.attnum;
This returns all columns if at least one publication has a NULL prattrs,
or only the union of columns listed in all publications, if all
publications have a list of columns.
(I was worried about obtaining the list of publications, but it turns
out that it's already as a convenient list of OIDs in the MySubscription
struct.)
With this, we can remove the second query added by Rahila's original patch to
filter out nonpublished columns.
I still need to add pg_partition_tree() in order to search for
publications containing a partition ancestor. I'm not yet sure what
happens (and what *should* happen) if an ancestor is part of a
publication and the partition is also part of a publication, and the
column lists differ.
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
Al principio era UNIX, y UNIX habló y dijo: "Hello world\n".
No dijo "Hello New Jersey\n", ni "Hello USA\n".
Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
Determining that an array has a NULL element seems convoluted. I ended
up with this query, where comparing the result of array_positions() with
an empty array does that. If anybody knows of a simpler way, or any
situations in which this fails, I'm all ears.
Maybe better to rethink why we allow elements of prattrs to be null?
regards, tom lane
On 2021-Dec-27, Tom Lane wrote:
Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
Determining that an array has a NULL element seems convoluted. I ended
up with this query, where comparing the result of array_positions() with
an empty array does that. If anybody knows of a simpler way, or any
situations in which this fails, I'm all ears.Maybe better to rethink why we allow elements of prattrs to be null?
What I'm doing is an unnest of all arrays and then aggregating them
back into a single array. If one array is null, the resulting aggregate
contains a null element.
Hmm, maybe I can in parallel do a bool_or() aggregate of "array is null" to
avoid that. ... ah yes, that works:
with published_cols as (
select pg_catalog.bool_or(pr.prattrs is null) as all_columns,
pg_catalog.array_agg(distinct unnest order by unnest) AS attrs
from pg_catalog.pg_publication p join
pg_catalog.pg_publication_rel pr on (p.oid = pr.prpubid) left join
unnest(prattrs) on (true)
where prrelid = :table and p.pubname in ('pub1', 'pub2')
)
SELECT a.attname,
a.atttypid,
a.attnum = ANY(i.indkey)
FROM pg_catalog.pg_attribute a
LEFT JOIN pg_catalog.pg_index i
ON (i.indexrelid = pg_get_replica_identity_index(:table)),
published_cols
WHERE a.attnum > 0::pg_catalog.int2
AND NOT a.attisdropped and a.attgenerated = ''
AND a.attrelid = :table
AND (all_columns OR attnum = ANY(published_cols.attrs))
ORDER BY a.attnum ;
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
OK, getting closer now. I've fixed the code to filter them column list
during the initial sync, and added some more tests for code that wasn't
covered.
There are still some XXX comments. The one that bothers me most is the
lack of an implementation that allows changing the column list in a
publication without having to remove the table from the publication
first.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
"I'm always right, but sometimes I'm more right than other times."
(Linus Torvalds)
Attachments:
column-filtering-11.patchtext/x-diff; charset=utf-8Download
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 34a7034282..5bc2e7a591 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -6877,7 +6877,9 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column (except
+ generated columns and other columns that don't appear in the column
+ filter list, for tables that have one):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index bb4ef5e5e2..c86055b93c 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -30,7 +30,7 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -110,6 +110,8 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index d805e8e77a..73a23cbb02 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -78,6 +78,15 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
publication, so they are never explicitly added to the publication.
</para>
+ <para>
+ When a column list is specified, only the listed columns are replicated;
+ any other columns are ignored for the purpose of replication through
+ this publication. If no column list is specified, all columns of the
+ table are replicated through this publication, including any columns
+ added later. If a column list is specified, it must include the replica
+ identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 62f10bcbd2..88e94a7cda 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -46,12 +46,17 @@
#include "utils/syscache.h"
/*
- * Check if relation can be in given publication and throws appropriate
- * error if not.
+ * Check if relation can be in given publication and that the column
+ * filter is sensible, and throws appropriate error if not.
+ *
+ * targetcols is the bitmapset of column specified as column filter, or NULL if
+ * no column filter was specified.
*/
static void
-check_publication_add_relation(Relation targetrel)
+check_publication_add_relation(Relation targetrel, Bitmapset *columns)
{
+ bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
@@ -82,6 +87,55 @@ check_publication_add_relation(Relation targetrel)
errmsg("cannot add relation \"%s\" to publication",
RelationGetRelationName(targetrel)),
errdetail("This operation is not supported for unlogged tables.")));
+
+ /*
+ * Enforce that the column filter can only leave out columns that aren't
+ * forced to be sent.
+ *
+ * No column can be excluded if REPLICA IDENTITY is FULL (since all the
+ * columns need to be sent regardless); and in other cases, the columns in
+ * the REPLICA IDENTITY cannot be left out.
+ */
+ if (columns != NULL)
+ {
+ Bitmapset *idattrs;
+ int x;
+
+ /*
+ * Even if the user listed all columns in the column list, we cannot
+ * allow a column list to be specified when REPLICA IDENTITY is FULL;
+ * that would cause problems if a new column is added later, because
+ * that could would have to be included (because of being part of the
+ * replica identity) but it's technically not allowed (because of not
+ * being in the publication's column list yet). So reject this case
+ * altogether.
+ */
+ if (replidentfull)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter on relations with REPLICA IDENTITY FULL."));
+
+ idattrs = RelationGetIndexAttrBitmap(targetrel,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+ /*
+ * We have to test membership the hard way, because the values returned
+ * by RelationGetIndexAttrBitmap are offset.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ if (!bms_is_member(x + FirstLowInvalidHeapAttributeNumber, columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
+ }
+
+ bms_free(idattrs);
+ }
}
/*
@@ -289,9 +343,13 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
Oid relid = RelationGetRelid(targetrel->relation);
Oid prrelid;
Publication *pub = GetPublication(pubid);
+ Bitmapset *attmap = NULL;
+ AttrNumber *attarray;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
+ ListCell *lc;
rel = table_open(PublicationRelRelationId, RowExclusiveLock);
@@ -305,6 +363,17 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
{
table_close(rel, RowExclusiveLock);
+ /* FIXME need to handle the case of different column list */
+
+ /*
+ * XXX So what's the right behavior for ADD TABLE with different column
+ * list? I'd say we should allow that, and that it should be mostly the
+ * same thing as adding/removing columns to the list incrementally, i.e.
+ * we should replace the column lists. We could also prohibit, but that
+ * seems like a really annoying limitation, forcing people to remove/add
+ * the relation.
+ */
+
if (if_not_exists)
return InvalidObjectAddress;
@@ -314,7 +383,45 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * should be okay.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(targetrel->columns));
+ foreach(lc, targetrel->columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(relid, colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel->relation)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, attmap))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ attmap = bms_add_member(attmap, attnum);
+ attarray[natts++] = attnum;
+ }
+
+ check_publication_add_relation(targetrel->relation, attmap);
+
+ bms_free(attmap);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -327,6 +434,15 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ if (targetrel->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -334,8 +450,16 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
CatalogTupleInsert(rel, tup);
heap_freetuple(tup);
+ /* Register dependencies as needed */
ObjectAddressSet(myself, PublicationRelRelationId, prrelid);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ pfree(attarray);
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -470,6 +594,74 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all column-partial publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull)
+ continue;
+
+ pubs = lappend_oid(pubs,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 404bb5d0c8..cd2c6a0f70 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -561,7 +561,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
pubrel = palloc(sizeof(PublicationRelInfo));
pubrel->relation = oldrel;
-
+ /* This is not needed to delete a table */
+ pubrel->columns = NIL;
delrels = lappend(delrels, pubrel);
}
}
@@ -932,6 +933,8 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
@@ -965,8 +968,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
+
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
@@ -1074,6 +1080,14 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ /* XXX Shouldn't this be prevented by the grammar, ideally? Can it actually
+ * happen? It does not seem to be tested in the regression tests. */
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list may not be specified for relation \"%s\" in ALTER PUBLICATION ... SET/DROP command",
+ RelationGetRelationName(pubrel->relation)));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 45e59e3d5c..a9051eb5e7 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -40,8 +40,8 @@
#include "catalog/pg_inherits.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
-#include "catalog/pg_tablespace.h"
#include "catalog/pg_statistic_ext.h"
+#include "catalog/pg_tablespace.h"
#include "catalog/pg_trigger.h"
#include "catalog/pg_type.h"
#include "catalog/storage.h"
@@ -8347,6 +8347,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
bool missing_ok, LOCKMODE lockmode,
ObjectAddresses *addrs)
{
+ Oid relid = RelationGetRelid(rel);
HeapTuple tuple;
Form_pg_attribute targetatt;
AttrNumber attnum;
@@ -8366,7 +8367,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/*
* get the number of the attribute
*/
- tuple = SearchSysCacheAttName(RelationGetRelid(rel), colName);
+ tuple = SearchSysCacheAttName(relid, colName);
if (!HeapTupleIsValid(tuple))
{
if (!missing_ok)
@@ -8420,13 +8421,42 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * Also, if the column is used in the column list of a publication,
+ * disallow the drop if the DROP is RESTRICT. We don't do anything if the
+ * DROP is CASCADE, which means that the dependency mechanism will remove
+ * the relation from the publication.
+ */
+ if (behavior == DROP_RESTRICT)
+ {
+ List *pubs;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(relid);
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(relid, pubid);
+
+ if (list_member_oid(published_cols, attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot drop column \"%s\" because it is part of publication \"%s\"",
+ colName, get_publication_name(pubid, false)),
+ errhint("Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication."));
+ }
+ }
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
* use find_all_inheritors to do it in one pass.
*/
children =
- find_inheritance_children(RelationGetRelid(rel), lockmode);
+ find_inheritance_children(relid, lockmode);
if (children)
{
@@ -8514,7 +8544,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Add object to delete */
object.classId = RelationRelationId;
- object.objectId = RelationGetRelid(rel);
+ object.objectId = relid;
object.objectSubId = attnum;
add_exact_object_address(&object, addrs);
@@ -15603,6 +15633,11 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
Oid indexOid;
Relation indexRel;
int key;
+ List *pubs;
+ Bitmapset *indexed_cols = NULL;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(RelationGetRelid(rel));
if (stmt->identity_type == REPLICA_IDENTITY_DEFAULT)
{
@@ -15611,11 +15646,16 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
}
else if (stmt->identity_type == REPLICA_IDENTITY_FULL)
{
+ if (pubs != NIL)
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set REPLICA IDENTITY FULL when column-partial publications exist"));
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
else if (stmt->identity_type == REPLICA_IDENTITY_NOTHING)
{
+ /* XXX not sure what's the right check for publications here */
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
@@ -15626,7 +15666,6 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
else
elog(ERROR, "unexpected identity type %u", stmt->identity_type);
-
/* Check that the index exists */
indexOid = get_relname_relid(stmt->name, rel->rd_rel->relnamespace);
if (!OidIsValid(indexOid))
@@ -15701,6 +15740,38 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
errmsg("index \"%s\" cannot be used as replica identity because column \"%s\" is nullable",
RelationGetRelationName(indexRel),
NameStr(attr->attname))));
+
+ /*
+ * Collect columns used, in case we have any publications that we need
+ * to vet. System attributes are disallowed so no need to subtract
+ * FirstLowInvalidHeapAttributeNumber.
+ */
+ indexed_cols = bms_add_member(indexed_cols, attno);
+ }
+
+ /*
+ * Check column-partial publications. All publications have to include all
+ * key columns of the new index.
+ */
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(RelationGetRelid(rel), pubid);
+
+ for (key = 0; key < IndexRelationGetNumberOfKeyAttributes(indexRel); key++)
+ {
+ int16 attno = indexRel->rd_index->indkey.values[key];
+
+ if (!list_member_oid(published_cols, attno))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("index \"%s\" cannot be used because publication \"%s\" does not include all indexed columns",
+ RelationGetRelationName(indexRel),
+ get_publication_name(pubid, false)));
+ }
}
/* This index is suitable for use as a replica identity. Mark it. */
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index df0b747883..0ff4c1ceac 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4833,6 +4833,7 @@ _copyPublicationTable(const PublicationTable *from)
PublicationTable *newnode = makeNode(PublicationTable);
COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index cb7ddd463c..d786a688ac 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2312,6 +2312,7 @@ static bool
_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 3d4dd43e47..4dad6fedfb 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9742,12 +9742,13 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr
+ TABLE relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
+ $$->pubtable->columns = $3;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9762,28 +9763,38 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLE_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId
+ | ColId opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- $$->name = $1;
+ if ($2 != NULL)
+ {
+ $$->pubtable = makeNode(PublicationTable);
+ $$->pubtable->relation = makeRangeVar(NULL, $1, @1);
+ $$->pubtable->columns = $2;
+ $$->name = NULL;
+ }
+ else
+ $$->name = $1;
$$->location = @1;
}
- | ColId indirection
+ | ColId indirection opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
+ $$->pubtable->columns = $3;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr
+ | extended_relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
+ $$->pubtable->columns = $2;
}
| CURRENT_SCHEMA
{
@@ -17435,8 +17446,9 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
{
/* convert it to PublicationTable */
PublicationTable *pubtable = makeNode(PublicationTable);
- pubtable->relation = makeRangeVar(NULL, pubobj->name,
- pubobj->location);
+
+ pubtable->relation =
+ makeRangeVar(NULL, pubobj->name, pubobj->location);
pubobj->pubtable = pubtable;
pubobj->name = NULL;
}
@@ -17444,6 +17456,16 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
else if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLE_IN_SCHEMA ||
pubobj->pubobjtype == PUBLICATIONOBJ_TABLE_IN_CUR_SCHEMA)
{
+ /*
+ * This can happen if a column list is specified in a continuation
+ * for a schema entry; reject it.
+ */
+ if (pubobj->pubtable)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schemas"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 9f5bf4b639..3428984130 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,9 +29,11 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary,
+ Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +400,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +412,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -442,7 +444,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary,
+ Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -463,11 +466,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -536,7 +539,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -651,7 +654,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -673,7 +677,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -749,7 +753,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple,
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -761,7 +766,13 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ /* Don't count attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
continue;
nliveatts++;
}
@@ -783,6 +794,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
if (att->attisdropped || att->attgenerated)
continue;
+ /* Ignore attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +919,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -914,20 +929,24 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
desc = RelationGetDescr(rel);
- /* send number of live attributes */
- for (i = 0; i < desc->natts; i++)
- {
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
- continue;
- nliveatts++;
- }
- pq_sendint16(out, nliveatts);
-
/* fetch bitmap of REPLICATION IDENTITY attributes */
replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
if (!replidentfull)
idattrs = RelationGetIdentityKeyBitmap(rel);
+ /* send number of live attributes */
+ for (i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+ nliveatts++;
+ }
+ pq_sendint16(out, nliveatts);
+
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -936,7 +955,8 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
-
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index f07983a43c..1303e85851 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -111,6 +111,7 @@
#include "replication/origin.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -697,17 +698,20 @@ fetch_remote_table_info(char *nspname, char *relname,
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
bool isnull;
int natt;
+ ListCell *lc;
+ bool am_partition = false;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -737,14 +741,18 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetChar(slot_getattr(slot, 4, &isnull));
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -772,16 +780,92 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * In server versions 15 and higher, obtain the applicable column filter,
+ * if any.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData publications;
+ bool first = true;
+
+ initStringInfo(&publications);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&publications, ", ");
+ appendStringInfoString(&publications, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ " SELECT pg_catalog.unnest(prattrs)\n"
+ " FROM pg_catalog.pg_publication p JOIN\n"
+ " pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n"
+ " WHERE p.pubname IN (%s) AND\n",
+ publications.data);
+ if (!am_partition)
+ appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "prrelid IN (SELECT relid\n"
+ " FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch attribute info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ if (isnull)
+ continue;
+ included_cols = bms_add_member(included_cols, attnum);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+ pfree(publications.data);
+ walrcv_clear_result(pubres);
+ }
+
+ /*
+ * Store the column names only if they are contained in column filter
+ * LogicalRepRelation will only contain attributes corresponding to those
+ * specficied in column filters.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ continue;
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
+ Assert(!isnull);
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -791,12 +875,13 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecClearTuple(slot);
}
+
ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(res);
+ pfree(cmd.data);
lrel->natts = natt;
- walrcv_clear_result(res);
- pfree(cmd.data);
}
/*
@@ -829,8 +914,17 @@ copy_table(Relation rel)
/* Start copy on the publisher. */
initStringInfo(&cmd);
if (lrel.relkind == RELKIND_RELATION)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 6f6a203dea..34df5d4956 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,16 +15,19 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -81,7 +84,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -130,6 +134,13 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+
+ /*
+ * Set of columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this list are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -570,11 +581,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
}
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -587,7 +598,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -610,13 +622,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip if attribute is not present in column filter. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -693,7 +709,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -722,7 +738,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -1122,6 +1138,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
bool found;
+ Oid ancestor_id;
MemoryContext oldctx;
Assert(RelationSyncCache != NULL);
@@ -1142,6 +1159,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->map = NULL; /* will be set by maybe_send_schema() if
* needed */
}
@@ -1182,6 +1200,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
if (pub->alltables)
{
@@ -1192,8 +1211,6 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (!publish)
{
- bool ancestor_published = false;
-
/*
* For a partition, check if any of the ancestors are
* published. If so, note down the topmost ancestor that is
@@ -1219,6 +1236,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
pub->oid))
{
ancestor_published = true;
+ ancestor_id = ancestor;
if (pub->pubviaroot)
publish_as_relid = ancestor;
}
@@ -1239,15 +1257,47 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ Oid relid;
+ HeapTuple pub_rel_tuple;
+
+ relid = ancestor_published ? ancestor_id : publish_as_relid;
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ Datum pub_rel_cols;
+ bool isnull;
+
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP,
+ pub_rel_tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(pub_rel_cols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
+ entry->columns = bms_add_member(entry->columns,
+ elems[i]);
+ MemoryContextSwitchTo(oldctx);
+ }
+ ReleaseSysCache(pub_rel_tuple);
+ }
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
}
-
- if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
- break;
}
list_free(pubids);
@@ -1343,6 +1393,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
if (entry->map)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b52f3ccda2..d98b1b50c4 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4034,6 +4034,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_oid;
int i_prpubid;
int i_prrelid;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4045,8 +4046,13 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
/* Collect all publication membership info. */
appendPQExpBufferStr(query,
- "SELECT tableoid, oid, prpubid, prrelid "
- "FROM pg_catalog.pg_publication_rel");
+ "SELECT tableoid, oid, prpubid, prrelid");
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBufferStr(query, ", prattrs");
+ else
+ appendPQExpBufferStr(query, ", NULL as prattrs");
+ appendPQExpBufferStr(query,
+ " FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
ntups = PQntuples(res);
@@ -4055,6 +4061,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_oid = PQfnumber(res, "oid");
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4096,6 +4103,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
pubrinfo[j].publication = pubinfo;
pubrinfo[j].pubtable = tbinfo;
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[i].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4160,10 +4189,12 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s;\n",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+ appendPQExpBufferStr(query, ";\n");
/*
* There is no point in creating a drop query as the drop is done by table
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index f011ace8a8..3f7500accc 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -630,6 +630,7 @@ typedef struct _PublicationRelInfo
DumpableObject dobj;
PublicationInfo *publication;
TableInfo *pubtable;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index c28788e84f..b9d0ebf762 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5815,7 +5815,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5832,10 +5832,14 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (!as_schema) /* as table */
+ {
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
- else
+ if (!PQgetisnull(res, i, 2))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 2));
+ }
+ else /* as schema */
printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
@@ -5963,8 +5967,20 @@ describePublications(const char *pattern)
{
/* Get the tables for the specified publication */
printfPQExpBuffer(&buf,
- "SELECT n.nspname, c.relname\n"
- "FROM pg_catalog.pg_class c,\n"
+ "SELECT n.nspname, c.relname, \n");
+ if (pset.sversion >= 150000)
+ appendPQExpBufferStr(&buf,
+ " CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string"
+ "(ARRAY(SELECT attname\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::int[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END AS columns");
+ else
+ appendPQExpBufferStr(&buf, "NULL as columns");
+ appendPQExpBuffer(&buf,
+ "\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
" pg_catalog.pg_publication_rel pr\n"
"WHERE c.relnamespace = n.oid\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index cf30239f6d..25c7c08040 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1648,6 +1648,8 @@ psql_completion(const char *text, int start, int end)
/* ALTER PUBLICATION <name> ADD */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
+ else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD", "TABLE"))
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL);
/* ALTER PUBLICATION <name> DROP */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "DROP"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 902f2f2f0d..500991e696 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -86,6 +86,7 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -109,6 +110,8 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern List *GetPublicationSchemas(Oid pubid);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index b5d5504cbb..7ad285faae 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN
+ int2vector prattrs; /* Variable length field starts here */
+#endif
} FormData_pg_publication_rel;
/* ----------------
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 4c5a8a39bf..02b547d044 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3642,6 +3642,7 @@ typedef struct PublicationTable
{
NodeTag type;
RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 83741dcf42..7a5cb9871d 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -207,11 +207,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -228,7 +228,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 5ac2d666a2..ae99b99cc6 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -165,7 +165,22 @@ Publications:
regress_publication_user | t | t | t | f | f | f
(1 row)
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+ERROR: cannot drop column "c" because it is part of publication "testpub_fortable"
+HINT: Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication.
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl6"
+DETAIL: Cannot have column filter on relations with REPLICA IDENTITY FULL.
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
CREATE TABLE testpub_tbl3a (b text) INHERITS (testpub_tbl3);
@@ -669,6 +684,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schemas
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 56dd358554..b422e3e374 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -89,7 +89,17 @@ SELECT pubname, puballtables FROM pg_publication WHERE pubname = 'testpub_forall
\d+ testpub_tbl2
\dRp+ testpub_foralltables
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
@@ -362,6 +372,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/021_column_filter.pl b/src/test/subscription/t/021_column_filter.pl
new file mode 100644
index 0000000000..dfae6d8eac
--- /dev/null
+++ b/src/test/subscription/t/021_column_filter.pl
@@ -0,0 +1,164 @@
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test TRUNCATE
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 9;
+
+# setup
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+ INSERT INTO tab2 VALUES (2, 'foo', 2);");
+# Test with weird column names
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, B varchar, \"c'\" int)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+#Test replication with multi-level partition
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, \"c'\" int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+#Test create publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub1 FOR TABLE tab1(a, \"B\"), tab3(\"a'\",\"c'\"), test_part(a,b)");
+
+my $result = $node_publisher->safe_psql('postgres',
+ "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
+is($result, qq(tab1|1 2
+tab3|1 3
+test_part|1 2), 'publication relation updated');
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+);
+#Initial sync
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab1 VALUES (1,2,3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 VALUES (1,2,3)");
+#Test for replication of partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (1,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (2,'bcd', '2021-07-03 11:12:13')");
+#Test for replication of multi-level partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (4,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (5,'bcd', '2021-07-03 11:12:13')");
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'insert on column tab1.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab3");
+is($result, qq(1|3), 'insert on column tab3.b is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part");
+is($result, qq(1|abc\n2|bcd\n4|abc\n5|bcd), 'insert on all columns is replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab1 SET c = 5 where a = 1");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'update on column tab1.c is not replicated');
+
+# Verify user-defined types
+$node_publisher->safe_psql('postgres',
+ qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+ ALTER PUBLICATION pub1 ADD TABLE test_tab4 (a, b, d);
+ });
+$node_subscriber->safe_psql('postgres',
+ qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, d text);
+ });
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab4 VALUES (1, 'red', 3, 'oh my');");
+
+#Test alter publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION"
+);
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 VALUES (1,'abc',3)");
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET c = 5 where a = 2");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 WHERE a = 1");
+is($result, qq(1|abc), 'insert on column tab2.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 WHERE a = 2");
+is($result, qq(2|foo), 'update on column tab2.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_tab4");
+is($result, qq(1|red|oh my), 'insert on table with user-defined type');
+
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d)");
+$node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2, pub3");
+$node_publisher->wait_for_catchup('sub2');
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (1, 11, 111, 1111)");
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (2, 22, 222, 2222)");
+$node_publisher->wait_for_catchup('sub2');
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5;"),
+ qq(1|11|1111
+2|22|2222),
+ 'overlapping publications with overlapping column lists');
On 2021-Dec-28, Alvaro Herrera wrote:
There are still some XXX comments. The one that bothers me most is the
lack of an implementation that allows changing the column list in a
publication without having to remove the table from the publication
first.
OK, I made some progress on this front; I added new forms of ALTER
PUBLICATION to support it:
ALTER PUBLICATION pub1 ALTER TABLE tbl SET COLUMNS (a, b, c);
ALTER PUBLICATION pub1 ALTER TABLE tbl SET COLUMNS ALL;
(not wedded to this syntax; other suggestions welcome)
In order to implement it I changed the haphazardly chosen use of
DEFELEM actions to a new enum. I also noticed that the division of
labor between pg_publication.c and publicationcmds.c is quite broken
(code to translate column names to numbers is in the former, should be
in the latter; some code that deals with pg_publication tuples is in the
latter, should be in the former, such as CreatePublication,
AlterPublicationOptions).
This new stuff is not yet finished. For example I didn't refactor
handling of REPLICA IDENTITY, so the new command does not correctly
check everything, such as the REPLICA IDENTITY FULL stuff. Also, no
tests have been added yet. In manual tests it seems to behave as
expected.
I noticed that prattrs is inserted in user-specified order instead of
catalog order, which is innocuous but quite weird.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"No renuncies a nada. No te aferres a nada."
Attachments:
column-filtering-12.patchtext/x-diff; charset=utf-8Download
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 34a7034282..5bc2e7a591 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -6877,7 +6877,9 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column (except
+ generated columns and other columns that don't appear in the column
+ filter list, for tables that have one):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index bb4ef5e5e2..4951343f6f 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -25,12 +25,13 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD <replace
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ALTER TABLE <replaceable class="parameter">publication_object</replaceable> SET COLUMNS { ( <replaceable class="parameter">name</replaceable> [, ...] ) | ALL
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -62,6 +63,11 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
command retain their previous settings.
</para>
+ <para>
+ The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows to change
+ the set of columns that are included in the publication.
+ </para>
+
<para>
The remaining variants change the owner and the name of the publication.
</para>
@@ -110,6 +116,8 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
</para>
</listitem>
</varlistentry>
@@ -164,9 +172,15 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
</para>
<para>
- Add some tables to the publication:
+ Add tables to the publication:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
+<programlisting>
+ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS (user_id, firstname, lastname);
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index d805e8e77a..73a23cbb02 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -78,6 +78,15 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
publication, so they are never explicitly added to the publication.
</para>
+ <para>
+ When a column list is specified, only the listed columns are replicated;
+ any other columns are ignored for the purpose of replication through
+ this publication. If no column list is specified, all columns of the
+ table are replicated through this publication, including any columns
+ added later. If a column list is specified, it must include the replica
+ identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 62f10bcbd2..322bfc2a82 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,13 +45,23 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+
+static void check_publication_columns(Relation targetrel, Bitmapset *columns);
+static AttrNumber *publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, Bitmapset **attset);
+
/*
- * Check if relation can be in given publication and throws appropriate
- * error if not.
+ * Check if relation can be in given publication and that the column
+ * filter is sensible, and throws appropriate error if not.
+ *
+ * targetcols is the bitmapset of column specified as column filter, or NULL if
+ * no column filter was specified.
*/
static void
-check_publication_add_relation(Relation targetrel)
+check_publication_add_relation(Relation targetrel, Bitmapset *columns)
{
+ bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
@@ -82,6 +92,62 @@ check_publication_add_relation(Relation targetrel)
errmsg("cannot add relation \"%s\" to publication",
RelationGetRelationName(targetrel)),
errdetail("This operation is not supported for unlogged tables.")));
+
+ /* Make sure the column list checks out */
+ if (columns != NULL)
+ {
+ /*
+ * Even if the user listed all columns in the column list, we cannot
+ * allow a column list to be specified when REPLICA IDENTITY is FULL;
+ * that would cause problems if a new column is added later, because
+ * that could would have to be included (because of being part of the
+ * replica identity) but it's technically not allowed (because of not
+ * being in the publication's column list yet). So reject this case
+ * altogether.
+ */
+ if (replidentfull)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter on relations with REPLICA IDENTITY FULL."));
+
+ check_publication_columns(targetrel, columns);
+ }
+}
+
+/*
+ * Enforce that the column filter can only leave out columns that aren't
+ * forced to be sent.
+ *
+ * No column can be excluded if REPLICA IDENTITY is FULL (since all the
+ * columns need to be sent regardless); and in other cases, the columns in
+ * the REPLICA IDENTITY cannot be left out.
+ */
+static void
+check_publication_columns(Relation targetrel, Bitmapset *columns)
+{
+ Bitmapset *idattrs;
+ int x;
+
+ idattrs = RelationGetIndexAttrBitmap(targetrel,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+ /*
+ * We have to test membership the hard way, because the values returned
+ * by RelationGetIndexAttrBitmap are offset.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ if (!bms_is_member(x + FirstLowInvalidHeapAttributeNumber, columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
+ }
+
+ bms_free(idattrs);
}
/*
@@ -287,8 +353,11 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
Datum values[Natts_pg_publication_rel];
bool nulls[Natts_pg_publication_rel];
Oid relid = RelationGetRelid(targetrel->relation);
- Oid prrelid;
+ Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ Bitmapset *attset = NULL;
+ AttrNumber *attarray;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -314,19 +383,35 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ /* Translate column names to numbers and verify suitability */
+ attarray = publication_translate_columns(targetrel->relation,
+ targetrel->columns,
+ &natts, &attset);
+
+ check_publication_add_relation(targetrel->relation, attset);
+
+ bms_free(attset);
/* Form a tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- prrelid = GetNewOidWithIndex(rel, PublicationRelObjectIndexId,
- Anum_pg_publication_rel_oid);
- values[Anum_pg_publication_rel_oid - 1] = ObjectIdGetDatum(prrelid);
+ pubreloid = GetNewOidWithIndex(rel, PublicationRelObjectIndexId,
+ Anum_pg_publication_rel_oid);
+ values[Anum_pg_publication_rel_oid - 1] = ObjectIdGetDatum(pubreloid);
values[Anum_pg_publication_rel_prpubid - 1] =
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ if (targetrel->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -334,8 +419,16 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
CatalogTupleInsert(rel, tup);
heap_freetuple(tup);
- ObjectAddressSet(myself, PublicationRelRelationId, prrelid);
+ /* Register dependencies as needed */
+ ObjectAddressSet(myself, PublicationRelRelationId, pubreloid);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ pfree(attarray);
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -363,6 +456,132 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
return myself;
}
+void
+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns)
+{
+ Bitmapset *attset;
+ AttrNumber *attarray;
+ HeapTuple copytup;
+ int natts;
+ bool nulls[Natts_pg_publication_rel];
+ bool replaces[Natts_pg_publication_rel];
+ Datum values[Natts_pg_publication_rel];
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ replaces[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ deleteDependencyRecordsForClass(PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid,
+ RelationRelationId,
+ DEPENDENCY_AUTO);
+
+ if (columns == NULL)
+ {
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+ }
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ int2vector *prattrs;
+
+ attarray = publication_translate_columns(targetrel, columns,
+ &natts, &attset);
+
+ /* make sure the column list checks out */
+ /* XXX missing to check for the REPLICA IDENTITY FULL case */
+ /* XXX this should occur at caller in publicationcmds.c, not here */
+ check_publication_columns(targetrel, attset);
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+
+ /* Add dependencies on the new list of columns */
+ ObjectAddressSet(myself, PublicationRelRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid);
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId,
+ RelationGetRelid(targetrel), attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ }
+
+ copytup = heap_modify_tuple(pubreltup, RelationGetDescr(pubrel),
+ values, nulls, replaces);
+
+ CatalogTupleUpdate(pubrel, &pubreltup->t_self, copytup);
+
+ heap_freetuple(copytup);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list. Other checks are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ */
+static AttrNumber *
+publication_translate_columns(Relation targetrel, List *columns, int *natts,
+ Bitmapset **attset)
+{
+ AttrNumber *attarray;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ *
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /*
+ * XXX qsort the array here, or maybe build just the bitmapset above and
+ * then scan that in order to produce the array? Do we care about the
+ * array being unsorted?
+ */
+
+ *natts = n;
+ *attset = set;
+ return attarray;
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -470,6 +689,74 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all column-partial publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull)
+ continue;
+
+ pubs = lappend_oid(pubs,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 404bb5d0c8..aefae8b3c4 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -48,7 +48,7 @@
#include "utils/syscache.h"
#include "utils/varlena.h"
-static List *OpenReliIdList(List *relids);
+static List *OpenRelIdList(List *relids);
static List *OpenTableList(List *tables);
static void CloseTableList(List *rels);
static void LockSchemaList(List *schemalist);
@@ -376,6 +376,46 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
return myself;
}
+/*
+ * Change the column list of a relation in a publication
+ */
+static void
+PublicationSetColumns(AlterPublicationStmt *stmt,
+ Form_pg_publication pubform, PublicationTable *table)
+{
+ Relation rel,
+ urel;
+ HeapTuple tup;
+ ObjectAddress obj,
+ secondary;
+
+ rel = table_open(PublicationRelRelationId, RowExclusiveLock);
+ urel = table_openrv(table->relation, ShareUpdateExclusiveLock);
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(RelationGetRelid(urel)),
+ ObjectIdGetDatum(pubform->oid));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errmsg("relation \"%s\" is not already in publication \"%s\"",
+ table->relation->relname,
+ NameStr(pubform->pubname)));
+
+ publication_set_table_columns(rel, tup, urel, table->columns);
+
+ ObjectAddressSet(obj, PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->oid);
+ ObjectAddressSet(secondary, RelationRelationId, RelationGetRelid(urel));
+ EventTriggerCollectSimpleCommand(obj, secondary, (Node *) stmt);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+ table_close(urel, NoLock);
+
+ InvokeObjectPostAlterHook(PublicationRelationId, pubform->oid, 0);
+}
+
/*
* Change options of a publication.
*/
@@ -499,15 +539,16 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
Oid pubid = pubform->oid;
/*
- * It is quite possible that for the SET case user has not specified any
- * tables in which case we need to remove all the existing tables.
+ * Nothing to do if no objects, except in SET: for that it is quite
+ * possible that user has not specified any schemas in which case we need
+ * to remove all the existing schemas.
*/
- if (!tables && stmt->action != DEFELEM_SET)
+ if (!tables && stmt->action != AP_SetObjects)
return;
rels = OpenTableList(tables);
- if (stmt->action == DEFELEM_ADD)
+ if (stmt->action == AP_AddObjects)
{
List *schemas = NIL;
@@ -520,9 +561,17 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
PUBLICATIONOBJ_TABLE);
PublicationAddTables(pubid, rels, false, stmt);
}
- else if (stmt->action == DEFELEM_DROP)
+ else if (stmt->action == AP_DropObjects)
PublicationDropTables(pubid, rels, false);
- else /* DEFELEM_SET */
+ else if (stmt->action == AP_SetColumns)
+ {
+ Assert(schemaidlist == NIL);
+ Assert(list_length(tables) == 1);
+
+ PublicationSetColumns(stmt, pubform,
+ linitial_node(PublicationTable, tables));
+ }
+ else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
PUBLICATION_PART_ROOT);
@@ -561,7 +610,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
pubrel = palloc(sizeof(PublicationRelInfo));
pubrel->relation = oldrel;
-
+ /* This is not needed to delete a table */
+ pubrel->columns = NIL;
delrels = lappend(delrels, pubrel);
}
}
@@ -593,10 +643,11 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
Form_pg_publication pubform = (Form_pg_publication) GETSTRUCT(tup);
/*
- * It is quite possible that for the SET case user has not specified any
- * schemas in which case we need to remove all the existing schemas.
+ * Nothing to do if no objects, except in SET: for that it is quite
+ * possible that user has not specified any schemas in which case we need
+ * to remove all the existing schemas.
*/
- if (!schemaidlist && stmt->action != DEFELEM_SET)
+ if (!schemaidlist && stmt->action != AP_SetObjects)
return;
/*
@@ -604,13 +655,13 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
* concurrent schema deletion.
*/
LockSchemaList(schemaidlist);
- if (stmt->action == DEFELEM_ADD)
+ if (stmt->action == AP_AddObjects)
{
List *rels;
List *reloids;
reloids = GetPublicationRelations(pubform->oid, PUBLICATION_PART_ROOT);
- rels = OpenReliIdList(reloids);
+ rels = OpenRelIdList(reloids);
CheckObjSchemaNotAlreadyInPublication(rels, schemaidlist,
PUBLICATIONOBJ_TABLE_IN_SCHEMA);
@@ -618,9 +669,9 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
CloseTableList(rels);
PublicationAddSchemas(pubform->oid, schemaidlist, false, stmt);
}
- else if (stmt->action == DEFELEM_DROP)
+ else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* DEFELEM_SET */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -643,6 +694,10 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
*/
PublicationAddSchemas(pubform->oid, schemaidlist, true, stmt);
}
+ else
+ {
+ /* Nothing to do for AP_SetColumns */
+ }
}
/*
@@ -655,7 +710,7 @@ CheckAlterPublication(AlterPublicationStmt *stmt, HeapTuple tup,
{
Form_pg_publication pubform = (Form_pg_publication) GETSTRUCT(tup);
- if ((stmt->action == DEFELEM_ADD || stmt->action == DEFELEM_SET) &&
+ if ((stmt->action == AP_AddObjects || stmt->action == AP_SetObjects) &&
schemaidlist && !superuser())
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
@@ -868,7 +923,7 @@ RemovePublicationSchemaById(Oid psoid)
* add them to a publication.
*/
static List *
-OpenReliIdList(List *relids)
+OpenRelIdList(List *relids)
{
ListCell *lc;
List *rels = NIL;
@@ -932,6 +987,8 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
@@ -965,8 +1022,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
+
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
@@ -1074,6 +1134,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 45e59e3d5c..a9051eb5e7 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -40,8 +40,8 @@
#include "catalog/pg_inherits.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
-#include "catalog/pg_tablespace.h"
#include "catalog/pg_statistic_ext.h"
+#include "catalog/pg_tablespace.h"
#include "catalog/pg_trigger.h"
#include "catalog/pg_type.h"
#include "catalog/storage.h"
@@ -8347,6 +8347,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
bool missing_ok, LOCKMODE lockmode,
ObjectAddresses *addrs)
{
+ Oid relid = RelationGetRelid(rel);
HeapTuple tuple;
Form_pg_attribute targetatt;
AttrNumber attnum;
@@ -8366,7 +8367,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/*
* get the number of the attribute
*/
- tuple = SearchSysCacheAttName(RelationGetRelid(rel), colName);
+ tuple = SearchSysCacheAttName(relid, colName);
if (!HeapTupleIsValid(tuple))
{
if (!missing_ok)
@@ -8420,13 +8421,42 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * Also, if the column is used in the column list of a publication,
+ * disallow the drop if the DROP is RESTRICT. We don't do anything if the
+ * DROP is CASCADE, which means that the dependency mechanism will remove
+ * the relation from the publication.
+ */
+ if (behavior == DROP_RESTRICT)
+ {
+ List *pubs;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(relid);
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(relid, pubid);
+
+ if (list_member_oid(published_cols, attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot drop column \"%s\" because it is part of publication \"%s\"",
+ colName, get_publication_name(pubid, false)),
+ errhint("Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication."));
+ }
+ }
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
* use find_all_inheritors to do it in one pass.
*/
children =
- find_inheritance_children(RelationGetRelid(rel), lockmode);
+ find_inheritance_children(relid, lockmode);
if (children)
{
@@ -8514,7 +8544,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Add object to delete */
object.classId = RelationRelationId;
- object.objectId = RelationGetRelid(rel);
+ object.objectId = relid;
object.objectSubId = attnum;
add_exact_object_address(&object, addrs);
@@ -15603,6 +15633,11 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
Oid indexOid;
Relation indexRel;
int key;
+ List *pubs;
+ Bitmapset *indexed_cols = NULL;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(RelationGetRelid(rel));
if (stmt->identity_type == REPLICA_IDENTITY_DEFAULT)
{
@@ -15611,11 +15646,16 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
}
else if (stmt->identity_type == REPLICA_IDENTITY_FULL)
{
+ if (pubs != NIL)
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set REPLICA IDENTITY FULL when column-partial publications exist"));
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
else if (stmt->identity_type == REPLICA_IDENTITY_NOTHING)
{
+ /* XXX not sure what's the right check for publications here */
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
@@ -15626,7 +15666,6 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
else
elog(ERROR, "unexpected identity type %u", stmt->identity_type);
-
/* Check that the index exists */
indexOid = get_relname_relid(stmt->name, rel->rd_rel->relnamespace);
if (!OidIsValid(indexOid))
@@ -15701,6 +15740,38 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
errmsg("index \"%s\" cannot be used as replica identity because column \"%s\" is nullable",
RelationGetRelationName(indexRel),
NameStr(attr->attname))));
+
+ /*
+ * Collect columns used, in case we have any publications that we need
+ * to vet. System attributes are disallowed so no need to subtract
+ * FirstLowInvalidHeapAttributeNumber.
+ */
+ indexed_cols = bms_add_member(indexed_cols, attno);
+ }
+
+ /*
+ * Check column-partial publications. All publications have to include all
+ * key columns of the new index.
+ */
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(RelationGetRelid(rel), pubid);
+
+ for (key = 0; key < IndexRelationGetNumberOfKeyAttributes(indexRel); key++)
+ {
+ int16 attno = indexRel->rd_index->indkey.values[key];
+
+ if (!list_member_oid(published_cols, attno))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("index \"%s\" cannot be used because publication \"%s\" does not include all indexed columns",
+ RelationGetRelationName(indexRel),
+ get_publication_name(pubid, false)));
+ }
}
/* This index is suitable for use as a replica identity. Mark it. */
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index df0b747883..0ff4c1ceac 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4833,6 +4833,7 @@ _copyPublicationTable(const PublicationTable *from)
PublicationTable *newnode = makeNode(PublicationTable);
COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index cb7ddd463c..d786a688ac 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2312,6 +2312,7 @@ static bool
_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 3d4dd43e47..68b1136788 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9742,12 +9742,13 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr
+ TABLE relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
+ $$->pubtable->columns = $3;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9762,28 +9763,38 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLE_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId
+ | ColId opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- $$->name = $1;
+ if ($2 != NULL)
+ {
+ $$->pubtable = makeNode(PublicationTable);
+ $$->pubtable->relation = makeRangeVar(NULL, $1, @1);
+ $$->pubtable->columns = $2;
+ $$->name = NULL;
+ }
+ else
+ $$->name = $1;
$$->location = @1;
}
- | ColId indirection
+ | ColId indirection opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
+ $$->pubtable->columns = $3;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr
+ | extended_relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
+ $$->pubtable->columns = $2;
}
| CURRENT_SCHEMA
{
@@ -9809,6 +9820,9 @@ pub_obj_list: PublicationObjSpec
*
* ALTER PUBLICATION name SET pub_obj [, ...]
*
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
* pub_obj is one of:
*
* TABLE table_name [, ...]
@@ -9830,7 +9844,7 @@ AlterPublicationStmt:
n->pubname = $3;
n->pubobjects = $5;
preprocess_pubobj_list(n->pubobjects, yyscanner);
- n->action = DEFELEM_ADD;
+ n->action = AP_AddObjects;
$$ = (Node *)n;
}
| ALTER PUBLICATION name SET pub_obj_list
@@ -9839,16 +9853,42 @@ AlterPublicationStmt:
n->pubname = $3;
n->pubobjects = $5;
preprocess_pubobj_list(n->pubobjects, yyscanner);
- n->action = DEFELEM_SET;
+ n->action = AP_SetObjects;
$$ = (Node *)n;
}
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS '(' columnList ')'
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = $10;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS ALL
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = NIL;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
| ALTER PUBLICATION name DROP pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
n->pubname = $3;
n->pubobjects = $5;
preprocess_pubobj_list(n->pubobjects, yyscanner);
- n->action = DEFELEM_DROP;
+ n->action = AP_DropObjects;
$$ = (Node *)n;
}
;
@@ -17435,8 +17475,9 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
{
/* convert it to PublicationTable */
PublicationTable *pubtable = makeNode(PublicationTable);
- pubtable->relation = makeRangeVar(NULL, pubobj->name,
- pubobj->location);
+
+ pubtable->relation =
+ makeRangeVar(NULL, pubobj->name, pubobj->location);
pubobj->pubtable = pubtable;
pubobj->name = NULL;
}
@@ -17444,6 +17485,16 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
else if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLE_IN_SCHEMA ||
pubobj->pubobjtype == PUBLICATIONOBJ_TABLE_IN_CUR_SCHEMA)
{
+ /*
+ * This can happen if a column list is specified in a continuation
+ * for a schema entry; reject it.
+ */
+ if (pubobj->pubtable)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schemas"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 9f5bf4b639..3428984130 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,9 +29,11 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary,
+ Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +400,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +412,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -442,7 +444,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary,
+ Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -463,11 +466,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -536,7 +539,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -651,7 +654,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -673,7 +677,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -749,7 +753,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple,
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -761,7 +766,13 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ /* Don't count attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
continue;
nliveatts++;
}
@@ -783,6 +794,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
if (att->attisdropped || att->attgenerated)
continue;
+ /* Ignore attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +919,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -914,20 +929,24 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
desc = RelationGetDescr(rel);
- /* send number of live attributes */
- for (i = 0; i < desc->natts; i++)
- {
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
- continue;
- nliveatts++;
- }
- pq_sendint16(out, nliveatts);
-
/* fetch bitmap of REPLICATION IDENTITY attributes */
replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
if (!replidentfull)
idattrs = RelationGetIdentityKeyBitmap(rel);
+ /* send number of live attributes */
+ for (i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+ nliveatts++;
+ }
+ pq_sendint16(out, nliveatts);
+
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -936,7 +955,8 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
-
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index f07983a43c..1303e85851 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -111,6 +111,7 @@
#include "replication/origin.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -697,17 +698,20 @@ fetch_remote_table_info(char *nspname, char *relname,
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
bool isnull;
int natt;
+ ListCell *lc;
+ bool am_partition = false;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -737,14 +741,18 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetChar(slot_getattr(slot, 4, &isnull));
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -772,16 +780,92 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * In server versions 15 and higher, obtain the applicable column filter,
+ * if any.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData publications;
+ bool first = true;
+
+ initStringInfo(&publications);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&publications, ", ");
+ appendStringInfoString(&publications, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ " SELECT pg_catalog.unnest(prattrs)\n"
+ " FROM pg_catalog.pg_publication p JOIN\n"
+ " pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n"
+ " WHERE p.pubname IN (%s) AND\n",
+ publications.data);
+ if (!am_partition)
+ appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "prrelid IN (SELECT relid\n"
+ " FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch attribute info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ if (isnull)
+ continue;
+ included_cols = bms_add_member(included_cols, attnum);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+ pfree(publications.data);
+ walrcv_clear_result(pubres);
+ }
+
+ /*
+ * Store the column names only if they are contained in column filter
+ * LogicalRepRelation will only contain attributes corresponding to those
+ * specficied in column filters.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ continue;
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
+ Assert(!isnull);
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -791,12 +875,13 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecClearTuple(slot);
}
+
ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(res);
+ pfree(cmd.data);
lrel->natts = natt;
- walrcv_clear_result(res);
- pfree(cmd.data);
}
/*
@@ -829,8 +914,17 @@ copy_table(Relation rel)
/* Start copy on the publisher. */
initStringInfo(&cmd);
if (lrel.relkind == RELKIND_RELATION)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 6f6a203dea..34df5d4956 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,16 +15,19 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -81,7 +84,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -130,6 +134,13 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+
+ /*
+ * Set of columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this list are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -570,11 +581,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
}
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -587,7 +598,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -610,13 +622,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip if attribute is not present in column filter. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -693,7 +709,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -722,7 +738,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -1122,6 +1138,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
bool found;
+ Oid ancestor_id;
MemoryContext oldctx;
Assert(RelationSyncCache != NULL);
@@ -1142,6 +1159,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->map = NULL; /* will be set by maybe_send_schema() if
* needed */
}
@@ -1182,6 +1200,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
if (pub->alltables)
{
@@ -1192,8 +1211,6 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (!publish)
{
- bool ancestor_published = false;
-
/*
* For a partition, check if any of the ancestors are
* published. If so, note down the topmost ancestor that is
@@ -1219,6 +1236,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
pub->oid))
{
ancestor_published = true;
+ ancestor_id = ancestor;
if (pub->pubviaroot)
publish_as_relid = ancestor;
}
@@ -1239,15 +1257,47 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ Oid relid;
+ HeapTuple pub_rel_tuple;
+
+ relid = ancestor_published ? ancestor_id : publish_as_relid;
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ Datum pub_rel_cols;
+ bool isnull;
+
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP,
+ pub_rel_tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(pub_rel_cols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
+ entry->columns = bms_add_member(entry->columns,
+ elems[i]);
+ MemoryContextSwitchTo(oldctx);
+ }
+ ReleaseSysCache(pub_rel_tuple);
+ }
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
}
-
- if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
- break;
}
list_free(pubids);
@@ -1343,6 +1393,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
if (entry->map)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b52f3ccda2..d98b1b50c4 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4034,6 +4034,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_oid;
int i_prpubid;
int i_prrelid;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4045,8 +4046,13 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
/* Collect all publication membership info. */
appendPQExpBufferStr(query,
- "SELECT tableoid, oid, prpubid, prrelid "
- "FROM pg_catalog.pg_publication_rel");
+ "SELECT tableoid, oid, prpubid, prrelid");
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBufferStr(query, ", prattrs");
+ else
+ appendPQExpBufferStr(query, ", NULL as prattrs");
+ appendPQExpBufferStr(query,
+ " FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
ntups = PQntuples(res);
@@ -4055,6 +4061,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_oid = PQfnumber(res, "oid");
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4096,6 +4103,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
pubrinfo[j].publication = pubinfo;
pubrinfo[j].pubtable = tbinfo;
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[i].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4160,10 +4189,12 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s;\n",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+ appendPQExpBufferStr(query, ";\n");
/*
* There is no point in creating a drop query as the drop is done by table
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index f011ace8a8..3f7500accc 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -630,6 +630,7 @@ typedef struct _PublicationRelInfo
DumpableObject dobj;
PublicationInfo *publication;
TableInfo *pubtable;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index c28788e84f..b9d0ebf762 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5815,7 +5815,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5832,10 +5832,14 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (!as_schema) /* as table */
+ {
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
- else
+ if (!PQgetisnull(res, i, 2))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 2));
+ }
+ else /* as schema */
printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
@@ -5963,8 +5967,20 @@ describePublications(const char *pattern)
{
/* Get the tables for the specified publication */
printfPQExpBuffer(&buf,
- "SELECT n.nspname, c.relname\n"
- "FROM pg_catalog.pg_class c,\n"
+ "SELECT n.nspname, c.relname, \n");
+ if (pset.sversion >= 150000)
+ appendPQExpBufferStr(&buf,
+ " CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string"
+ "(ARRAY(SELECT attname\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::int[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END AS columns");
+ else
+ appendPQExpBufferStr(&buf, "NULL as columns");
+ appendPQExpBuffer(&buf,
+ "\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
" pg_catalog.pg_publication_rel pr\n"
"WHERE c.relnamespace = n.oid\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index cf30239f6d..25c7c08040 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1648,6 +1648,8 @@ psql_completion(const char *text, int start, int end)
/* ALTER PUBLICATION <name> ADD */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
+ else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD", "TABLE"))
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL);
/* ALTER PUBLICATION <name> DROP */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "DROP"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 902f2f2f0d..edd4f0c63c 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -86,6 +86,7 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -109,6 +110,8 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern List *GetPublicationSchemas(Oid pubid);
@@ -127,6 +130,8 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *tar
bool if_not_exists);
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern void publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index b5d5504cbb..7ad285faae 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN
+ int2vector prattrs; /* Variable length field starts here */
+#endif
} FormData_pg_publication_rel;
/* ----------------
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 4c5a8a39bf..91ea815e14 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3642,6 +3642,7 @@ typedef struct PublicationTable
{
NodeTag type;
RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
@@ -3674,6 +3675,14 @@ typedef struct CreatePublicationStmt
bool for_all_tables; /* Special publication for all tables in db */
} CreatePublicationStmt;
+typedef enum AlterPublicationAction
+{
+ AP_AddObjects, /* add objects to publication */
+ AP_DropObjects, /* remove objects from publication */
+ AP_SetObjects, /* set list of objects */
+ AP_SetColumns /* change list of columns for a table */
+} AlterPublicationAction;
+
typedef struct AlterPublicationStmt
{
NodeTag type;
@@ -3688,7 +3697,7 @@ typedef struct AlterPublicationStmt
*/
List *pubobjects; /* Optional list of publication objects */
bool for_all_tables; /* Special publication for all tables in db */
- DefElemAction action; /* What action to perform with the
+ AlterPublicationAction action; /* What action to perform with the
* tables/schemas */
} AlterPublicationStmt;
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 83741dcf42..7a5cb9871d 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -207,11 +207,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -228,7 +228,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 5ac2d666a2..9f540e1144 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -165,7 +165,24 @@ Publications:
regress_publication_user | t | t | t | f | f | f
(1 row)
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+ERROR: cannot drop column "c" because it is part of publication "testpub_fortable"
+HINT: Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5 (a);
+ERROR: column list must not be specified in ALTER PUBLICATION ... DROP
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl6"
+DETAIL: Cannot have column filter on relations with REPLICA IDENTITY FULL.
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
CREATE TABLE testpub_tbl3a (b text) INHERITS (testpub_tbl3);
@@ -669,6 +686,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schemas
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 56dd358554..d82b034efd 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -89,7 +89,18 @@ SELECT pubname, puballtables FROM pg_publication WHERE pubname = 'testpub_forall
\d+ testpub_tbl2
\dRp+ testpub_foralltables
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5 (a);
+
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
@@ -362,6 +373,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
On 2021-Dec-29, Alvaro Herrera wrote:
This new stuff is not yet finished. For example I didn't refactor
handling of REPLICA IDENTITY, so the new command does not correctly
check everything, such as the REPLICA IDENTITY FULL stuff. Also, no
tests have been added yet. In manual tests it seems to behave as
expected.
Fixing the lack of check for replica identity full didn't really require
much refactoring, so I did it that way.
I split it with some trivial fixes that can be committed separately
ahead of time. I'm thinking in committing 0001 later today, perhaps
0002 tomorrow. The interesting part is 0003.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
Attachments:
v13-0001-Small-cleanups-for-publicationcmds.c-and-pg_publ.patchtext/x-diff; charset=utf-8Download
From 0453eb6397803ce4dd607fd3a17a12d573eb2c90 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Thu, 30 Dec 2021 16:48:09 -0300
Subject: [PATCH v13 1/3] Small cleanups for publicationcmds.c and
pg_publication.c
This fixes a typo in a local function name, completes an existing comment,
and renames an unhappily named local variable.
---
src/backend/catalog/pg_publication.c | 11 ++++++-----
src/backend/commands/publicationcmds.c | 16 +++++++++-------
src/backend/commands/tablecmds.c | 3 +--
src/backend/parser/gram.y | 5 +++--
4 files changed, 19 insertions(+), 16 deletions(-)
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 62f10bcbd2..b307bc2ed5 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -287,7 +287,7 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
Datum values[Natts_pg_publication_rel];
bool nulls[Natts_pg_publication_rel];
Oid relid = RelationGetRelid(targetrel->relation);
- Oid prrelid;
+ Oid pubreloid;
Publication *pub = GetPublication(pubid);
ObjectAddress myself,
referenced;
@@ -320,9 +320,9 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- prrelid = GetNewOidWithIndex(rel, PublicationRelObjectIndexId,
- Anum_pg_publication_rel_oid);
- values[Anum_pg_publication_rel_oid - 1] = ObjectIdGetDatum(prrelid);
+ pubreloid = GetNewOidWithIndex(rel, PublicationRelObjectIndexId,
+ Anum_pg_publication_rel_oid);
+ values[Anum_pg_publication_rel_oid - 1] = ObjectIdGetDatum(pubreloid);
values[Anum_pg_publication_rel_prpubid - 1] =
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
@@ -334,7 +334,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
CatalogTupleInsert(rel, tup);
heap_freetuple(tup);
- ObjectAddressSet(myself, PublicationRelRelationId, prrelid);
+ /* Register dependencies as needed */
+ ObjectAddressSet(myself, PublicationRelRelationId, pubreloid);
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 404bb5d0c8..3466c57dc0 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -48,7 +48,7 @@
#include "utils/syscache.h"
#include "utils/varlena.h"
-static List *OpenReliIdList(List *relids);
+static List *OpenRelIdList(List *relids);
static List *OpenTableList(List *tables);
static void CloseTableList(List *rels);
static void LockSchemaList(List *schemalist);
@@ -499,8 +499,9 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
Oid pubid = pubform->oid;
/*
- * It is quite possible that for the SET case user has not specified any
- * tables in which case we need to remove all the existing tables.
+ * Nothing to do if no objects, except in SET: for that it is quite
+ * possible that user has not specified any schemas in which case we need
+ * to remove all the existing schemas.
*/
if (!tables && stmt->action != DEFELEM_SET)
return;
@@ -593,8 +594,9 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
Form_pg_publication pubform = (Form_pg_publication) GETSTRUCT(tup);
/*
- * It is quite possible that for the SET case user has not specified any
- * schemas in which case we need to remove all the existing schemas.
+ * Nothing to do if no objects, except in SET: for that it is quite
+ * possible that user has not specified any schemas in which case we need
+ * to remove all the existing schemas.
*/
if (!schemaidlist && stmt->action != DEFELEM_SET)
return;
@@ -610,7 +612,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
List *reloids;
reloids = GetPublicationRelations(pubform->oid, PUBLICATION_PART_ROOT);
- rels = OpenReliIdList(reloids);
+ rels = OpenRelIdList(reloids);
CheckObjSchemaNotAlreadyInPublication(rels, schemaidlist,
PUBLICATIONOBJ_TABLE_IN_SCHEMA);
@@ -868,7 +870,7 @@ RemovePublicationSchemaById(Oid psoid)
* add them to a publication.
*/
static List *
-OpenReliIdList(List *relids)
+OpenRelIdList(List *relids)
{
ListCell *lc;
List *rels = NIL;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 45e59e3d5c..3631b8a929 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -40,8 +40,8 @@
#include "catalog/pg_inherits.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
-#include "catalog/pg_tablespace.h"
#include "catalog/pg_statistic_ext.h"
+#include "catalog/pg_tablespace.h"
#include "catalog/pg_trigger.h"
#include "catalog/pg_type.h"
#include "catalog/storage.h"
@@ -15626,7 +15626,6 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
else
elog(ERROR, "unexpected identity type %u", stmt->identity_type);
-
/* Check that the index exists */
indexOid = get_relname_relid(stmt->name, rel->rd_rel->relnamespace);
if (!OidIsValid(indexOid))
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 2a319eecda..4415ba00fa 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -17433,8 +17433,9 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
{
/* convert it to PublicationTable */
PublicationTable *pubtable = makeNode(PublicationTable);
- pubtable->relation = makeRangeVar(NULL, pubobj->name,
- pubobj->location);
+
+ pubtable->relation =
+ makeRangeVar(NULL, pubobj->name, pubobj->location);
pubobj->pubtable = pubtable;
pubobj->name = NULL;
}
--
2.30.2
v13-0002-change-use-of-DEFELEM-enum-to-a-new-one.patchtext/x-diff; charset=utf-8Download
From b30bba79a8bf7cef972d45d0e5a3bdd27d555cd0 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Thu, 30 Dec 2021 16:52:04 -0300
Subject: [PATCH v13 2/3] change use of DEFELEM enum to a new one
---
src/backend/commands/publicationcmds.c | 18 +++++++++---------
src/backend/parser/gram.y | 6 +++---
src/include/nodes/parsenodes.h | 9 ++++++++-
3 files changed, 20 insertions(+), 13 deletions(-)
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 3466c57dc0..f63132d2ba 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -503,12 +503,12 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
* possible that user has not specified any schemas in which case we need
* to remove all the existing schemas.
*/
- if (!tables && stmt->action != DEFELEM_SET)
+ if (!tables && stmt->action != AP_SetObjects)
return;
rels = OpenTableList(tables);
- if (stmt->action == DEFELEM_ADD)
+ if (stmt->action == AP_AddObjects)
{
List *schemas = NIL;
@@ -521,9 +521,9 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
PUBLICATIONOBJ_TABLE);
PublicationAddTables(pubid, rels, false, stmt);
}
- else if (stmt->action == DEFELEM_DROP)
+ else if (stmt->action == AP_DropObjects)
PublicationDropTables(pubid, rels, false);
- else /* DEFELEM_SET */
+ else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
PUBLICATION_PART_ROOT);
@@ -598,7 +598,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
* possible that user has not specified any schemas in which case we need
* to remove all the existing schemas.
*/
- if (!schemaidlist && stmt->action != DEFELEM_SET)
+ if (!schemaidlist && stmt->action != AP_SetObjects)
return;
/*
@@ -606,7 +606,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
* concurrent schema deletion.
*/
LockSchemaList(schemaidlist);
- if (stmt->action == DEFELEM_ADD)
+ if (stmt->action == AP_AddObjects)
{
List *rels;
List *reloids;
@@ -620,9 +620,9 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
CloseTableList(rels);
PublicationAddSchemas(pubform->oid, schemaidlist, false, stmt);
}
- else if (stmt->action == DEFELEM_DROP)
+ else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* DEFELEM_SET */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -657,7 +657,7 @@ CheckAlterPublication(AlterPublicationStmt *stmt, HeapTuple tup,
{
Form_pg_publication pubform = (Form_pg_publication) GETSTRUCT(tup);
- if ((stmt->action == DEFELEM_ADD || stmt->action == DEFELEM_SET) &&
+ if ((stmt->action == AP_AddObjects || stmt->action == AP_SetObjects) &&
schemaidlist && !superuser())
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 4415ba00fa..539fb2d03b 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9828,7 +9828,7 @@ AlterPublicationStmt:
n->pubname = $3;
n->pubobjects = $5;
preprocess_pubobj_list(n->pubobjects, yyscanner);
- n->action = DEFELEM_ADD;
+ n->action = AP_AddObjects;
$$ = (Node *)n;
}
| ALTER PUBLICATION name SET pub_obj_list
@@ -9837,7 +9837,7 @@ AlterPublicationStmt:
n->pubname = $3;
n->pubobjects = $5;
preprocess_pubobj_list(n->pubobjects, yyscanner);
- n->action = DEFELEM_SET;
+ n->action = AP_SetObjects;
$$ = (Node *)n;
}
| ALTER PUBLICATION name DROP pub_obj_list
@@ -9846,7 +9846,7 @@ AlterPublicationStmt:
n->pubname = $3;
n->pubobjects = $5;
preprocess_pubobj_list(n->pubobjects, yyscanner);
- n->action = DEFELEM_DROP;
+ n->action = AP_DropObjects;
$$ = (Node *)n;
}
;
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 4c5a8a39bf..ced2835d33 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3674,6 +3674,13 @@ typedef struct CreatePublicationStmt
bool for_all_tables; /* Special publication for all tables in db */
} CreatePublicationStmt;
+typedef enum AlterPublicationAction
+{
+ AP_AddObjects, /* add objects to publication */
+ AP_DropObjects, /* remove objects from publication */
+ AP_SetObjects /* set list of objects */
+} AlterPublicationAction;
+
typedef struct AlterPublicationStmt
{
NodeTag type;
@@ -3688,7 +3695,7 @@ typedef struct AlterPublicationStmt
*/
List *pubobjects; /* Optional list of publication objects */
bool for_all_tables; /* Special publication for all tables in db */
- DefElemAction action; /* What action to perform with the
+ AlterPublicationAction action; /* What action to perform with the
* tables/schemas */
} AlterPublicationStmt;
--
2.30.2
v13-0003-Add-column-filtering-to-logical-replication.patchtext/x-diff; charset=utf-8Download
From 756cc058ca0b6102e273729f87efec34fdd69c00 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 6 Sep 2021 10:34:29 -0300
Subject: [PATCH v13 3/3] Add column filtering to logical replication
Add capability to specifiy column names while linking
the table to a publication, at the time of CREATE or ALTER
publication. This will allow replicating only the specified
columns. Other columns, if any, on the subscriber will be populated
locally or NULL will be inserted if no value is supplied for the column
by the upstream during INSERT.
This facilitates replication to a table on subscriber
containing only the subscribed/filtered columns.
If no filter is specified, all the columns are replicated.
REPLICA IDENTITY columns are always replicated.
Thus, prohibit adding relation to publication, if column filters
do not contain REPLICA IDENTITY.
Add a tap test for the same in src/test/subscription.
Author: Rahila Syed <rahilasyed90@gmail.com>
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/protocol.sgml | 4 +-
doc/src/sgml/ref/alter_publication.sgml | 20 +-
doc/src/sgml/ref/create_publication.sgml | 11 +-
src/backend/catalog/pg_publication.c | 306 ++++++++++++++++++-
src/backend/commands/publicationcmds.c | 65 +++-
src/backend/commands/tablecmds.c | 78 ++++-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 60 +++-
src/backend/replication/logical/proto.c | 66 ++--
src/backend/replication/logical/tablesync.c | 118 ++++++-
src/backend/replication/pgoutput/pgoutput.c | 78 ++++-
src/bin/pg_dump/pg_dump.c | 41 ++-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 26 +-
src/bin/psql/tab-complete.c | 2 +
src/include/catalog/pg_publication.h | 5 +
src/include/catalog/pg_publication_rel.h | 3 +
src/include/nodes/parsenodes.h | 4 +-
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 33 +-
src/test/regress/sql/publication.sql | 20 +-
src/test/subscription/t/028_column_filter.pl | 164 ++++++++++
23 files changed, 1031 insertions(+), 82 deletions(-)
create mode 100644 src/test/subscription/t/028_column_filter.pl
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 34a7034282..5bc2e7a591 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -6877,7 +6877,9 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column (except
+ generated columns and other columns that don't appear in the column
+ filter list, for tables that have one):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index bb4ef5e5e2..16a12b44b9 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -25,12 +25,13 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD <replace
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ALTER TABLE <replaceable class="parameter">publication_object</replaceable> SET COLUMNS { ( <replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -62,6 +63,11 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
command retain their previous settings.
</para>
+ <para>
+ The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows to change
+ the set of columns that are included in the publication.
+ </para>
+
<para>
The remaining variants change the owner and the name of the publication.
</para>
@@ -110,6 +116,8 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
</para>
</listitem>
</varlistentry>
@@ -164,9 +172,15 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
</para>
<para>
- Add some tables to the publication:
+ Add tables to the publication:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
+<programlisting>
+ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS (user_id, firstname, lastname);
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index d805e8e77a..73a23cbb02 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -78,6 +78,15 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
publication, so they are never explicitly added to the publication.
</para>
+ <para>
+ When a column list is specified, only the listed columns are replicated;
+ any other columns are ignored for the purpose of replication through
+ this publication. If no column list is specified, all columns of the
+ table are replicated through this publication, including any columns
+ added later. If a column list is specified, it must include the replica
+ identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index b307bc2ed5..783ee74c6b 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,13 +45,23 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+
+static void check_publication_columns(Relation targetrel, Bitmapset *columns);
+static AttrNumber *publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, Bitmapset **attset);
+
/*
- * Check if relation can be in given publication and throws appropriate
- * error if not.
+ * Check if relation can be in given publication and that the column
+ * filter is sensible, and throws appropriate error if not.
+ *
+ * targetcols is the bitmapset of column specified as column filter, or NULL if
+ * no column filter was specified.
*/
static void
-check_publication_add_relation(Relation targetrel)
+check_publication_add_relation(Relation targetrel, Bitmapset *columns)
{
+ bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
@@ -82,6 +92,62 @@ check_publication_add_relation(Relation targetrel)
errmsg("cannot add relation \"%s\" to publication",
RelationGetRelationName(targetrel)),
errdetail("This operation is not supported for unlogged tables.")));
+
+ /* Make sure the column list checks out */
+ if (columns != NULL)
+ {
+ /*
+ * Even if the user listed all columns in the column list, we cannot
+ * allow a column list to be specified when REPLICA IDENTITY is FULL;
+ * that would cause problems if a new column is added later, because
+ * that could would have to be included (because of being part of the
+ * replica identity) but it's technically not allowed (because of not
+ * being in the publication's column list yet). So reject this case
+ * altogether.
+ */
+ if (replidentfull)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter on relations with REPLICA IDENTITY FULL."));
+
+ check_publication_columns(targetrel, columns);
+ }
+}
+
+/*
+ * Enforce that the column filter can only leave out columns that aren't
+ * forced to be sent.
+ *
+ * No column can be excluded if REPLICA IDENTITY is FULL (since all the
+ * columns need to be sent regardless); and in other cases, the columns in
+ * the REPLICA IDENTITY cannot be left out.
+ */
+static void
+check_publication_columns(Relation targetrel, Bitmapset *columns)
+{
+ Bitmapset *idattrs;
+ int x;
+
+ idattrs = RelationGetIndexAttrBitmap(targetrel,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+ /*
+ * We have to test membership the hard way, because the values returned
+ * by RelationGetIndexAttrBitmap are offset.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ if (!bms_is_member(x + FirstLowInvalidHeapAttributeNumber, columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
+ }
+
+ bms_free(idattrs);
}
/*
@@ -289,6 +355,9 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
Oid relid = RelationGetRelid(targetrel->relation);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ Bitmapset *attset = NULL;
+ AttrNumber *attarray;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -314,7 +383,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ /* Translate column names to numbers and verify suitability */
+ attarray = publication_translate_columns(targetrel->relation,
+ targetrel->columns,
+ &natts, &attset);
+
+ check_publication_add_relation(targetrel->relation, attset);
+
+ bms_free(attset);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -327,6 +403,15 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ if (targetrel->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -337,6 +422,13 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
/* Register dependencies as needed */
ObjectAddressSet(myself, PublicationRelRelationId, pubreloid);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ pfree(attarray);
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -364,6 +456,144 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
return myself;
}
+/*
+ * Update the column list for a relation in a publication.
+ */
+void
+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns)
+{
+ Bitmapset *attset;
+ AttrNumber *attarray;
+ HeapTuple copytup;
+ int natts;
+ bool nulls[Natts_pg_publication_rel];
+ bool replaces[Natts_pg_publication_rel];
+ Datum values[Natts_pg_publication_rel];
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ replaces[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ deleteDependencyRecordsForClass(PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid,
+ RelationRelationId,
+ DEPENDENCY_AUTO);
+
+ if (columns == NULL)
+ {
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+ }
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ int2vector *prattrs;
+
+ if (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot change column set for relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter on relations with REPLICA IDENTITY FULL."));
+
+ attarray = publication_translate_columns(targetrel, columns,
+ &natts, &attset);
+
+ /*
+ * Make sure the column list checks out. XXX this should occur at
+ * caller in publicationcmds.c, not here.
+ */
+ check_publication_columns(targetrel, attset);
+ bms_free(attset);
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+
+ /* Add dependencies on the new list of columns */
+ ObjectAddressSet(myself, PublicationRelRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid);
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId,
+ RelationGetRelid(targetrel), attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ }
+
+ copytup = heap_modify_tuple(pubreltup, RelationGetDescr(pubrel),
+ values, nulls, replaces);
+
+ CatalogTupleUpdate(pubrel, &pubreltup->t_self, copytup);
+
+ heap_freetuple(copytup);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list. Other checks are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ */
+static AttrNumber *
+publication_translate_columns(Relation targetrel, List *columns, int *natts,
+ Bitmapset **attset)
+{
+ AttrNumber *attarray;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ *
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /*
+ * XXX qsort the array here, or maybe build just the bitmapset above and
+ * then scan that in order to produce the array? Do we care about the
+ * array being unsorted?
+ */
+
+ *natts = n;
+ *attset = set;
+ return attarray;
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -471,6 +701,74 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all column-partial publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull)
+ continue;
+
+ pubs = lappend_oid(pubs,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index f63132d2ba..aefae8b3c4 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -376,6 +376,46 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
return myself;
}
+/*
+ * Change the column list of a relation in a publication
+ */
+static void
+PublicationSetColumns(AlterPublicationStmt *stmt,
+ Form_pg_publication pubform, PublicationTable *table)
+{
+ Relation rel,
+ urel;
+ HeapTuple tup;
+ ObjectAddress obj,
+ secondary;
+
+ rel = table_open(PublicationRelRelationId, RowExclusiveLock);
+ urel = table_openrv(table->relation, ShareUpdateExclusiveLock);
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(RelationGetRelid(urel)),
+ ObjectIdGetDatum(pubform->oid));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errmsg("relation \"%s\" is not already in publication \"%s\"",
+ table->relation->relname,
+ NameStr(pubform->pubname)));
+
+ publication_set_table_columns(rel, tup, urel, table->columns);
+
+ ObjectAddressSet(obj, PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->oid);
+ ObjectAddressSet(secondary, RelationRelationId, RelationGetRelid(urel));
+ EventTriggerCollectSimpleCommand(obj, secondary, (Node *) stmt);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+ table_close(urel, NoLock);
+
+ InvokeObjectPostAlterHook(PublicationRelationId, pubform->oid, 0);
+}
+
/*
* Change options of a publication.
*/
@@ -523,6 +563,14 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
}
else if (stmt->action == AP_DropObjects)
PublicationDropTables(pubid, rels, false);
+ else if (stmt->action == AP_SetColumns)
+ {
+ Assert(schemaidlist == NIL);
+ Assert(list_length(tables) == 1);
+
+ PublicationSetColumns(stmt, pubform,
+ linitial_node(PublicationTable, tables));
+ }
else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
@@ -562,7 +610,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
pubrel = palloc(sizeof(PublicationRelInfo));
pubrel->relation = oldrel;
-
+ /* This is not needed to delete a table */
+ pubrel->columns = NIL;
delrels = lappend(delrels, pubrel);
}
}
@@ -645,6 +694,10 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
*/
PublicationAddSchemas(pubform->oid, schemaidlist, true, stmt);
}
+ else
+ {
+ /* Nothing to do for AP_SetColumns */
+ }
}
/*
@@ -934,6 +987,8 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
@@ -967,8 +1022,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
+
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
@@ -1076,6 +1134,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 3631b8a929..a9051eb5e7 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -8347,6 +8347,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
bool missing_ok, LOCKMODE lockmode,
ObjectAddresses *addrs)
{
+ Oid relid = RelationGetRelid(rel);
HeapTuple tuple;
Form_pg_attribute targetatt;
AttrNumber attnum;
@@ -8366,7 +8367,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/*
* get the number of the attribute
*/
- tuple = SearchSysCacheAttName(RelationGetRelid(rel), colName);
+ tuple = SearchSysCacheAttName(relid, colName);
if (!HeapTupleIsValid(tuple))
{
if (!missing_ok)
@@ -8420,13 +8421,42 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * Also, if the column is used in the column list of a publication,
+ * disallow the drop if the DROP is RESTRICT. We don't do anything if the
+ * DROP is CASCADE, which means that the dependency mechanism will remove
+ * the relation from the publication.
+ */
+ if (behavior == DROP_RESTRICT)
+ {
+ List *pubs;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(relid);
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(relid, pubid);
+
+ if (list_member_oid(published_cols, attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot drop column \"%s\" because it is part of publication \"%s\"",
+ colName, get_publication_name(pubid, false)),
+ errhint("Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication."));
+ }
+ }
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
* use find_all_inheritors to do it in one pass.
*/
children =
- find_inheritance_children(RelationGetRelid(rel), lockmode);
+ find_inheritance_children(relid, lockmode);
if (children)
{
@@ -8514,7 +8544,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Add object to delete */
object.classId = RelationRelationId;
- object.objectId = RelationGetRelid(rel);
+ object.objectId = relid;
object.objectSubId = attnum;
add_exact_object_address(&object, addrs);
@@ -15603,6 +15633,11 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
Oid indexOid;
Relation indexRel;
int key;
+ List *pubs;
+ Bitmapset *indexed_cols = NULL;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(RelationGetRelid(rel));
if (stmt->identity_type == REPLICA_IDENTITY_DEFAULT)
{
@@ -15611,11 +15646,16 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
}
else if (stmt->identity_type == REPLICA_IDENTITY_FULL)
{
+ if (pubs != NIL)
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set REPLICA IDENTITY FULL when column-partial publications exist"));
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
else if (stmt->identity_type == REPLICA_IDENTITY_NOTHING)
{
+ /* XXX not sure what's the right check for publications here */
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
@@ -15700,6 +15740,38 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
errmsg("index \"%s\" cannot be used as replica identity because column \"%s\" is nullable",
RelationGetRelationName(indexRel),
NameStr(attr->attname))));
+
+ /*
+ * Collect columns used, in case we have any publications that we need
+ * to vet. System attributes are disallowed so no need to subtract
+ * FirstLowInvalidHeapAttributeNumber.
+ */
+ indexed_cols = bms_add_member(indexed_cols, attno);
+ }
+
+ /*
+ * Check column-partial publications. All publications have to include all
+ * key columns of the new index.
+ */
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(RelationGetRelid(rel), pubid);
+
+ for (key = 0; key < IndexRelationGetNumberOfKeyAttributes(indexRel); key++)
+ {
+ int16 attno = indexRel->rd_index->indkey.values[key];
+
+ if (!list_member_oid(published_cols, attno))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("index \"%s\" cannot be used because publication \"%s\" does not include all indexed columns",
+ RelationGetRelationName(indexRel),
+ get_publication_name(pubid, false)));
+ }
}
/* This index is suitable for use as a replica identity. Mark it. */
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index df0b747883..0ff4c1ceac 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4833,6 +4833,7 @@ _copyPublicationTable(const PublicationTable *from)
PublicationTable *newnode = makeNode(PublicationTable);
COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index cb7ddd463c..d786a688ac 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2312,6 +2312,7 @@ static bool
_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 539fb2d03b..150c23df1b 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9740,12 +9740,13 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr
+ TABLE relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
+ $$->pubtable->columns = $3;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9760,28 +9761,38 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLE_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId
+ | ColId opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- $$->name = $1;
+ if ($2 != NULL)
+ {
+ $$->pubtable = makeNode(PublicationTable);
+ $$->pubtable->relation = makeRangeVar(NULL, $1, @1);
+ $$->pubtable->columns = $2;
+ $$->name = NULL;
+ }
+ else
+ $$->name = $1;
$$->location = @1;
}
- | ColId indirection
+ | ColId indirection opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
+ $$->pubtable->columns = $3;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr
+ | extended_relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
+ $$->pubtable->columns = $2;
}
| CURRENT_SCHEMA
{
@@ -9807,6 +9818,9 @@ pub_obj_list: PublicationObjSpec
*
* ALTER PUBLICATION name SET pub_obj [, ...]
*
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
* pub_obj is one of:
*
* TABLE table_name [, ...]
@@ -9840,6 +9854,32 @@ AlterPublicationStmt:
n->action = AP_SetObjects;
$$ = (Node *)n;
}
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS '(' columnList ')'
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = $10;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS ALL
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = NIL;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
| ALTER PUBLICATION name DROP pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
@@ -17443,6 +17483,16 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
else if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLE_IN_SCHEMA ||
pubobj->pubobjtype == PUBLICATIONOBJ_TABLE_IN_CUR_SCHEMA)
{
+ /*
+ * This can happen if a column list is specified in a continuation
+ * for a schema entry; reject it.
+ */
+ if (pubobj->pubtable)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schemas"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 9f5bf4b639..3428984130 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,9 +29,11 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary,
+ Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +400,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +412,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -442,7 +444,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary,
+ Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -463,11 +466,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -536,7 +539,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -651,7 +654,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -673,7 +677,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -749,7 +753,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple,
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -761,7 +766,13 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ /* Don't count attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
continue;
nliveatts++;
}
@@ -783,6 +794,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
if (att->attisdropped || att->attgenerated)
continue;
+ /* Ignore attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +919,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -914,20 +929,24 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
desc = RelationGetDescr(rel);
- /* send number of live attributes */
- for (i = 0; i < desc->natts; i++)
- {
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
- continue;
- nliveatts++;
- }
- pq_sendint16(out, nliveatts);
-
/* fetch bitmap of REPLICATION IDENTITY attributes */
replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
if (!replidentfull)
idattrs = RelationGetIdentityKeyBitmap(rel);
+ /* send number of live attributes */
+ for (i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+ nliveatts++;
+ }
+ pq_sendint16(out, nliveatts);
+
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -936,7 +955,8 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
-
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index f07983a43c..1303e85851 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -111,6 +111,7 @@
#include "replication/origin.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -697,17 +698,20 @@ fetch_remote_table_info(char *nspname, char *relname,
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
bool isnull;
int natt;
+ ListCell *lc;
+ bool am_partition = false;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -737,14 +741,18 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetChar(slot_getattr(slot, 4, &isnull));
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -772,16 +780,92 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * In server versions 15 and higher, obtain the applicable column filter,
+ * if any.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData publications;
+ bool first = true;
+
+ initStringInfo(&publications);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&publications, ", ");
+ appendStringInfoString(&publications, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ " SELECT pg_catalog.unnest(prattrs)\n"
+ " FROM pg_catalog.pg_publication p JOIN\n"
+ " pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n"
+ " WHERE p.pubname IN (%s) AND\n",
+ publications.data);
+ if (!am_partition)
+ appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "prrelid IN (SELECT relid\n"
+ " FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch attribute info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ if (isnull)
+ continue;
+ included_cols = bms_add_member(included_cols, attnum);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+ pfree(publications.data);
+ walrcv_clear_result(pubres);
+ }
+
+ /*
+ * Store the column names only if they are contained in column filter
+ * LogicalRepRelation will only contain attributes corresponding to those
+ * specficied in column filters.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ continue;
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
+ Assert(!isnull);
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -791,12 +875,13 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecClearTuple(slot);
}
+
ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(res);
+ pfree(cmd.data);
lrel->natts = natt;
- walrcv_clear_result(res);
- pfree(cmd.data);
}
/*
@@ -829,8 +914,17 @@ copy_table(Relation rel)
/* Start copy on the publisher. */
initStringInfo(&cmd);
if (lrel.relkind == RELKIND_RELATION)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 6f6a203dea..34df5d4956 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,16 +15,19 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -81,7 +84,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -130,6 +134,13 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+
+ /*
+ * Set of columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this list are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -570,11 +581,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
}
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -587,7 +598,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -610,13 +622,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip if attribute is not present in column filter. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -693,7 +709,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -722,7 +738,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -1122,6 +1138,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
bool found;
+ Oid ancestor_id;
MemoryContext oldctx;
Assert(RelationSyncCache != NULL);
@@ -1142,6 +1159,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->map = NULL; /* will be set by maybe_send_schema() if
* needed */
}
@@ -1182,6 +1200,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
if (pub->alltables)
{
@@ -1192,8 +1211,6 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (!publish)
{
- bool ancestor_published = false;
-
/*
* For a partition, check if any of the ancestors are
* published. If so, note down the topmost ancestor that is
@@ -1219,6 +1236,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
pub->oid))
{
ancestor_published = true;
+ ancestor_id = ancestor;
if (pub->pubviaroot)
publish_as_relid = ancestor;
}
@@ -1239,15 +1257,47 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ Oid relid;
+ HeapTuple pub_rel_tuple;
+
+ relid = ancestor_published ? ancestor_id : publish_as_relid;
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ Datum pub_rel_cols;
+ bool isnull;
+
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP,
+ pub_rel_tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(pub_rel_cols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
+ entry->columns = bms_add_member(entry->columns,
+ elems[i]);
+ MemoryContextSwitchTo(oldctx);
+ }
+ ReleaseSysCache(pub_rel_tuple);
+ }
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
}
-
- if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
- break;
}
list_free(pubids);
@@ -1343,6 +1393,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
if (entry->map)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 94f1f32558..1f8b965fd5 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4033,6 +4033,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_oid;
int i_prpubid;
int i_prrelid;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4044,8 +4045,13 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
/* Collect all publication membership info. */
appendPQExpBufferStr(query,
- "SELECT tableoid, oid, prpubid, prrelid "
- "FROM pg_catalog.pg_publication_rel");
+ "SELECT tableoid, oid, prpubid, prrelid");
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBufferStr(query, ", prattrs");
+ else
+ appendPQExpBufferStr(query, ", NULL as prattrs");
+ appendPQExpBufferStr(query,
+ " FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
ntups = PQntuples(res);
@@ -4054,6 +4060,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_oid = PQfnumber(res, "oid");
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4095,6 +4102,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
pubrinfo[j].publication = pubinfo;
pubrinfo[j].pubtable = tbinfo;
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[i].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4159,10 +4188,12 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s;\n",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+ appendPQExpBufferStr(query, ";\n");
/*
* There is no point in creating a drop query as the drop is done by table
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index f011ace8a8..3f7500accc 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -630,6 +630,7 @@ typedef struct _PublicationRelInfo
DumpableObject dobj;
PublicationInfo *publication;
TableInfo *pubtable;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index c28788e84f..b9d0ebf762 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5815,7 +5815,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5832,10 +5832,14 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (!as_schema) /* as table */
+ {
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
- else
+ if (!PQgetisnull(res, i, 2))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 2));
+ }
+ else /* as schema */
printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
@@ -5963,8 +5967,20 @@ describePublications(const char *pattern)
{
/* Get the tables for the specified publication */
printfPQExpBuffer(&buf,
- "SELECT n.nspname, c.relname\n"
- "FROM pg_catalog.pg_class c,\n"
+ "SELECT n.nspname, c.relname, \n");
+ if (pset.sversion >= 150000)
+ appendPQExpBufferStr(&buf,
+ " CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string"
+ "(ARRAY(SELECT attname\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::int[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END AS columns");
+ else
+ appendPQExpBufferStr(&buf, "NULL as columns");
+ appendPQExpBuffer(&buf,
+ "\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
" pg_catalog.pg_publication_rel pr\n"
"WHERE c.relnamespace = n.oid\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index cf30239f6d..25c7c08040 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1648,6 +1648,8 @@ psql_completion(const char *text, int start, int end)
/* ALTER PUBLICATION <name> ADD */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
+ else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD", "TABLE"))
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL);
/* ALTER PUBLICATION <name> DROP */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "DROP"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 902f2f2f0d..edd4f0c63c 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -86,6 +86,7 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -109,6 +110,8 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern List *GetPublicationSchemas(Oid pubid);
@@ -127,6 +130,8 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *tar
bool if_not_exists);
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern void publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index b5d5504cbb..7ad285faae 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN
+ int2vector prattrs; /* Variable length field starts here */
+#endif
} FormData_pg_publication_rel;
/* ----------------
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index ced2835d33..91ea815e14 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3642,6 +3642,7 @@ typedef struct PublicationTable
{
NodeTag type;
RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
@@ -3678,7 +3679,8 @@ typedef enum AlterPublicationAction
{
AP_AddObjects, /* add objects to publication */
AP_DropObjects, /* remove objects from publication */
- AP_SetObjects /* set list of objects */
+ AP_SetObjects, /* set list of objects */
+ AP_SetColumns /* change list of columns for a table */
} AlterPublicationAction;
typedef struct AlterPublicationStmt
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 83741dcf42..7a5cb9871d 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -207,11 +207,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -228,7 +228,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 5ac2d666a2..60cb242c5a 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -165,7 +165,29 @@ Publications:
regress_publication_user | t | t | t | f | f | f
(1 row)
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+ERROR: cannot drop column "c" because it is part of publication "testpub_fortable"
+HINT: Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5 (a);
+ERROR: column list must not be specified in ALTER PUBLICATION ... DROP
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl6"
+DETAIL: Cannot have column filter on relations with REPLICA IDENTITY FULL.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+ALTER PUBLICATION testpub_fortable
+ ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c); -- error
+ERROR: cannot change column set for relation "testpub_tbl6"
+DETAIL: Cannot have column filter on relations with REPLICA IDENTITY FULL.
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
CREATE TABLE testpub_tbl3a (b text) INHERITS (testpub_tbl3);
@@ -669,6 +691,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schemas
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 56dd358554..b625d161cb 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -89,7 +89,21 @@ SELECT pubname, puballtables FROM pg_publication WHERE pubname = 'testpub_forall
\d+ testpub_tbl2
\dRp+ testpub_foralltables
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5 (a);
+
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+ALTER PUBLICATION testpub_fortable
+ ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c); -- error
+
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
@@ -362,6 +376,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/028_column_filter.pl b/src/test/subscription/t/028_column_filter.pl
new file mode 100644
index 0000000000..dfae6d8eac
--- /dev/null
+++ b/src/test/subscription/t/028_column_filter.pl
@@ -0,0 +1,164 @@
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test TRUNCATE
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 9;
+
+# setup
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+ INSERT INTO tab2 VALUES (2, 'foo', 2);");
+# Test with weird column names
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, B varchar, \"c'\" int)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+#Test replication with multi-level partition
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, \"c'\" int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+#Test create publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub1 FOR TABLE tab1(a, \"B\"), tab3(\"a'\",\"c'\"), test_part(a,b)");
+
+my $result = $node_publisher->safe_psql('postgres',
+ "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
+is($result, qq(tab1|1 2
+tab3|1 3
+test_part|1 2), 'publication relation updated');
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+);
+#Initial sync
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab1 VALUES (1,2,3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 VALUES (1,2,3)");
+#Test for replication of partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (1,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (2,'bcd', '2021-07-03 11:12:13')");
+#Test for replication of multi-level partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (4,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (5,'bcd', '2021-07-03 11:12:13')");
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'insert on column tab1.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab3");
+is($result, qq(1|3), 'insert on column tab3.b is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part");
+is($result, qq(1|abc\n2|bcd\n4|abc\n5|bcd), 'insert on all columns is replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab1 SET c = 5 where a = 1");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'update on column tab1.c is not replicated');
+
+# Verify user-defined types
+$node_publisher->safe_psql('postgres',
+ qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+ ALTER PUBLICATION pub1 ADD TABLE test_tab4 (a, b, d);
+ });
+$node_subscriber->safe_psql('postgres',
+ qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, d text);
+ });
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab4 VALUES (1, 'red', 3, 'oh my');");
+
+#Test alter publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION"
+);
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 VALUES (1,'abc',3)");
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET c = 5 where a = 2");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 WHERE a = 1");
+is($result, qq(1|abc), 'insert on column tab2.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 WHERE a = 2");
+is($result, qq(2|foo), 'update on column tab2.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_tab4");
+is($result, qq(1|red|oh my), 'insert on table with user-defined type');
+
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d)");
+$node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2, pub3");
+$node_publisher->wait_for_catchup('sub2');
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (1, 11, 111, 1111)");
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (2, 22, 222, 2222)");
+$node_publisher->wait_for_catchup('sub2');
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5;"),
+ qq(1|11|1111
+2|22|2222),
+ 'overlapping publications with overlapping column lists');
--
2.30.2
+ bool am_partition = false; ... Assert(!isnull); lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull)); Assert(!isnull); + am_partition = DatumGetChar(slot_getattr(slot, 4, &isnull));
I think this needs to be GetBool.
You should Assert(!isnull) like the others.
Also, I think it doesn't need to be initialized to "false".
+ /* + * Even if the user listed all columns in the column list, we cannot + * allow a column list to be specified when REPLICA IDENTITY is FULL; + * that would cause problems if a new column is added later, because + * that could would have to be included (because of being part of the
could would is wrong
+ /* + * Translate list of columns to attnums. We prohibit system attributes and + * make sure there are no duplicate columns. + * + */
extraneous line
+/* + * Gets a list of OIDs of all column-partial publications of the given + * relation, that is, those that specify a column list.
I would call this a "partial-column" publication.
+ errmsg("cannot set REPLICA IDENTITY FULL when column-partial publications exist")); + * Check column-partial publications. All publications have to include all
same
+ /* + * Store the column names only if they are contained in column filter
period(.)
+ * LogicalRepRelation will only contain attributes corresponding to those + * specficied in column filters.
specified
--- a/src/include/catalog/pg_publication_rel.h +++ b/src/include/catalog/pg_publication_rel.h @@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId) Oid oid; /* oid */ Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */ Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */ +#ifdef CATALOG_VARLEN + int2vector prattrs; /* Variable length field starts here */ +#endif
The language in the pre-existing comments is better:
/* variable-length fields start here */
@@ -791,12 +875,13 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecClearTuple(slot); } + ExecDropSingleTupleTableSlot(slot); + walrcv_clear_result(res); + pfree(cmd.data);lrel->natts = natt;
- walrcv_clear_result(res);
- pfree(cmd.data);
}
The blank line after "lrel->natts = natt;" should be removed.
On 2021-Dec-30, Justin Pryzby wrote:
Thank you! I've incorporated your proposed fixes.
+ /* + * Even if the user listed all columns in the column list, we cannot + * allow a column list to be specified when REPLICA IDENTITY is FULL; + * that would cause problems if a new column is added later, because + * that could would have to be included (because of being part of thecould would is wrong
Hah, yeah, this was "that column would".
+ * Gets a list of OIDs of all column-partial publications of the given + * relation, that is, those that specify a column list.I would call this a "partial-column" publication.
OK, done that way.
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
Attachments:
v14-0001-Avoid-use-of-DEFELEM-enum-in-AlterPublicationStm.patchtext/x-diff; charset=utf-8Download
From dd2515ee7e0b37f82c76edc4fe890bb7be1abb3e Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Thu, 30 Dec 2021 19:49:31 -0300
Subject: [PATCH v14 1/2] Avoid use of DEFELEM enum in AlterPublicationStmt
This allows to add new values for future functionality.
Discussion: https://postgr.es/m/202112302021.ca7ihogysgh3@alvherre.pgsql
---
src/backend/commands/publicationcmds.c | 18 +++++++++---------
src/backend/parser/gram.y | 6 +++---
src/include/nodes/parsenodes.h | 11 +++++++++--
3 files changed, 21 insertions(+), 14 deletions(-)
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index f932f47a08..0f04969fd6 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -503,12 +503,12 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
* possible that user has not specified any tables in which case we need
* to remove all the existing tables.
*/
- if (!tables && stmt->action != DEFELEM_SET)
+ if (!tables && stmt->action != AP_SetObjects)
return;
rels = OpenTableList(tables);
- if (stmt->action == DEFELEM_ADD)
+ if (stmt->action == AP_AddObjects)
{
List *schemas = NIL;
@@ -521,9 +521,9 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
PUBLICATIONOBJ_TABLE);
PublicationAddTables(pubid, rels, false, stmt);
}
- else if (stmt->action == DEFELEM_DROP)
+ else if (stmt->action == AP_DropObjects)
PublicationDropTables(pubid, rels, false);
- else /* DEFELEM_SET */
+ else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
PUBLICATION_PART_ROOT);
@@ -598,7 +598,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
* possible that user has not specified any schemas in which case we need
* to remove all the existing schemas.
*/
- if (!schemaidlist && stmt->action != DEFELEM_SET)
+ if (!schemaidlist && stmt->action != AP_SetObjects)
return;
/*
@@ -606,7 +606,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
* concurrent schema deletion.
*/
LockSchemaList(schemaidlist);
- if (stmt->action == DEFELEM_ADD)
+ if (stmt->action == AP_AddObjects)
{
List *rels;
List *reloids;
@@ -620,9 +620,9 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
CloseTableList(rels);
PublicationAddSchemas(pubform->oid, schemaidlist, false, stmt);
}
- else if (stmt->action == DEFELEM_DROP)
+ else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* DEFELEM_SET */
+ else /* AP_SetObjects */
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -657,7 +657,7 @@ CheckAlterPublication(AlterPublicationStmt *stmt, HeapTuple tup,
{
Form_pg_publication pubform = (Form_pg_publication) GETSTRUCT(tup);
- if ((stmt->action == DEFELEM_ADD || stmt->action == DEFELEM_SET) &&
+ if ((stmt->action == AP_AddObjects || stmt->action == AP_SetObjects) &&
schemaidlist && !superuser())
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index f3c232842d..6dddc07947 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9828,7 +9828,7 @@ AlterPublicationStmt:
n->pubname = $3;
n->pubobjects = $5;
preprocess_pubobj_list(n->pubobjects, yyscanner);
- n->action = DEFELEM_ADD;
+ n->action = AP_AddObjects;
$$ = (Node *)n;
}
| ALTER PUBLICATION name SET pub_obj_list
@@ -9837,7 +9837,7 @@ AlterPublicationStmt:
n->pubname = $3;
n->pubobjects = $5;
preprocess_pubobj_list(n->pubobjects, yyscanner);
- n->action = DEFELEM_SET;
+ n->action = AP_SetObjects;
$$ = (Node *)n;
}
| ALTER PUBLICATION name DROP pub_obj_list
@@ -9846,7 +9846,7 @@ AlterPublicationStmt:
n->pubname = $3;
n->pubobjects = $5;
preprocess_pubobj_list(n->pubobjects, yyscanner);
- n->action = DEFELEM_DROP;
+ n->action = AP_DropObjects;
$$ = (Node *)n;
}
;
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 784164b32a..593e301f7a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3674,6 +3674,13 @@ typedef struct CreatePublicationStmt
bool for_all_tables; /* Special publication for all tables in db */
} CreatePublicationStmt;
+typedef enum AlterPublicationAction
+{
+ AP_AddObjects, /* add objects to publication */
+ AP_DropObjects, /* remove objects from publication */
+ AP_SetObjects /* set list of objects */
+} AlterPublicationAction;
+
typedef struct AlterPublicationStmt
{
NodeTag type;
@@ -3688,8 +3695,8 @@ typedef struct AlterPublicationStmt
*/
List *pubobjects; /* Optional list of publication objects */
bool for_all_tables; /* Special publication for all tables in db */
- DefElemAction action; /* What action to perform with the
- * tables/schemas */
+ AlterPublicationAction action; /* What action to perform with the given
+ * objects */
} AlterPublicationStmt;
typedef struct CreateSubscriptionStmt
--
2.30.2
v14-0002-Support-column-lists-for-logical-replication-of-.patchtext/x-diff; charset=utf-8Download
From c0651cb9cce58508344282a445aca90ab93c318f Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 6 Sep 2021 10:34:29 -0300
Subject: [PATCH v14 2/2] Support column lists for logical replication of
tables
Add the capability of specifying a column list for individual tables as
part of a publication. Columns not in the list are not published. This
enables replicating to a table with only a subset of the columns.
If no column list is specified, all the columns are replicated, as
previously
Author: Rahila Syed <rahilasyed90@gmail.com>
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/protocol.sgml | 4 +-
doc/src/sgml/ref/alter_publication.sgml | 20 +-
doc/src/sgml/ref/create_publication.sgml | 11 +-
src/backend/catalog/pg_publication.c | 306 +++++++++++++++++-
src/backend/commands/publicationcmds.c | 67 +++-
src/backend/commands/tablecmds.c | 79 ++++-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 60 +++-
src/backend/replication/logical/proto.c | 66 ++--
src/backend/replication/logical/tablesync.c | 120 ++++++-
src/backend/replication/pgoutput/pgoutput.c | 78 ++++-
src/bin/pg_dump/pg_dump.c | 41 ++-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 26 +-
src/bin/psql/tab-complete.c | 2 +
src/include/catalog/pg_publication.h | 5 +
src/include/catalog/pg_publication_rel.h | 3 +
src/include/nodes/parsenodes.h | 4 +-
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 33 +-
src/test/regress/sql/publication.sql | 20 +-
src/test/subscription/t/028_column_list.patch | 164 ++++++++++
23 files changed, 1034 insertions(+), 84 deletions(-)
create mode 100644 src/test/subscription/t/028_column_list.patch
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 34a7034282..5bc2e7a591 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -6877,7 +6877,9 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column (except
+ generated columns and other columns that don't appear in the column
+ filter list, for tables that have one):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index bb4ef5e5e2..16a12b44b9 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -25,12 +25,13 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD <replace
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ALTER TABLE <replaceable class="parameter">publication_object</replaceable> SET COLUMNS { ( <replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -62,6 +63,11 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
command retain their previous settings.
</para>
+ <para>
+ The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows to change
+ the set of columns that are included in the publication.
+ </para>
+
<para>
The remaining variants change the owner and the name of the publication.
</para>
@@ -110,6 +116,8 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
</para>
</listitem>
</varlistentry>
@@ -164,9 +172,15 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
</para>
<para>
- Add some tables to the publication:
+ Add tables to the publication:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
+<programlisting>
+ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS (user_id, firstname, lastname);
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index d805e8e77a..73a23cbb02 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -78,6 +78,15 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
publication, so they are never explicitly added to the publication.
</para>
+ <para>
+ When a column list is specified, only the listed columns are replicated;
+ any other columns are ignored for the purpose of replication through
+ this publication. If no column list is specified, all columns of the
+ table are replicated through this publication, including any columns
+ added later. If a column list is specified, it must include the replica
+ identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index b307bc2ed5..af5d1a281f 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,13 +45,23 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+
+static void check_publication_columns(Relation targetrel, Bitmapset *columns);
+static AttrNumber *publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, Bitmapset **attset);
+
/*
- * Check if relation can be in given publication and throws appropriate
- * error if not.
+ * Check if relation can be in given publication and that the column
+ * filter is sensible, and throws appropriate error if not.
+ *
+ * targetcols is the bitmapset of column specified as column filter, or NULL if
+ * no column filter was specified.
*/
static void
-check_publication_add_relation(Relation targetrel)
+check_publication_add_relation(Relation targetrel, Bitmapset *columns)
{
+ bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
@@ -82,6 +92,63 @@ check_publication_add_relation(Relation targetrel)
errmsg("cannot add relation \"%s\" to publication",
RelationGetRelationName(targetrel)),
errdetail("This operation is not supported for unlogged tables.")));
+
+ /* Make sure the column list checks out */
+ if (columns != NULL)
+ {
+ /*
+ * Even if the user listed all columns in the column list, we cannot
+ * allow a column list to be specified when REPLICA IDENTITY is FULL;
+ * that would cause problems if a new column is added later, because
+ * the new column would have to be included (because of being part of
+ * the replica identity) but it's technically not allowed (because of
+ * not being in the publication's column list yet). So reject this
+ * case altogether.
+ */
+ if (replidentfull)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter on relations with REPLICA IDENTITY FULL."));
+
+ check_publication_columns(targetrel, columns);
+ }
+}
+
+/*
+ * Enforce that the column filter can only leave out columns that aren't
+ * forced to be sent.
+ *
+ * No column can be excluded if REPLICA IDENTITY is FULL (since all the
+ * columns need to be sent regardless); and in other cases, the columns in
+ * the REPLICA IDENTITY cannot be left out.
+ */
+static void
+check_publication_columns(Relation targetrel, Bitmapset *columns)
+{
+ Bitmapset *idattrs;
+ int x;
+
+ idattrs = RelationGetIndexAttrBitmap(targetrel,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * We have to test membership the hard way, because the values returned by
+ * RelationGetIndexAttrBitmap are offset.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ if (!bms_is_member(x + FirstLowInvalidHeapAttributeNumber, columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
+ }
+
+ bms_free(idattrs);
}
/*
@@ -289,6 +356,9 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
Oid relid = RelationGetRelid(targetrel->relation);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ Bitmapset *attset = NULL;
+ AttrNumber *attarray;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -314,7 +384,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ /* Translate column names to numbers and verify suitability */
+ attarray = publication_translate_columns(targetrel->relation,
+ targetrel->columns,
+ &natts, &attset);
+
+ check_publication_add_relation(targetrel->relation, attset);
+
+ bms_free(attset);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -327,6 +404,15 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ if (targetrel->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -337,6 +423,13 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
/* Register dependencies as needed */
ObjectAddressSet(myself, PublicationRelRelationId, pubreloid);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ pfree(attarray);
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -364,6 +457,143 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
return myself;
}
+/*
+ * Update the column list for a relation in a publication.
+ */
+void
+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns)
+{
+ Bitmapset *attset;
+ AttrNumber *attarray;
+ HeapTuple copytup;
+ int natts;
+ bool nulls[Natts_pg_publication_rel];
+ bool replaces[Natts_pg_publication_rel];
+ Datum values[Natts_pg_publication_rel];
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ replaces[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ deleteDependencyRecordsForClass(PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid,
+ RelationRelationId,
+ DEPENDENCY_AUTO);
+
+ if (columns == NULL)
+ {
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+ }
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ int2vector *prattrs;
+
+ if (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot change column set for relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter on relations with REPLICA IDENTITY FULL."));
+
+ attarray = publication_translate_columns(targetrel, columns,
+ &natts, &attset);
+
+ /*
+ * Make sure the column list checks out. XXX this should occur at
+ * caller in publicationcmds.c, not here.
+ */
+ check_publication_columns(targetrel, attset);
+ bms_free(attset);
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+
+ /* Add dependencies on the new list of columns */
+ ObjectAddressSet(myself, PublicationRelRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid);
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId,
+ RelationGetRelid(targetrel), attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ }
+
+ copytup = heap_modify_tuple(pubreltup, RelationGetDescr(pubrel),
+ values, nulls, replaces);
+
+ CatalogTupleUpdate(pubrel, &pubreltup->t_self, copytup);
+
+ heap_freetuple(copytup);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list. Other checks are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ */
+static AttrNumber *
+publication_translate_columns(Relation targetrel, List *columns, int *natts,
+ Bitmapset **attset)
+{
+ AttrNumber *attarray;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /*
+ * XXX qsort the array here, or maybe build just the bitmapset above and
+ * then scan that in order to produce the array? Do we care about the
+ * array being unsorted?
+ */
+
+ *natts = n;
+ *attset = set;
+ return attarray;
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -471,6 +701,74 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull)
+ continue;
+
+ pubs = lappend_oid(pubs,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 0f04969fd6..657374c0d1 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -376,6 +376,46 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
return myself;
}
+/*
+ * Change the column list of a relation in a publication
+ */
+static void
+PublicationSetColumns(AlterPublicationStmt *stmt,
+ Form_pg_publication pubform, PublicationTable *table)
+{
+ Relation rel,
+ urel;
+ HeapTuple tup;
+ ObjectAddress obj,
+ secondary;
+
+ rel = table_open(PublicationRelRelationId, RowExclusiveLock);
+ urel = table_openrv(table->relation, ShareUpdateExclusiveLock);
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(RelationGetRelid(urel)),
+ ObjectIdGetDatum(pubform->oid));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errmsg("relation \"%s\" is not already in publication \"%s\"",
+ table->relation->relname,
+ NameStr(pubform->pubname)));
+
+ publication_set_table_columns(rel, tup, urel, table->columns);
+
+ ObjectAddressSet(obj, PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->oid);
+ ObjectAddressSet(secondary, RelationRelationId, RelationGetRelid(urel));
+ EventTriggerCollectSimpleCommand(obj, secondary, (Node *) stmt);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+ table_close(urel, NoLock);
+
+ InvokeObjectPostAlterHook(PublicationRelationId, pubform->oid, 0);
+}
+
/*
* Change options of a publication.
*/
@@ -523,6 +563,14 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
}
else if (stmt->action == AP_DropObjects)
PublicationDropTables(pubid, rels, false);
+ else if (stmt->action == AP_SetColumns)
+ {
+ Assert(schemaidlist == NIL);
+ Assert(list_length(tables) == 1);
+
+ PublicationSetColumns(stmt, pubform,
+ linitial_node(PublicationTable, tables));
+ }
else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
@@ -562,7 +610,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
pubrel = palloc(sizeof(PublicationRelInfo));
pubrel->relation = oldrel;
-
+ /* This is not needed to delete a table */
+ pubrel->columns = NIL;
delrels = lappend(delrels, pubrel);
}
}
@@ -622,7 +671,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
}
else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* AP_SetObjects */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -645,6 +694,10 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
*/
PublicationAddSchemas(pubform->oid, schemaidlist, true, stmt);
}
+ else
+ {
+ /* Nothing to do for AP_SetColumns */
+ }
}
/*
@@ -934,6 +987,8 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
@@ -967,8 +1022,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
+
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
@@ -1076,6 +1134,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 3631b8a929..232a068613 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -8347,6 +8347,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
bool missing_ok, LOCKMODE lockmode,
ObjectAddresses *addrs)
{
+ Oid relid = RelationGetRelid(rel);
HeapTuple tuple;
Form_pg_attribute targetatt;
AttrNumber attnum;
@@ -8366,7 +8367,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/*
* get the number of the attribute
*/
- tuple = SearchSysCacheAttName(RelationGetRelid(rel), colName);
+ tuple = SearchSysCacheAttName(relid, colName);
if (!HeapTupleIsValid(tuple))
{
if (!missing_ok)
@@ -8420,13 +8421,42 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * Also, if the column is used in the column list of a publication,
+ * disallow the drop if the DROP is RESTRICT. We don't do anything if the
+ * DROP is CASCADE, which means that the dependency mechanism will remove
+ * the relation from the publication.
+ */
+ if (behavior == DROP_RESTRICT)
+ {
+ List *pubs;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(relid);
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(relid, pubid);
+
+ if (list_member_oid(published_cols, attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot drop column \"%s\" because it is part of publication \"%s\"",
+ colName, get_publication_name(pubid, false)),
+ errhint("Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication."));
+ }
+ }
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
* use find_all_inheritors to do it in one pass.
*/
children =
- find_inheritance_children(RelationGetRelid(rel), lockmode);
+ find_inheritance_children(relid, lockmode);
if (children)
{
@@ -8514,7 +8544,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Add object to delete */
object.classId = RelationRelationId;
- object.objectId = RelationGetRelid(rel);
+ object.objectId = relid;
object.objectSubId = attnum;
add_exact_object_address(&object, addrs);
@@ -15581,6 +15611,7 @@ relation_mark_replica_identity(Relation rel, char ri_type, Oid indexOid,
CatalogTupleUpdate(pg_index, &pg_index_tuple->t_self, pg_index_tuple);
InvokeObjectPostAlterHookArg(IndexRelationId, thisIndexOid, 0,
InvalidOid, is_internal);
+
/*
* Invalidate the relcache for the table, so that after we commit
* all sessions will refresh the table's replica identity index
@@ -15603,6 +15634,11 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
Oid indexOid;
Relation indexRel;
int key;
+ List *pubs;
+ Bitmapset *indexed_cols = NULL;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(RelationGetRelid(rel));
if (stmt->identity_type == REPLICA_IDENTITY_DEFAULT)
{
@@ -15611,11 +15647,16 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
}
else if (stmt->identity_type == REPLICA_IDENTITY_FULL)
{
+ if (pubs != NIL)
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set REPLICA IDENTITY FULL when publications contain relations that specify column lists"));
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
else if (stmt->identity_type == REPLICA_IDENTITY_NOTHING)
{
+ /* XXX not sure what's the right check for publications here */
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
@@ -15700,6 +15741,38 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
errmsg("index \"%s\" cannot be used as replica identity because column \"%s\" is nullable",
RelationGetRelationName(indexRel),
NameStr(attr->attname))));
+
+ /*
+ * Collect columns used, in case we have any publications that we need
+ * to vet. System attributes are disallowed so no need to subtract
+ * FirstLowInvalidHeapAttributeNumber.
+ */
+ indexed_cols = bms_add_member(indexed_cols, attno);
+ }
+
+ /*
+ * Check partial-column publications. All publications have to include
+ * all key columns of the new index.
+ */
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(RelationGetRelid(rel), pubid);
+
+ for (key = 0; key < IndexRelationGetNumberOfKeyAttributes(indexRel); key++)
+ {
+ int16 attno = indexRel->rd_index->indkey.values[key];
+
+ if (!list_member_oid(published_cols, attno))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("index \"%s\" cannot be used because publication \"%s\" does not include all indexed columns",
+ RelationGetRelationName(indexRel),
+ get_publication_name(pubid, false)));
+ }
}
/* This index is suitable for use as a replica identity. Mark it. */
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index df0b747883..0ff4c1ceac 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4833,6 +4833,7 @@ _copyPublicationTable(const PublicationTable *from)
PublicationTable *newnode = makeNode(PublicationTable);
COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index cb7ddd463c..d786a688ac 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2312,6 +2312,7 @@ static bool
_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 6dddc07947..068f67998c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9740,12 +9740,13 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr
+ TABLE relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
+ $$->pubtable->columns = $3;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9760,28 +9761,38 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId
+ | ColId opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- $$->name = $1;
+ if ($2 != NULL)
+ {
+ $$->pubtable = makeNode(PublicationTable);
+ $$->pubtable->relation = makeRangeVar(NULL, $1, @1);
+ $$->pubtable->columns = $2;
+ $$->name = NULL;
+ }
+ else
+ $$->name = $1;
$$->location = @1;
}
- | ColId indirection
+ | ColId indirection opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
+ $$->pubtable->columns = $3;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr
+ | extended_relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
+ $$->pubtable->columns = $2;
}
| CURRENT_SCHEMA
{
@@ -9807,6 +9818,9 @@ pub_obj_list: PublicationObjSpec
*
* ALTER PUBLICATION name SET pub_obj [, ...]
*
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
* pub_obj is one of:
*
* TABLE table_name [, ...]
@@ -9840,6 +9854,32 @@ AlterPublicationStmt:
n->action = AP_SetObjects;
$$ = (Node *)n;
}
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS '(' columnList ')'
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = $10;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS ALL
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = NIL;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
| ALTER PUBLICATION name DROP pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
@@ -17444,6 +17484,16 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
else if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLES_IN_SCHEMA ||
pubobj->pubobjtype == PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA)
{
+ /*
+ * This can happen if a column list is specified in a continuation
+ * for a schema entry; reject it.
+ */
+ if (pubobj->pubtable)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schemas"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 9f5bf4b639..3428984130 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,9 +29,11 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary,
+ Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +400,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +412,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -442,7 +444,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary,
+ Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -463,11 +466,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -536,7 +539,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -651,7 +654,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -673,7 +677,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -749,7 +753,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple,
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -761,7 +766,13 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ /* Don't count attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
continue;
nliveatts++;
}
@@ -783,6 +794,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
if (att->attisdropped || att->attgenerated)
continue;
+ /* Ignore attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +919,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -914,20 +929,24 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
desc = RelationGetDescr(rel);
- /* send number of live attributes */
- for (i = 0; i < desc->natts; i++)
- {
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
- continue;
- nliveatts++;
- }
- pq_sendint16(out, nliveatts);
-
/* fetch bitmap of REPLICATION IDENTITY attributes */
replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
if (!replidentfull)
idattrs = RelationGetIdentityKeyBitmap(rel);
+ /* send number of live attributes */
+ for (i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+ nliveatts++;
+ }
+ pq_sendint16(out, nliveatts);
+
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -936,7 +955,8 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
-
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index f07983a43c..35f1294ae4 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -111,6 +111,7 @@
#include "replication/origin.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -697,17 +698,20 @@ fetch_remote_table_info(char *nspname, char *relname,
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
bool isnull;
int natt;
+ ListCell *lc;
+ bool am_partition;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -737,14 +741,19 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetBool(slot_getattr(slot, 4, &isnull));
+ Assert(!isnull);
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -772,16 +781,92 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * In server versions 15 and higher, obtain the applicable column filter,
+ * if any.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData publications;
+ bool first = true;
+
+ initStringInfo(&publications);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&publications, ", ");
+ appendStringInfoString(&publications, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ " SELECT pg_catalog.unnest(prattrs)\n"
+ " FROM pg_catalog.pg_publication p JOIN\n"
+ " pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n"
+ " WHERE p.pubname IN (%s) AND\n",
+ publications.data);
+ if (!am_partition)
+ appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "prrelid IN (SELECT relid\n"
+ " FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch attribute info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ if (isnull)
+ continue;
+ included_cols = bms_add_member(included_cols, attnum);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+ pfree(publications.data);
+ walrcv_clear_result(pubres);
+ }
+
+ /*
+ * Store the column names only if they are contained in column filter.
+ * LogicalRepRelation will only contain attributes corresponding to those
+ * specified in column filters.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ continue;
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
+ Assert(!isnull);
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -791,12 +876,12 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecClearTuple(slot);
}
+
ExecDropSingleTupleTableSlot(slot);
-
- lrel->natts = natt;
-
walrcv_clear_result(res);
pfree(cmd.data);
+
+ lrel->natts = natt;
}
/*
@@ -829,8 +914,17 @@ copy_table(Relation rel)
/* Start copy on the publisher. */
initStringInfo(&cmd);
if (lrel.relkind == RELKIND_RELATION)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 6f6a203dea..34df5d4956 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,16 +15,19 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -81,7 +84,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -130,6 +134,13 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+
+ /*
+ * Set of columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this list are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -570,11 +581,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
}
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -587,7 +598,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -610,13 +622,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip if attribute is not present in column filter. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -693,7 +709,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -722,7 +738,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -1122,6 +1138,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
bool found;
+ Oid ancestor_id;
MemoryContext oldctx;
Assert(RelationSyncCache != NULL);
@@ -1142,6 +1159,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->map = NULL; /* will be set by maybe_send_schema() if
* needed */
}
@@ -1182,6 +1200,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
if (pub->alltables)
{
@@ -1192,8 +1211,6 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (!publish)
{
- bool ancestor_published = false;
-
/*
* For a partition, check if any of the ancestors are
* published. If so, note down the topmost ancestor that is
@@ -1219,6 +1236,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
pub->oid))
{
ancestor_published = true;
+ ancestor_id = ancestor;
if (pub->pubviaroot)
publish_as_relid = ancestor;
}
@@ -1239,15 +1257,47 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ Oid relid;
+ HeapTuple pub_rel_tuple;
+
+ relid = ancestor_published ? ancestor_id : publish_as_relid;
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ Datum pub_rel_cols;
+ bool isnull;
+
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP,
+ pub_rel_tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(pub_rel_cols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
+ entry->columns = bms_add_member(entry->columns,
+ elems[i]);
+ MemoryContextSwitchTo(oldctx);
+ }
+ ReleaseSysCache(pub_rel_tuple);
+ }
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
}
-
- if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
- break;
}
list_free(pubids);
@@ -1343,6 +1393,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
if (entry->map)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 94f1f32558..1f8b965fd5 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4033,6 +4033,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_oid;
int i_prpubid;
int i_prrelid;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4044,8 +4045,13 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
/* Collect all publication membership info. */
appendPQExpBufferStr(query,
- "SELECT tableoid, oid, prpubid, prrelid "
- "FROM pg_catalog.pg_publication_rel");
+ "SELECT tableoid, oid, prpubid, prrelid");
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBufferStr(query, ", prattrs");
+ else
+ appendPQExpBufferStr(query, ", NULL as prattrs");
+ appendPQExpBufferStr(query,
+ " FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
ntups = PQntuples(res);
@@ -4054,6 +4060,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_oid = PQfnumber(res, "oid");
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4095,6 +4102,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
pubrinfo[j].publication = pubinfo;
pubrinfo[j].pubtable = tbinfo;
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[i].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4159,10 +4188,12 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s;\n",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+ appendPQExpBufferStr(query, ";\n");
/*
* There is no point in creating a drop query as the drop is done by table
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index f011ace8a8..3f7500accc 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -630,6 +630,7 @@ typedef struct _PublicationRelInfo
DumpableObject dobj;
PublicationInfo *publication;
TableInfo *pubtable;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index c28788e84f..b9d0ebf762 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5815,7 +5815,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5832,10 +5832,14 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (!as_schema) /* as table */
+ {
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
- else
+ if (!PQgetisnull(res, i, 2))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 2));
+ }
+ else /* as schema */
printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
@@ -5963,8 +5967,20 @@ describePublications(const char *pattern)
{
/* Get the tables for the specified publication */
printfPQExpBuffer(&buf,
- "SELECT n.nspname, c.relname\n"
- "FROM pg_catalog.pg_class c,\n"
+ "SELECT n.nspname, c.relname, \n");
+ if (pset.sversion >= 150000)
+ appendPQExpBufferStr(&buf,
+ " CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string"
+ "(ARRAY(SELECT attname\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::int[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END AS columns");
+ else
+ appendPQExpBufferStr(&buf, "NULL as columns");
+ appendPQExpBuffer(&buf,
+ "\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
" pg_catalog.pg_publication_rel pr\n"
"WHERE c.relnamespace = n.oid\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index cf30239f6d..25c7c08040 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1648,6 +1648,8 @@ psql_completion(const char *text, int start, int end)
/* ALTER PUBLICATION <name> ADD */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
+ else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD", "TABLE"))
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL);
/* ALTER PUBLICATION <name> DROP */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "DROP"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 902f2f2f0d..edd4f0c63c 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -86,6 +86,7 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -109,6 +110,8 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern List *GetPublicationSchemas(Oid pubid);
@@ -127,6 +130,8 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *tar
bool if_not_exists);
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern void publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index b5d5504cbb..51946cce59 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ int2vector prattrs;
+#endif
} FormData_pg_publication_rel;
/* ----------------
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 593e301f7a..f98a78c016 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3642,6 +3642,7 @@ typedef struct PublicationTable
{
NodeTag type;
RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
@@ -3678,7 +3679,8 @@ typedef enum AlterPublicationAction
{
AP_AddObjects, /* add objects to publication */
AP_DropObjects, /* remove objects from publication */
- AP_SetObjects /* set list of objects */
+ AP_SetObjects, /* set list of objects */
+ AP_SetColumns /* change list of columns for a table */
} AlterPublicationAction;
typedef struct AlterPublicationStmt
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 83741dcf42..7a5cb9871d 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -207,11 +207,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -228,7 +228,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 12c5f67080..44de5fa8a2 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -165,7 +165,29 @@ Publications:
regress_publication_user | t | t | t | f | f | f
(1 row)
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+ERROR: cannot drop column "c" because it is part of publication "testpub_fortable"
+HINT: Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5 (a);
+ERROR: column list must not be specified in ALTER PUBLICATION ... DROP
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl6"
+DETAIL: Cannot have column filter on relations with REPLICA IDENTITY FULL.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+ALTER PUBLICATION testpub_fortable
+ ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c); -- error
+ERROR: cannot change column set for relation "testpub_tbl6"
+DETAIL: Cannot have column filter on relations with REPLICA IDENTITY FULL.
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
CREATE TABLE testpub_tbl3a (b text) INHERITS (testpub_tbl3);
@@ -670,6 +692,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schemas
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 56dd358554..b625d161cb 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -89,7 +89,21 @@ SELECT pubname, puballtables FROM pg_publication WHERE pubname = 'testpub_forall
\d+ testpub_tbl2
\dRp+ testpub_foralltables
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5 (a);
+
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+ALTER PUBLICATION testpub_fortable
+ ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c); -- error
+
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
@@ -362,6 +376,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/028_column_list.patch b/src/test/subscription/t/028_column_list.patch
new file mode 100644
index 0000000000..f2b26176ed
--- /dev/null
+++ b/src/test/subscription/t/028_column_list.patch
@@ -0,0 +1,164 @@
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# Test partial-column publication of tables
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 9;
+
+# setup
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+ INSERT INTO tab2 VALUES (2, 'foo', 2);");
+# Test with weird column names
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, B varchar, \"c'\" int)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+#Test replication with multi-level partition
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, \"c'\" int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+#Test create publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub1 FOR TABLE tab1(a, \"B\"), tab3(\"a'\",\"c'\"), test_part(a,b)");
+
+my $result = $node_publisher->safe_psql('postgres',
+ "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
+is($result, qq(tab1|1 2
+tab3|1 3
+test_part|1 2), 'publication relation updated');
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+);
+#Initial sync
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab1 VALUES (1,2,3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 VALUES (1,2,3)");
+#Test for replication of partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (1,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (2,'bcd', '2021-07-03 11:12:13')");
+#Test for replication of multi-level partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (4,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (5,'bcd', '2021-07-03 11:12:13')");
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'insert on column tab1.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab3");
+is($result, qq(1|3), 'insert on column tab3.b is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part");
+is($result, qq(1|abc\n2|bcd\n4|abc\n5|bcd), 'insert on all columns is replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab1 SET c = 5 where a = 1");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'update on column tab1.c is not replicated');
+
+# Verify user-defined types
+$node_publisher->safe_psql('postgres',
+ qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+ ALTER PUBLICATION pub1 ADD TABLE test_tab4 (a, b, d);
+ });
+$node_subscriber->safe_psql('postgres',
+ qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, d text);
+ });
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab4 VALUES (1, 'red', 3, 'oh my');");
+
+#Test alter publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION"
+);
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 VALUES (1,'abc',3)");
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET c = 5 where a = 2");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 WHERE a = 1");
+is($result, qq(1|abc), 'insert on column tab2.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 WHERE a = 2");
+is($result, qq(2|foo), 'update on column tab2.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_tab4");
+is($result, qq(1|red|oh my), 'insert on table with user-defined type');
+
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d)");
+$node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2, pub3");
+$node_publisher->wait_for_catchup('sub2');
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (1, 11, 111, 1111)");
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (2, 22, 222, 2222)");
+$node_publisher->wait_for_catchup('sub2');
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5;"),
+ qq(1|11|1111
+2|22|2222),
+ 'overlapping publications with overlapping column lists');
--
2.30.2
@@ -5963,8 +5967,20 @@ describePublications(const char *pattern) { /* Get the tables for the specified publication */ printfPQExpBuffer(&buf, - "SELECT n.nspname, c.relname\n" - "FROM pg_catalog.pg_class c,\n" + "SELECT n.nspname, c.relname, \n"); + if (pset.sversion >= 150000) + appendPQExpBufferStr(&buf, + " CASE WHEN pr.prattrs IS NOT NULL THEN\n" + " pg_catalog.array_to_string" + "(ARRAY(SELECT attname\n" + " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::int[], 1)) s,\n" + " pg_catalog.pg_attribute\n" + " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n" + " ELSE NULL END AS columns"); + else + appendPQExpBufferStr(&buf, "NULL as columns"); + appendPQExpBuffer(&buf, + "\nFROM pg_catalog.pg_class c,\n" " pg_catalog.pg_namespace n,\n" " pg_catalog.pg_publication_rel pr\n" "WHERE c.relnamespace = n.oid\n"
I suppose this should use pr.prattrs::pg_catalog.int2[] ?
Did the DatumGetBool issue expose a deficiency in testing ?
I think the !am_partition path was never being hit.
--
Justin
On 2021-Dec-31, Justin Pryzby wrote:
@@ -5963,8 +5967,20 @@ describePublications(const char *pattern) { + " CASE WHEN pr.prattrs IS NOT NULL THEN\n" + " pg_catalog.array_to_string" + "(ARRAY(SELECT attname\n" + " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::int[], 1)) s,\n" + " pg_catalog.pg_attribute\n" + " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n" + " ELSE NULL END AS columns");
I suppose this should use pr.prattrs::pg_catalog.int2[] ?
True. Changed that.
Another change in this v15 is that I renamed the test file from ".patch"
to ".pl". I suppose I mistyped the extension when renumbering from 021
to 028.
Did the DatumGetBool issue expose a deficiency in testing ?
I think the !am_partition path was never being hit.
Hmm, the TAP test creates a subscription that contains both types of
tables. I tried adding an assert for each case, and they were both hit
on running the test.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
"La persona que no quería pecar / estaba obligada a sentarse
en duras y empinadas sillas / desprovistas, por cierto
de blandos atenuantes" (Patricio Vogel)
Attachments:
v15-0001-Support-column-lists-for-logical-replication-of-.patchtext/x-diff; charset=utf-8Download
From ad2766290e7011481813ce24c1947bff70415211 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 6 Sep 2021 10:34:29 -0300
Subject: [PATCH v15] Support column lists for logical replication of tables
Add the capability of specifying a column list for individual tables as
part of a publication. Columns not in the list are not published. This
enables replicating to a table with only a subset of the columns.
If no column list is specified, all the columns are replicated, as
previously
Author: Rahila Syed <rahilasyed90@gmail.com>
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/protocol.sgml | 4 +-
doc/src/sgml/ref/alter_publication.sgml | 20 +-
doc/src/sgml/ref/create_publication.sgml | 11 +-
src/backend/catalog/pg_publication.c | 306 +++++++++++++++++++-
src/backend/commands/publicationcmds.c | 67 ++++-
src/backend/commands/tablecmds.c | 79 ++++-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 60 +++-
src/backend/replication/logical/proto.c | 66 +++--
src/backend/replication/logical/tablesync.c | 120 +++++++-
src/backend/replication/pgoutput/pgoutput.c | 78 ++++-
src/bin/pg_dump/pg_dump.c | 41 ++-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 26 +-
src/bin/psql/tab-complete.c | 2 +
src/include/catalog/pg_publication.h | 5 +
src/include/catalog/pg_publication_rel.h | 3 +
src/include/nodes/parsenodes.h | 4 +-
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 33 ++-
src/test/regress/sql/publication.sql | 20 +-
src/test/subscription/t/028_column_list.pl | 164 +++++++++++
23 files changed, 1034 insertions(+), 84 deletions(-)
create mode 100644 src/test/subscription/t/028_column_list.pl
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 34a7034282..5bc2e7a591 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -6877,7 +6877,9 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column (except
+ generated columns and other columns that don't appear in the column
+ filter list, for tables that have one):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index bb4ef5e5e2..16a12b44b9 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -25,12 +25,13 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD <replace
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ALTER TABLE <replaceable class="parameter">publication_object</replaceable> SET COLUMNS { ( <replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -62,6 +63,11 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
command retain their previous settings.
</para>
+ <para>
+ The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows to change
+ the set of columns that are included in the publication.
+ </para>
+
<para>
The remaining variants change the owner and the name of the publication.
</para>
@@ -110,6 +116,8 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
</para>
</listitem>
</varlistentry>
@@ -164,9 +172,15 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
</para>
<para>
- Add some tables to the publication:
+ Add tables to the publication:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
+<programlisting>
+ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS (user_id, firstname, lastname);
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index d805e8e77a..73a23cbb02 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -78,6 +78,15 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
publication, so they are never explicitly added to the publication.
</para>
+ <para>
+ When a column list is specified, only the listed columns are replicated;
+ any other columns are ignored for the purpose of replication through
+ this publication. If no column list is specified, all columns of the
+ table are replicated through this publication, including any columns
+ added later. If a column list is specified, it must include the replica
+ identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index b307bc2ed5..af5d1a281f 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,13 +45,23 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+
+static void check_publication_columns(Relation targetrel, Bitmapset *columns);
+static AttrNumber *publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, Bitmapset **attset);
+
/*
- * Check if relation can be in given publication and throws appropriate
- * error if not.
+ * Check if relation can be in given publication and that the column
+ * filter is sensible, and throws appropriate error if not.
+ *
+ * targetcols is the bitmapset of column specified as column filter, or NULL if
+ * no column filter was specified.
*/
static void
-check_publication_add_relation(Relation targetrel)
+check_publication_add_relation(Relation targetrel, Bitmapset *columns)
{
+ bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
@@ -82,6 +92,63 @@ check_publication_add_relation(Relation targetrel)
errmsg("cannot add relation \"%s\" to publication",
RelationGetRelationName(targetrel)),
errdetail("This operation is not supported for unlogged tables.")));
+
+ /* Make sure the column list checks out */
+ if (columns != NULL)
+ {
+ /*
+ * Even if the user listed all columns in the column list, we cannot
+ * allow a column list to be specified when REPLICA IDENTITY is FULL;
+ * that would cause problems if a new column is added later, because
+ * the new column would have to be included (because of being part of
+ * the replica identity) but it's technically not allowed (because of
+ * not being in the publication's column list yet). So reject this
+ * case altogether.
+ */
+ if (replidentfull)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter on relations with REPLICA IDENTITY FULL."));
+
+ check_publication_columns(targetrel, columns);
+ }
+}
+
+/*
+ * Enforce that the column filter can only leave out columns that aren't
+ * forced to be sent.
+ *
+ * No column can be excluded if REPLICA IDENTITY is FULL (since all the
+ * columns need to be sent regardless); and in other cases, the columns in
+ * the REPLICA IDENTITY cannot be left out.
+ */
+static void
+check_publication_columns(Relation targetrel, Bitmapset *columns)
+{
+ Bitmapset *idattrs;
+ int x;
+
+ idattrs = RelationGetIndexAttrBitmap(targetrel,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * We have to test membership the hard way, because the values returned by
+ * RelationGetIndexAttrBitmap are offset.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ if (!bms_is_member(x + FirstLowInvalidHeapAttributeNumber, columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
+ }
+
+ bms_free(idattrs);
}
/*
@@ -289,6 +356,9 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
Oid relid = RelationGetRelid(targetrel->relation);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ Bitmapset *attset = NULL;
+ AttrNumber *attarray;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -314,7 +384,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ /* Translate column names to numbers and verify suitability */
+ attarray = publication_translate_columns(targetrel->relation,
+ targetrel->columns,
+ &natts, &attset);
+
+ check_publication_add_relation(targetrel->relation, attset);
+
+ bms_free(attset);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -327,6 +404,15 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ if (targetrel->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -337,6 +423,13 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
/* Register dependencies as needed */
ObjectAddressSet(myself, PublicationRelRelationId, pubreloid);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ pfree(attarray);
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -364,6 +457,143 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
return myself;
}
+/*
+ * Update the column list for a relation in a publication.
+ */
+void
+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns)
+{
+ Bitmapset *attset;
+ AttrNumber *attarray;
+ HeapTuple copytup;
+ int natts;
+ bool nulls[Natts_pg_publication_rel];
+ bool replaces[Natts_pg_publication_rel];
+ Datum values[Natts_pg_publication_rel];
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ replaces[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ deleteDependencyRecordsForClass(PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid,
+ RelationRelationId,
+ DEPENDENCY_AUTO);
+
+ if (columns == NULL)
+ {
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+ }
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ int2vector *prattrs;
+
+ if (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot change column set for relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot have column filter on relations with REPLICA IDENTITY FULL."));
+
+ attarray = publication_translate_columns(targetrel, columns,
+ &natts, &attset);
+
+ /*
+ * Make sure the column list checks out. XXX this should occur at
+ * caller in publicationcmds.c, not here.
+ */
+ check_publication_columns(targetrel, attset);
+ bms_free(attset);
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+
+ /* Add dependencies on the new list of columns */
+ ObjectAddressSet(myself, PublicationRelRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid);
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId,
+ RelationGetRelid(targetrel), attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ }
+
+ copytup = heap_modify_tuple(pubreltup, RelationGetDescr(pubrel),
+ values, nulls, replaces);
+
+ CatalogTupleUpdate(pubrel, &pubreltup->t_self, copytup);
+
+ heap_freetuple(copytup);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list. Other checks are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ */
+static AttrNumber *
+publication_translate_columns(Relation targetrel, List *columns, int *natts,
+ Bitmapset **attset)
+{
+ AttrNumber *attarray;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /*
+ * XXX qsort the array here, or maybe build just the bitmapset above and
+ * then scan that in order to produce the array? Do we care about the
+ * array being unsorted?
+ */
+
+ *natts = n;
+ *attset = set;
+ return attarray;
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -471,6 +701,74 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull)
+ continue;
+
+ pubs = lappend_oid(pubs,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 0f04969fd6..657374c0d1 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -376,6 +376,46 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
return myself;
}
+/*
+ * Change the column list of a relation in a publication
+ */
+static void
+PublicationSetColumns(AlterPublicationStmt *stmt,
+ Form_pg_publication pubform, PublicationTable *table)
+{
+ Relation rel,
+ urel;
+ HeapTuple tup;
+ ObjectAddress obj,
+ secondary;
+
+ rel = table_open(PublicationRelRelationId, RowExclusiveLock);
+ urel = table_openrv(table->relation, ShareUpdateExclusiveLock);
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(RelationGetRelid(urel)),
+ ObjectIdGetDatum(pubform->oid));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errmsg("relation \"%s\" is not already in publication \"%s\"",
+ table->relation->relname,
+ NameStr(pubform->pubname)));
+
+ publication_set_table_columns(rel, tup, urel, table->columns);
+
+ ObjectAddressSet(obj, PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->oid);
+ ObjectAddressSet(secondary, RelationRelationId, RelationGetRelid(urel));
+ EventTriggerCollectSimpleCommand(obj, secondary, (Node *) stmt);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+ table_close(urel, NoLock);
+
+ InvokeObjectPostAlterHook(PublicationRelationId, pubform->oid, 0);
+}
+
/*
* Change options of a publication.
*/
@@ -523,6 +563,14 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
}
else if (stmt->action == AP_DropObjects)
PublicationDropTables(pubid, rels, false);
+ else if (stmt->action == AP_SetColumns)
+ {
+ Assert(schemaidlist == NIL);
+ Assert(list_length(tables) == 1);
+
+ PublicationSetColumns(stmt, pubform,
+ linitial_node(PublicationTable, tables));
+ }
else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
@@ -562,7 +610,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
pubrel = palloc(sizeof(PublicationRelInfo));
pubrel->relation = oldrel;
-
+ /* This is not needed to delete a table */
+ pubrel->columns = NIL;
delrels = lappend(delrels, pubrel);
}
}
@@ -622,7 +671,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
}
else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* AP_SetObjects */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -645,6 +694,10 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
*/
PublicationAddSchemas(pubform->oid, schemaidlist, true, stmt);
}
+ else
+ {
+ /* Nothing to do for AP_SetColumns */
+ }
}
/*
@@ -934,6 +987,8 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
@@ -967,8 +1022,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
+
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
@@ -1076,6 +1134,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 3631b8a929..232a068613 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -8347,6 +8347,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
bool missing_ok, LOCKMODE lockmode,
ObjectAddresses *addrs)
{
+ Oid relid = RelationGetRelid(rel);
HeapTuple tuple;
Form_pg_attribute targetatt;
AttrNumber attnum;
@@ -8366,7 +8367,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/*
* get the number of the attribute
*/
- tuple = SearchSysCacheAttName(RelationGetRelid(rel), colName);
+ tuple = SearchSysCacheAttName(relid, colName);
if (!HeapTupleIsValid(tuple))
{
if (!missing_ok)
@@ -8420,13 +8421,42 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * Also, if the column is used in the column list of a publication,
+ * disallow the drop if the DROP is RESTRICT. We don't do anything if the
+ * DROP is CASCADE, which means that the dependency mechanism will remove
+ * the relation from the publication.
+ */
+ if (behavior == DROP_RESTRICT)
+ {
+ List *pubs;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(relid);
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(relid, pubid);
+
+ if (list_member_oid(published_cols, attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot drop column \"%s\" because it is part of publication \"%s\"",
+ colName, get_publication_name(pubid, false)),
+ errhint("Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication."));
+ }
+ }
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
* use find_all_inheritors to do it in one pass.
*/
children =
- find_inheritance_children(RelationGetRelid(rel), lockmode);
+ find_inheritance_children(relid, lockmode);
if (children)
{
@@ -8514,7 +8544,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Add object to delete */
object.classId = RelationRelationId;
- object.objectId = RelationGetRelid(rel);
+ object.objectId = relid;
object.objectSubId = attnum;
add_exact_object_address(&object, addrs);
@@ -15581,6 +15611,7 @@ relation_mark_replica_identity(Relation rel, char ri_type, Oid indexOid,
CatalogTupleUpdate(pg_index, &pg_index_tuple->t_self, pg_index_tuple);
InvokeObjectPostAlterHookArg(IndexRelationId, thisIndexOid, 0,
InvalidOid, is_internal);
+
/*
* Invalidate the relcache for the table, so that after we commit
* all sessions will refresh the table's replica identity index
@@ -15603,6 +15634,11 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
Oid indexOid;
Relation indexRel;
int key;
+ List *pubs;
+ Bitmapset *indexed_cols = NULL;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(RelationGetRelid(rel));
if (stmt->identity_type == REPLICA_IDENTITY_DEFAULT)
{
@@ -15611,11 +15647,16 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
}
else if (stmt->identity_type == REPLICA_IDENTITY_FULL)
{
+ if (pubs != NIL)
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set REPLICA IDENTITY FULL when publications contain relations that specify column lists"));
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
else if (stmt->identity_type == REPLICA_IDENTITY_NOTHING)
{
+ /* XXX not sure what's the right check for publications here */
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
@@ -15700,6 +15741,38 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
errmsg("index \"%s\" cannot be used as replica identity because column \"%s\" is nullable",
RelationGetRelationName(indexRel),
NameStr(attr->attname))));
+
+ /*
+ * Collect columns used, in case we have any publications that we need
+ * to vet. System attributes are disallowed so no need to subtract
+ * FirstLowInvalidHeapAttributeNumber.
+ */
+ indexed_cols = bms_add_member(indexed_cols, attno);
+ }
+
+ /*
+ * Check partial-column publications. All publications have to include
+ * all key columns of the new index.
+ */
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(RelationGetRelid(rel), pubid);
+
+ for (key = 0; key < IndexRelationGetNumberOfKeyAttributes(indexRel); key++)
+ {
+ int16 attno = indexRel->rd_index->indkey.values[key];
+
+ if (!list_member_oid(published_cols, attno))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("index \"%s\" cannot be used because publication \"%s\" does not include all indexed columns",
+ RelationGetRelationName(indexRel),
+ get_publication_name(pubid, false)));
+ }
}
/* This index is suitable for use as a replica identity. Mark it. */
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index df0b747883..0ff4c1ceac 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4833,6 +4833,7 @@ _copyPublicationTable(const PublicationTable *from)
PublicationTable *newnode = makeNode(PublicationTable);
COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index cb7ddd463c..d786a688ac 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2312,6 +2312,7 @@ static bool
_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 6dddc07947..068f67998c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9740,12 +9740,13 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr
+ TABLE relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
+ $$->pubtable->columns = $3;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9760,28 +9761,38 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId
+ | ColId opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- $$->name = $1;
+ if ($2 != NULL)
+ {
+ $$->pubtable = makeNode(PublicationTable);
+ $$->pubtable->relation = makeRangeVar(NULL, $1, @1);
+ $$->pubtable->columns = $2;
+ $$->name = NULL;
+ }
+ else
+ $$->name = $1;
$$->location = @1;
}
- | ColId indirection
+ | ColId indirection opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
+ $$->pubtable->columns = $3;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr
+ | extended_relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
+ $$->pubtable->columns = $2;
}
| CURRENT_SCHEMA
{
@@ -9807,6 +9818,9 @@ pub_obj_list: PublicationObjSpec
*
* ALTER PUBLICATION name SET pub_obj [, ...]
*
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
* pub_obj is one of:
*
* TABLE table_name [, ...]
@@ -9840,6 +9854,32 @@ AlterPublicationStmt:
n->action = AP_SetObjects;
$$ = (Node *)n;
}
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS '(' columnList ')'
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = $10;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS ALL
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = NIL;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
| ALTER PUBLICATION name DROP pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
@@ -17444,6 +17484,16 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
else if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLES_IN_SCHEMA ||
pubobj->pubobjtype == PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA)
{
+ /*
+ * This can happen if a column list is specified in a continuation
+ * for a schema entry; reject it.
+ */
+ if (pubobj->pubtable)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schemas"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 9f5bf4b639..3428984130 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,9 +29,11 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary,
+ Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +400,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +412,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -442,7 +444,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary,
+ Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -463,11 +466,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -536,7 +539,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -651,7 +654,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -673,7 +677,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -749,7 +753,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple,
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -761,7 +766,13 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ /* Don't count attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
continue;
nliveatts++;
}
@@ -783,6 +794,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
if (att->attisdropped || att->attgenerated)
continue;
+ /* Ignore attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +919,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -914,20 +929,24 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
desc = RelationGetDescr(rel);
- /* send number of live attributes */
- for (i = 0; i < desc->natts; i++)
- {
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
- continue;
- nliveatts++;
- }
- pq_sendint16(out, nliveatts);
-
/* fetch bitmap of REPLICATION IDENTITY attributes */
replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
if (!replidentfull)
idattrs = RelationGetIdentityKeyBitmap(rel);
+ /* send number of live attributes */
+ for (i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+ nliveatts++;
+ }
+ pq_sendint16(out, nliveatts);
+
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -936,7 +955,8 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
-
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index f07983a43c..35f1294ae4 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -111,6 +111,7 @@
#include "replication/origin.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -697,17 +698,20 @@ fetch_remote_table_info(char *nspname, char *relname,
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
bool isnull;
int natt;
+ ListCell *lc;
+ bool am_partition;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -737,14 +741,19 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetBool(slot_getattr(slot, 4, &isnull));
+ Assert(!isnull);
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -772,16 +781,92 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * In server versions 15 and higher, obtain the applicable column filter,
+ * if any.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData publications;
+ bool first = true;
+
+ initStringInfo(&publications);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&publications, ", ");
+ appendStringInfoString(&publications, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ " SELECT pg_catalog.unnest(prattrs)\n"
+ " FROM pg_catalog.pg_publication p JOIN\n"
+ " pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n"
+ " WHERE p.pubname IN (%s) AND\n",
+ publications.data);
+ if (!am_partition)
+ appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "prrelid IN (SELECT relid\n"
+ " FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch attribute info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ if (isnull)
+ continue;
+ included_cols = bms_add_member(included_cols, attnum);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+ pfree(publications.data);
+ walrcv_clear_result(pubres);
+ }
+
+ /*
+ * Store the column names only if they are contained in column filter.
+ * LogicalRepRelation will only contain attributes corresponding to those
+ * specified in column filters.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ continue;
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
+ Assert(!isnull);
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -791,12 +876,12 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecClearTuple(slot);
}
+
ExecDropSingleTupleTableSlot(slot);
-
- lrel->natts = natt;
-
walrcv_clear_result(res);
pfree(cmd.data);
+
+ lrel->natts = natt;
}
/*
@@ -829,8 +914,17 @@ copy_table(Relation rel)
/* Start copy on the publisher. */
initStringInfo(&cmd);
if (lrel.relkind == RELKIND_RELATION)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 6f6a203dea..34df5d4956 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,16 +15,19 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -81,7 +84,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -130,6 +134,13 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+
+ /*
+ * Set of columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this list are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -570,11 +581,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
}
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -587,7 +598,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -610,13 +622,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip if attribute is not present in column filter. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -693,7 +709,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -722,7 +738,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -1122,6 +1138,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
bool found;
+ Oid ancestor_id;
MemoryContext oldctx;
Assert(RelationSyncCache != NULL);
@@ -1142,6 +1159,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->map = NULL; /* will be set by maybe_send_schema() if
* needed */
}
@@ -1182,6 +1200,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
if (pub->alltables)
{
@@ -1192,8 +1211,6 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (!publish)
{
- bool ancestor_published = false;
-
/*
* For a partition, check if any of the ancestors are
* published. If so, note down the topmost ancestor that is
@@ -1219,6 +1236,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
pub->oid))
{
ancestor_published = true;
+ ancestor_id = ancestor;
if (pub->pubviaroot)
publish_as_relid = ancestor;
}
@@ -1239,15 +1257,47 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ Oid relid;
+ HeapTuple pub_rel_tuple;
+
+ relid = ancestor_published ? ancestor_id : publish_as_relid;
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ Datum pub_rel_cols;
+ bool isnull;
+
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP,
+ pub_rel_tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(pub_rel_cols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
+ entry->columns = bms_add_member(entry->columns,
+ elems[i]);
+ MemoryContextSwitchTo(oldctx);
+ }
+ ReleaseSysCache(pub_rel_tuple);
+ }
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
}
-
- if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
- break;
}
list_free(pubids);
@@ -1343,6 +1393,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
if (entry->map)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7af6dfa575..c7c5f3de66 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4055,6 +4055,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_oid;
int i_prpubid;
int i_prrelid;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4066,8 +4067,13 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
/* Collect all publication membership info. */
appendPQExpBufferStr(query,
- "SELECT tableoid, oid, prpubid, prrelid "
- "FROM pg_catalog.pg_publication_rel");
+ "SELECT tableoid, oid, prpubid, prrelid");
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBufferStr(query, ", prattrs");
+ else
+ appendPQExpBufferStr(query, ", NULL as prattrs");
+ appendPQExpBufferStr(query,
+ " FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
ntups = PQntuples(res);
@@ -4076,6 +4082,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_oid = PQfnumber(res, "oid");
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4117,6 +4124,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
pubrinfo[j].publication = pubinfo;
pubrinfo[j].pubtable = tbinfo;
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[i].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4191,10 +4220,12 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s;\n",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+ appendPQExpBufferStr(query, ";\n");
/*
* There is no point in creating a drop query as the drop is done by table
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index f9deb321ac..f3d3689ac9 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -630,6 +630,7 @@ typedef struct _PublicationRelInfo
DumpableObject dobj;
PublicationInfo *publication;
TableInfo *pubtable;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index c28788e84f..e9c2650b49 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5815,7 +5815,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5832,10 +5832,14 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (!as_schema) /* as table */
+ {
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
- else
+ if (!PQgetisnull(res, i, 2))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 2));
+ }
+ else /* as schema */
printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
@@ -5963,8 +5967,20 @@ describePublications(const char *pattern)
{
/* Get the tables for the specified publication */
printfPQExpBuffer(&buf,
- "SELECT n.nspname, c.relname\n"
- "FROM pg_catalog.pg_class c,\n"
+ "SELECT n.nspname, c.relname, \n");
+ if (pset.sversion >= 150000)
+ appendPQExpBufferStr(&buf,
+ " CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string"
+ "(ARRAY(SELECT attname\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END AS columns");
+ else
+ appendPQExpBufferStr(&buf, "NULL as columns");
+ appendPQExpBuffer(&buf,
+ "\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
" pg_catalog.pg_publication_rel pr\n"
"WHERE c.relnamespace = n.oid\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index cf30239f6d..25c7c08040 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1648,6 +1648,8 @@ psql_completion(const char *text, int start, int end)
/* ALTER PUBLICATION <name> ADD */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
+ else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD", "TABLE"))
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL);
/* ALTER PUBLICATION <name> DROP */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "DROP"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 902f2f2f0d..edd4f0c63c 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -86,6 +86,7 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -109,6 +110,8 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern List *GetPublicationSchemas(Oid pubid);
@@ -127,6 +130,8 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *tar
bool if_not_exists);
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern void publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index b5d5504cbb..51946cce59 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ int2vector prattrs;
+#endif
} FormData_pg_publication_rel;
/* ----------------
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 593e301f7a..f98a78c016 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3642,6 +3642,7 @@ typedef struct PublicationTable
{
NodeTag type;
RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
@@ -3678,7 +3679,8 @@ typedef enum AlterPublicationAction
{
AP_AddObjects, /* add objects to publication */
AP_DropObjects, /* remove objects from publication */
- AP_SetObjects /* set list of objects */
+ AP_SetObjects, /* set list of objects */
+ AP_SetColumns /* change list of columns for a table */
} AlterPublicationAction;
typedef struct AlterPublicationStmt
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 83741dcf42..7a5cb9871d 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -207,11 +207,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -228,7 +228,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 12c5f67080..44de5fa8a2 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -165,7 +165,29 @@ Publications:
regress_publication_user | t | t | t | f | f | f
(1 row)
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+ERROR: cannot drop column "c" because it is part of publication "testpub_fortable"
+HINT: Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5 (a);
+ERROR: column list must not be specified in ALTER PUBLICATION ... DROP
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl6"
+DETAIL: Cannot have column filter on relations with REPLICA IDENTITY FULL.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+ALTER PUBLICATION testpub_fortable
+ ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c); -- error
+ERROR: cannot change column set for relation "testpub_tbl6"
+DETAIL: Cannot have column filter on relations with REPLICA IDENTITY FULL.
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
CREATE TABLE testpub_tbl3a (b text) INHERITS (testpub_tbl3);
@@ -670,6 +692,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schemas
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 56dd358554..b625d161cb 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -89,7 +89,21 @@ SELECT pubname, puballtables FROM pg_publication WHERE pubname = 'testpub_forall
\d+ testpub_tbl2
\dRp+ testpub_foralltables
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5 (a);
+
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+ALTER PUBLICATION testpub_fortable
+ ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c); -- error
+
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
@@ -362,6 +376,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/028_column_list.pl b/src/test/subscription/t/028_column_list.pl
new file mode 100644
index 0000000000..f2b26176ed
--- /dev/null
+++ b/src/test/subscription/t/028_column_list.pl
@@ -0,0 +1,164 @@
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test partial-column publication of tables
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 9;
+
+# setup
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+ INSERT INTO tab2 VALUES (2, 'foo', 2);");
+# Test with weird column names
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, B varchar, \"c'\" int)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+#Test replication with multi-level partition
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, \"c'\" int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+#Test create publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub1 FOR TABLE tab1(a, \"B\"), tab3(\"a'\",\"c'\"), test_part(a,b)");
+
+my $result = $node_publisher->safe_psql('postgres',
+ "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
+is($result, qq(tab1|1 2
+tab3|1 3
+test_part|1 2), 'publication relation updated');
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+);
+#Initial sync
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab1 VALUES (1,2,3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 VALUES (1,2,3)");
+#Test for replication of partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (1,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (2,'bcd', '2021-07-03 11:12:13')");
+#Test for replication of multi-level partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (4,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (5,'bcd', '2021-07-03 11:12:13')");
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'insert on column tab1.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab3");
+is($result, qq(1|3), 'insert on column tab3.b is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part");
+is($result, qq(1|abc\n2|bcd\n4|abc\n5|bcd), 'insert on all columns is replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab1 SET c = 5 where a = 1");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'update on column tab1.c is not replicated');
+
+# Verify user-defined types
+$node_publisher->safe_psql('postgres',
+ qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+ ALTER PUBLICATION pub1 ADD TABLE test_tab4 (a, b, d);
+ });
+$node_subscriber->safe_psql('postgres',
+ qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, d text);
+ });
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab4 VALUES (1, 'red', 3, 'oh my');");
+
+#Test alter publication with column filtering
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION"
+);
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 VALUES (1,'abc',3)");
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET c = 5 where a = 2");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 WHERE a = 1");
+is($result, qq(1|abc), 'insert on column tab2.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 WHERE a = 2");
+is($result, qq(2|foo), 'update on column tab2.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_tab4");
+is($result, qq(1|red|oh my), 'insert on table with user-defined type');
+
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d)");
+$node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2, pub3");
+$node_publisher->wait_for_catchup('sub2');
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (1, 11, 111, 1111)");
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (2, 22, 222, 2222)");
+$node_publisher->wait_for_catchup('sub2');
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5;"),
+ qq(1|11|1111
+2|22|2222),
+ 'overlapping publications with overlapping column lists');
--
2.30.2
On Mon, Jan 03, 2022 at 11:31:39AM -0300, Alvaro Herrera wrote:
Did the DatumGetBool issue expose a deficiency in testing ?
I think the !am_partition path was never being hit.Hmm, the TAP test creates a subscription that contains both types of
tables. I tried adding an assert for each case, and they were both hit
on running the test.
Yes, I know both paths are hit now that it uses GetBool.
What I'm wondering is why tests didn't fail when one path wasn't hit - when it
said am_partition=DatumGetChar(); if (!am_partition){}
I suppose it's because the am_partition=true case correctly handles
nonpartitions.
Maybe the !am_partition case should be removed, and add a comment that
pg_partition_tree(pg_partition_root(%u))) also handles non-partitions.
Or maybe that's inefficient...
--
Justin
On 2022-Jan-03, Justin Pryzby wrote:
Yes, I know both paths are hit now that it uses GetBool.
What I'm wondering is why tests didn't fail when one path wasn't hit - when it
said am_partition=DatumGetChar(); if (!am_partition){}
Ah!
I suppose it's because the am_partition=true case correctly handles
nonpartitions.Maybe the !am_partition case should be removed, and add a comment that
pg_partition_tree(pg_partition_root(%u))) also handles non-partitions.
Or maybe that's inefficient...
Hmm, that doesn't sound true. Running the query manually, you get an
empty list if you use pg_partition_tree(pg_partition_root) with a
non-partition. Maybe what was happening is that all columns were being
transmitted instead of only the required columns. Maybe you're right
that the test isn't complete enough.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
On Mon, Jan 3, 2022 at 8:01 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
fetch_remote_table_info()
{
..
+ appendStringInfo(&cmd,
+ " SELECT pg_catalog.unnest(prattrs)\n"
+ " FROM pg_catalog.pg_publication p JOIN\n"
+ " pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n"
+ " WHERE p.pubname IN (%s) AND\n",
+ publications.data);
+ if (!am_partition)
+ appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "prrelid IN (SELECT relid\n"
+ " FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);
IIUC, this doesn't deal with cases when some publication has not
specified table attrs. In those cases, I think it should return all
attrs? Also, it is not very clear to me what exactly we want to do
with partitions?
--
With Regards,
Amit Kapila.
On Mon, Dec 27, 2021 at 10:36 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
Determining that an array has a NULL element seems convoluted. I ended
up with this query, where comparing the result of array_positions() with
an empty array does that. If anybody knows of a simpler way, or any
situations in which this fails, I'm all ears.with published_cols as (
select case when
pg_catalog.array_positions(pg_catalog.array_agg(unnest), null) <> '{}' then null else
pg_catalog.array_agg(distinct unnest order by unnest) end AS attrs
from pg_catalog.pg_publication p join
pg_catalog.pg_publication_rel pr on (p.oid = pr.prpubid) left join
unnest(prattrs) on (true)
where prrelid = 38168 and p.pubname in ('pub1', 'pub2')
)
SELECT a.attname,
a.atttypid,
a.attnum = ANY(i.indkey)
FROM pg_catalog.pg_attribute a
LEFT JOIN pg_catalog.pg_index i
ON (i.indexrelid = pg_get_replica_identity_index(38168)),
published_cols
WHERE a.attnum > 0::pg_catalog.int2
AND NOT a.attisdropped and a.attgenerated = ''
AND a.attrelid = 38168
AND (published_cols.attrs IS NULL OR attnum = ANY(published_cols.attrs))
ORDER BY a.attnum;This returns all columns if at least one publication has a NULL prattrs,
or only the union of columns listed in all publications, if all
publications have a list of columns.
Considering this, don't we need to deal with "For All Tables" and "For
All Tables In Schema .." Publications in this query? The row filter
patch deal with such cases. The row filter patch handles the NULL case
via C code which makes the query relatively simpler. I am not sure if
the same logic can be used here but having a simple query here have
merit that if we want to use a single query to fetch both column and
row filters then we should be able to enhance it without making it
further complicated.
(I was worried about obtaining the list of publications, but it turns
out that it's already as a convenient list of OIDs in the MySubscription
struct.)With this, we can remove the second query added by Rahila's original patch to
filter out nonpublished columns.I still need to add pg_partition_tree() in order to search for
publications containing a partition ancestor. I'm not yet sure what
happens (and what *should* happen) if an ancestor is part of a
publication and the partition is also part of a publication, and the
column lists differ.
Shouldn't we try to have a behavior similar to the row filter patch
for this case? The row filter patch behavior is as follows: "If your
publication contains a partitioned table, the publication parameter
publish_via_partition_root determines if it uses the partition row
filter (if the parameter is false, the default) or the root
partitioned table row filter. During initial tablesync, it doesn't do
any special handling for partitions.
--
With Regards,
Amit Kapila.
On 2022-Jan-06, Amit Kapila wrote:
On Mon, Jan 3, 2022 at 8:01 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
fetch_remote_table_info() { .. + appendStringInfo(&cmd, + " SELECT pg_catalog.unnest(prattrs)\n" + " FROM pg_catalog.pg_publication p JOIN\n" + " pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n" + " WHERE p.pubname IN (%s) AND\n", + publications.data); + if (!am_partition) + appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid); + else + appendStringInfo(&cmd, + "prrelid IN (SELECT relid\n" + " FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))", + lrel->remoteid);IIUC, this doesn't deal with cases when some publication has not
specified table attrs. In those cases, I think it should return all
attrs?
Hmm, no, the idea here is that the list of columns should be null; the
code that uses this result is supposed to handle a null result to mean
hat all columns are included.
Also, it is not very clear to me what exactly we want to do
with partitions?
... Hmm, maybe there is a gap in testing here, I'll check; but the idea
is that we would use the column list of the most immediate ancestor that
has one, if the partition itself doesn't have one. (I see we're missing
a check for "pubviaroot", which should represent an override. Need more
tests here.)
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Uno puede defenderse de los ataques; contra los elogios se esta indefenso"
On 2022-Jan-06, Amit Kapila wrote:
Considering this, don't we need to deal with "For All Tables" and "For
All Tables In Schema .." Publications in this query? The row filter
patch deal with such cases. The row filter patch handles the NULL case
via C code which makes the query relatively simpler.
Yes. I realized after sending that email that the need to handle schema
publications would make a single query very difficult, so I ended up
splitting it again in two queries, which is what you see in the latest
version submitted.
I am not sure if the same logic can be used here but having a simple
query here have merit that if we want to use a single query to fetch
both column and row filters then we should be able to enhance it
without making it further complicated.
I have looked the row filter code a couple of times to make sure we're
somewhat compatible, but didn't look closely enough to see if we can
make the queries added by both patches into a single one.
Shouldn't we try to have a behavior similar to the row filter patch
for this case? The row filter patch behavior is as follows: "If your
publication contains a partitioned table, the publication parameter
publish_via_partition_root determines if it uses the partition row
filter (if the parameter is false, the default) or the root
partitioned table row filter. During initial tablesync, it doesn't do
any special handling for partitions.
I'll have a look.
Thanks for looking!
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
I think this is getting pretty good now. I like the overall behavior now.
Some details:
There are still a few references to "filter", but I see most of the
patch now uses column list or something. Maybe do another cleanup
pass before finalizing the patch.
doc/src/sgml/catalogs.sgml needs to be updated.
doc/src/sgml/ref/alter_publication.sgml:
"allows to change" -> "allows changing"
src/backend/catalog/pg_publication.c:
publication_translate_columns(): I find the style of having a couple
of output arguments plus a return value that is actually another
output value confusing. (It would be different if the return value
was some kind of success value.) Let's make it all output arguments.
About the XXX question there: I would make the column numbers always
sorted. I don't have a strong reason for this, but otherwise we might
get version differences, unstable dumps etc. It doesn't seem
complicated to keep this a bit cleaner.
I think publication_translate_columns() also needs to prohibit
generated columns. We already exclude those implicitly throughout the
logical replication code, but if a user explicitly set one here,
things would probably break.
src/backend/commands/tablecmds.c:
ATExecReplicaIdentity(): Regarding the question of how to handle
REPLICA_IDENTITY_NOTHING: I see two ways to do this. Right now, the
approach is that the user can set the replica identity freely, and we
decide later based on that what we can replicate (e.g., no updates).
For this patch, that would mean we don't restrict what columns can be
in the column list, but we check what actions we can replicate based
on the column list. The alternative is that we require the column
list to include the replica identity, as the patch is currently doing,
which would mean that REPLICA_IDENTITY_NOTHING can be allowed since
it's essentially a set of zero columns.
I find the current behavior a bit weird on reflection. If a user
wants to replicate on some columns and only INSERTs, that should be
allowed regardless of what the replica identity columns are.
src/backend/replication/pgoutput/pgoutput.c:
In get_rel_sync_entry(), why did you remove the block
- if (entry->pubactions.pubinsert &&
entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete &&
entry->pubactions.pubtruncate)
- break;
Maybe this is intentional, but it's not clear to me.
On Fri, Jan 7, 2022 at 5:16 PM Peter Eisentraut
<peter.eisentraut@enterprisedb.com> wrote:
src/backend/commands/tablecmds.c:
ATExecReplicaIdentity(): Regarding the question of how to handle
REPLICA_IDENTITY_NOTHING: I see two ways to do this. Right now, the
approach is that the user can set the replica identity freely, and we
decide later based on that what we can replicate (e.g., no updates).
+1. This is what we are trying to do with the row filter patch. It
seems Hou-San has also mentioned the same on this thread [1]/messages/by-id/OS0PR01MB5716330FFE3803DF887D073C94789@OS0PR01MB5716.jpnprd01.prod.outlook.com.
For this patch, that would mean we don't restrict what columns can be
in the column list, but we check what actions we can replicate based
on the column list. The alternative is that we require the column
list to include the replica identity, as the patch is currently doing,
which would mean that REPLICA_IDENTITY_NOTHING can be allowed since
it's essentially a set of zero columns.I find the current behavior a bit weird on reflection. If a user
wants to replicate on some columns and only INSERTs, that should be
allowed regardless of what the replica identity columns are.
Right, I also raised the same point [2]/messages/by-id/CAA4eK1+FoJ-J7wUG5s8zCtY0iBuN9LcjQcYhV4BD17xhuHfoug@mail.gmail.com related to INSERTs.
[1]: /messages/by-id/OS0PR01MB5716330FFE3803DF887D073C94789@OS0PR01MB5716.jpnprd01.prod.outlook.com
[2]: /messages/by-id/CAA4eK1+FoJ-J7wUG5s8zCtY0iBuN9LcjQcYhV4BD17xhuHfoug@mail.gmail.com
--
With Regards,
Amit Kapila.
In this version I have addressed these points, except the REPLICA
IDENTITY NOTHING stuff.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
"The eagle never lost so much time, as
when he submitted to learn of the crow." (William Blake)
Attachments:
v16-0001-Support-column-lists-for-logical-replication-of-.patchtext/x-diff; charset=utf-8Download
From e907582e4cd249850dc3784fa4e21b8c1448e99f Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 6 Sep 2021 10:34:29 -0300
Subject: [PATCH v16] Support column lists for logical replication of tables
Add the capability of specifying a column list for individual tables as
part of a publication. Columns not in the list are not published. This
enables replicating to a table with only a subset of the columns.
If no column list is specified, all the columns are replicated, as
previously
Author: Rahila Syed <rahilasyed90@gmail.com>
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 13 +
doc/src/sgml/protocol.sgml | 4 +-
doc/src/sgml/ref/alter_publication.sgml | 20 +-
doc/src/sgml/ref/create_publication.sgml | 11 +-
src/backend/catalog/pg_publication.c | 321 +++++++++++++++++++-
src/backend/commands/publicationcmds.c | 67 +++-
src/backend/commands/tablecmds.c | 79 ++++-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 60 +++-
src/backend/replication/logical/proto.c | 66 ++--
src/backend/replication/logical/tablesync.c | 119 +++++++-
src/backend/replication/pgoutput/pgoutput.c | 118 ++++++-
src/bin/pg_dump/pg_dump.c | 41 ++-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 26 +-
src/bin/psql/tab-complete.c | 2 +
src/include/catalog/pg_publication.h | 5 +
src/include/catalog/pg_publication_rel.h | 3 +
src/include/nodes/parsenodes.h | 4 +-
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 36 ++-
src/test/regress/sql/publication.sql | 22 +-
src/test/subscription/t/028_column_list.pl | 164 ++++++++++
24 files changed, 1105 insertions(+), 85 deletions(-)
create mode 100644 src/test/subscription/t/028_column_list.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 03e2537b07..b7b75f64a2 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6311,6 +6311,19 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to relation
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>prattrs</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+ </para>
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all attributes are published.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 34a7034282..5bc2e7a591 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -6877,7 +6877,9 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column (except
+ generated columns and other columns that don't appear in the column
+ filter list, for tables that have one):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index bb4ef5e5e2..d0e97243b8 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -25,12 +25,13 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD <replace
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ALTER TABLE <replaceable class="parameter">publication_object</replaceable> SET COLUMNS { ( <replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -62,6 +63,11 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
command retain their previous settings.
</para>
+ <para>
+ The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows changing
+ the set of columns that are included in the publication.
+ </para>
+
<para>
The remaining variants change the owner and the name of the publication.
</para>
@@ -110,6 +116,8 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
</para>
</listitem>
</varlistentry>
@@ -164,9 +172,15 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
</para>
<para>
- Add some tables to the publication:
+ Add tables to the publication:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
+<programlisting>
+ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS (user_id, firstname, lastname);
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index d805e8e77a..73a23cbb02 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -78,6 +78,15 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
publication, so they are never explicitly added to the publication.
</para>
+ <para>
+ When a column list is specified, only the listed columns are replicated;
+ any other columns are ignored for the purpose of replication through
+ this publication. If no column list is specified, all columns of the
+ table are replicated through this publication, including any columns
+ added later. If a column list is specified, it must include the replica
+ identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 2992a2e0c6..ef4e9b6ab0 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,13 +45,23 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+
+static void check_publication_columns(Relation targetrel, Bitmapset *columns);
+static AttrNumber *publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, Bitmapset **attset);
+
/*
- * Check if relation can be in given publication and throws appropriate
- * error if not.
+ * Check if relation can be in given publication and that the column
+ * filter is sensible, and throws appropriate error if not.
+ *
+ * targetcols is the bitmapset of attribute numbers given in the column list,
+ * or NULL if it was not specified.
*/
static void
-check_publication_add_relation(Relation targetrel)
+check_publication_add_relation(Relation targetrel, Bitmapset *columns)
{
+ bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
@@ -82,6 +92,63 @@ check_publication_add_relation(Relation targetrel)
errmsg("cannot add relation \"%s\" to publication",
RelationGetRelationName(targetrel)),
errdetail("This operation is not supported for unlogged tables.")));
+
+ /* Make sure the column list checks out */
+ if (columns != NULL)
+ {
+ /*
+ * Even if the user listed all columns in the column list, we cannot
+ * allow a column list to be specified when REPLICA IDENTITY is FULL;
+ * that would cause problems if a new column is added later, because
+ * the new column would have to be included (because of being part of
+ * the replica identity) but it's technically not allowed (because of
+ * not being in the publication's column list yet). So reject this
+ * case altogether.
+ */
+ if (replidentfull)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot specify a column list on relations with REPLICA IDENTITY FULL."));
+
+ check_publication_columns(targetrel, columns);
+ }
+}
+
+/*
+ * Enforce that the column list can only leave out columns that aren't
+ * forced to be sent.
+ *
+ * No column can be excluded if REPLICA IDENTITY is FULL (since all the
+ * columns need to be sent regardless); and in other cases, the columns in
+ * the REPLICA IDENTITY cannot be left out.
+ */
+static void
+check_publication_columns(Relation targetrel, Bitmapset *columns)
+{
+ Bitmapset *idattrs;
+ int x;
+
+ idattrs = RelationGetIndexAttrBitmap(targetrel,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * We have to test membership the hard way, because the values returned by
+ * RelationGetIndexAttrBitmap are offset.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ if (!bms_is_member(x + FirstLowInvalidHeapAttributeNumber, columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
+ }
+
+ bms_free(idattrs);
}
/*
@@ -289,6 +356,9 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
Oid relid = RelationGetRelid(targetrel->relation);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ Bitmapset *attset = NULL;
+ AttrNumber *attarray;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -314,7 +384,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ /* Translate column names to numbers and verify suitability */
+ attarray = publication_translate_columns(targetrel->relation,
+ targetrel->columns,
+ &natts, &attset);
+
+ check_publication_add_relation(targetrel->relation, attset);
+
+ bms_free(attset);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -327,6 +404,15 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ if (targetrel->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -337,6 +423,13 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
/* Register dependencies as needed */
ObjectAddressSet(myself, PublicationRelRelationId, pubreloid);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ pfree(attarray);
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -364,6 +457,158 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
return myself;
}
+/*
+ * Update the column list for a relation in a publication.
+ */
+void
+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns)
+{
+ Bitmapset *attset;
+ AttrNumber *attarray;
+ HeapTuple copytup;
+ int natts;
+ bool nulls[Natts_pg_publication_rel];
+ bool replaces[Natts_pg_publication_rel];
+ Datum values[Natts_pg_publication_rel];
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ replaces[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ deleteDependencyRecordsForClass(PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid,
+ RelationRelationId,
+ DEPENDENCY_AUTO);
+
+ if (columns == NULL)
+ {
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+ }
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ int2vector *prattrs;
+
+ if (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot change column set for relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot specify column list on relations with REPLICA IDENTITY FULL."));
+
+ attarray = publication_translate_columns(targetrel, columns,
+ &natts, &attset);
+
+ /*
+ * Make sure the column list checks out. XXX this should occur at
+ * caller in publicationcmds.c, not here.
+ */
+ check_publication_columns(targetrel, attset);
+ bms_free(attset);
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+
+ /* Add dependencies on the new list of columns */
+ ObjectAddressSet(myself, PublicationRelRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid);
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId,
+ RelationGetRelid(targetrel), attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ }
+
+ copytup = heap_modify_tuple(pubreltup, RelationGetDescr(pubrel),
+ values, nulls, replaces);
+
+ CatalogTupleUpdate(pubrel, &pubreltup->t_self, copytup);
+
+ heap_freetuple(copytup);
+}
+
+/* qsort comparator for attnums */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list. Other checks are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ */
+static AttrNumber *
+publication_translate_columns(Relation targetrel, List *columns, int *natts,
+ Bitmapset **attset)
+{
+ AttrNumber *attarray;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference generated column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /* Be tidy, so that the catalog representation is always sorted */
+ qsort(attarray, n, sizeof(AttrNumber), compare_int16);
+
+ *natts = n;
+ *attset = set;
+ return attarray;
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -471,6 +716,74 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull)
+ continue;
+
+ pubs = lappend_oid(pubs,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 3ab1bdeae1..aa41940d1b 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -376,6 +376,46 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
return myself;
}
+/*
+ * Change the column list of a relation in a publication
+ */
+static void
+PublicationSetColumns(AlterPublicationStmt *stmt,
+ Form_pg_publication pubform, PublicationTable *table)
+{
+ Relation rel,
+ urel;
+ HeapTuple tup;
+ ObjectAddress obj,
+ secondary;
+
+ rel = table_open(PublicationRelRelationId, RowExclusiveLock);
+ urel = table_openrv(table->relation, ShareUpdateExclusiveLock);
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(RelationGetRelid(urel)),
+ ObjectIdGetDatum(pubform->oid));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errmsg("relation \"%s\" is not already in publication \"%s\"",
+ table->relation->relname,
+ NameStr(pubform->pubname)));
+
+ publication_set_table_columns(rel, tup, urel, table->columns);
+
+ ObjectAddressSet(obj, PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->oid);
+ ObjectAddressSet(secondary, RelationRelationId, RelationGetRelid(urel));
+ EventTriggerCollectSimpleCommand(obj, secondary, (Node *) stmt);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+ table_close(urel, NoLock);
+
+ InvokeObjectPostAlterHook(PublicationRelationId, pubform->oid, 0);
+}
+
/*
* Change options of a publication.
*/
@@ -523,6 +563,14 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
}
else if (stmt->action == AP_DropObjects)
PublicationDropTables(pubid, rels, false);
+ else if (stmt->action == AP_SetColumns)
+ {
+ Assert(schemaidlist == NIL);
+ Assert(list_length(tables) == 1);
+
+ PublicationSetColumns(stmt, pubform,
+ linitial_node(PublicationTable, tables));
+ }
else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
@@ -562,7 +610,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
pubrel = palloc(sizeof(PublicationRelInfo));
pubrel->relation = oldrel;
-
+ /* This is not needed to delete a table */
+ pubrel->columns = NIL;
delrels = lappend(delrels, pubrel);
}
}
@@ -622,7 +671,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
}
else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* AP_SetObjects */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -645,6 +694,10 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
*/
PublicationAddSchemas(pubform->oid, schemaidlist, true, stmt);
}
+ else
+ {
+ /* Nothing to do for AP_SetColumns */
+ }
}
/*
@@ -934,6 +987,8 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
@@ -967,8 +1022,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
+
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
@@ -1076,6 +1134,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 1f0654c2f5..aba319431e 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -8364,6 +8364,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
bool missing_ok, LOCKMODE lockmode,
ObjectAddresses *addrs)
{
+ Oid relid = RelationGetRelid(rel);
HeapTuple tuple;
Form_pg_attribute targetatt;
AttrNumber attnum;
@@ -8383,7 +8384,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/*
* get the number of the attribute
*/
- tuple = SearchSysCacheAttName(RelationGetRelid(rel), colName);
+ tuple = SearchSysCacheAttName(relid, colName);
if (!HeapTupleIsValid(tuple))
{
if (!missing_ok)
@@ -8437,13 +8438,42 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * Also, if the column is used in the column list of a publication,
+ * disallow the drop if the DROP is RESTRICT. We don't do anything if the
+ * DROP is CASCADE, which means that the dependency mechanism will remove
+ * the relation from the publication.
+ */
+ if (behavior == DROP_RESTRICT)
+ {
+ List *pubs;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(relid);
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(relid, pubid);
+
+ if (list_member_oid(published_cols, attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot drop column \"%s\" because it is part of publication \"%s\"",
+ colName, get_publication_name(pubid, false)),
+ errhint("Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication."));
+ }
+ }
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
* use find_all_inheritors to do it in one pass.
*/
children =
- find_inheritance_children(RelationGetRelid(rel), lockmode);
+ find_inheritance_children(relid, lockmode);
if (children)
{
@@ -8531,7 +8561,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Add object to delete */
object.classId = RelationRelationId;
- object.objectId = RelationGetRelid(rel);
+ object.objectId = relid;
object.objectSubId = attnum;
add_exact_object_address(&object, addrs);
@@ -15841,6 +15871,7 @@ relation_mark_replica_identity(Relation rel, char ri_type, Oid indexOid,
CatalogTupleUpdate(pg_index, &pg_index_tuple->t_self, pg_index_tuple);
InvokeObjectPostAlterHookArg(IndexRelationId, thisIndexOid, 0,
InvalidOid, is_internal);
+
/*
* Invalidate the relcache for the table, so that after we commit
* all sessions will refresh the table's replica identity index
@@ -15863,6 +15894,11 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
Oid indexOid;
Relation indexRel;
int key;
+ List *pubs;
+ Bitmapset *indexed_cols = NULL;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(RelationGetRelid(rel));
if (stmt->identity_type == REPLICA_IDENTITY_DEFAULT)
{
@@ -15871,11 +15907,16 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
}
else if (stmt->identity_type == REPLICA_IDENTITY_FULL)
{
+ if (pubs != NIL)
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set REPLICA IDENTITY FULL when publications contain relations that specify column lists"));
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
else if (stmt->identity_type == REPLICA_IDENTITY_NOTHING)
{
+ /* XXX not sure what's the right check for publications here */
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
@@ -15960,6 +16001,38 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
errmsg("index \"%s\" cannot be used as replica identity because column \"%s\" is nullable",
RelationGetRelationName(indexRel),
NameStr(attr->attname))));
+
+ /*
+ * Collect columns used, in case we have any publications that we need
+ * to vet. System attributes are disallowed so no need to subtract
+ * FirstLowInvalidHeapAttributeNumber.
+ */
+ indexed_cols = bms_add_member(indexed_cols, attno);
+ }
+
+ /*
+ * Check partial-column publications. All publications have to include
+ * all key columns of the new index.
+ */
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(RelationGetRelid(rel), pubid);
+
+ for (key = 0; key < IndexRelationGetNumberOfKeyAttributes(indexRel); key++)
+ {
+ int16 attno = indexRel->rd_index->indkey.values[key];
+
+ if (!list_member_oid(published_cols, attno))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("index \"%s\" cannot be used because publication \"%s\" does not include all indexed columns",
+ RelationGetRelationName(indexRel),
+ get_publication_name(pubid, false)));
+ }
}
/* This index is suitable for use as a replica identity. Mark it. */
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 456d563f34..aa333fcdf5 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4834,6 +4834,7 @@ _copyPublicationTable(const PublicationTable *from)
PublicationTable *newnode = makeNode(PublicationTable);
COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 53beef1488..3119f7836c 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2312,6 +2312,7 @@ static bool
_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 879018377b..c1f3d6a8c8 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9740,12 +9740,13 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr
+ TABLE relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
+ $$->pubtable->columns = $3;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9760,28 +9761,38 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId
+ | ColId opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- $$->name = $1;
+ if ($2 != NULL)
+ {
+ $$->pubtable = makeNode(PublicationTable);
+ $$->pubtable->relation = makeRangeVar(NULL, $1, @1);
+ $$->pubtable->columns = $2;
+ $$->name = NULL;
+ }
+ else
+ $$->name = $1;
$$->location = @1;
}
- | ColId indirection
+ | ColId indirection opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
+ $$->pubtable->columns = $3;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr
+ | extended_relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
+ $$->pubtable->columns = $2;
}
| CURRENT_SCHEMA
{
@@ -9807,6 +9818,9 @@ pub_obj_list: PublicationObjSpec
*
* ALTER PUBLICATION name SET pub_obj [, ...]
*
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
* pub_obj is one of:
*
* TABLE table_name [, ...]
@@ -9840,6 +9854,32 @@ AlterPublicationStmt:
n->action = AP_SetObjects;
$$ = (Node *)n;
}
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS '(' columnList ')'
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = $10;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS ALL
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = NIL;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
| ALTER PUBLICATION name DROP pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
@@ -17444,6 +17484,16 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
else if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLES_IN_SCHEMA ||
pubobj->pubobjtype == PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA)
{
+ /*
+ * This can happen if a column list is specified in a continuation
+ * for a schema entry; reject it.
+ */
+ if (pubobj->pubtable)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schemas"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 953942692c..e6da46d83e 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,9 +29,11 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary,
+ Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +400,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +412,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -442,7 +444,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary,
+ Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -463,11 +466,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -536,7 +539,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -651,7 +654,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -673,7 +677,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -749,7 +753,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple,
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -761,7 +766,13 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ /* Don't count attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
continue;
nliveatts++;
}
@@ -783,6 +794,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
if (att->attisdropped || att->attgenerated)
continue;
+ /* Ignore attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +919,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -914,20 +929,24 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
desc = RelationGetDescr(rel);
- /* send number of live attributes */
- for (i = 0; i < desc->natts; i++)
- {
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
- continue;
- nliveatts++;
- }
- pq_sendint16(out, nliveatts);
-
/* fetch bitmap of REPLICATION IDENTITY attributes */
replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
if (!replidentfull)
idattrs = RelationGetIdentityKeyBitmap(rel);
+ /* send number of live attributes */
+ for (i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+ nliveatts++;
+ }
+ pq_sendint16(out, nliveatts);
+
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -936,7 +955,8 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
-
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index e596b69d46..a7befd712a 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -112,6 +112,7 @@
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "utils/acl.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -699,17 +700,20 @@ fetch_remote_table_info(char *nspname, char *relname,
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
bool isnull;
int natt;
+ ListCell *lc;
+ bool am_partition;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -739,14 +743,19 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetBool(slot_getattr(slot, 4, &isnull));
+ Assert(!isnull);
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -774,16 +783,91 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * In server versions 15 and higher, obtain the publication column list,
+ * if any.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData publications;
+ bool first = true;
+
+ initStringInfo(&publications);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&publications, ", ");
+ appendStringInfoString(&publications, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ " SELECT pg_catalog.unnest(prattrs)\n"
+ " FROM pg_catalog.pg_publication p JOIN\n"
+ " pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n"
+ " WHERE p.pubname IN (%s) AND\n",
+ publications.data);
+ if (!am_partition)
+ appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "prrelid IN (SELECT relid\n"
+ " FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch attribute info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ if (isnull)
+ continue;
+ included_cols = bms_add_member(included_cols, attnum);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+ pfree(publications.data);
+ walrcv_clear_result(pubres);
+ }
+
+ /*
+ * Store the columns as a list of names. Ignore those that are not
+ * present in the column list, if there is one.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ continue;
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
+ Assert(!isnull);
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -793,12 +877,12 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecClearTuple(slot);
}
+
ExecDropSingleTupleTableSlot(slot);
-
- lrel->natts = natt;
-
walrcv_clear_result(res);
pfree(cmd.data);
+
+ lrel->natts = natt;
}
/*
@@ -831,8 +915,17 @@ copy_table(Relation rel)
/* Start copy on the publisher. */
initStringInfo(&cmd);
if (lrel.relkind == RELKIND_RELATION)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index af8d51aee9..bdab0b1c8d 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,16 +15,19 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -81,7 +84,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -130,6 +134,13 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+
+ /*
+ * Set of columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this list are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -570,11 +581,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
}
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -587,7 +598,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -610,13 +622,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip this attribute if it's not present in the column list */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -693,7 +709,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -722,7 +738,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -1120,6 +1136,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
{
RelationSyncEntry *entry;
bool found;
+ Oid ancestor_id;
MemoryContext oldctx;
Assert(RelationSyncCache != NULL);
@@ -1140,6 +1157,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->map = NULL; /* will be set by maybe_send_schema() if
* needed */
}
@@ -1175,13 +1193,16 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
/*
* Build publication cache. We can't use one provided by relcache as
- * relcache considers all publications given relation is in, but here
- * we only need to consider ones that the subscriber requested.
+ * relcache considers all publications that the given relation is in,
+ * but here we only need to consider ones that the subscriber
+ * requested.
*/
foreach(lc, data->publications)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
+ bool all_columns = false;
if (pub->alltables)
{
@@ -1192,8 +1213,6 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (!publish)
{
- bool ancestor_published = false;
-
/*
* For a partition, check if any of the ancestors are
* published. If so, note down the topmost ancestor that is
@@ -1219,6 +1238,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
pub->oid))
{
ancestor_published = true;
+ ancestor_id = ancestor;
if (pub->pubviaroot)
publish_as_relid = ancestor;
}
@@ -1232,9 +1252,13 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
* Don't publish changes for partitioned tables, because
- * publishing those of its partitions suffices, unless partition
- * changes won't be published due to pubviaroot being set.
+ * publishing those of its partitions suffices. (However, ignore
+ * this if partition changes are not to published due to
+ * pubviaroot being set.)
*/
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
@@ -1243,10 +1267,74 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
+
+ /*
+ * Obtain columns published by this publication, and add them
+ * to the list for this rel. Note that if at least one
+ * publication has a empty column list, that means to publish
+ * everything; so if we saw a publication that includes all
+ * columns, skip this.
+ */
+ if (!all_columns)
+ {
+ HeapTuple pub_rel_tuple;
+ Oid relid;
+
+ relid = ancestor_published ? ancestor_id : publish_as_relid;
+
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ Datum pub_rel_cols;
+ bool isnull;
+
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP,
+ pub_rel_tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull)
+ {
+ /*
+ * If we see a publication with no columns, reset the
+ * list and ignore further ones.
+ */
+ all_columns = true;
+ bms_free(entry->columns);
+ entry->columns = NULL;
+ }
+ else if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(pub_rel_cols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
+ entry->columns = bms_add_member(entry->columns,
+ elems[i]);
+ MemoryContextSwitchTo(oldctx);
+ }
+
+ ReleaseSysCache(pub_rel_tuple);
+ }
+ }
}
+ /*
+ * If we've seen all action bits, and we know that all columns are
+ * published, there's no reason to look at further publications.
+ */
if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
+ entry->pubactions.pubdelete && entry->pubactions.pubtruncate &&
+ all_columns)
break;
}
@@ -1343,6 +1431,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
if (entry->map)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 92ab95724d..d13570f5aa 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4042,6 +4042,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_oid;
int i_prpubid;
int i_prrelid;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4053,8 +4054,13 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
/* Collect all publication membership info. */
appendPQExpBufferStr(query,
- "SELECT tableoid, oid, prpubid, prrelid "
- "FROM pg_catalog.pg_publication_rel");
+ "SELECT tableoid, oid, prpubid, prrelid");
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBufferStr(query, ", prattrs");
+ else
+ appendPQExpBufferStr(query, ", NULL as prattrs");
+ appendPQExpBufferStr(query,
+ " FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
ntups = PQntuples(res);
@@ -4063,6 +4069,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_oid = PQfnumber(res, "oid");
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4104,6 +4111,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
pubrinfo[j].publication = pubinfo;
pubrinfo[j].pubtable = tbinfo;
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[i].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4178,10 +4207,12 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s;\n",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+ appendPQExpBufferStr(query, ";\n");
/*
* There is no point in creating a drop query as the drop is done by table
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 066a129ee5..857f2891fc 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -630,6 +630,7 @@ typedef struct _PublicationRelInfo
DumpableObject dobj;
PublicationInfo *publication;
TableInfo *pubtable;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 8587b19160..37faf4bef4 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5823,7 +5823,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5840,10 +5840,14 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (!as_schema) /* as table */
+ {
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
- else
+ if (!PQgetisnull(res, i, 2))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 2));
+ }
+ else /* as schema */
printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
@@ -5971,8 +5975,20 @@ describePublications(const char *pattern)
{
/* Get the tables for the specified publication */
printfPQExpBuffer(&buf,
- "SELECT n.nspname, c.relname\n"
- "FROM pg_catalog.pg_class c,\n"
+ "SELECT n.nspname, c.relname, \n");
+ if (pset.sversion >= 150000)
+ appendPQExpBufferStr(&buf,
+ " CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string"
+ "(ARRAY(SELECT attname\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END AS columns");
+ else
+ appendPQExpBufferStr(&buf, "NULL as columns");
+ appendPQExpBuffer(&buf,
+ "\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
" pg_catalog.pg_publication_rel pr\n"
"WHERE c.relnamespace = n.oid\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 39be6f556a..20c852cdf9 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1657,6 +1657,8 @@ psql_completion(const char *text, int start, int end)
/* ALTER PUBLICATION <name> ADD */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
+ else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD", "TABLE"))
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL);
/* ALTER PUBLICATION <name> DROP */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "DROP"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 841b9b6c25..5ca0f0f3fc 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -86,6 +86,7 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -109,6 +110,8 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
extern List *GetPublicationSchemas(Oid pubid);
@@ -127,6 +130,8 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *tar
bool if_not_exists);
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern void publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index 0ff3716225..151644b870 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ int2vector prattrs;
+#endif
} FormData_pg_publication_rel;
/* ----------------
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 413e7c85a1..096a3c1fe0 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3642,6 +3642,7 @@ typedef struct PublicationTable
{
NodeTag type;
RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
@@ -3678,7 +3679,8 @@ typedef enum AlterPublicationAction
{
AP_AddObjects, /* add objects to publication */
AP_DropObjects, /* remove objects from publication */
- AP_SetObjects /* set list of objects */
+ AP_SetObjects, /* set list of objects */
+ AP_SetColumns /* change list of columns for a table */
} AlterPublicationAction;
typedef struct AlterPublicationStmt
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 22fffaca62..fcbed4ed2d 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -207,11 +207,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -228,7 +228,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 12c5f67080..defff6b081 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -165,7 +165,32 @@ Publications:
regress_publication_user | t | t | t | f | f | f
(1 row)
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d); -- error
+ERROR: cannot reference generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+ERROR: cannot drop column "c" because it is part of publication "testpub_fortable"
+HINT: Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5 (a);
+ERROR: column list must not be specified in ALTER PUBLICATION ... DROP
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl6"
+DETAIL: Cannot specify a column list on relations with REPLICA IDENTITY FULL.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+ALTER PUBLICATION testpub_fortable
+ ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c); -- error
+ERROR: cannot change column set for relation "testpub_tbl6"
+DETAIL: Cannot specify column list on relations with REPLICA IDENTITY FULL.
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
CREATE TABLE testpub_tbl3a (b text) INHERITS (testpub_tbl3);
@@ -670,6 +695,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schemas
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 56dd358554..2acf717d3d 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -89,7 +89,23 @@ SELECT pubname, puballtables FROM pg_publication WHERE pubname = 'testpub_forall
\d+ testpub_tbl2
\dRp+ testpub_foralltables
-DROP TABLE testpub_tbl2;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5 (a);
+
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+ALTER PUBLICATION testpub_fortable
+ ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c); -- error
+
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
CREATE TABLE testpub_tbl3 (a int);
@@ -362,6 +378,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/028_column_list.pl b/src/test/subscription/t/028_column_list.pl
new file mode 100644
index 0000000000..5a4f022f26
--- /dev/null
+++ b/src/test/subscription/t/028_column_list.pl
@@ -0,0 +1,164 @@
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test partial-column publication of tables
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 9;
+
+# setup
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+ INSERT INTO tab2 VALUES (2, 'foo', 2);");
+# Test with weird column names
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, B varchar, \"c'\" int)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+# Test replication with multi-level partition
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, \"c'\" int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+# Test create publication with a column list
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub1 FOR TABLE tab1(a, \"B\"), tab3(\"a'\",\"c'\"), test_part(a,b)");
+
+my $result = $node_publisher->safe_psql('postgres',
+ "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
+is($result, qq(tab1|1 2
+tab3|1 3
+test_part|1 2), 'publication relation updated');
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+);
+# Initial sync
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab1 VALUES (1,2,3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 VALUES (1,2,3)");
+# Test for replication of partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (1,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (2,'bcd', '2021-07-03 11:12:13')");
+# Test for replication of multi-level partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (4,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (5,'bcd', '2021-07-03 11:12:13')");
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'insert on column tab1.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab3");
+is($result, qq(1|3), 'insert on column tab3.b is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part");
+is($result, qq(1|abc\n2|bcd\n4|abc\n5|bcd), 'insert on all columns is replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab1 SET c = 5 where a = 1");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'update on column tab1.c is not replicated');
+
+# Verify user-defined types
+$node_publisher->safe_psql('postgres',
+ qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+ ALTER PUBLICATION pub1 ADD TABLE test_tab4 (a, b, d);
+ });
+$node_subscriber->safe_psql('postgres',
+ qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, d text);
+ });
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab4 VALUES (1, 'red', 3, 'oh my');");
+
+# Test alter publication with a column list
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION"
+);
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 VALUES (1,'abc',3)");
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET c = 5 where a = 2");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 WHERE a = 1");
+is($result, qq(1|abc), 'insert on column tab2.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 WHERE a = 2");
+is($result, qq(2|foo), 'update on column tab2.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_tab4");
+is($result, qq(1|red|oh my), 'insert on table with user-defined type');
+
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d)");
+$node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2, pub3");
+$node_publisher->wait_for_catchup('sub2');
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (1, 11, 111, 1111)");
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (2, 22, 222, 2222)");
+$node_publisher->wait_for_catchup('sub2');
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5;"),
+ qq(1|11|1111
+2|22|2222),
+ 'overlapping publications with overlapping column lists');
--
2.30.2
On 2022-Jan-07, Peter Eisentraut wrote:
ATExecReplicaIdentity(): Regarding the question of how to handle
REPLICA_IDENTITY_NOTHING: I see two ways to do this. Right now, the
approach is that the user can set the replica identity freely, and we
decide later based on that what we can replicate (e.g., no updates).
For this patch, that would mean we don't restrict what columns can be
in the column list, but we check what actions we can replicate based
on the column list. The alternative is that we require the column
list to include the replica identity, as the patch is currently doing,
which would mean that REPLICA_IDENTITY_NOTHING can be allowed since
it's essentially a set of zero columns.I find the current behavior a bit weird on reflection. If a user
wants to replicate on some columns and only INSERTs, that should be
allowed regardless of what the replica identity columns are.
Hmm. So you're saying that we should only raise errors about the column
list if we are publishing UPDATE or DELETE, but otherwise let the
replica identity be anything. OK, I'll see if I can come up with a
reasonable set of rules ...
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Before you were born your parents weren't as boring as they are now. They
got that way paying your bills, cleaning up your room and listening to you
tell them how idealistic you are." -- Charles J. Sykes' advice to teenagers
On 2022-Jan-10, Alvaro Herrera wrote:
Hmm. So you're saying that we should only raise errors about the column
list if we are publishing UPDATE or DELETE, but otherwise let the
replica identity be anything. OK, I'll see if I can come up with a
reasonable set of rules ...
This is an attempt to do it that way. Now you can add a table to a
publication without regards for how column filter compares to the
replica identity, as long as the publication does not include updates
and inserts.
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
"La fuerza no está en los medios físicos
sino que reside en una voluntad indomable" (Gandhi)
Attachments:
v17-0001-Support-column-lists-for-logical-replication-of-.patchtext/x-diff; charset=utf-8Download
From e3350af9023b5181b70ce480d039b3df46e9c019 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 11 Jan 2022 15:46:07 -0300
Subject: [PATCH v17] Support column lists for logical replication of tables
Add the capability of specifying a column list for individual tables as
part of a publication. Columns not in the list are not published. This
enables replicating to a table with only a subset of the columns.
If no column list is specified, all the columns are replicated, as
previously
Author: Rahila Syed <rahilasyed90@gmail.com>
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 13 +
doc/src/sgml/protocol.sgml | 4 +-
doc/src/sgml/ref/alter_publication.sgml | 20 +-
doc/src/sgml/ref/create_publication.sgml | 11 +-
src/backend/catalog/pg_publication.c | 353 +++++++++++++++++++-
src/backend/commands/publicationcmds.c | 67 +++-
src/backend/commands/tablecmds.c | 87 ++++-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 60 +++-
src/backend/replication/logical/proto.c | 66 ++--
src/backend/replication/logical/tablesync.c | 119 ++++++-
src/backend/replication/pgoutput/pgoutput.c | 118 ++++++-
src/bin/pg_dump/pg_dump.c | 41 ++-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 26 +-
src/bin/psql/tab-complete.c | 2 +
src/include/catalog/pg_publication.h | 6 +
src/include/catalog/pg_publication_rel.h | 3 +
src/include/nodes/parsenodes.h | 4 +-
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 59 +++-
src/test/regress/sql/publication.sql | 39 ++-
src/test/subscription/t/028_column_list.pl | 164 +++++++++
24 files changed, 1184 insertions(+), 87 deletions(-)
create mode 100644 src/test/subscription/t/028_column_list.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 03e2537b07..b7b75f64a2 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6311,6 +6311,19 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to relation
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>prattrs</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+ </para>
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all attributes are published.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 34a7034282..5bc2e7a591 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -6877,7 +6877,9 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column (except
+ generated columns and other columns that don't appear in the column
+ filter list, for tables that have one):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index bb4ef5e5e2..d0e97243b8 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -25,12 +25,13 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD <replace
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ALTER TABLE <replaceable class="parameter">publication_object</replaceable> SET COLUMNS { ( <replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -62,6 +63,11 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
command retain their previous settings.
</para>
+ <para>
+ The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows changing
+ the set of columns that are included in the publication.
+ </para>
+
<para>
The remaining variants change the owner and the name of the publication.
</para>
@@ -110,6 +116,8 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
</para>
</listitem>
</varlistentry>
@@ -164,9 +172,15 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
</para>
<para>
- Add some tables to the publication:
+ Add tables to the publication:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
+<programlisting>
+ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS (user_id, firstname, lastname);
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index d805e8e77a..73a23cbb02 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -78,6 +78,15 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
publication, so they are never explicitly added to the publication.
</para>
+ <para>
+ When a column list is specified, only the listed columns are replicated;
+ any other columns are ignored for the purpose of replication through
+ this publication. If no column list is specified, all columns of the
+ table are replicated through this publication, including any columns
+ added later. If a column list is specified, it must include the replica
+ identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 2992a2e0c6..ea12b52b83 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,13 +45,26 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+
+static void check_publication_columns(Publication *pub, Relation targetrel,
+ Bitmapset *columns);
+static void publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs,
+ Bitmapset **attset);
+
/*
- * Check if relation can be in given publication and throws appropriate
- * error if not.
+ * Check if relation can be in given publication and that the column
+ * filter is sensible, and throws appropriate error if not.
+ *
+ * targetcols is the bitmapset of attribute numbers given in the column list,
+ * or NULL if it was not specified.
*/
static void
-check_publication_add_relation(Relation targetrel)
+check_publication_add_relation(Publication *pub, Relation targetrel,
+ Bitmapset *columns)
{
+ bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
@@ -82,6 +95,73 @@ check_publication_add_relation(Relation targetrel)
errmsg("cannot add relation \"%s\" to publication",
RelationGetRelationName(targetrel)),
errdetail("This operation is not supported for unlogged tables.")));
+
+ /* Make sure the column list checks out */
+ if (columns != NULL)
+ {
+ /*
+ * Even if the user listed all columns in the column list, we cannot
+ * allow a column list to be specified when REPLICA IDENTITY is FULL;
+ * that would cause problems if a new column is added later, because
+ * the new column would have to be included (because of being part of
+ * the replica identity) but it's technically not allowed (because of
+ * not being in the publication's column list yet). So reject this
+ * case altogether.
+ */
+ if (replidentfull)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot specify a column list on relations with REPLICA IDENTITY FULL."));
+
+ check_publication_columns(pub, targetrel, columns);
+ }
+}
+
+/*
+ * Enforce that the column list can only leave out columns that aren't
+ * forced to be sent.
+ *
+ * No column can be excluded if REPLICA IDENTITY is FULL (since all the
+ * columns need to be sent regardless); and in other cases, the columns in
+ * the REPLICA IDENTITY cannot be left out.
+ */
+static void
+check_publication_columns(Publication *pub, Relation targetrel, Bitmapset *columns)
+{
+ if (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot change column set for relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot specify column list on relations with REPLICA IDENTITY FULL."));
+
+ if (pub->pubactions.pubupdate || pub->pubactions.pubdelete)
+ {
+ Bitmapset *idattrs;
+ int x;
+
+ idattrs = RelationGetIndexAttrBitmap(targetrel,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * We have to test membership the hard way, because the values returned by
+ * RelationGetIndexAttrBitmap are offset.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ if (!bms_is_member(x + FirstLowInvalidHeapAttributeNumber, columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
+ }
+
+ bms_free(idattrs);
+ }
}
/*
@@ -289,6 +369,9 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
Oid relid = RelationGetRelid(targetrel->relation);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ Bitmapset *attset = NULL;
+ AttrNumber *attarray;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -314,7 +397,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ /* Translate column names to numbers and verify suitability */
+ publication_translate_columns(targetrel->relation,
+ targetrel->columns,
+ &natts, &attarray, &attset);
+
+ check_publication_add_relation(pub, targetrel->relation, attset);
+
+ bms_free(attset);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -327,6 +417,15 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ if (targetrel->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -337,6 +436,13 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
/* Register dependencies as needed */
ObjectAddressSet(myself, PublicationRelRelationId, pubreloid);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ pfree(attarray);
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -364,6 +470,155 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
return myself;
}
+/*
+ * Update the column list for a relation in a publication.
+ */
+void
+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns)
+{
+ Bitmapset *attset;
+ AttrNumber *attarray;
+ HeapTuple copytup;
+ int natts;
+ bool nulls[Natts_pg_publication_rel];
+ bool replaces[Natts_pg_publication_rel];
+ Datum values[Natts_pg_publication_rel];
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ replaces[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ deleteDependencyRecordsForClass(PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid,
+ RelationRelationId,
+ DEPENDENCY_AUTO);
+
+ if (columns == NULL)
+ {
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+ }
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ int2vector *prattrs;
+ Publication *pub;
+
+ pub = GetPublication(((Form_pg_publication_rel) GETSTRUCT(pubreltup))->prpubid);
+
+ publication_translate_columns(targetrel, columns,
+ &natts, &attarray, &attset);
+
+ /*
+ * Make sure the column list checks out. XXX this should occur at
+ * caller in publicationcmds.c, not here.
+ */
+ check_publication_columns(pub, targetrel, attset);
+ bms_free(attset);
+ /* XXX "pub" is leaked here */
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+
+ /* Add dependencies on the new list of columns */
+ ObjectAddressSet(myself, PublicationRelRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid);
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId,
+ RelationGetRelid(targetrel), attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ }
+
+ copytup = heap_modify_tuple(pubreltup, RelationGetDescr(pubrel),
+ values, nulls, replaces);
+
+ CatalogTupleUpdate(pubrel, &pubreltup->t_self, copytup);
+
+ heap_freetuple(copytup);
+}
+
+/* qsort comparator for attnums */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list. Other checks are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ */
+static void
+publication_translate_columns(Relation targetrel, List *columns, int *natts,
+ AttrNumber **attrs, Bitmapset **attset)
+{
+ AttrNumber *attarray;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference generated column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /* Be tidy, so that the catalog representation is always sorted */
+ qsort(attarray, n, sizeof(AttrNumber), compare_int16);
+
+ *natts = n;
+ *attrs = attarray;
+ *attset = set;
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -471,6 +726,96 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull)
+ continue;
+
+ pubs = lappend_oid(pubs,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+/* FIXME maybe these two routines should be in lsyscache.c */
+/* Return the set of actions that the given publication includes */
+void
+GetActionsInPublication(Oid pubid, PublicationActions *actions)
+{
+ HeapTuple pub;
+ Form_pg_publication pubForm;
+
+ pub = SearchSysCache1(PUBLICATIONOID,
+ ObjectIdGetDatum(pubid));
+ if (!HeapTupleIsValid(pub))
+ elog(ERROR, "cache lookup failed for publication %u", pubid);
+
+ pubForm = (Form_pg_publication) GETSTRUCT(pub);
+ actions->pubinsert = pubForm->pubinsert;
+ actions->pubupdate = pubForm->pubupdate;
+ actions->pubdelete = pubForm->pubdelete;
+ actions->pubtruncate = pubForm->pubtruncate;
+
+ ReleaseSysCache(pub);
+}
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 3ab1bdeae1..aa41940d1b 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -376,6 +376,46 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
return myself;
}
+/*
+ * Change the column list of a relation in a publication
+ */
+static void
+PublicationSetColumns(AlterPublicationStmt *stmt,
+ Form_pg_publication pubform, PublicationTable *table)
+{
+ Relation rel,
+ urel;
+ HeapTuple tup;
+ ObjectAddress obj,
+ secondary;
+
+ rel = table_open(PublicationRelRelationId, RowExclusiveLock);
+ urel = table_openrv(table->relation, ShareUpdateExclusiveLock);
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(RelationGetRelid(urel)),
+ ObjectIdGetDatum(pubform->oid));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errmsg("relation \"%s\" is not already in publication \"%s\"",
+ table->relation->relname,
+ NameStr(pubform->pubname)));
+
+ publication_set_table_columns(rel, tup, urel, table->columns);
+
+ ObjectAddressSet(obj, PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->oid);
+ ObjectAddressSet(secondary, RelationRelationId, RelationGetRelid(urel));
+ EventTriggerCollectSimpleCommand(obj, secondary, (Node *) stmt);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+ table_close(urel, NoLock);
+
+ InvokeObjectPostAlterHook(PublicationRelationId, pubform->oid, 0);
+}
+
/*
* Change options of a publication.
*/
@@ -523,6 +563,14 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
}
else if (stmt->action == AP_DropObjects)
PublicationDropTables(pubid, rels, false);
+ else if (stmt->action == AP_SetColumns)
+ {
+ Assert(schemaidlist == NIL);
+ Assert(list_length(tables) == 1);
+
+ PublicationSetColumns(stmt, pubform,
+ linitial_node(PublicationTable, tables));
+ }
else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
@@ -562,7 +610,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
pubrel = palloc(sizeof(PublicationRelInfo));
pubrel->relation = oldrel;
-
+ /* This is not needed to delete a table */
+ pubrel->columns = NIL;
delrels = lappend(delrels, pubrel);
}
}
@@ -622,7 +671,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
}
else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* AP_SetObjects */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -645,6 +694,10 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
*/
PublicationAddSchemas(pubform->oid, schemaidlist, true, stmt);
}
+ else
+ {
+ /* Nothing to do for AP_SetColumns */
+ }
}
/*
@@ -934,6 +987,8 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
@@ -967,8 +1022,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
+
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
@@ -1076,6 +1134,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 1f0654c2f5..893223b437 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -8364,6 +8364,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
bool missing_ok, LOCKMODE lockmode,
ObjectAddresses *addrs)
{
+ Oid relid = RelationGetRelid(rel);
HeapTuple tuple;
Form_pg_attribute targetatt;
AttrNumber attnum;
@@ -8383,7 +8384,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/*
* get the number of the attribute
*/
- tuple = SearchSysCacheAttName(RelationGetRelid(rel), colName);
+ tuple = SearchSysCacheAttName(relid, colName);
if (!HeapTupleIsValid(tuple))
{
if (!missing_ok)
@@ -8437,13 +8438,42 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * Also, if the column is used in the column list of a publication,
+ * disallow the drop if the DROP is RESTRICT. We don't do anything if the
+ * DROP is CASCADE, which means that the dependency mechanism will remove
+ * the relation from the publication.
+ */
+ if (behavior == DROP_RESTRICT)
+ {
+ List *pubs;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(relid);
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(relid, pubid);
+
+ if (list_member_oid(published_cols, attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot drop column \"%s\" because it is part of publication \"%s\"",
+ colName, get_publication_name(pubid, false)),
+ errhint("Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication."));
+ }
+ }
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
* use find_all_inheritors to do it in one pass.
*/
children =
- find_inheritance_children(RelationGetRelid(rel), lockmode);
+ find_inheritance_children(relid, lockmode);
if (children)
{
@@ -8531,7 +8561,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Add object to delete */
object.classId = RelationRelationId;
- object.objectId = RelationGetRelid(rel);
+ object.objectId = relid;
object.objectSubId = attnum;
add_exact_object_address(&object, addrs);
@@ -15841,6 +15871,7 @@ relation_mark_replica_identity(Relation rel, char ri_type, Oid indexOid,
CatalogTupleUpdate(pg_index, &pg_index_tuple->t_self, pg_index_tuple);
InvokeObjectPostAlterHookArg(IndexRelationId, thisIndexOid, 0,
InvalidOid, is_internal);
+
/*
* Invalidate the relcache for the table, so that after we commit
* all sessions will refresh the table's replica identity index
@@ -15863,6 +15894,11 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
Oid indexOid;
Relation indexRel;
int key;
+ List *pubs;
+ Bitmapset *indexed_cols = NULL;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(RelationGetRelid(rel));
if (stmt->identity_type == REPLICA_IDENTITY_DEFAULT)
{
@@ -15871,11 +15907,16 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
}
else if (stmt->identity_type == REPLICA_IDENTITY_FULL)
{
+ if (pubs != NIL)
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set REPLICA IDENTITY FULL when publications contain relations that specify column lists"));
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
else if (stmt->identity_type == REPLICA_IDENTITY_NOTHING)
{
+ /* XXX not sure what's the right check for publications here */
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
@@ -15960,6 +16001,46 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
errmsg("index \"%s\" cannot be used as replica identity because column \"%s\" is nullable",
RelationGetRelationName(indexRel),
NameStr(attr->attname))));
+
+ /*
+ * Collect columns used, in case we have any publications that we need
+ * to vet. System attributes are disallowed so no need to subtract
+ * FirstLowInvalidHeapAttributeNumber.
+ */
+ indexed_cols = bms_add_member(indexed_cols, attno);
+ }
+
+ /*
+ * Check partial-column publications. For those that include UPDATE and
+ * DELETE, we must enforce that the columns in the replica identity are
+ * included in the column list. For publications that only include INSERT
+ * and TRUNCATE, we don't need to restrict the replica identity.
+ */
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+ PublicationActions actions;
+
+ GetActionsInPublication(pubid, &actions);
+ /* No need to worry about this one */
+ if (!actions.pubupdate && !actions.pubdelete)
+ continue;
+
+ published_cols =
+ GetRelationColumnListInPublication(RelationGetRelid(rel), pubid);
+
+ for (key = 0; key < IndexRelationGetNumberOfKeyAttributes(indexRel); key++)
+ {
+ int16 attno = indexRel->rd_index->indkey.values[key];
+
+ if (!list_member_oid(published_cols, attno))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("index \"%s\" cannot be used because publication \"%s\" does not include all indexed columns",
+ RelationGetRelationName(indexRel),
+ get_publication_name(pubid, false)));
+ }
}
/* This index is suitable for use as a replica identity. Mark it. */
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 456d563f34..aa333fcdf5 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4834,6 +4834,7 @@ _copyPublicationTable(const PublicationTable *from)
PublicationTable *newnode = makeNode(PublicationTable);
COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 53beef1488..3119f7836c 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2312,6 +2312,7 @@ static bool
_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 879018377b..c1f3d6a8c8 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9740,12 +9740,13 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr
+ TABLE relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
+ $$->pubtable->columns = $3;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9760,28 +9761,38 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId
+ | ColId opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- $$->name = $1;
+ if ($2 != NULL)
+ {
+ $$->pubtable = makeNode(PublicationTable);
+ $$->pubtable->relation = makeRangeVar(NULL, $1, @1);
+ $$->pubtable->columns = $2;
+ $$->name = NULL;
+ }
+ else
+ $$->name = $1;
$$->location = @1;
}
- | ColId indirection
+ | ColId indirection opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
+ $$->pubtable->columns = $3;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr
+ | extended_relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
+ $$->pubtable->columns = $2;
}
| CURRENT_SCHEMA
{
@@ -9807,6 +9818,9 @@ pub_obj_list: PublicationObjSpec
*
* ALTER PUBLICATION name SET pub_obj [, ...]
*
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
* pub_obj is one of:
*
* TABLE table_name [, ...]
@@ -9840,6 +9854,32 @@ AlterPublicationStmt:
n->action = AP_SetObjects;
$$ = (Node *)n;
}
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS '(' columnList ')'
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = $10;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS ALL
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = NIL;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
| ALTER PUBLICATION name DROP pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
@@ -17444,6 +17484,16 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
else if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLES_IN_SCHEMA ||
pubobj->pubobjtype == PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA)
{
+ /*
+ * This can happen if a column list is specified in a continuation
+ * for a schema entry; reject it.
+ */
+ if (pubobj->pubtable)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schemas"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 953942692c..e6da46d83e 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,9 +29,11 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary,
+ Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +400,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +412,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -442,7 +444,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary,
+ Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -463,11 +466,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -536,7 +539,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -651,7 +654,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -673,7 +677,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -749,7 +753,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple,
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -761,7 +766,13 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ /* Don't count attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
continue;
nliveatts++;
}
@@ -783,6 +794,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
if (att->attisdropped || att->attgenerated)
continue;
+ /* Ignore attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +919,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -914,20 +929,24 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
desc = RelationGetDescr(rel);
- /* send number of live attributes */
- for (i = 0; i < desc->natts; i++)
- {
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
- continue;
- nliveatts++;
- }
- pq_sendint16(out, nliveatts);
-
/* fetch bitmap of REPLICATION IDENTITY attributes */
replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
if (!replidentfull)
idattrs = RelationGetIdentityKeyBitmap(rel);
+ /* send number of live attributes */
+ for (i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+ nliveatts++;
+ }
+ pq_sendint16(out, nliveatts);
+
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -936,7 +955,8 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
-
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index e596b69d46..a7befd712a 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -112,6 +112,7 @@
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "utils/acl.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -699,17 +700,20 @@ fetch_remote_table_info(char *nspname, char *relname,
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
bool isnull;
int natt;
+ ListCell *lc;
+ bool am_partition;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -739,14 +743,19 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetBool(slot_getattr(slot, 4, &isnull));
+ Assert(!isnull);
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -774,16 +783,91 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * In server versions 15 and higher, obtain the publication column list,
+ * if any.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData publications;
+ bool first = true;
+
+ initStringInfo(&publications);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&publications, ", ");
+ appendStringInfoString(&publications, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ " SELECT pg_catalog.unnest(prattrs)\n"
+ " FROM pg_catalog.pg_publication p JOIN\n"
+ " pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n"
+ " WHERE p.pubname IN (%s) AND\n",
+ publications.data);
+ if (!am_partition)
+ appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "prrelid IN (SELECT relid\n"
+ " FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch attribute info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ if (isnull)
+ continue;
+ included_cols = bms_add_member(included_cols, attnum);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+ pfree(publications.data);
+ walrcv_clear_result(pubres);
+ }
+
+ /*
+ * Store the columns as a list of names. Ignore those that are not
+ * present in the column list, if there is one.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ continue;
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
+ Assert(!isnull);
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -793,12 +877,12 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecClearTuple(slot);
}
+
ExecDropSingleTupleTableSlot(slot);
-
- lrel->natts = natt;
-
walrcv_clear_result(res);
pfree(cmd.data);
+
+ lrel->natts = natt;
}
/*
@@ -831,8 +915,17 @@ copy_table(Relation rel)
/* Start copy on the publisher. */
initStringInfo(&cmd);
if (lrel.relkind == RELKIND_RELATION)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index af8d51aee9..bdab0b1c8d 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,16 +15,19 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -81,7 +84,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -130,6 +134,13 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+
+ /*
+ * Set of columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this list are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -570,11 +581,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
}
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -587,7 +598,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -610,13 +622,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip this attribute if it's not present in the column list */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -693,7 +709,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -722,7 +738,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -1120,6 +1136,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
{
RelationSyncEntry *entry;
bool found;
+ Oid ancestor_id;
MemoryContext oldctx;
Assert(RelationSyncCache != NULL);
@@ -1140,6 +1157,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->map = NULL; /* will be set by maybe_send_schema() if
* needed */
}
@@ -1175,13 +1193,16 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
/*
* Build publication cache. We can't use one provided by relcache as
- * relcache considers all publications given relation is in, but here
- * we only need to consider ones that the subscriber requested.
+ * relcache considers all publications that the given relation is in,
+ * but here we only need to consider ones that the subscriber
+ * requested.
*/
foreach(lc, data->publications)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
+ bool all_columns = false;
if (pub->alltables)
{
@@ -1192,8 +1213,6 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (!publish)
{
- bool ancestor_published = false;
-
/*
* For a partition, check if any of the ancestors are
* published. If so, note down the topmost ancestor that is
@@ -1219,6 +1238,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
pub->oid))
{
ancestor_published = true;
+ ancestor_id = ancestor;
if (pub->pubviaroot)
publish_as_relid = ancestor;
}
@@ -1232,9 +1252,13 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
* Don't publish changes for partitioned tables, because
- * publishing those of its partitions suffices, unless partition
- * changes won't be published due to pubviaroot being set.
+ * publishing those of its partitions suffices. (However, ignore
+ * this if partition changes are not to published due to
+ * pubviaroot being set.)
*/
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
@@ -1243,10 +1267,74 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
+
+ /*
+ * Obtain columns published by this publication, and add them
+ * to the list for this rel. Note that if at least one
+ * publication has a empty column list, that means to publish
+ * everything; so if we saw a publication that includes all
+ * columns, skip this.
+ */
+ if (!all_columns)
+ {
+ HeapTuple pub_rel_tuple;
+ Oid relid;
+
+ relid = ancestor_published ? ancestor_id : publish_as_relid;
+
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ Datum pub_rel_cols;
+ bool isnull;
+
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP,
+ pub_rel_tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull)
+ {
+ /*
+ * If we see a publication with no columns, reset the
+ * list and ignore further ones.
+ */
+ all_columns = true;
+ bms_free(entry->columns);
+ entry->columns = NULL;
+ }
+ else if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(pub_rel_cols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
+ entry->columns = bms_add_member(entry->columns,
+ elems[i]);
+ MemoryContextSwitchTo(oldctx);
+ }
+
+ ReleaseSysCache(pub_rel_tuple);
+ }
+ }
}
+ /*
+ * If we've seen all action bits, and we know that all columns are
+ * published, there's no reason to look at further publications.
+ */
if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
+ entry->pubactions.pubdelete && entry->pubactions.pubtruncate &&
+ all_columns)
break;
}
@@ -1343,6 +1431,8 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
if (entry->map)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 92ab95724d..d13570f5aa 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4042,6 +4042,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_oid;
int i_prpubid;
int i_prrelid;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4053,8 +4054,13 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
/* Collect all publication membership info. */
appendPQExpBufferStr(query,
- "SELECT tableoid, oid, prpubid, prrelid "
- "FROM pg_catalog.pg_publication_rel");
+ "SELECT tableoid, oid, prpubid, prrelid");
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBufferStr(query, ", prattrs");
+ else
+ appendPQExpBufferStr(query, ", NULL as prattrs");
+ appendPQExpBufferStr(query,
+ " FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
ntups = PQntuples(res);
@@ -4063,6 +4069,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_oid = PQfnumber(res, "oid");
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4104,6 +4111,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
pubrinfo[j].publication = pubinfo;
pubrinfo[j].pubtable = tbinfo;
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[i].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4178,10 +4207,12 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s;\n",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+ appendPQExpBufferStr(query, ";\n");
/*
* There is no point in creating a drop query as the drop is done by table
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 066a129ee5..857f2891fc 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -630,6 +630,7 @@ typedef struct _PublicationRelInfo
DumpableObject dobj;
PublicationInfo *publication;
TableInfo *pubtable;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 8587b19160..37faf4bef4 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5823,7 +5823,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5840,10 +5840,14 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (!as_schema) /* as table */
+ {
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
- else
+ if (!PQgetisnull(res, i, 2))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 2));
+ }
+ else /* as schema */
printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
@@ -5971,8 +5975,20 @@ describePublications(const char *pattern)
{
/* Get the tables for the specified publication */
printfPQExpBuffer(&buf,
- "SELECT n.nspname, c.relname\n"
- "FROM pg_catalog.pg_class c,\n"
+ "SELECT n.nspname, c.relname, \n");
+ if (pset.sversion >= 150000)
+ appendPQExpBufferStr(&buf,
+ " CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string"
+ "(ARRAY(SELECT attname\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END AS columns");
+ else
+ appendPQExpBufferStr(&buf, "NULL as columns");
+ appendPQExpBuffer(&buf,
+ "\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
" pg_catalog.pg_publication_rel pr\n"
"WHERE c.relnamespace = n.oid\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 39be6f556a..20c852cdf9 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1657,6 +1657,8 @@ psql_completion(const char *text, int start, int end)
/* ALTER PUBLICATION <name> ADD */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
+ else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD", "TABLE"))
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL);
/* ALTER PUBLICATION <name> DROP */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "DROP"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 841b9b6c25..70f73d01dc 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -86,6 +86,7 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -109,8 +110,11 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
+extern void GetActionsInPublication(Oid pubid, PublicationActions *actions);
extern List *GetPublicationSchemas(Oid pubid);
extern List *GetSchemaPublications(Oid schemaid);
extern List *GetSchemaPublicationRelations(Oid schemaid,
@@ -127,6 +131,8 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *tar
bool if_not_exists);
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern void publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index 0ff3716225..151644b870 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ int2vector prattrs;
+#endif
} FormData_pg_publication_rel;
/* ----------------
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 413e7c85a1..096a3c1fe0 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3642,6 +3642,7 @@ typedef struct PublicationTable
{
NodeTag type;
RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
@@ -3678,7 +3679,8 @@ typedef enum AlterPublicationAction
{
AP_AddObjects, /* add objects to publication */
AP_DropObjects, /* remove objects from publication */
- AP_SetObjects /* set list of objects */
+ AP_SetObjects, /* set list of objects */
+ AP_SetColumns /* change list of columns for a table */
} AlterPublicationAction;
typedef struct AlterPublicationStmt
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 22fffaca62..fcbed4ed2d 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -207,11 +207,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -228,7 +228,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 12c5f67080..af98090f8f 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -165,8 +165,54 @@ Publications:
regress_publication_user | t | t | t | f | f | f
(1 row)
-DROP TABLE testpub_tbl2;
-DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d); -- error
+ERROR: cannot reference generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+ERROR: cannot drop column "c" because it is part of publication "testpub_fortable"
+HINT: Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication.
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key; -- nope
+ERROR: index "testpub_tbl5_b_key" cannot be used because publication "testpub_fortable" does not include all indexed columns
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key; -- ok, but ...
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- no dice
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+ Publication testpub_table_ins
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | t | f
+Tables:
+ "public.testpub_tbl5" (a)
+
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl6"
+DETAIL: Cannot specify a column list on relations with REPLICA IDENTITY FULL.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+ALTER PUBLICATION testpub_fortable
+ ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c); -- error
+ERROR: cannot change column set for relation "testpub_tbl6"
+DETAIL: Cannot specify column list on relations with REPLICA IDENTITY FULL.
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
+DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema, testpub_table_ins;
CREATE TABLE testpub_tbl3 (a int);
CREATE TABLE testpub_tbl3a (b text) INHERITS (testpub_tbl3);
SET client_min_messages = 'ERROR';
@@ -670,6 +716,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schemas
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 56dd358554..e0d50e3f69 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -89,8 +89,39 @@ SELECT pubname, puballtables FROM pg_publication WHERE pubname = 'testpub_forall
\d+ testpub_tbl2
\dRp+ testpub_foralltables
-DROP TABLE testpub_tbl2;
-DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key; -- nope
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key; -- ok, but ...
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- no dice
+
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+ALTER PUBLICATION testpub_fortable
+ ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c); -- error
+
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
+DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema, testpub_table_ins;
CREATE TABLE testpub_tbl3 (a int);
CREATE TABLE testpub_tbl3a (b text) INHERITS (testpub_tbl3);
@@ -362,6 +393,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/028_column_list.pl b/src/test/subscription/t/028_column_list.pl
new file mode 100644
index 0000000000..5a4f022f26
--- /dev/null
+++ b/src/test/subscription/t/028_column_list.pl
@@ -0,0 +1,164 @@
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test partial-column publication of tables
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 9;
+
+# setup
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+ INSERT INTO tab2 VALUES (2, 'foo', 2);");
+# Test with weird column names
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, B varchar, \"c'\" int)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+# Test replication with multi-level partition
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, \"c'\" int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+# Test create publication with a column list
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub1 FOR TABLE tab1(a, \"B\"), tab3(\"a'\",\"c'\"), test_part(a,b)");
+
+my $result = $node_publisher->safe_psql('postgres',
+ "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
+is($result, qq(tab1|1 2
+tab3|1 3
+test_part|1 2), 'publication relation updated');
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+);
+# Initial sync
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab1 VALUES (1,2,3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 VALUES (1,2,3)");
+# Test for replication of partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (1,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (2,'bcd', '2021-07-03 11:12:13')");
+# Test for replication of multi-level partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (4,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (5,'bcd', '2021-07-03 11:12:13')");
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'insert on column tab1.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab3");
+is($result, qq(1|3), 'insert on column tab3.b is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part");
+is($result, qq(1|abc\n2|bcd\n4|abc\n5|bcd), 'insert on all columns is replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab1 SET c = 5 where a = 1");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'update on column tab1.c is not replicated');
+
+# Verify user-defined types
+$node_publisher->safe_psql('postgres',
+ qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+ ALTER PUBLICATION pub1 ADD TABLE test_tab4 (a, b, d);
+ });
+$node_subscriber->safe_psql('postgres',
+ qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, d text);
+ });
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab4 VALUES (1, 'red', 3, 'oh my');");
+
+# Test alter publication with a column list
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION"
+);
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 VALUES (1,'abc',3)");
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET c = 5 where a = 2");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 WHERE a = 1");
+is($result, qq(1|abc), 'insert on column tab2.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 WHERE a = 2");
+is($result, qq(2|foo), 'update on column tab2.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_tab4");
+is($result, qq(1|red|oh my), 'insert on table with user-defined type');
+
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d)");
+$node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2, pub3");
+$node_publisher->wait_for_catchup('sub2');
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (1, 11, 111, 1111)");
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (2, 22, 222, 2222)");
+$node_publisher->wait_for_catchup('sub2');
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5;"),
+ qq(1|11|1111
+2|22|2222),
+ 'overlapping publications with overlapping column lists');
--
2.30.2
Is there any coordination between the "column filter" patch and the "row
filter" patch ? Are they both on track for PG15 ? Has anybody run them
together ?
Whichever patch is merged 2nd should include tests involving a subset of
columns along with a WHERE clause.
I have a suggestion: for the functions for which both patches are adding
additional argument types, define a filtering structure for both patches to
use. Similar to what we did for some utility statements in a3dc92600.
I'm referring to:
logicalrep_write_update()
logicalrep_write_tuple()
That would avoid avoid some rebase conflicts on april 9, and avoid functions
with 7,8,9 arguments, and maybe simplify adding arguments in the future.
--
Justin
On 2022-Jan-11, Alvaro Herrera wrote:
On 2022-Jan-10, Alvaro Herrera wrote:
Hmm. So you're saying that we should only raise errors about the column
list if we are publishing UPDATE or DELETE, but otherwise let the
replica identity be anything. OK, I'll see if I can come up with a
reasonable set of rules ...This is an attempt to do it that way. Now you can add a table to a
publication without regards for how column filter compares to the
replica identity, as long as the publication does not include updates
and inserts.
I discovered a big hole in this, which is that ALTER PUBLICATION SET
(publish='insert,update') can add UPDATE publishing to a publication
that was only publishing INSERTs. It's easy to implement a fix: in
AlterPublicationOptions, scan the list of tables and raise an error if
any of them has a column list that doesn't include all the columns in
the replica identity.
However, that proposal has an ugly flaw: there is no index on
pg_publication_rel.prpubid, which means that the only way to find the
relations we need to inspect is to seqscan pg_publication_rel.
Also, psql's query for \dRp+ uses a seqscan in pg_publication_rel.
Therefore, I propose to add an index on pg_publication_rel.prpubid.
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
"¿Qué importan los años? Lo que realmente importa es comprobar que
a fin de cuentas la mejor edad de la vida es estar vivo" (Mafalda)
On 12.01.22 01:41, Alvaro Herrera wrote:
Therefore, I propose to add an index on pg_publication_rel.prpubid.
That seems very reasonable.
On 2022-Jan-11, Justin Pryzby wrote:
Is there any coordination between the "column filter" patch and the "row
filter" patch ?
Not beyond the grammar, which I tested.
Are they both on track for PG15 ?
I think they're both on track, yes.
Has anybody run them together ?
Not me.
I have a suggestion: for the functions for which both patches are adding
additional argument types, define a filtering structure for both patches to
use. Similar to what we did for some utility statements in a3dc92600.I'm referring to:
logicalrep_write_update()
logicalrep_write_tuple()
Fixed: the row filter patch no longer adds extra arguments to those
functions.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
"Tiene valor aquel que admite que es un cobarde" (Fernandel)
On Wed, Jan 12, 2022 at 2:40 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
Is there any coordination between the "column filter" patch and the "row
filter" patch ? Are they both on track for PG15 ? Has anybody run them
together ?
The few things where I think we might need to define some common
behavior are as follows:
1. Replica Identity handling: Currently the column filter patch gives
an error during create/alter subscription if the specified column list
is invalid (Replica Identity columns are missing). It also gives an
error if the user tries to change the replica identity. However, it
doesn't deal with cases where the user drops and adds a different
primary key that has a different set of columns which can lead to
failure during apply on the subscriber.
I think another issue w.r.t column filter patch is that even while
creating publication (even for 'insert' publications) it should check
that all primary key columns must be part of published columns,
otherwise, it can fail while applying on subscriber as it will try to
insert NULL for the primary key column.
2. Handling of partitioned tables vs. Replica Identity (RI): When
adding a partitioned table with a column list to the publication (with
publish_via_partition_root = false), we should check the Replica
Identity of all its leaf partition as the RI on the partition is the
one actually takes effect when publishing DML changes. We need to
check RI while attaching the partition as well, as the newly added
partitions will automatically become part of publication if the
partitioned table is part of the publication. If we don't do this the
later deletes/updates can fail.
All these cases are dealt with in row filter patch because of the
on-the-fly check which means we check the validation of columns in row
filters while actual operation update/delete via
CheckCmdReplicaIdentity and cache the result of same for future use.
This is inline with existing checks of RI vs. operations on tables.
The primary reason for this was we didn't want to handle validation of
row filters at so many places.
3. Tablesync.c handling: Ideally, it would be good if we have a single
query to fetch both row filters and column filters but even if that is
not possible in the first version, the behavior should be same for
both queries w.r.t partitioned tables, For ALL Tables and For All
Tables In Schema cases.
Currently, the column filter patch doesn't seem to respect For ALL
Tables and For All Tables In Schema cases, basically, it just copies
the columns it finds through some of the publications even if one of
the publications is defined as For All Tables. The row filter patch
ignores the row filters if one of the publications is defined as For
ALL Tables and For All Tables In Schema.
For row filter patch, if the publication contains a partitioned table,
the publication parameter publish_via_partition_root determines if it
uses the partition row filter (if the parameter is false, the default)
or the root partitioned table row filter and this is taken care of
even during the initial tablesync.
For column filter patch, if the publication contains a partitioned
table, it seems that it finds all columns that the tables in its
partition tree specified in the publications, whether
publish_via_partition_root is true or false.
We have done some testing w.r.t above cases with both patches and my
colleague will share the results.
--
With Regards,
Amit Kapila.
On Friday, January 14, 2022 7:52 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jan 12, 2022 at 2:40 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
Is there any coordination between the "column filter" patch and the "row
filter" patch ? Are they both on track for PG15 ? Has anybody run them
together ?The few things where I think we might need to define some common
behavior are as follows:
I tried some cases about the points you mentions, which can be taken as
reference.
1. Replica Identity handling: Currently the column filter patch gives
an error during create/alter subscription if the specified column list
is invalid (Replica Identity columns are missing). It also gives an
error if the user tries to change the replica identity. However, it
doesn't deal with cases where the user drops and adds a different
primary key that has a different set of columns which can lead to
failure during apply on the subscriber.
An example for this scenario:
-- publisher --
create table tbl(a int primary key, b int);
create publication pub for table tbl(a);
alter table tbl drop CONSTRAINT tbl_pkey;
alter table tbl add primary key (b);
-- subscriber --
create table tbl(a int, b int);
create subscription sub connection 'port=5432 dbname=postgres' publication pub;
-- publisher --
insert into tbl values (1,1);
-- subscriber --
postgres=# select * from tbl;
a | b
---+---
1 |
(1 row)
update tbl set b=1 where a=1;
alter table tbl add primary key (b);
-- publisher --
delete from tbl;
The subscriber reported the following error message and DELETE failed in subscriber.
ERROR: publisher did not send replica identity column expected by the logical replication target relation "public.tbl"
CONTEXT: processing remote data during "DELETE" for replication target relation "public.tbl" in transaction 723 at 2022-01-14 13:11:51.514261+08
-- subscriber
postgres=# select * from tbl;
a | b
---+---
1 | 1
(1 row)
I think another issue w.r.t column filter patch is that even while
creating publication (even for 'insert' publications) it should check
that all primary key columns must be part of published columns,
otherwise, it can fail while applying on subscriber as it will try to
insert NULL for the primary key column.
For example:
-- publisher --
create table tbl(a int primary key, b int);
create publication pub for table tbl(a);
alter table tbl drop CONSTRAINT tbl_pkey;
alter table tbl add primary key (b);
-- subscriber --
create table tbl(a int, b int primary key);
create subscription sub connection 'port=5432 dbname=postgres' publication pub;
-- publisher --
insert into tbl values (1,1);
The subscriber reported the following error message and INSERT failed in subscriber.
ERROR: null value in column "b" of relation "tbl" violates not-null constraint
DETAIL: Failing row contains (1, null).
-- subscriber --
postgres=# select * from tbl;
a | b
---+---
(0 rows)
2. Handling of partitioned tables vs. Replica Identity (RI): When
adding a partitioned table with a column list to the publication (with
publish_via_partition_root = false), we should check the Replica
Identity of all its leaf partition as the RI on the partition is the
one actually takes effect when publishing DML changes. We need to
check RI while attaching the partition as well, as the newly added
partitions will automatically become part of publication if the
partitioned table is part of the publication. If we don't do this the
later deletes/updates can fail.
Please see the following 3 cases about partition.
Case1 (publish a parent table which has a partition table):
----------------------------
-- publisher --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create unique INDEX ON child (a,b);
alter table child alter a set not null;
alter table child alter b set not null;
alter table child replica identity using INDEX child_a_b_idx;
create publication pub for table parent(a) with(publish_via_partition_root=false);
-- subscriber --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create subscription sub connection 'port=5432 dbname=postgres' publication pub;
-- publisher --
insert into parent values (1,1);
-- subscriber --
postgres=# select * from parent;
a | b
---+---
1 |
(1 row)
-- add RI in subscriber to avoid other errors
update child set b=1 where a=1;
create unique INDEX ON child (a,b);
alter table child alter a set not null;
alter table child alter b set not null;
alter table child replica identity using INDEX child_a_b_idx;
-- publisher --
delete from parent;
The subscriber reported the following error message and DELETE failed in subscriber.
ERROR: publisher did not send replica identity column expected by the logical replication target relation "public.child"
CONTEXT: processing remote data during "DELETE" for replication target relation "public.child" in transaction 727 at 2022-01-14 20:29:46.50784+08
-- subscriber --
postgres=# select * from parent;
a | b
---+---
1 | 1
(1 row)
Case2 (create publication for parent table, then alter table to attach partition):
----------------------------
-- publisher --
create table parent (a int, b int) partition by range (a);
create table child (a int, b int);
create unique INDEX ON child (a,b);
alter table child alter a set not null;
alter table child alter b set not null;
alter table child replica identity using INDEX child_a_b_idx;
create publication pub for table parent(a) with(publish_via_partition_root=false);
alter table parent attach partition child default;
insert into parent values (1,1);
-- subscriber --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create subscription sub connection 'port=5432 dbname=postgres' publication pub;
postgres=# select * from parent;
a | b
---+---
1 |
(1 row)
-- add RI in subscriber to avoid other errors
update child set b=1 where a=1;
create unique INDEX ON child (a,b);
alter table child alter a set not null;
alter table child alter b set not null;
alter table child replica identity using INDEX child_a_b_idx;
-- publisher --
delete from parent;
The subscriber reported the following error message and DELETE failed in subscriber.
ERROR: publisher did not send replica identity column expected by the logical replication target relation "public.child"
CONTEXT: processing remote data during "DELETE" for replication target relation "public.child" in transaction 728 at 2022-01-14 20:42:16.483878+08
-- subscriber --
postgres=# select * from parent;
a | b
---+---
1 | 1
(1 row)
Case3 (create publication for parent table, then using "create table partition
of", and specify primary key when creating partition table):
----------------------------
-- publisher --
create table parent (a int, b int) partition by range (a);
create publication pub for table parent(a) with(publish_via_partition_root=false);
create table child partition of parent (primary key (a,b)) default;
-- subscriber --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create subscription sub connection 'port=5432 dbname=postgres' publication pub;
-- publisher --
insert into parent values (1,1);
-- subscriber --
postgres=# select * from parent;
a | b
---+---
1 |
(1 row)
-- add PK in subscriber to avoid other errors
update child set b=1 where a=1;
alter table child add primary key (a,b);
-- publisher --
delete from parent;
The subscriber reported the following error message and DELETE failed in subscriber.
ERROR: publisher did not send replica identity column expected by the logical replication target relation "public.child"
CONTEXT: processing remote data during "DELETE" for replication target relation "public.child" in transaction 723 at 2022-01-14 20:45:33.622168+08
-- subscriber --
postgres=# select * from parent;
a | b
---+---
1 | 1
(1 row)
3. Tablesync.c handling: Ideally, it would be good if we have a single
query to fetch both row filters and column filters but even if that is
not possible in the first version, the behavior should be same for
both queries w.r.t partitioned tables, For ALL Tables and For All
Tables In Schema cases.Currently, the column filter patch doesn't seem to respect For ALL
Tables and For All Tables In Schema cases, basically, it just copies
the columns it finds through some of the publications even if one of
the publications is defined as For All Tables. The row filter patch
ignores the row filters if one of the publications is defined as For
ALL Tables and For All Tables In Schema.
A case for the publications is defined as For ALL Tables and For All Tables In
Schema:
-- publisher --
create schema s1;
create table s1.t1 (a int, b int);
create publication p1 for table s1.t1 (a);
create publication p2 for all tables;
insert into s1.t1 values (1,1);
-- subscriber --
create schema s1;
create table s1.t1 (a int, b int);
create subscription sub connection 'port=5432 dbname=postgres' publication p1, p2;
postgres=# select * from s1.t1;
a | b
---+---
1 |
(1 row)
(I got the same result when p2 is specified as "FOR ALL TABLES IN SCHEMA s1")
For row filter patch, if the publication contains a partitioned table,
the publication parameter publish_via_partition_root determines if it
uses the partition row filter (if the parameter is false, the default)
or the root partitioned table row filter and this is taken care of
even during the initial tablesync.For column filter patch, if the publication contains a partitioned
table, it seems that it finds all columns that the tables in its
partition tree specified in the publications, whether
publish_via_partition_root is true or false.
Please see the following cases.
Column filter
----------------------------------------
-- publisher --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create publication p1 for table parent (a) with(publish_via_partition_root=false);
create publication p2 for table parent (a) with(publish_via_partition_root=true);
insert into parent values (1,1);
-- subscriber --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create subscription sub connection 'port=5432 dbname=postgres' publication p1;
postgres=# select * from parent; -- column filter works when publish_via_partition_root=false
a | b
---+---
1 |
(1 row)
drop subscription sub;
delete from parent;
create subscription sub connection 'port=5432 dbname=postgres' publication p2;
postgres=# select * from parent; -- column filter also works when publish_via_partition_root=true
a | b
---+---
1 |
(1 row)
Row filter
----------------------------------------
-- publisher --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create publication p1 for table parent where (a>10) with(publish_via_partition_root=false);
create publication p2 for table parent where (a>10) with(publish_via_partition_root=true);
insert into parent values (1,1);
insert into parent values (11,11);
-- subscriber
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create subscription sub connection 'port=5432 dbname=postgres' publication p1;
postgres=# select * from parent; -- row filter doesn't work when publish_via_partition_root=false
a | b
----+----
1 | 1
11 | 11
(2 rows)
drop subscription sub;
delete from parent;
create subscription sub connection 'port=5432 dbname=postgres' publication p2;
postgres=# select * from parent; -- row filter works when publish_via_partition_root=true
a | b
----+----
11 | 11
(1 row)
Regards,
Tang
On 2022-Jan-14, Amit Kapila wrote:
1. Replica Identity handling: Currently the column filter patch gives
an error during create/alter subscription if the specified column list
is invalid (Replica Identity columns are missing). It also gives an
error if the user tries to change the replica identity. However, it
doesn't deal with cases where the user drops and adds a different
primary key that has a different set of columns which can lead to
failure during apply on the subscriber.
Hmm, yeah, I suppose we should check that the primary key is compatible
with the column list in all publications. (I wonder what happens in the
interim, that is, what happens to tuples modified after the initial PK
is dropped and before the new PK is installed. Are these considered to
have "replica identiy nothing"?)
I think another issue w.r.t column filter patch is that even while
creating publication (even for 'insert' publications) it should check
that all primary key columns must be part of published columns,
otherwise, it can fail while applying on subscriber as it will try to
insert NULL for the primary key column.
I'm not so sure about the primary key aspects, actually; keep in mind
that the replica can have a different table definition, and it might
have even a completely different primary key. I think this part is up
to the user to set up correctly; we have enough with just trying to make
the replica identity correct.
2. Handling of partitioned tables vs. Replica Identity (RI): When
adding a partitioned table with a column list to the publication (with
publish_via_partition_root = false), we should check the Replica
Identity of all its leaf partition as the RI on the partition is the
one actually takes effect when publishing DML changes. We need to
check RI while attaching the partition as well, as the newly added
partitions will automatically become part of publication if the
partitioned table is part of the publication. If we don't do this the
later deletes/updates can fail.
Hmm, yeah.
3. Tablesync.c handling: Ideally, it would be good if we have a single
query to fetch both row filters and column filters but even if that is
not possible in the first version, the behavior should be same for
both queries w.r.t partitioned tables, For ALL Tables and For All
Tables In Schema cases.Currently, the column filter patch doesn't seem to respect For ALL
Tables and For All Tables In Schema cases, basically, it just copies
the columns it finds through some of the publications even if one of
the publications is defined as For All Tables. The row filter patch
ignores the row filters if one of the publications is defined as For
ALL Tables and For All Tables In Schema.
Oh, yeah, if a table appears in two publications and one of them is ALL
TABLES [IN SCHEMA], then we don't consider it as an all-columns
publication. You're right, that should be corrected.
We have done some testing w.r.t above cases with both patches and my
colleague will share the results.
Great, thanks.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
On Fri, Jan 14, 2022 at 7:08 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2022-Jan-14, Amit Kapila wrote:
1. Replica Identity handling: Currently the column filter patch gives
an error during create/alter subscription if the specified column list
is invalid (Replica Identity columns are missing). It also gives an
error if the user tries to change the replica identity. However, it
doesn't deal with cases where the user drops and adds a different
primary key that has a different set of columns which can lead to
failure during apply on the subscriber.Hmm, yeah, I suppose we should check that the primary key is compatible
with the column list in all publications. (I wonder what happens in the
interim, that is, what happens to tuples modified after the initial PK
is dropped and before the new PK is installed. Are these considered to
have "replica identiy nothing"?)
I think so.
I think another issue w.r.t column filter patch is that even while
creating publication (even for 'insert' publications) it should check
that all primary key columns must be part of published columns,
otherwise, it can fail while applying on subscriber as it will try to
insert NULL for the primary key column.I'm not so sure about the primary key aspects, actually; keep in mind
that the replica can have a different table definition, and it might
have even a completely different primary key. I think this part is up
to the user to set up correctly; we have enough with just trying to make
the replica identity correct.
But OTOH, the primary key is also considered default replica identity,
so I think users will expect it to work. You are right this problem
can also happen if the user defined a different primary key on a
replica but that is even a problem in HEAD (simple inserts will fail)
but I am worried about the case where both the publisher and
subscriber have the same primary key as that works in HEAD.
--
With Regards,
Amit Kapila.
On 15.01.22 04:45, Amit Kapila wrote:
I think another issue w.r.t column filter patch is that even while
creating publication (even for 'insert' publications) it should check
that all primary key columns must be part of published columns,
otherwise, it can fail while applying on subscriber as it will try to
insert NULL for the primary key column.I'm not so sure about the primary key aspects, actually; keep in mind
that the replica can have a different table definition, and it might
have even a completely different primary key. I think this part is up
to the user to set up correctly; we have enough with just trying to make
the replica identity correct.But OTOH, the primary key is also considered default replica identity,
so I think users will expect it to work. You are right this problem
can also happen if the user defined a different primary key on a
replica but that is even a problem in HEAD (simple inserts will fail)
but I am worried about the case where both the publisher and
subscriber have the same primary key as that works in HEAD.
This would seem to be a departure from the current design of logical
replication. It's up to the user to arrange things so that data can be
applied in general. Otherwise, if the default assumption is that the
schema is the same on both sides, then column filtering shouldn't exist
at all, since that will necessarily break that assumption.
Maybe there could be a strict mode or something that has more checks,
but that would be a separate feature. The existing behavior is that you
can publish anything you want and it's up to you to make sure the
receiving side can store it.
Here are some review comments for the v17-0001 patch.
~~~
1. Commit message
If no column list is specified, all the columns are replicated, as
previously
Missing period (.) at the end of that sentence.
~~~
2. doc/src/sgml/catalogs.sgml
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all attributes are published.
+ </para></entry>
Missing comma:
"For example" --> "For example,"
Terms:
The text seems to jump between "columns" and "attributes". Perhaps,
for consistency, that last sentence should say: "A null value
indicates that all columns are published."
~~~
3. doc/src/sgml/protocol.sgml
</variablelist>
- Next, the following message part appears for each column
(except generated columns):
+ Next, the following message part appears for each column (except
+ generated columns and other columns that don't appear in the column
+ filter list, for tables that have one):
<variablelist>
Perhaps that can be expressed more simply, like:
Next, the following message part appears for each column (except
generated columns and other columns not present in the optional column
filter list):
~~~
4. doc/src/sgml/ref/alter_publication.sgml
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable>
ALTER TABLE <replaceable
class="parameter">publication_object</replaceable> SET COLUMNS { (
<replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
The syntax chart looks strange because there is already a "TABLE" and
a column_name list within the "publication_object" definition, so do
ALTER TABLE and publication_object co-exist?
According to the current documentation it suggests nonsense like below is valid:
ALTER PUBLICATION mypublication ALTER TABLE TABLE t1 (a,b,c) SET
COLUMNS (a,b,c);
--
But more fundamentally, I don't see why any new syntax is even needed at all.
Instead of:
ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS
(user_id, firstname, lastname);
Why not just:
ALTER PUBLICATION mypublication ALTER TABLE users (user_id, firstname,
lastname);
Then, if the altered table defines a *different* column list then it
would be functionally equivalent to whatever your SET COLUMNS is doing
now. AFAIK this is how the Row-Filter [1]/messages/by-id/CAHut+PtNWXPba0h=do_UiwaEziePNr7Z+58+-ctpyP2Pq1VkPw@mail.gmail.com works, so that altering an
existing table to have a different Row-Filter just overwrites that
table's filter. IMO the Col-Filter behaviour should work the same as
that - "SET COLUMNS" is redundant.
~~~
5. doc/src/sgml/ref/alter_publication.sgml
- TABLE [ ONLY ] <replaceable
class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable
class="parameter">table_name</replaceable> [ * ] [ ( <replaceable
class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
That extra comma after the "column_name" seems wrong because there is
one already in "[, ... ]".
~~~
6. doc/src/sgml/ref/create_publication.sgml
- TABLE [ ONLY ] <replaceable
class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable
class="parameter">table_name</replaceable> [ * ] [ ( <replaceable
class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
(Same as comment #5).
That extra comma after the "column_name" seems wrong because there is
one already in "[, ... ]".
~~~
7. doc/src/sgml/ref/create_publication.sgml
+ <para>
+ When a column list is specified, only the listed columns are replicated;
+ any other columns are ignored for the purpose of replication through
+ this publication. If no column list is specified, all columns of the
+ table are replicated through this publication, including any columns
+ added later. If a column list is specified, it must include the replica
+ identity columns.
+ </para>
Suggest to re-word this a bit simpler:
e.g.
- "listed columns" --> "named columns"
- I don't think it is necessary to say the unlisted columns are ignored.
- I didn't think it is necessary to say "though this publication"
AFTER
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated,
including any columns added later. If a column list is specified, it must
include the replica identity columns.
~~~
8. doc/src/sgml/ref/create_publication.sgml
Consider adding another example showing a CREATE PUBLICATION which has
a column list.
~~~
9. src/backend/catalog/pg_publication.c - check_publication_add_relation
/*
- * Check if relation can be in given publication and throws appropriate
- * error if not.
+ * Check if relation can be in given publication and that the column
+ * filter is sensible, and throws appropriate error if not.
+ *
+ * targetcols is the bitmapset of attribute numbers given in the column list,
+ * or NULL if it was not specified.
*/
Typo: "targetcols" --> "columns" ??
~~~
10. src/backend/catalog/pg_publication.c - check_publication_add_relation
+
+ /* Make sure the column list checks out */
+ if (columns != NULL)
+ {
Perhaps "checks out" could be worded better.
~~~
11. src/backend/catalog/pg_publication.c - check_publication_add_relation
+ /* Make sure the column list checks out */
+ if (columns != NULL)
+ {
+ /*
+ * Even if the user listed all columns in the column list, we cannot
+ * allow a column list to be specified when REPLICA IDENTITY is FULL;
+ * that would cause problems if a new column is added later, because
+ * the new column would have to be included (because of being part of
+ * the replica identity) but it's technically not allowed (because of
+ * not being in the publication's column list yet). So reject this
+ * case altogether.
+ */
+ if (replidentfull)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot specify a column list on relations with REPLICA
IDENTITY FULL."));
+
+ check_publication_columns(pub, targetrel, columns);
+ }
IIUC almost all of the above comment and code is redundant because by
calling the check_publication_columns function it will do exactly the
same check...
So, that entire slab might be replaced by 2 lines:
if (columns != NULL)
check_publication_columns(pub, targetrel, columns);
~~~
12. src/backend/catalog/pg_publication.c - publication_set_table_columns
+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns)
+{
+ Bitmapset *attset;
+ AttrNumber *attarray;
+ HeapTuple copytup;
+ int natts;
+ bool nulls[Natts_pg_publication_rel];
+ bool replaces[Natts_pg_publication_rel];
+ Datum values[Natts_pg_publication_rel];
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
It seemed curious to use memset false for "replaces" but memset 0 for
"nulls", since they are both bool arrays (??)
~~~
13. src/backend/catalog/pg_publication.c - compare_int16
+/* qsort comparator for attnums */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}
This comparator seems common with another one already in the PG
source. Perhaps it would be better for generic comparators (like this
one) to be in some common code instead of scattered cut/paste copies
of the same thing.
~~~
14. src/backend/commands/publicationcmds.c - AlterPublicationTables
+ else if (stmt->action == AP_SetColumns)
+ {
+ Assert(schemaidlist == NIL);
+ Assert(list_length(tables) == 1);
+
+ PublicationSetColumns(stmt, pubform,
+ linitial_node(PublicationTable, tables));
+ }
(Same as my earlier review comment #4)
Suggest to call this PublicationSetColumns based on some smarter
detection logic of a changed column list. Please refer to the
Row-Filter patch [1]/messages/by-id/CAHut+PtNWXPba0h=do_UiwaEziePNr7Z+58+-ctpyP2Pq1VkPw@mail.gmail.com for this same function.
~~~
15. src/backend/commands/publicationcmds.c - AlterPublicationTables
+ /* This is not needed to delete a table */
+ pubrel->columns = NIL;
Perhaps a more explanatory comment would be better there?
~~~
16. src/backend/commands/tablecmds.c - relation_mark_replica_identity
@@ -15841,6 +15871,7 @@ relation_mark_replica_identity(Relation rel,
char ri_type, Oid indexOid,
CatalogTupleUpdate(pg_index, &pg_index_tuple->t_self, pg_index_tuple);
InvokeObjectPostAlterHookArg(IndexRelationId, thisIndexOid, 0,
InvalidOid, is_internal);
+
/*
* Invalidate the relcache for the table, so that after we commit
* all sessions will refresh the table's replica identity index
Spurious whitespace change seemed unrelated to the Col-Filter patch.
~~~
17. src/backend/parser/gram.y
*
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
(Same as my earlier review comment #4)
IMO there was no need for the new syntax of SET COLUMNS.
~~~
18. src/backend/replication/logical/proto.c - logicalrep_write_attrs
- /* send number of live attributes */
- for (i = 0; i < desc->natts; i++)
- {
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc,
i)->attgenerated)
- continue;
- nliveatts++;
- }
- pq_sendint16(out, nliveatts);
-
/* fetch bitmap of REPLICATION IDENTITY attributes */
replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
if (!replidentfull)
idattrs = RelationGetIdentityKeyBitmap(rel);
+ /* send number of live attributes */
+ for (i = 0; i < desc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+ nliveatts++;
+ }
+ pq_sendint16(out, nliveatts);
+
This change seemed to have the effect of moving that 4 lines of
"replidentfull" code from below the loop to above the loop. But moving
that code seems unrelated to the Col-Filter patch. (??).
~~~
19. src/backend/replication/logical/tablesync.c - fetch_remote_table_info
@@ -793,12 +877,12 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecClearTuple(slot);
}
+
ExecDropSingleTupleTableSlot(slot);
-
- lrel->natts = natt;
-
walrcv_clear_result(res);
pfree(cmd.data);
+
+ lrel->natts = natt;
}
The shuffling of those few lines seems unrelated to any requirement of
the Col-Filter patch (??)
~~~
20. src/backend/replication/logical/tablesync.c - copy_table
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
Perhaps that could be expressed more simply if the other way around like:
for (int i = 0; i < lrel.natts; i++)
{
if (i)
appendStringInfoString(&cmd, ", ");
appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
}
~~~
21. src/backend/replication/pgoutput/pgoutput.c
+
+ /*
+ * Set of columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this list are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
Typo: "in this list" --> "in this set" (??)
~~~
22. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry
* Don't publish changes for partitioned tables, because
- * publishing those of its partitions suffices, unless partition
- * changes won't be published due to pubviaroot being set.
+ * publishing those of its partitions suffices. (However, ignore
+ * this if partition changes are not to published due to
+ * pubviaroot being set.)
*/
This change seems unrelated to the Col-Filter patch, so perhaps it
should not be here at all.
Also, typo: "are not to published"
~~~
23. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry
+ /*
+ * Obtain columns published by this publication, and add them
+ * to the list for this rel. Note that if at least one
+ * publication has a empty column list, that means to publish
+ * everything; so if we saw a publication that includes all
+ * columns, skip this.
+ */
Typo: "a empty" --> "an empty"
~~~
24. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry
+ if (isnull)
+ {
+ /*
+ * If we see a publication with no columns, reset the
+ * list and ignore further ones.
+ */
Perhaps that comment is meant to say "with no column filter" instead
of "with no columns"?
~~~
25. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry
+ if (isnull)
+ {
...
+ }
+ else if (!isnull)
+ {
...
+ }
Is the "if (!isnull)" in the else just to be really REALLY sure it is not null?
~~~
26. src/bin/pg_dump/pg_dump.c - getPublicationTables
+ pubrinfo[i].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
I got confused reading this code. Are those different indices 'i' and
'j' correct?
~~~
27. src/bin/psql/describe.c
The Row-Filter [1]/messages/by-id/CAHut+PtNWXPba0h=do_UiwaEziePNr7Z+58+-ctpyP2Pq1VkPw@mail.gmail.com displays filter information not only for the psql
\dRp+ command but also for the psql \d <tablename> command. Perhaps
the Col-Filter patch should do that too.
~~~
28. src/bin/psql/tab-complete.c
@@ -1657,6 +1657,8 @@ psql_completion(const char *text, int start, int end)
/* ALTER PUBLICATION <name> ADD */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD"))
COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
+ else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD", "TABLE"))
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL);
/* ALTER PUBLICATION <name> DROP */
I am not sure about this one- is that change even related to the
Col-Filter patch or is this some unrelated bugfix?
~~~
29. src/include/catalog/pg_publication.h
@@ -86,6 +86,7 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+ List *columns;
} PublicationRelInfo;
Perhaps that needs some comment. e.g. do you need to mention that a
NIL List means all columns?
~~~
30. src/include/nodes/parsenodes.h
@@ -3642,6 +3642,7 @@ typedef struct PublicationTable
{
NodeTag type;
RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
That comment "List of columns in a publication table" doesn't really
say anything helpful.
Perhaps it should mention that a NIL List means all table columns?
~~~
31. src/test/regress/sql/publication.sql
The regression test file has an uncommon mixture of /* */ and -- style comments.
Perhaps change all the /* */ ones?
~~~
32. src/test/regress/sql/publication.sql
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
For all these tests (and more) there seems not sufficient explanation
comments to say exactly what each test case is testing, e.g. *why* is
an "error" expected for some cases but "ok" for others.
~~~
33. src/test/regress/sql/publication.sql
"-- no dice"
(??) confusing comment.
~~~
34. src/test/subscription/t/028_column_list.pl
I think a few more comments in this TAP file would help to make the
purpose of the tests more clear.
------
[1]: /messages/by-id/CAHut+PtNWXPba0h=do_UiwaEziePNr7Z+58+-ctpyP2Pq1VkPw@mail.gmail.com
Kind Regards,
Peter Smith.
Fujitsu Australia
On 12.01.22 01:41, Alvaro Herrera wrote:
I discovered a big hole in this, which is that ALTER PUBLICATION SET
(publish='insert,update') can add UPDATE publishing to a publication
that was only publishing INSERTs. It's easy to implement a fix: in
AlterPublicationOptions, scan the list of tables and raise an error if
any of them has a column list that doesn't include all the columns in
the replica identity.
Right now, we are not checking the publication options and the replica
identity combinations at all at DDL time. This is only checked at
execution time in CheckCmdReplicaIdentity(). So under that scheme I
don't think the check you describe is actually necessary. Let the user
set whatever combination they want, and check at execution time if it's
an UPDATE or DELETE command whether the replica identity is sufficient.
Hi,
Here's an updated version of the patch, rebased to current master. Parts
0002 and 0003 include various improvements based on review by me and
another one by Peter Smith [1]/messages/by-id/CAHut+Ptc7Rh187eQKrxdUmUNWyfxz7OkhYAX=AW411Qwxya0LQ@mail.gmail.com.
Part 0003 reworks and significantly extends the TAP test, to exercise
various cases related to changes of replica identity etc. discussed in
this thread. Some of the tests however still fail, because the behavior
was not updated - I'll work on that once we agree what the expected
behavior is.
1) partitioning with pubviaroot=true
The main set of failures is related to partitions with different replica
identities and (pubviaroot=true), some of which may be mismatching the
column list. There are multiple such test cases, depending on how the
inconsistency is introduced - it may be there from the beginning, the
column filter may be modified after adding the partitioned table to the
publication, etc.
I think the expected behavior is to prohibit such cases from happening,
by cross-checking the column filter when adding the partitioned table to
publication, attaching a partition or changing a column filter.
2) merging multiple column filters
When the table has multiple column filters (in different publications),
we need to merge them. Which works, except that FOR ALL TABLES [IN
SCHEMA] needs to be handled as "has no column filter" (and replicates
everything).
3) partitioning with pubivaroot=false
When a partitioned table is added with (pubviaroot=false), it should not
be subject to column filter on the parent relation, which is the same
behavior used by the row filtering patch.
regards
[1]: /messages/by-id/CAHut+Ptc7Rh187eQKrxdUmUNWyfxz7OkhYAX=AW411Qwxya0LQ@mail.gmail.com
/messages/by-id/CAHut+Ptc7Rh187eQKrxdUmUNWyfxz7OkhYAX=AW411Qwxya0LQ@mail.gmail.com
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
0001-Support-column-lists-for-logical-replicatio-20220216.patchtext/x-patch; charset=UTF-8; name=0001-Support-column-lists-for-logical-replicatio-20220216.patchDownload
From 1d5951c26a2821d20c995978eaa4028a39f56565 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Mon, 14 Feb 2022 17:13:04 +0100
Subject: [PATCH 1/4] Support column lists for logical replication of tables
Add the capability of specifying a column list for individual tables as
part of a publication. Columns not in the list are not published. This
enables replicating to a table with only a subset of the columns.
If no column list is specified, all the columns are replicated, as
previously
Author: Rahila Syed <rahilasyed90@gmail.com>
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 13 +
doc/src/sgml/protocol.sgml | 4 +-
doc/src/sgml/ref/alter_publication.sgml | 20 +-
doc/src/sgml/ref/create_publication.sgml | 11 +-
src/backend/catalog/pg_publication.c | 353 +++++++++++++++++++-
src/backend/commands/publicationcmds.c | 67 +++-
src/backend/commands/tablecmds.c | 87 ++++-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 60 +++-
src/backend/replication/logical/proto.c | 60 ++--
src/backend/replication/logical/tablesync.c | 119 ++++++-
src/backend/replication/pgoutput/pgoutput.c | 118 ++++++-
src/bin/pg_dump/pg_dump.c | 41 ++-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 26 +-
src/include/catalog/pg_publication.h | 6 +
src/include/catalog/pg_publication_rel.h | 3 +
src/include/nodes/parsenodes.h | 4 +-
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 59 +++-
src/test/regress/sql/publication.sql | 39 ++-
src/test/subscription/t/028_column_list.pl | 164 +++++++++
23 files changed, 1179 insertions(+), 84 deletions(-)
create mode 100644 src/test/subscription/t/028_column_list.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 5a1627a3941..f20cb2b78ab 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6325,6 +6325,19 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to relation
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>prattrs</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+ </para>
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all attributes are published.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 1c5ab008791..91541cd8cf7 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7005,7 +7005,9 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column (except
+ generated columns and other columns that don't appear in the column
+ filter list, for tables that have one):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 7c7c27bf7ce..a85574214a5 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -25,12 +25,13 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD <replace
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ALTER TABLE <replaceable class="parameter">publication_object</replaceable> SET COLUMNS { ( <replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -62,6 +63,11 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
command retain their previous settings.
</para>
+ <para>
+ The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows changing
+ the set of columns that are included in the publication.
+ </para>
+
<para>
The remaining variants change the owner and the name of the publication.
</para>
@@ -110,6 +116,8 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
</para>
</listitem>
</varlistentry>
@@ -164,9 +172,15 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
</para>
<para>
- Add some tables to the publication:
+ Add tables to the publication:
+<programlisting>
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS (user_id, firstname, lastname);
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 385975bfadd..a59cd3f532a 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -78,6 +78,15 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
publication, so they are never explicitly added to the publication.
</para>
+ <para>
+ When a column list is specified, only the listed columns are replicated;
+ any other columns are ignored for the purpose of replication through
+ this publication. If no column list is specified, all columns of the
+ table are replicated through this publication, including any columns
+ added later. If a column list is specified, it must include the replica
+ identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index e14ca2f5630..aa1655696f5 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,13 +45,26 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+
+static void check_publication_columns(Publication *pub, Relation targetrel,
+ Bitmapset *columns);
+static void publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs,
+ Bitmapset **attset);
+
/*
- * Check if relation can be in given publication and throws appropriate
- * error if not.
+ * Check if relation can be in given publication and that the column
+ * filter is sensible, and throws appropriate error if not.
+ *
+ * targetcols is the bitmapset of attribute numbers given in the column list,
+ * or NULL if it was not specified.
*/
static void
-check_publication_add_relation(Relation targetrel)
+check_publication_add_relation(Publication *pub, Relation targetrel,
+ Bitmapset *columns)
{
+ bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
@@ -82,6 +95,73 @@ check_publication_add_relation(Relation targetrel)
errmsg("cannot add relation \"%s\" to publication",
RelationGetRelationName(targetrel)),
errdetail("This operation is not supported for unlogged tables.")));
+
+ /* Make sure the column list checks out */
+ if (columns != NULL)
+ {
+ /*
+ * Even if the user listed all columns in the column list, we cannot
+ * allow a column list to be specified when REPLICA IDENTITY is FULL;
+ * that would cause problems if a new column is added later, because
+ * the new column would have to be included (because of being part of
+ * the replica identity) but it's technically not allowed (because of
+ * not being in the publication's column list yet). So reject this
+ * case altogether.
+ */
+ if (replidentfull)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot specify a column list on relations with REPLICA IDENTITY FULL."));
+
+ check_publication_columns(pub, targetrel, columns);
+ }
+}
+
+/*
+ * Enforce that the column list can only leave out columns that aren't
+ * forced to be sent.
+ *
+ * No column can be excluded if REPLICA IDENTITY is FULL (since all the
+ * columns need to be sent regardless); and in other cases, the columns in
+ * the REPLICA IDENTITY cannot be left out.
+ */
+static void
+check_publication_columns(Publication *pub, Relation targetrel, Bitmapset *columns)
+{
+ if (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot change column set for relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("Cannot specify column list on relations with REPLICA IDENTITY FULL."));
+
+ if (pub->pubactions.pubupdate || pub->pubactions.pubdelete)
+ {
+ Bitmapset *idattrs;
+ int x;
+
+ idattrs = RelationGetIndexAttrBitmap(targetrel,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * We have to test membership the hard way, because the values returned by
+ * RelationGetIndexAttrBitmap are offset.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ if (!bms_is_member(x + FirstLowInvalidHeapAttributeNumber, columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)),
+ errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
+ }
+
+ bms_free(idattrs);
+ }
}
/*
@@ -289,6 +369,9 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
Oid relid = RelationGetRelid(targetrel->relation);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ Bitmapset *attset = NULL;
+ AttrNumber *attarray;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -314,7 +397,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
RelationGetRelationName(targetrel->relation), pub->name)));
}
- check_publication_add_relation(targetrel->relation);
+ /* Translate column names to numbers and verify suitability */
+ publication_translate_columns(targetrel->relation,
+ targetrel->columns,
+ &natts, &attarray, &attset);
+
+ check_publication_add_relation(pub, targetrel->relation, attset);
+
+ bms_free(attset);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -327,6 +417,15 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ if (targetrel->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -337,6 +436,13 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
/* Register dependencies as needed */
ObjectAddressSet(myself, PublicationRelRelationId, pubreloid);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ pfree(attarray);
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -364,6 +470,155 @@ publication_add_relation(Oid pubid, PublicationRelInfo *targetrel,
return myself;
}
+/*
+ * Update the column list for a relation in a publication.
+ */
+void
+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns)
+{
+ Bitmapset *attset;
+ AttrNumber *attarray;
+ HeapTuple copytup;
+ int natts;
+ bool nulls[Natts_pg_publication_rel];
+ bool replaces[Natts_pg_publication_rel];
+ Datum values[Natts_pg_publication_rel];
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ replaces[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ deleteDependencyRecordsForClass(PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid,
+ RelationRelationId,
+ DEPENDENCY_AUTO);
+
+ if (columns == NULL)
+ {
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+ }
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ int2vector *prattrs;
+ Publication *pub;
+
+ pub = GetPublication(((Form_pg_publication_rel) GETSTRUCT(pubreltup))->prpubid);
+
+ publication_translate_columns(targetrel, columns,
+ &natts, &attarray, &attset);
+
+ /*
+ * Make sure the column list checks out. XXX this should occur at
+ * caller in publicationcmds.c, not here.
+ */
+ check_publication_columns(pub, targetrel, attset);
+ bms_free(attset);
+ /* XXX "pub" is leaked here */
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+
+ /* Add dependencies on the new list of columns */
+ ObjectAddressSet(myself, PublicationRelRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid);
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId,
+ RelationGetRelid(targetrel), attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ }
+
+ copytup = heap_modify_tuple(pubreltup, RelationGetDescr(pubrel),
+ values, nulls, replaces);
+
+ CatalogTupleUpdate(pubrel, &pubreltup->t_self, copytup);
+
+ heap_freetuple(copytup);
+}
+
+/* qsort comparator for attnums */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list. Other checks are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ */
+static void
+publication_translate_columns(Relation targetrel, List *columns, int *natts,
+ AttrNumber **attrs, Bitmapset **attset)
+{
+ AttrNumber *attarray;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference generated column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /* Be tidy, so that the catalog representation is always sorted */
+ qsort(attarray, n, sizeof(AttrNumber), compare_int16);
+
+ *natts = n;
+ *attrs = attarray;
+ *attset = set;
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -471,6 +726,96 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull)
+ continue;
+
+ pubs = lappend_oid(pubs,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+/* FIXME maybe these two routines should be in lsyscache.c */
+/* Return the set of actions that the given publication includes */
+void
+GetActionsInPublication(Oid pubid, PublicationActions *actions)
+{
+ HeapTuple pub;
+ Form_pg_publication pubForm;
+
+ pub = SearchSysCache1(PUBLICATIONOID,
+ ObjectIdGetDatum(pubid));
+ if (!HeapTupleIsValid(pub))
+ elog(ERROR, "cache lookup failed for publication %u", pubid);
+
+ pubForm = (Form_pg_publication) GETSTRUCT(pub);
+ actions->pubinsert = pubForm->pubinsert;
+ actions->pubupdate = pubForm->pubupdate;
+ actions->pubdelete = pubForm->pubdelete;
+ actions->pubtruncate = pubForm->pubtruncate;
+
+ ReleaseSysCache(pub);
+}
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 0e4bb97fb73..592f56dd61e 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -376,6 +376,46 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
return myself;
}
+/*
+ * Change the column list of a relation in a publication
+ */
+static void
+PublicationSetColumns(AlterPublicationStmt *stmt,
+ Form_pg_publication pubform, PublicationTable *table)
+{
+ Relation rel,
+ urel;
+ HeapTuple tup;
+ ObjectAddress obj,
+ secondary;
+
+ rel = table_open(PublicationRelRelationId, RowExclusiveLock);
+ urel = table_openrv(table->relation, ShareUpdateExclusiveLock);
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(RelationGetRelid(urel)),
+ ObjectIdGetDatum(pubform->oid));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errmsg("relation \"%s\" is not already in publication \"%s\"",
+ table->relation->relname,
+ NameStr(pubform->pubname)));
+
+ publication_set_table_columns(rel, tup, urel, table->columns);
+
+ ObjectAddressSet(obj, PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->oid);
+ ObjectAddressSet(secondary, RelationRelationId, RelationGetRelid(urel));
+ EventTriggerCollectSimpleCommand(obj, secondary, (Node *) stmt);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+ table_close(urel, NoLock);
+
+ InvokeObjectPostAlterHook(PublicationRelationId, pubform->oid, 0);
+}
+
/*
* Change options of a publication.
*/
@@ -523,6 +563,14 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
}
else if (stmt->action == AP_DropObjects)
PublicationDropTables(pubid, rels, false);
+ else if (stmt->action == AP_SetColumns)
+ {
+ Assert(schemaidlist == NIL);
+ Assert(list_length(tables) == 1);
+
+ PublicationSetColumns(stmt, pubform,
+ linitial_node(PublicationTable, tables));
+ }
else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
@@ -562,7 +610,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
pubrel = palloc(sizeof(PublicationRelInfo));
pubrel->relation = oldrel;
-
+ /* This is not needed to delete a table */
+ pubrel->columns = NIL;
delrels = lappend(delrels, pubrel);
}
}
@@ -622,7 +671,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
}
else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* AP_SetObjects */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -645,6 +694,10 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
*/
PublicationAddSchemas(pubform->oid, schemaidlist, true, stmt);
}
+ else
+ {
+ /* Nothing to do for AP_SetColumns */
+ }
}
/*
@@ -934,6 +987,8 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
@@ -967,8 +1022,11 @@ OpenTableList(List *tables)
/* find_all_inheritors already got lock */
rel = table_open(childrelid, NoLock);
+
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
+ pub_rel->columns = t->columns;
+
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
}
@@ -1076,6 +1134,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 3e83f375b55..28de9329800 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -8365,6 +8365,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
bool missing_ok, LOCKMODE lockmode,
ObjectAddresses *addrs)
{
+ Oid relid = RelationGetRelid(rel);
HeapTuple tuple;
Form_pg_attribute targetatt;
AttrNumber attnum;
@@ -8384,7 +8385,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/*
* get the number of the attribute
*/
- tuple = SearchSysCacheAttName(RelationGetRelid(rel), colName);
+ tuple = SearchSysCacheAttName(relid, colName);
if (!HeapTupleIsValid(tuple))
{
if (!missing_ok)
@@ -8438,13 +8439,42 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * Also, if the column is used in the column list of a publication,
+ * disallow the drop if the DROP is RESTRICT. We don't do anything if the
+ * DROP is CASCADE, which means that the dependency mechanism will remove
+ * the relation from the publication.
+ */
+ if (behavior == DROP_RESTRICT)
+ {
+ List *pubs;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(relid);
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(relid, pubid);
+
+ if (list_member_oid(published_cols, attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot drop column \"%s\" because it is part of publication \"%s\"",
+ colName, get_publication_name(pubid, false)),
+ errhint("Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication."));
+ }
+ }
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
* use find_all_inheritors to do it in one pass.
*/
children =
- find_inheritance_children(RelationGetRelid(rel), lockmode);
+ find_inheritance_children(relid, lockmode);
if (children)
{
@@ -8532,7 +8562,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Add object to delete */
object.classId = RelationRelationId;
- object.objectId = RelationGetRelid(rel);
+ object.objectId = relid;
object.objectSubId = attnum;
add_exact_object_address(&object, addrs);
@@ -15842,6 +15872,7 @@ relation_mark_replica_identity(Relation rel, char ri_type, Oid indexOid,
CatalogTupleUpdate(pg_index, &pg_index_tuple->t_self, pg_index_tuple);
InvokeObjectPostAlterHookArg(IndexRelationId, thisIndexOid, 0,
InvalidOid, is_internal);
+
/*
* Invalidate the relcache for the table, so that after we commit
* all sessions will refresh the table's replica identity index
@@ -15864,6 +15895,11 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
Oid indexOid;
Relation indexRel;
int key;
+ List *pubs;
+ Bitmapset *indexed_cols = NULL;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(RelationGetRelid(rel));
if (stmt->identity_type == REPLICA_IDENTITY_DEFAULT)
{
@@ -15872,11 +15908,16 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
}
else if (stmt->identity_type == REPLICA_IDENTITY_FULL)
{
+ if (pubs != NIL)
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set REPLICA IDENTITY FULL when publications contain relations that specify column lists"));
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
else if (stmt->identity_type == REPLICA_IDENTITY_NOTHING)
{
+ /* XXX not sure what's the right check for publications here */
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
return;
}
@@ -15961,6 +16002,46 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
errmsg("index \"%s\" cannot be used as replica identity because column \"%s\" is nullable",
RelationGetRelationName(indexRel),
NameStr(attr->attname))));
+
+ /*
+ * Collect columns used, in case we have any publications that we need
+ * to vet. System attributes are disallowed so no need to subtract
+ * FirstLowInvalidHeapAttributeNumber.
+ */
+ indexed_cols = bms_add_member(indexed_cols, attno);
+ }
+
+ /*
+ * Check partial-column publications. For those that include UPDATE and
+ * DELETE, we must enforce that the columns in the replica identity are
+ * included in the column list. For publications that only include INSERT
+ * and TRUNCATE, we don't need to restrict the replica identity.
+ */
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+ PublicationActions actions;
+
+ GetActionsInPublication(pubid, &actions);
+ /* No need to worry about this one */
+ if (!actions.pubupdate && !actions.pubdelete)
+ continue;
+
+ published_cols =
+ GetRelationColumnListInPublication(RelationGetRelid(rel), pubid);
+
+ for (key = 0; key < IndexRelationGetNumberOfKeyAttributes(indexRel); key++)
+ {
+ int16 attno = indexRel->rd_index->indkey.values[key];
+
+ if (!list_member_oid(published_cols, attno))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("index \"%s\" cannot be used because publication \"%s\" does not include all indexed columns",
+ RelationGetRelationName(indexRel),
+ get_publication_name(pubid, false)));
+ }
}
/* This index is suitable for use as a replica identity. Mark it. */
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index bc0d90b4b1b..58603ab18e1 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4849,6 +4849,7 @@ _copyPublicationTable(const PublicationTable *from)
PublicationTable *newnode = makeNode(PublicationTable);
COPY_NODE_FIELD(relation);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 2e7122ad2f6..f7f1b8be2ed 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2321,6 +2321,7 @@ static bool
_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 92f93cfc72d..3e7455dd5e9 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9751,12 +9751,13 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr
+ TABLE relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
+ $$->pubtable->columns = $3;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9771,28 +9772,38 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId
+ | ColId opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- $$->name = $1;
+ if ($2 != NULL)
+ {
+ $$->pubtable = makeNode(PublicationTable);
+ $$->pubtable->relation = makeRangeVar(NULL, $1, @1);
+ $$->pubtable->columns = $2;
+ $$->name = NULL;
+ }
+ else
+ $$->name = $1;
$$->location = @1;
}
- | ColId indirection
+ | ColId indirection opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
+ $$->pubtable->columns = $3;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr
+ | extended_relation_expr opt_column_list
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
+ $$->pubtable->columns = $2;
}
| CURRENT_SCHEMA
{
@@ -9818,6 +9829,9 @@ pub_obj_list: PublicationObjSpec
*
* ALTER PUBLICATION name SET pub_obj [, ...]
*
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
* pub_obj is one of:
*
* TABLE table_name [, ...]
@@ -9851,6 +9865,32 @@ AlterPublicationStmt:
n->action = AP_SetObjects;
$$ = (Node *)n;
}
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS '(' columnList ')'
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = $10;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS ALL
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = NIL;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
| ALTER PUBLICATION name DROP pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
@@ -17462,6 +17502,16 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
else if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLES_IN_SCHEMA ||
pubobj->pubobjtype == PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA)
{
+ /*
+ * This can happen if a column list is specified in a continuation
+ * for a schema entry; reject it.
+ */
+ if (pubobj->pubtable)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schemas"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 953942692ce..e6da46d83e5 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,9 +29,11 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple, bool binary);
+ HeapTuple tuple, bool binary,
+ Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +400,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple newtuple, bool binary)
+ HeapTuple newtuple, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +412,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -442,7 +444,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
- HeapTuple oldtuple, HeapTuple newtuple, bool binary)
+ HeapTuple oldtuple, HeapTuple newtuple, bool binary,
+ Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -463,11 +466,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple, binary);
+ logicalrep_write_tuple(out, rel, newtuple, binary, columns);
}
/*
@@ -536,7 +539,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple, binary);
+ logicalrep_write_tuple(out, rel, oldtuple, binary, NULL);
}
/*
@@ -651,7 +654,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -673,7 +677,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -749,7 +753,8 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple,
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -761,7 +766,13 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ /* Don't count attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
continue;
nliveatts++;
}
@@ -783,6 +794,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
if (att->attisdropped || att->attgenerated)
continue;
+ /* Ignore attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +919,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -914,20 +929,24 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
desc = RelationGetDescr(rel);
+ /* fetch bitmap of REPLICATION IDENTITY attributes */
+ replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ if (!replidentfull)
+ idattrs = RelationGetIdentityKeyBitmap(rel);
+
/* send number of live attributes */
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
continue;
nliveatts++;
}
pq_sendint16(out, nliveatts);
- /* fetch bitmap of REPLICATION IDENTITY attributes */
- replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
- if (!replidentfull)
- idattrs = RelationGetIdentityKeyBitmap(rel);
-
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -936,7 +955,8 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
-
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index e596b69d466..a7befd712a0 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -112,6 +112,7 @@
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "utils/acl.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -699,17 +700,20 @@ fetch_remote_table_info(char *nspname, char *relname,
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
bool isnull;
int natt;
+ ListCell *lc;
+ bool am_partition;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -739,14 +743,19 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetBool(slot_getattr(slot, 4, &isnull));
+ Assert(!isnull);
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -774,16 +783,91 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * In server versions 15 and higher, obtain the publication column list,
+ * if any.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData publications;
+ bool first = true;
+
+ initStringInfo(&publications);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&publications, ", ");
+ appendStringInfoString(&publications, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ " SELECT pg_catalog.unnest(prattrs)\n"
+ " FROM pg_catalog.pg_publication p JOIN\n"
+ " pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n"
+ " WHERE p.pubname IN (%s) AND\n",
+ publications.data);
+ if (!am_partition)
+ appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "prrelid IN (SELECT relid\n"
+ " FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch attribute info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ if (isnull)
+ continue;
+ included_cols = bms_add_member(included_cols, attnum);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+ pfree(publications.data);
+ walrcv_clear_result(pubres);
+ }
+
+ /*
+ * Store the columns as a list of names. Ignore those that are not
+ * present in the column list, if there is one.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ continue;
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
+ Assert(!isnull);
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -793,12 +877,12 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecClearTuple(slot);
}
- ExecDropSingleTupleTableSlot(slot);
-
- lrel->natts = natt;
+ ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
pfree(cmd.data);
+
+ lrel->natts = natt;
}
/*
@@ -831,8 +915,17 @@ copy_table(Relation rel)
/* Start copy on the publisher. */
initStringInfo(&cmd);
if (lrel.relkind == RELKIND_RELATION)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ if (i < lrel.natts - 1)
+ appendStringInfoString(&cmd, ", ");
+ }
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 6df705f90ff..c7b28d70d23 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,16 +15,19 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel_d.h"
#include "commands/defrem.h"
#include "fmgr.h"
#include "replication/logical.h"
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -81,7 +84,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -132,6 +136,13 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
TupleConversionMap *map;
+
+ /*
+ * Set of columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this list are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -572,11 +583,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
}
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -589,7 +600,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -612,13 +624,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip this attribute if it's not present in the column list */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -695,7 +711,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, relation, tuple,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -724,7 +740,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, xid, relation, oldtuple,
- newtuple, data->binary);
+ newtuple, data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
}
@@ -1124,6 +1140,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
{
RelationSyncEntry *entry;
bool found;
+ Oid ancestor_id;
MemoryContext oldctx;
Assert(RelationSyncCache != NULL);
@@ -1143,6 +1160,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->map = NULL; /* will be set by maybe_send_schema() if
* needed */
}
@@ -1187,6 +1205,8 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
entry->pubactions.pubinsert = false;
entry->pubactions.pubupdate = false;
entry->pubactions.pubdelete = false;
@@ -1205,13 +1225,16 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
/*
* Build publication cache. We can't use one provided by relcache as
- * relcache considers all publications given relation is in, but here
- * we only need to consider ones that the subscriber requested.
+ * relcache considers all publications that the given relation is in,
+ * but here we only need to consider ones that the subscriber
+ * requested.
*/
foreach(lc, data->publications)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
+ bool all_columns = false;
if (pub->alltables)
{
@@ -1222,8 +1245,6 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (!publish)
{
- bool ancestor_published = false;
-
/*
* For a partition, check if any of the ancestors are
* published. If so, note down the topmost ancestor that is
@@ -1249,6 +1270,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
list_member_oid(aschemaPubids, pub->oid))
{
ancestor_published = true;
+ ancestor_id = ancestor;
if (pub->pubviaroot)
publish_as_relid = ancestor;
}
@@ -1264,9 +1286,13 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
* Don't publish changes for partitioned tables, because
- * publishing those of its partitions suffices, unless partition
- * changes won't be published due to pubviaroot being set.
+ * publishing those of its partitions suffices. (However, ignore
+ * this if partition changes are not to published due to
+ * pubviaroot being set.)
*/
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
@@ -1275,10 +1301,74 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
+
+ /*
+ * Obtain columns published by this publication, and add them
+ * to the list for this rel. Note that if at least one
+ * publication has a empty column list, that means to publish
+ * everything; so if we saw a publication that includes all
+ * columns, skip this.
+ */
+ if (!all_columns)
+ {
+ HeapTuple pub_rel_tuple;
+ Oid relid;
+
+ relid = ancestor_published ? ancestor_id : publish_as_relid;
+
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ Datum pub_rel_cols;
+ bool isnull;
+
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP,
+ pub_rel_tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull)
+ {
+ /*
+ * If we see a publication with no columns, reset the
+ * list and ignore further ones.
+ */
+ all_columns = true;
+ bms_free(entry->columns);
+ entry->columns = NULL;
+ }
+ else if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(pub_rel_cols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
+ entry->columns = bms_add_member(entry->columns,
+ elems[i]);
+ MemoryContextSwitchTo(oldctx);
+ }
+
+ ReleaseSysCache(pub_rel_tuple);
+ }
+ }
}
+ /*
+ * If we've seen all action bits, and we know that all columns are
+ * published, there's no reason to look at further publications.
+ */
if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
- entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
+ entry->pubactions.pubdelete && entry->pubactions.pubtruncate &&
+ all_columns)
break;
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 4485ea83b1e..2c347e80ee0 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4074,6 +4074,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_oid;
int i_prpubid;
int i_prrelid;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4085,8 +4086,13 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
/* Collect all publication membership info. */
appendPQExpBufferStr(query,
- "SELECT tableoid, oid, prpubid, prrelid "
- "FROM pg_catalog.pg_publication_rel");
+ "SELECT tableoid, oid, prpubid, prrelid");
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBufferStr(query, ", prattrs");
+ else
+ appendPQExpBufferStr(query, ", NULL as prattrs");
+ appendPQExpBufferStr(query,
+ " FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
ntups = PQntuples(res);
@@ -4095,6 +4101,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_oid = PQfnumber(res, "oid");
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4136,6 +4143,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
pubrinfo[j].publication = pubinfo;
pubrinfo[j].pubtable = tbinfo;
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[i].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4210,10 +4239,12 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s;\n",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+ appendPQExpBufferStr(query, ";\n");
/*
* There is no point in creating a drop query as the drop is done by table
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 9965ac2518b..54df6996428 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -631,6 +631,7 @@ typedef struct _PublicationRelInfo
DumpableObject dobj;
PublicationInfo *publication;
TableInfo *pubtable;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 654ef2d7c30..1d1474a8c29 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5856,7 +5856,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5873,10 +5873,14 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (!as_schema) /* as table */
+ {
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
- else
+ if (!PQgetisnull(res, i, 2))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 2));
+ }
+ else /* as schema */
printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
@@ -6004,8 +6008,20 @@ describePublications(const char *pattern)
{
/* Get the tables for the specified publication */
printfPQExpBuffer(&buf,
- "SELECT n.nspname, c.relname\n"
- "FROM pg_catalog.pg_class c,\n"
+ "SELECT n.nspname, c.relname, \n");
+ if (pset.sversion >= 150000)
+ appendPQExpBufferStr(&buf,
+ " CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string"
+ "(ARRAY(SELECT attname\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END AS columns");
+ else
+ appendPQExpBufferStr(&buf, "NULL as columns");
+ appendPQExpBuffer(&buf,
+ "\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
" pg_catalog.pg_publication_rel pr\n"
"WHERE c.relnamespace = n.oid\n"
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 841b9b6c253..70f73d01dc0 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -86,6 +86,7 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -109,8 +110,11 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
+extern void GetActionsInPublication(Oid pubid, PublicationActions *actions);
extern List *GetPublicationSchemas(Oid pubid);
extern List *GetSchemaPublications(Oid schemaid);
extern List *GetSchemaPublicationRelations(Oid schemaid,
@@ -127,6 +131,8 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *tar
bool if_not_exists);
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern void publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index 117a1d67e56..291ab73fee7 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,9 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
Oid oid; /* oid */
Oid prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid prrelid BKI_LOOKUP(pg_class); /* Oid of the relation */
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ int2vector prattrs;
+#endif
} FormData_pg_publication_rel;
/* ----------------
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 34218b718c1..1becb5b78ea 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3651,6 +3651,7 @@ typedef struct PublicationTable
{
NodeTag type;
RangeVar *relation; /* relation to be published */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
@@ -3687,7 +3688,8 @@ typedef enum AlterPublicationAction
{
AP_AddObjects, /* add objects to publication */
AP_DropObjects, /* remove objects from publication */
- AP_SetObjects /* set list of objects */
+ AP_SetObjects, /* set list of objects */
+ AP_SetColumns /* change list of columns for a table */
} AlterPublicationAction;
typedef struct AlterPublicationStmt
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 22fffaca62d..fcbed4ed2d6 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -207,11 +207,11 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel, HeapTuple newtuple,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple, bool binary);
+ HeapTuple newtuple, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -228,7 +228,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index b97f98cda72..861eaf6f695 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -165,8 +165,54 @@ Publications:
regress_publication_user | t | t | t | f | f | f
(1 row)
-DROP TABLE testpub_tbl2;
-DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d); -- error
+ERROR: cannot reference generated column "d" in publication column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+ERROR: cannot drop column "c" because it is part of publication "testpub_fortable"
+HINT: Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication.
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key; -- nope
+ERROR: index "testpub_tbl5_b_key" cannot be used because publication "testpub_fortable" does not include all indexed columns
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key; -- ok, but ...
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- no dice
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+ Publication testpub_table_ins
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | t | f
+Tables:
+ "public.testpub_tbl5" (a)
+
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ERROR: invalid column list for publishing relation "testpub_tbl6"
+DETAIL: Cannot specify a column list on relations with REPLICA IDENTITY FULL.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+ALTER PUBLICATION testpub_fortable
+ ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c); -- error
+ERROR: cannot change column set for relation "testpub_tbl6"
+DETAIL: Cannot specify column list on relations with REPLICA IDENTITY FULL.
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
+DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema, testpub_table_ins;
CREATE TABLE testpub_tbl3 (a int);
CREATE TABLE testpub_tbl3a (b text) INHERITS (testpub_tbl3);
SET client_min_messages = 'ERROR';
@@ -684,6 +730,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schemas
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 86c019bddb4..d0e9f2b11ba 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -89,8 +89,39 @@ SELECT pubname, puballtables FROM pg_publication WHERE pubname = 'testpub_forall
\d+ testpub_tbl2
\dRp+ testpub_foralltables
-DROP TABLE testpub_tbl2;
-DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- ok
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key; -- nope
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key; -- ok, but ...
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- no dice
+
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c); -- error
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+ALTER PUBLICATION testpub_fortable
+ ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c); -- error
+
+DROP TABLE testpub_tbl2, testpub_tbl5, testpub_tbl6;
+DROP PUBLICATION testpub_foralltables, testpub_fortable, testpub_forschema, testpub_table_ins;
CREATE TABLE testpub_tbl3 (a int);
CREATE TABLE testpub_tbl3a (b text) INHERITS (testpub_tbl3);
@@ -375,6 +406,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/028_column_list.pl b/src/test/subscription/t/028_column_list.pl
new file mode 100644
index 00000000000..5a4f022f268
--- /dev/null
+++ b/src/test/subscription/t/028_column_list.pl
@@ -0,0 +1,164 @@
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test partial-column publication of tables
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 9;
+
+# setup
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+ INSERT INTO tab2 VALUES (2, 'foo', 2);");
+# Test with weird column names
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, B varchar, \"c'\" int)");
+
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+# Test replication with multi-level partition
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, \"c'\" int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+
+# Test create publication with a column list
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub1 FOR TABLE tab1(a, \"B\"), tab3(\"a'\",\"c'\"), test_part(a,b)");
+
+my $result = $node_publisher->safe_psql('postgres',
+ "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
+is($result, qq(tab1|1 2
+tab3|1 3
+test_part|1 2), 'publication relation updated');
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+);
+# Initial sync
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab1 VALUES (1,2,3)");
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab3 VALUES (1,2,3)");
+# Test for replication of partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (1,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (2,'bcd', '2021-07-03 11:12:13')");
+# Test for replication of multi-level partition data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (4,'abc', '2021-07-04 12:00:00')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_part VALUES (5,'bcd', '2021-07-03 11:12:13')");
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'insert on column tab1.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab3");
+is($result, qq(1|3), 'insert on column tab3.b is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part");
+is($result, qq(1|abc\n2|bcd\n4|abc\n5|bcd), 'insert on all columns is replicated');
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab1 SET c = 5 where a = 1");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|), 'update on column tab1.c is not replicated');
+
+# Verify user-defined types
+$node_publisher->safe_psql('postgres',
+ qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+ ALTER PUBLICATION pub1 ADD TABLE test_tab4 (a, b, d);
+ });
+$node_subscriber->safe_psql('postgres',
+ qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, d text);
+ });
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab4 VALUES (1, 'red', 3, 'oh my');");
+
+# Test alter publication with a column list
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2(a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION"
+);
+
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab2 VALUES (1,'abc',3)");
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET c = 5 where a = 2");
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 WHERE a = 1");
+is($result, qq(1|abc), 'insert on column tab2.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 WHERE a = 2");
+is($result, qq(2|foo), 'update on column tab2.c is not replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_tab4");
+is($result, qq(1|red|oh my), 'insert on table with user-defined type');
+
+$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int)");
+$node_subscriber->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b)");
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d)");
+$node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2, pub3");
+$node_publisher->wait_for_catchup('sub2');
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (1, 11, 111, 1111)");
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (2, 22, 222, 2222)");
+$node_publisher->wait_for_catchup('sub2');
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5;"),
+ qq(1|11|1111
+2|22|2222),
+ 'overlapping publications with overlapping column lists');
--
2.34.1
0002-commit-tweaks-and-reviews-20220216.patchtext/x-patch; charset=UTF-8; name=0002-commit-tweaks-and-reviews-20220216.patchDownload
From 777e2cd1b9dea015c32bf606c76f205561106928 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Tue, 15 Feb 2022 11:04:05 +0100
Subject: [PATCH 2/4] commit tweaks and reviews
---
doc/src/sgml/ref/alter_publication.sgml | 3 +-
src/backend/catalog/pg_publication.c | 50 +++++++++++++++++--------
src/backend/commands/publicationcmds.c | 6 +++
src/backend/replication/logical/proto.c | 12 +++++-
4 files changed, 54 insertions(+), 17 deletions(-)
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index a85574214a5..6a0c7722c7d 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -65,7 +65,8 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
<para>
The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows changing
- the set of columns that are included in the publication.
+ the set of columns that are included in the publication. If a column list
+ is specified, it must include the replica identity columns.
</para>
<para>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index aa1655696f5..f8e0a728a8c 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -56,14 +56,14 @@ static void publication_translate_columns(Relation targetrel, List *columns,
* Check if relation can be in given publication and that the column
* filter is sensible, and throws appropriate error if not.
*
- * targetcols is the bitmapset of attribute numbers given in the column list,
- * or NULL if it was not specified.
+ * columns is the bitmapset of attribute numbers included in the column list,
+ * or NULL if no column list was specified (i.e. all columns are replicated)
*/
static void
check_publication_add_relation(Publication *pub, Relation targetrel,
Bitmapset *columns)
{
- bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
@@ -96,17 +96,24 @@ check_publication_add_relation(Publication *pub, Relation targetrel,
RelationGetRelationName(targetrel)),
errdetail("This operation is not supported for unlogged tables.")));
- /* Make sure the column list checks out */
+ /*
+ * If a column filter was specified, check it's sensible with respect to
+ * replica identity - the list has to include all columns in the replica
+ * identity.
+ */
if (columns != NULL)
{
/*
- * Even if the user listed all columns in the column list, we cannot
- * allow a column list to be specified when REPLICA IDENTITY is FULL;
- * that would cause problems if a new column is added later, because
- * the new column would have to be included (because of being part of
- * the replica identity) but it's technically not allowed (because of
- * not being in the publication's column list yet). So reject this
- * case altogether.
+ *
+ * If the relation uses REPLICA IDENTITY FULL, we can't allow any column
+ * list even if it lists all columns of the relation - it'd cause issues
+ * if a column is added later. The column would become part of a replica
+ * identity, violating the rule that the column list includes the whole
+ * replica identity. We could add the column to the column list too, of
+ * course, but it seems rather useles - the column list would always
+ * include all columns, i.e. as if there's no column filter.
+ *
+ * So just reject this case altogether.
*/
if (replidentfull)
ereport(ERROR,
@@ -126,6 +133,10 @@ check_publication_add_relation(Publication *pub, Relation targetrel,
* No column can be excluded if REPLICA IDENTITY is FULL (since all the
* columns need to be sent regardless); and in other cases, the columns in
* the REPLICA IDENTITY cannot be left out.
+ *
+ * Then we need to cross-check the replica identity and column list. For
+ * UPDATE/DELETE we need to ensure the replica identity is a subset of the
+ * column filter. For INSERT, we don't need to check anything.
*/
static void
check_publication_columns(Publication *pub, Relation targetrel, Bitmapset *columns)
@@ -513,11 +524,15 @@ publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
&natts, &attarray, &attset);
/*
- * Make sure the column list checks out. XXX this should occur at
- * caller in publicationcmds.c, not here.
+ * Make sure the column list checks out.
+ *
+ * XXX this should occur at caller in publicationcmds.c, not here.
+ * XXX How come this does not check replica identity? Should this prevent
+ * replica identity full, just like check_publication_add_relation?
*/
check_publication_columns(pub, targetrel, attset);
bms_free(attset);
+
/* XXX "pub" is leaked here */
prattrs = buildint2vector(attarray, natts);
@@ -542,7 +557,11 @@ publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
heap_freetuple(copytup);
}
-/* qsort comparator for attnums */
+/*
+ * qsort comparator for attnums
+ *
+ * XXX We already have compare_int16, so maybe let's share that, somehow?
+ */
static int
compare_int16(const void *a, const void *b)
{
@@ -556,7 +575,8 @@ compare_int16(const void *a, const void *b)
/*
* Translate a list of column names to an array of attribute numbers
* and a Bitmapset with them; verify that each attribute is appropriate
- * to have in a publication column list. Other checks are done later;
+ * to have in a publication column list (no system or generated attributes,
+ * no duplicates). Additional checks with replica identity are done later;
* see check_publication_columns.
*
* Note that the attribute numbers are *not* offset by
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 592f56dd61e..e6a8f53912d 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -439,6 +439,12 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
&publish_via_partition_root_given,
&publish_via_partition_root);
+ /*
+ * FIXME check pubactions vs. replica identity, to ensure the replica
+ * identity is included in the column filter. Only do this for update
+ * and delete publications. See check_publication_columns.
+ */
+
/* Everything ok, form a new tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e6da46d83e5..28c10e4d1e4 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -794,7 +794,11 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple,
if (att->attisdropped || att->attgenerated)
continue;
- /* Ignore attributes that are not to be sent. */
+ /* Ignore attributes that are not to be sent.
+ *
+ * XXX Do we need the (columns != NULL) check? I don't think so, because
+ * such bitmap has no members.
+ */
if (columns != NULL && !bms_is_member(att->attnum, columns))
continue;
@@ -941,8 +945,11 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
if (att->attisdropped || att->attgenerated)
continue;
+
+ /* XXX we should have a function/macro for this check */
if (columns != NULL && !bms_is_member(att->attnum, columns))
continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -955,8 +962,11 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
if (att->attisdropped || att->attgenerated)
continue;
+
+ /* XXX we should have a function/macro for this check */
if (columns != NULL && !bms_is_member(att->attnum, columns))
continue;
+
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
--
2.34.1
0003-Peter-Smith-review-20220216.patchtext/x-patch; charset=UTF-8; name=0003-Peter-Smith-review-20220216.patchDownload
From a268e9a2eb13ecd8bc5f1f9053f13605bc4a3629 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Tue, 15 Feb 2022 23:08:11 +0100
Subject: [PATCH 3/4] Peter Smith review
---
doc/src/sgml/catalogs.sgml | 4 +-
doc/src/sgml/ref/alter_publication.sgml | 4 +-
doc/src/sgml/ref/create_publication.sgml | 20 +++++---
src/backend/catalog/pg_publication.c | 57 ++++++++-------------
src/backend/commands/tablecmds.c | 1 -
src/backend/replication/logical/proto.c | 10 ++--
src/backend/replication/logical/tablesync.c | 11 ++--
src/backend/replication/pgoutput/pgoutput.c | 16 +++---
src/bin/pg_dump/pg_dump.c | 2 +-
src/include/catalog/pg_publication.h | 2 +
10 files changed, 61 insertions(+), 66 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index f20cb2b78ab..adb0819c120 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -4392,7 +4392,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para>
<para>
This is an array of <structfield>indnatts</structfield> values that
- indicate which table columns this index indexes. For example a value
+ indicate which table columns this index indexes. For example, a value
of <literal>1 3</literal> would mean that the first and the third table
columns make up the index entries. Key columns come before non-key
(included) columns. A zero in this array indicates that the
@@ -6335,7 +6335,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
This is an array of values that indicates which table columns are
part of the publication. For example a value of <literal>1 3</literal>
would mean that the first and the third table columns are published.
- A null value indicates that all attributes are published.
+ A null value indicates that all columns are published.
</para></entry>
</row>
</tbody>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 6a0c7722c7d..5eae5cde499 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -25,13 +25,13 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD <replace
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
-ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ALTER TABLE <replaceable class="parameter">publication_object</replaceable> SET COLUMNS { ( <replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ALTER TABLE <replaceable class="parameter">table_name</replaceable> SET COLUMNS { ( <replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index a59cd3f532a..da4d929e02a 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -79,12 +79,10 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
</para>
<para>
- When a column list is specified, only the listed columns are replicated;
- any other columns are ignored for the purpose of replication through
- this publication. If no column list is specified, all columns of the
- table are replicated through this publication, including any columns
- added later. If a column list is specified, it must include the replica
- identity columns.
+ When a column list is specified, only the named columns are replicated.
+ If no column list is specified, all columns of the table are replicated
+ through this publication, including any columns added later. If a column
+ list is specified, it must include the replica identity columns.
</para>
<para>
@@ -300,6 +298,14 @@ CREATE PUBLICATION production_publication FOR TABLE users, departments, ALL TABL
<structname>sales</structname>:
<programlisting>
CREATE PUBLICATION sales_publication FOR ALL TABLES IN SCHEMA marketing, sales;
+</programlisting></para>
+
+ <para>
+ Create a publication that publishes all changes for table <structname>users</structname>,
+ but replicates only columns <structname>user_id</structname> and
+ <structname>firstname</structname>:
+<programlisting>
+CREATE PUBLICATION users_filtered FOR TABLE users (user_id, firstname);
</programlisting></para>
</refsect1>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index f8e0a728a8c..ab5a345b3de 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -63,8 +63,6 @@ static void
check_publication_add_relation(Publication *pub, Relation targetrel,
Bitmapset *columns)
{
- bool replidentfull = (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
-
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
@@ -97,50 +95,39 @@ check_publication_add_relation(Publication *pub, Relation targetrel,
errdetail("This operation is not supported for unlogged tables.")));
/*
- * If a column filter was specified, check it's sensible with respect to
- * replica identity - the list has to include all columns in the replica
- * identity.
+ * Ensure the column filter is compatible with the replica identity and the
+ * actions the publication is replicating.
*/
- if (columns != NULL)
- {
- /*
- *
- * If the relation uses REPLICA IDENTITY FULL, we can't allow any column
- * list even if it lists all columns of the relation - it'd cause issues
- * if a column is added later. The column would become part of a replica
- * identity, violating the rule that the column list includes the whole
- * replica identity. We could add the column to the column list too, of
- * course, but it seems rather useles - the column list would always
- * include all columns, i.e. as if there's no column filter.
- *
- * So just reject this case altogether.
- */
- if (replidentfull)
- ereport(ERROR,
- errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("invalid column list for publishing relation \"%s\"",
- RelationGetRelationName(targetrel)),
- errdetail("Cannot specify a column list on relations with REPLICA IDENTITY FULL."));
-
- check_publication_columns(pub, targetrel, columns);
- }
+ check_publication_columns(pub, targetrel, columns);
}
/*
* Enforce that the column list can only leave out columns that aren't
* forced to be sent.
*
- * No column can be excluded if REPLICA IDENTITY is FULL (since all the
- * columns need to be sent regardless); and in other cases, the columns in
- * the REPLICA IDENTITY cannot be left out.
+ * If the relation uses REPLICA IDENTITY FULL, we can't allow any column
+ * list even if it lists all columns of the relation - it'd cause issues
+ * if a column is added later. The column would become part of a replica
+ * identity, violating the rule that the column list includes the whole
+ * replica identity. We could add the column to the column list too, of
+ * course, but it seems rather useles - the column list would always
+ * include all columns, i.e. as if there's no column filter.
*
- * Then we need to cross-check the replica identity and column list. For
- * UPDATE/DELETE we need to ensure the replica identity is a subset of the
- * column filter. For INSERT, we don't need to check anything.
+ * In other cases, the columns in the REPLICA IDENTITY cannot be left out,
+ * except when the publication replicates only inserts. So we check that
+ * for UPDATE/DELETE the replica identity is a subset of the column filter.
*/
static void
check_publication_columns(Publication *pub, Relation targetrel, Bitmapset *columns)
{
+ /*
+ * If there is no column list, we treat it as if the list contains all columns. In
+ * which case there's nothing to check so we're done.
+ */
+ if (!columns)
+ return;
+
+ /* With REPLICA IDENTITY FULL no column filter is allowed. */
if (targetrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
ereport(ERROR,
errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -497,7 +484,7 @@ publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
Datum values[Natts_pg_publication_rel];
memset(values, 0, sizeof(values));
- memset(nulls, 0, sizeof(nulls));
+ memset(nulls, false, sizeof(nulls));
memset(replaces, false, sizeof(replaces));
replaces[Anum_pg_publication_rel_prattrs - 1] = true;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 28de9329800..2bfb165e45d 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -15872,7 +15872,6 @@ relation_mark_replica_identity(Relation rel, char ri_type, Oid indexOid,
CatalogTupleUpdate(pg_index, &pg_index_tuple->t_self, pg_index_tuple);
InvokeObjectPostAlterHookArg(IndexRelationId, thisIndexOid, 0,
InvalidOid, is_internal);
-
/*
* Invalidate the relcache for the table, so that after we commit
* all sessions will refresh the table's replica identity index
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 28c10e4d1e4..ec5a787cc3a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -933,11 +933,6 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
desc = RelationGetDescr(rel);
- /* fetch bitmap of REPLICATION IDENTITY attributes */
- replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
- if (!replidentfull)
- idattrs = RelationGetIdentityKeyBitmap(rel);
-
/* send number of live attributes */
for (i = 0; i < desc->natts; i++)
{
@@ -954,6 +949,11 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
}
pq_sendint16(out, nliveatts);
+ /* fetch bitmap of REPLICATION IDENTITY attributes */
+ replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
+ if (!replidentfull)
+ idattrs = RelationGetIdentityKeyBitmap(rel);
+
/* send the attributes */
for (i = 0; i < desc->natts; i++)
{
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index a7befd712a0..d2dbf5f7b83 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -877,12 +877,12 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecClearTuple(slot);
}
-
ExecDropSingleTupleTableSlot(slot);
- walrcv_clear_result(res);
- pfree(cmd.data);
lrel->natts = natt;
+
+ walrcv_clear_result(res);
+ pfree(cmd.data);
}
/*
@@ -920,9 +920,10 @@ copy_table(Relation rel)
quote_qualified_identifier(lrel.nspname, lrel.relname));
for (int i = 0; i < lrel.natts; i++)
{
- appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
- if (i < lrel.natts - 1)
+ if (i > 0)
appendStringInfoString(&cmd, ", ");
+
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
}
appendStringInfo(&cmd, ") TO STDOUT");
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index c7b28d70d23..0068ae25d3e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -138,8 +138,8 @@ typedef struct RelationSyncEntry
TupleConversionMap *map;
/*
- * Set of columns included in the publication, or NULL if all columns are
- * included implicitly. Note that the attnums in this list are not
+ * Columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this bitmap are not
* shifted by FirstLowInvalidHeapAttributeNumber.
*/
Bitmapset *columns;
@@ -1290,9 +1290,8 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
* publish, and list of columns, if appropriate.
*
* Don't publish changes for partitioned tables, because
- * publishing those of its partitions suffices. (However, ignore
- * this if partition changes are not to published due to
- * pubviaroot being set.)
+ * publishing those of its partitions suffices, unless partition
+ * changes won't be published due to pubviaroot being set.
*/
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
@@ -1305,7 +1304,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
/*
* Obtain columns published by this publication, and add them
* to the list for this rel. Note that if at least one
- * publication has a empty column list, that means to publish
+ * publication has an empty column list, that means to publish
* everything; so if we saw a publication that includes all
* columns, skip this.
*/
@@ -1332,14 +1331,15 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
if (isnull)
{
/*
- * If we see a publication with no columns, reset the
+ * If we see a publication with no column filter, it
+ * means we need to publish all columns, so reset the
* list and ignore further ones.
*/
all_columns = true;
bms_free(entry->columns);
entry->columns = NULL;
}
- else if (!isnull)
+ else
{
ArrayType *arr;
int nelems;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 2c347e80ee0..2f29e0d7189 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4160,7 +4160,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
appendPQExpBufferStr(attribs, fmtId(attnames[k]));
}
- pubrinfo[i].pubrattrs = attribs->data;
+ pubrinfo[j].pubrattrs = attribs->data;
}
else
pubrinfo[j].pubrattrs = NULL;
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 70f73d01dc0..ff9d9a43984 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -86,6 +86,8 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+
+ /* List of columns to replicate (or NIL to replicate all columns) */
List *columns;
} PublicationRelInfo;
--
2.34.1
0004-rework-028_column_list.pl-20220216.patchtext/x-patch; charset=UTF-8; name=0004-rework-028_column_list.pl-20220216.patchDownload
From eabd329888df71f18467f6c6be53ae58de3a8d9e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Tue, 15 Feb 2022 14:14:40 +0100
Subject: [PATCH 4/4] rework 028_column_list.pl
---
src/test/subscription/t/028_column_list.pl | 917 ++++++++++++++++++---
1 file changed, 820 insertions(+), 97 deletions(-)
diff --git a/src/test/subscription/t/028_column_list.pl b/src/test/subscription/t/028_column_list.pl
index 5a4f022f268..8b109402af6 100644
--- a/src/test/subscription/t/028_column_list.pl
+++ b/src/test/subscription/t/028_column_list.pl
@@ -5,7 +5,7 @@ use strict;
use warnings;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
-use Test::More tests => 9;
+use Test::More tests => 31;
# setup
@@ -21,144 +21,867 @@ $node_subscriber->start;
my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
-$node_publisher->safe_psql('postgres',
- "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
+# setup tables on both nodes
-$node_subscriber->safe_psql('postgres',
- "CREATE TABLE tab1 (a int PRIMARY KEY, \"B\" int, c int)");
-$node_publisher->safe_psql('postgres',
- "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
- INSERT INTO tab2 VALUES (2, 'foo', 2);");
-# Test with weird column names
-$node_publisher->safe_psql('postgres',
- "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, B varchar, \"c'\" int)");
+# tab1: simple 1:1 replication
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
-$node_publisher->safe_psql('postgres',
- "CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a)");
-$node_publisher->safe_psql('postgres',
- "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
-# Test replication with multi-level partition
-$node_publisher->safe_psql('postgres',
- "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
-$node_publisher->safe_psql('postgres',
- "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
-$node_subscriber->safe_psql('postgres',
- "CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a)");
-$node_subscriber->safe_psql('postgres',
- "CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3)");
-$node_subscriber->safe_psql('postgres',
- "CREATE TABLE tab3 (\"a'\" int PRIMARY KEY, \"c'\" int)");
-$node_subscriber->safe_psql('postgres',
- "CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)");
-$node_subscriber->safe_psql('postgres',
- "CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a)");
-$node_subscriber->safe_psql('postgres',
- "CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5)");
+# tab2: replication from regular to table with fewer columns
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+));
-# Test create publication with a column list
-$node_publisher->safe_psql('postgres',
- "CREATE PUBLICATION pub1 FOR TABLE tab1(a, \"B\"), tab3(\"a'\",\"c'\"), test_part(a,b)");
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)
+));
+
+# tab3: simple 1:1 replication with weird column names
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "B" varchar, "c'" int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "c'" int)
+));
+
+# test_part: partitioned tables, with partitioning (including multi-level
+# partitioning, and fewer columns on the subscriber)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5);
+));
+
+# tab4: table with user-defined enum types
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, d text);
+));
+
+
+# TEST: create publication and subscription for some of the tables with
+# column lists
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub1
+ FOR TABLE tab1 (a, "B"), tab3 ("a'", "c'"), test_part (a, b), tab4 (a, b, d);
+));
+
+# check that we got the right prattrs values for the publication in the
+# pg_publication_rel catalog (order by relname, to get stable ordering)
+my $result = $node_publisher->safe_psql('postgres', qq(
+ SELECT relname, prattrs
+ FROM pg_publication_rel pb JOIN pg_class pc ON(pb.prrelid = pc.oid)
+ ORDER BY relname
+));
-my $result = $node_publisher->safe_psql('postgres',
- "select relname, prattrs from pg_publication_rel pb, pg_class pc where pb.prrelid = pc.oid;");
is($result, qq(tab1|1 2
tab3|1 3
+tab4|1 2 4
test_part|1 2), 'publication relation updated');
-$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
-);
-# Initial sync
+# create subscription for the publication, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1
+));
+
$node_publisher->wait_for_catchup('sub1');
-$node_publisher->safe_psql('postgres',
- "INSERT INTO tab1 VALUES (1,2,3)");
+# TEST: insert data into the tables, and see we got replication of just
+# the filtered columns
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (1, 2, 3);
+ INSERT INTO tab1 VALUES (4, 5, 6);
+));
-$node_publisher->safe_psql('postgres',
- "INSERT INTO tab3 VALUES (1,2,3)");
-# Test for replication of partition data
-$node_publisher->safe_psql('postgres',
- "INSERT INTO test_part VALUES (1,'abc', '2021-07-04 12:00:00')");
-$node_publisher->safe_psql('postgres',
- "INSERT INTO test_part VALUES (2,'bcd', '2021-07-03 11:12:13')");
-# Test for replication of multi-level partition data
-$node_publisher->safe_psql('postgres',
- "INSERT INTO test_part VALUES (4,'abc', '2021-07-04 12:00:00')");
-$node_publisher->safe_psql('postgres',
- "INSERT INTO test_part VALUES (5,'bcd', '2021-07-03 11:12:13')");
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (1, 2, 3);
+ INSERT INTO tab3 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (1, 'red', 3, 'oh my');
+ INSERT INTO tab4 VALUES (2, 'blue', 4, 'hello');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (1, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (2, 'bcd', '2021-07-03 11:12:13');
+ INSERT INTO test_part VALUES (4, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (5, 'bcd', '2021-07-03 11:12:13');
+));
+
+# wait for catchup before checking the subscriber
+$node_publisher->wait_for_catchup('sub1');
+# tab1: only (a,b) is replicated
$result = $node_subscriber->safe_psql('postgres',
"SELECT * FROM tab1");
-is($result, qq(1|2|), 'insert on column tab1.c is not replicated');
+is($result, qq(1|2|
+4|5|), 'insert on column tab1.c is not replicated');
+# tab3: only (a,c) is replicated
$result = $node_subscriber->safe_psql('postgres',
"SELECT * FROM tab3");
-is($result, qq(1|3), 'insert on column tab3.b is not replicated');
+is($result, qq(1|3
+4|6), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4");
+is($result, qq(1|red|oh my
+2|blue|hello), 'insert on column tab4.c is not replicated');
+# test_part: (a,b) is replicated
$result = $node_subscriber->safe_psql('postgres',
"SELECT * FROM test_part");
-is($result, qq(1|abc\n2|bcd\n4|abc\n5|bcd), 'insert on all columns is replicated');
+is($result, qq(1|abc
+2|bcd
+4|abc
+5|bcd), 'insert on column test_part.c columns is not replicated');
+
+# TEST: do some updated on some of the tables, both on columns included
+# in the column list and other
+# tab1: update of replicated column
$node_publisher->safe_psql('postgres',
- "UPDATE tab1 SET c = 5 where a = 1");
+ qq(UPDATE tab1 SET "B" = 2 * "B" where a = 1));
-$node_publisher->wait_for_catchup('sub1');
+# tab1: update of non-replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET c = 2*c where a = 4));
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT * FROM tab1");
-is($result, qq(1|2|), 'update on column tab1.c is not replicated');
+# tab3: update of non-replicated
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "B" = "B" || ' updated' where "a'" = 4));
-# Verify user-defined types
+# tab3: update of replicated column
$node_publisher->safe_psql('postgres',
- qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
- CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
- ALTER PUBLICATION pub1 ADD TABLE test_tab4 (a, b, d);
- });
-$node_subscriber->safe_psql('postgres',
- qq{CREATE TYPE test_typ AS ENUM ('blue', 'red');
- CREATE TABLE test_tab4 (a INT PRIMARY KEY, b test_typ, d text);
- });
+ qq(UPDATE tab3 SET "c'" = 2 * "c'" where "a'" = 1));
+
+# tab4
$node_publisher->safe_psql('postgres',
- "INSERT INTO test_tab4 VALUES (1, 'red', 3, 'oh my');");
+ qq(UPDATE tab4 SET b = 'blue', c = c * 2, d = d || ' updated' where a = 1));
-# Test alter publication with a column list
+# tab4
$node_publisher->safe_psql('postgres',
- "ALTER PUBLICATION pub1 ADD TABLE tab2(a, b)");
+ qq(UPDATE tab4 SET b = 'red', c = c * 2, d = d || ' updated' where a = 2));
-$node_subscriber->safe_psql('postgres',
- "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION"
-);
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab1 ORDER BY a));
+is($result,
+qq(1|4|
+4|5|), 'only update on column tab1.b is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result,
+qq(1|6
+4|6), 'only update on column tab3.c is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab4 ORDER BY a));
+
+is($result, qq(1|blue|oh my updated
+2|red|hello updated), 'update on column tab4.c is not replicated');
+
+
+# TEST: add table with a column list, insert data, replicate
+
+# insert some data before adding it to the publication
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (1, 'abc', 3);
+));
$node_publisher->safe_psql('postgres',
- "INSERT INTO tab2 VALUES (1,'abc',3)");
-$node_publisher->safe_psql('postgres',
- "UPDATE tab2 SET c = 5 where a = 2");
+ "ALTER PUBLICATION pub1 ADD TABLE tab2 (a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION");
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (2, 'def', 6);
+));
$node_publisher->wait_for_catchup('sub1');
$result = $node_subscriber->safe_psql('postgres',
- "SELECT * FROM tab2 WHERE a = 1");
-is($result, qq(1|abc), 'insert on column tab2.c is not replicated');
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|def), 'insert on column tab2.c is not replicated');
-$result = $node_subscriber->safe_psql('postgres',
- "SELECT * FROM tab2 WHERE a = 2");
-is($result, qq(2|foo), 'update on column tab2.c is not replicated');
+$node_publisher->safe_psql('postgres', qq(
+ UPDATE tab2 SET c = 5 where a = 1;
+ UPDATE tab2 SET b = 'xyz' where a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
$result = $node_subscriber->safe_psql('postgres',
- "SELECT * FROM test_tab4");
-is($result, qq(1|red|oh my), 'insert on table with user-defined type');
-
-$node_publisher->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int)");
-$node_subscriber->safe_psql('postgres', "CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int)");
-$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b)");
-$node_publisher->safe_psql('postgres', "CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d)");
-$node_subscriber->safe_psql('postgres', "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2, pub3");
-$node_publisher->wait_for_catchup('sub2');
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|xyz), 'update on column tab2.c is not replicated');
+
+# TEST: add a table to two publications with different column lists, and
+# create a single subscription replicating both publications
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 ADD PUBLICATION pub2, pub3
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+# TEST: insert data and make sure all the columns (union of the columns lists)
+# were replicated
$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (1, 11, 111, 1111)");
$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (2, 22, 222, 2222)");
-$node_publisher->wait_for_catchup('sub2');
-is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5;"),
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
qq(1|11|1111
2|22|2222),
'overlapping publications with overlapping column lists');
+
+# and finally, set the column filter to ALL for one of the publications,
+# which means replicating all columns (removing the column filter), but
+# first add the missing column to the table on subscriber
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub3 ALTER TABLE tab5 SET COLUMNS ALL;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER TABLE tab5 ADD COLUMN c INT;
+));
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (3, 33, 333, 3333)");
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111|
+2|22|2222|
+3|33|3333|333),
+ 'overlapping publications with overlapping column lists');
+
+# TEST: create a table with a column filter, then change the replica
+# identity by replacing a primary key (but use a different column in
+# the column filter)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub4 FOR TABLE tab6 (a, b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 ADD PUBLICATION pub4
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (1, 22, 333, 4444);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 1;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ "1|44||", 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is still covered by the column list, though)
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+# we need to do the same thing on the subscriber
+# XXX What would happen if this happens before the publisher ALTER? Or
+# interleaved, somehow? But that seems unrelated to column lists.
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (2, 55, 666, 8888);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|44||
+2|110||),
+ 'replication with the modified primary key');
+
+# now switch the primary key back, but also generate writes between the
+# drop and creation of the new constraint
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ INSERT INTO tab6 VALUES (3, 33, 999, 7777);
+ -- update/delete is not allowed for tables without RI
+ ALTER TABLE tab6 ADD PRIMARY KEY (a);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 3;
+ DELETE FROM tab6 WHERE a = 1;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (a);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(2|110||
+3|66||),
+ 'replication with the modified primary key');
+
+
+# TEST: create a table with a column filter, then change the replica
+# identity by replacing a primary key with a key on multiple columns
+# (all of them covered by the column filter)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub5 FOR TABLE tab7 (a, b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 ADD PUBLICATION pub5
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (1, 22, 333, 4444);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 1;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ "1|44||", 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is not covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ ALTER TABLE tab7 ADD PRIMARY KEY (a, b);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (2, 55, 666, 8888);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|44||
+2|110||),
+ 'replication with the modified primary key');
+
+# now switch the primary key again to another columns not covered by the
+# column filter, but also generate writes between the drop and creation
+# of the new constraint
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ INSERT INTO tab7 VALUES (3, 33, 999, 7777);
+ -- update/delete is not allowed for tables without RI
+ ALTER TABLE tab7 ADD PRIMARY KEY (b, a);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 3;
+ DELETE FROM tab7 WHERE a = 1;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(2|110||
+3|66||),
+ 'replication with the modified primary key');
+
+
+# TEST: partitioned tables (with publish_via_partition_root = false)
+# and replica identity. The (leaf) partitions may have different RI, so
+# we need to check the partition RI (with respect to the column filter)
+# while attaching the partition.
+
+# First, let's create a partitioned table with two partitions, each with
+# a different RI, but a column filter not covering all those RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (2);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (2);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+));
+
+# create a publication replicating just the column "a", which is not enough
+# for the second partition
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub6 FOR TABLE test_part_a (a) WITH (publish_via_partition_root = false);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 ADD PUBLICATION pub6
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_a VALUES (1, 1);
+ -- FIXME: This fails, because it does not send (b) as needed for the RI
+ -- INSERT INTO test_part_a VALUES (2, 2);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_a ORDER BY a, b"),
+ qq(1|1
+2|2),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# This time start with a column filter covering RI for all partitions, but
+# then update the column filter to not cover column "b" (needed by the
+# second partition)
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,3);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (2,4);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,3);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (2,4);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub7 FOR TABLE test_part_b (a, b) WITH (publish_via_partition_root = false);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 ADD PUBLICATION pub7
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_b VALUES (1, 1);
+ INSERT INTO test_part_b VALUES (2, 2);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_b ORDER BY a, b"),
+ qq(1|1
+2|2),
+ 'partitions with different replica identities not replicated correctly');
+
+# now alter the publication to only replicate column "a"
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub7 ALTER TABLE test_part_b SET COLUMNS (a);
+));
+
+# now retry the inserts
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_b VALUES (3, 2);
+ -- FIXME: This fails, because it does not send (b) as needed for the RI
+ -- INSERT INTO test_part_b VALUES (4, 3);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_b ORDER BY a, b"),
+ qq(1|1
+2|2
+3|2
+4|3),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: This time start with a column filter covering RI for all partitions,
+# but then update RI for one of the partitions to not be covered by the
+# column filter anymore.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c (a) WITH (publish_via_partition_root = false);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 ADD PUBLICATION pub8
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_c VALUES (1, 1);
+ INSERT INTO test_part_c VALUES (2, 2);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1|1
+2|2),
+ 'partitions with different replica identities not replicated correctly');
+
+# now alter the publication to only replicate column "a"
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE test_part_c_2 DROP CONSTRAINT test_part_c_2_pkey;
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+));
+
+# now retry the inserts
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_c VALUES (3, 2);
+ -- FIXME: This fails, because it does not send (b) as needed for the RI
+ -- INSERT INTO test_part_c VALUES (4, 3);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1|1
+2|2
+3|2
+4|3),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: Start with a single partition, with RI compatible with the column
+# filter, and then attach a partition with incompatible RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+));
+
+# do the same thing on the subscriber (in fact, create both partitions right
+# away, no need to delay that)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ CREATE TABLE test_part_d_2 PARTITION OF test_part_d FOR VALUES IN (2,4);
+ ALTER TABLE test_part_d_2 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_2 REPLICA IDENTITY USING INDEX test_part_d_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub9 FOR TABLE test_part_d (a) WITH (publish_via_partition_root = false);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 ADD PUBLICATION pub9
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_d VALUES (1, 1);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_d ORDER BY a, b"),
+ qq(1|1),
+ 'partitions with different replica identities not replicated correctly');
+
+# now add the second partition, with a mismatching RI
+# FIXME this should fail, complaining about the column filter
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d_2 PARTITION OF test_part_d FOR VALUES IN (2,4);
+ ALTER TABLE test_part_d_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_d_2 REPLICA IDENTITY USING INDEX test_part_d_2_pkey;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_d ORDER BY a, b"),
+ qq(1|1
+2|2
+3|2
+4|3),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column filters. So with column filters (a,b) and (a,c) we
+# should replicate (a,b,c).
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 ADD PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_1 VALUES (1, 2, 3);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_1"),
+ qq(1|2|3),
+ 'a mix of publications should use a union of column filter');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column filters. If any of the publications is FOR ALL
+# TABLES, we should replicate all columns.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 ADD PUBLICATION pub_mix_3, pub_mix_4;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_2 VALUES (1, 2, 3);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_2"),
+ qq(1|2|3),
+ 'a mix of publications should use a union of column filter');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column filters. If any of the publications is FOR ALL
+# TABLES IN SCHEMA, we should replicate all columns.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 ADD PUBLICATION pub_mix_5, pub_mix_6;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_3 VALUES (1, 2, 3);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_3"),
+ qq(1|2|3),
+ 'a mix of publications should use a union of column filter');
+
+
+# TEST: Check handling of publish_via_partition_root - if a partition is
+# not published through partition root, we should not apply the column
+# filter defined for the whole table - both during the initial sync and
+# when replicating changes. This is what we do for row filters.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+
+ CREATE PUBLICATION pub_root_true FOR TABLE test_root (a) WITH (publish_via_partition_root = true);
+ CREATE PUBLICATION pub_root_false FOR TABLE test_root (a) WITH (publish_via_partition_root = false);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 ADD PUBLICATION pub_root_true;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_root VALUES (1, 2, 3);
+ INSERT INTO test_root VALUES (10, 20, 30);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_root ORDER BY a, b, c"),
+ qq(1||
+10||),
+ 'publication via partition root applies column filter');
+
+# now switch to using the publication with (publish_via_partition_root = false),
+# which should not apply the column filter
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 DROP PUBLICATION pub_root_true;
+ ALTER SUBSCRIPTION sub1 ADD PUBLICATION pub_root_false;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_root VALUES (2, 4, 6);
+ INSERT INTO test_root VALUES (11, 22, 33);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_root ORDER BY a, b, c"),
+ qq(1||
+2|4|6
+10||
+11|22|33),
+ 'publication via partition root applies column filter');
+
+# and finally add both publications to the subscription - in this case we
+# should not apply the column filter either, because the "false" case is
+# handled as column filter with all columns
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 ADD PUBLICATION pub_root_true;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_root VALUES (3, 6, 9);
+ INSERT INTO test_root VALUES (12, 24, 36);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_root ORDER BY a, b, c"),
+ qq(1||
+2|4|6
+3|6|9
+10||
+11|22|33
+12|24|36),
+ 'publication via partition root applies column filter');
--
2.34.1
Hi Peter,
Thanks for the review and sorry for taking so long.
I've addressed most of the comments in the patch I sent a couple minutes
ago. More comments in-line:
On 1/28/22 09:39, Peter Smith wrote:
Here are some review comments for the v17-0001 patch.
~~~
1. Commit message
If no column list is specified, all the columns are replicated, as
previouslyMissing period (.) at the end of that sentence.
I plan to reword that anyway.
~~~
2. doc/src/sgml/catalogs.sgml
+ <para> + This is an array of values that indicates which table columns are + part of the publication. For example a value of <literal>1 3</literal> + would mean that the first and the third table columns are published. + A null value indicates that all attributes are published. + </para></entry>Missing comma:
"For example" --> "For example,"
Fixed.
Terms:
The text seems to jump between "columns" and "attributes". Perhaps,
for consistency, that last sentence should say: "A null value
indicates that all columns are published."
Yeah, but that's a pre-existing problem. I've modified the parts added
by the patch to use "columns" though.
~~~
3. doc/src/sgml/protocol.sgml
</variablelist> - Next, the following message part appears for each column (except generated columns): + Next, the following message part appears for each column (except + generated columns and other columns that don't appear in the column + filter list, for tables that have one): <variablelist>Perhaps that can be expressed more simply, like:
Next, the following message part appears for each column (except
generated columns and other columns not present in the optional column
filter list):
Not sure. I'll think about it.
~~~
4. doc/src/sgml/ref/alter_publication.sgml
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable>
ALTER TABLE <replaceable
class="parameter">publication_object</replaceable> SET COLUMNS { (
<replaceable class="parameter">name</replaceable> [, ...] ) | ALL }The syntax chart looks strange because there is already a "TABLE" and
a column_name list within the "publication_object" definition, so do
ALTER TABLE and publication_object co-exist?
According to the current documentation it suggests nonsense like below is valid:
ALTER PUBLICATION mypublication ALTER TABLE TABLE t1 (a,b,c) SET
COLUMNS (a,b,c);
Yeah, I think that's wrong. I think "publication_object" is wrong in
this place, so I've used "table_name".
--
But more fundamentally, I don't see why any new syntax is even needed at all.
Instead of:
ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS
(user_id, firstname, lastname);
Why not just:
ALTER PUBLICATION mypublication ALTER TABLE users (user_id, firstname,
lastname);
I haven't modified the grammar yet, but I agree SET COLUMNS seems a bit
unnecessary. It also seems a bit inconsistent with ADD TABLE which
simply lists the columns right adter the table name.
Then, if the altered table defines a *different* column list then it
would be functionally equivalent to whatever your SET COLUMNS is doing
now. AFAIK this is how the Row-Filter [1] works, so that altering an
existing table to have a different Row-Filter just overwrites that
table's filter. IMO the Col-Filter behaviour should work the same as
that - "SET COLUMNS" is redundant.
I'm sorry, I don't understand what this is saying :-(
~~~
5. doc/src/sgml/ref/alter_publication.sgml
- TABLE [ ONLY ] <replaceable
class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable
class="parameter">table_name</replaceable> [ * ] [ ( <replaceable
class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ]That extra comma after the "column_name" seems wrong because there is
one already in "[, ... ]".
Fixed.
~~~
6. doc/src/sgml/ref/create_publication.sgml
- TABLE [ ONLY ] <replaceable
class="parameter">table_name</replaceable> [ * ] [, ... ]
+ TABLE [ ONLY ] <replaceable
class="parameter">table_name</replaceable> [ * ] [ ( <replaceable
class="parameter">column_name</replaceable>, [, ... ] ) ] [, ... ](Same as comment #5).
That extra comma after the "column_name" seems wrong because there is
one already in "[, ... ]".
Fixed.
~~~
7. doc/src/sgml/ref/create_publication.sgml
+ <para> + When a column list is specified, only the listed columns are replicated; + any other columns are ignored for the purpose of replication through + this publication. If no column list is specified, all columns of the + table are replicated through this publication, including any columns + added later. If a column list is specified, it must include the replica + identity columns. + </para>Suggest to re-word this a bit simpler:
e.g.
- "listed columns" --> "named columns"
- I don't think it is necessary to say the unlisted columns are ignored.
- I didn't think it is necessary to say "though this publication"AFTER
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated,
including any columns added later. If a column list is specified, it must
include the replica identity columns.
Fixed, seems reasonable.
~~~
8. doc/src/sgml/ref/create_publication.sgml
Consider adding another example showing a CREATE PUBLICATION which has
a column list.
Added.
~~~
9. src/backend/catalog/pg_publication.c - check_publication_add_relation
/* - * Check if relation can be in given publication and throws appropriate - * error if not. + * Check if relation can be in given publication and that the column + * filter is sensible, and throws appropriate error if not. + * + * targetcols is the bitmapset of attribute numbers given in the column list, + * or NULL if it was not specified. */Typo: "targetcols" --> "columns" ??
Right, I noticed that too.
~~~
10. src/backend/catalog/pg_publication.c - check_publication_add_relation
+ + /* Make sure the column list checks out */ + if (columns != NULL) + {Perhaps "checks out" could be worded better.
Right, I expanded that in my review.
~~~
11. src/backend/catalog/pg_publication.c - check_publication_add_relation
+ /* Make sure the column list checks out */ + if (columns != NULL) + { + /* + * Even if the user listed all columns in the column list, we cannot + * allow a column list to be specified when REPLICA IDENTITY is FULL; + * that would cause problems if a new column is added later, because + * the new column would have to be included (because of being part of + * the replica identity) but it's technically not allowed (because of + * not being in the publication's column list yet). So reject this + * case altogether. + */ + if (replidentfull) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("invalid column list for publishing relation \"%s\"", + RelationGetRelationName(targetrel)), + errdetail("Cannot specify a column list on relations with REPLICA IDENTITY FULL.")); + + check_publication_columns(pub, targetrel, columns); + }IIUC almost all of the above comment and code is redundant because by
calling the check_publication_columns function it will do exactly the
same check...So, that entire slab might be replaced by 2 lines:
if (columns != NULL)
check_publication_columns(pub, targetrel, columns);
You're right. But I think we can make that even simpler by moving even
the (columns!=NULL) check into the function.
~~~
12. src/backend/catalog/pg_publication.c - publication_set_table_columns
+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup, + Relation targetrel, List *columns) +{ + Bitmapset *attset; + AttrNumber *attarray; + HeapTuple copytup; + int natts; + bool nulls[Natts_pg_publication_rel]; + bool replaces[Natts_pg_publication_rel]; + Datum values[Natts_pg_publication_rel]; + + memset(values, 0, sizeof(values)); + memset(nulls, 0, sizeof(nulls)); + memset(replaces, false, sizeof(replaces));It seemed curious to use memset false for "replaces" but memset 0 for
"nulls", since they are both bool arrays (??)
Fixed.
~~~
13. src/backend/catalog/pg_publication.c - compare_int16
+/* qsort comparator for attnums */ +static int +compare_int16(const void *a, const void *b) +{ + int av = *(const int16 *) a; + int bv = *(const int16 *) b; + + /* this can't overflow if int is wider than int16 */ + return (av - bv); +}This comparator seems common with another one already in the PG
source. Perhaps it would be better for generic comparators (like this
one) to be in some common code instead of scattered cut/paste copies
of the same thing.
I thought about it, but it doesn't really seem worth the effort.
~~~
14. src/backend/commands/publicationcmds.c - AlterPublicationTables
+ else if (stmt->action == AP_SetColumns) + { + Assert(schemaidlist == NIL); + Assert(list_length(tables) == 1); + + PublicationSetColumns(stmt, pubform, + linitial_node(PublicationTable, tables)); + }(Same as my earlier review comment #4)
Suggest to call this PublicationSetColumns based on some smarter
detection logic of a changed column list. Please refer to the
Row-Filter patch [1] for this same function.
I don't understand. Comment #4 is about syntax, no?
~~~
15. src/backend/commands/publicationcmds.c - AlterPublicationTables
+ /* This is not needed to delete a table */ + pubrel->columns = NIL;Perhaps a more explanatory comment would be better there?
If I understand the comment, it says we don't actually need to set
columns to NIL. In which case we can just get rid of the change.
~~~
16. src/backend/commands/tablecmds.c - relation_mark_replica_identity
@@ -15841,6 +15871,7 @@ relation_mark_replica_identity(Relation rel,
char ri_type, Oid indexOid,
CatalogTupleUpdate(pg_index, &pg_index_tuple->t_self, pg_index_tuple);
InvokeObjectPostAlterHookArg(IndexRelationId, thisIndexOid, 0,
InvalidOid, is_internal);
+
/*
* Invalidate the relcache for the table, so that after we commit
* all sessions will refresh the table's replica identity indexSpurious whitespace change seemed unrelated to the Col-Filter patch.
Fixed.
~~~
17. src/backend/parser/gram.y
* + * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...]) + * ALTER PUBLICATION name SET COLUMNS table_name ALL + *(Same as my earlier review comment #4)
IMO there was no need for the new syntax of SET COLUMNS.
Not modified yet, we'll see about the syntax.
~~~
18. src/backend/replication/logical/proto.c - logicalrep_write_attrs
- /* send number of live attributes */
- for (i = 0; i < desc->natts; i++)
- {
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc,
i)->attgenerated)
- continue;
- nliveatts++;
- }
- pq_sendint16(out, nliveatts);
-
/* fetch bitmap of REPLICATION IDENTITY attributes */
replidentfull = (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL);
if (!replidentfull)
idattrs = RelationGetIdentityKeyBitmap(rel);+ /* send number of live attributes */ + for (i = 0; i < desc->natts; i++) + { + Form_pg_attribute att = TupleDescAttr(desc, i); + + if (att->attisdropped || att->attgenerated) + continue; + if (columns != NULL && !bms_is_member(att->attnum, columns)) + continue; + nliveatts++; + } + pq_sendint16(out, nliveatts); +This change seemed to have the effect of moving that 4 lines of
"replidentfull" code from below the loop to above the loop. But moving
that code seems unrelated to the Col-Filter patch. (??).
Right, restored the original code.
~~~
19. src/backend/replication/logical/tablesync.c - fetch_remote_table_info
@@ -793,12 +877,12 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecClearTuple(slot); } + ExecDropSingleTupleTableSlot(slot); - - lrel->natts = natt; - walrcv_clear_result(res); pfree(cmd.data); + + lrel->natts = natt; }The shuffling of those few lines seems unrelated to any requirement of
the Col-Filter patch (??)
Yep, undone. I'd bet this is simply due to older versions of the patch
touching this place, and then undoing some of it.
~~~
20. src/backend/replication/logical/tablesync.c - copy_table
+ for (int i = 0; i < lrel.natts; i++) + { + appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i])); + if (i < lrel.natts - 1) + appendStringInfoString(&cmd, ", "); + }Perhaps that could be expressed more simply if the other way around like:
for (int i = 0; i < lrel.natts; i++)
{
if (i)
appendStringInfoString(&cmd, ", ");
appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
}
I used a slightly different version.
~~~
21. src/backend/replication/pgoutput/pgoutput.c
+ + /* + * Set of columns included in the publication, or NULL if all columns are + * included implicitly. Note that the attnums in this list are not + * shifted by FirstLowInvalidHeapAttributeNumber. + */ + Bitmapset *columns;Typo: "in this list" --> "in this set" (??)
"bitmap" is what we call Bitmapset so I used that.
~~~
22. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry
* Don't publish changes for partitioned tables, because - * publishing those of its partitions suffices, unless partition - * changes won't be published due to pubviaroot being set. + * publishing those of its partitions suffices. (However, ignore + * this if partition changes are not to published due to + * pubviaroot being set.) */This change seems unrelated to the Col-Filter patch, so perhaps it
should not be here at all.Also, typo: "are not to published"
Yeah, unrelated. Reverted.
~~~
23. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry
+ /* + * Obtain columns published by this publication, and add them + * to the list for this rel. Note that if at least one + * publication has a empty column list, that means to publish + * everything; so if we saw a publication that includes all + * columns, skip this. + */Typo: "a empty" --> "an empty"
Fixed.
~~~
24. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry
+ if (isnull) + { + /* + * If we see a publication with no columns, reset the + * list and ignore further ones. + */Perhaps that comment is meant to say "with no column filter" instead
of "with no columns"?
Yep, fixed.
~~~
25. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry
+ if (isnull) + { ... + } + else if (!isnull) + { ... + }Is the "if (!isnull)" in the else just to be really REALLY sure it is not null?
Double-tap ;-) Removed the condition.
~~~
26. src/bin/pg_dump/pg_dump.c - getPublicationTables
+ pubrinfo[i].pubrattrs = attribs->data; + } + else + pubrinfo[j].pubrattrs = NULL;I got confused reading this code. Are those different indices 'i' and
'j' correct?
Good catch! I think you're right and it should be "j" in both places.
This'd only cause trouble in selective pg_dumps (when dumping selected
tables). The patch clearly needs some pg_dump tests.
~~~
27. src/bin/psql/describe.c
The Row-Filter [1] displays filter information not only for the psql
\dRp+ command but also for the psql \d <tablename> command. Perhaps
the Col-Filter patch should do that too.
Not sure.
~~~
28. src/bin/psql/tab-complete.c
@@ -1657,6 +1657,8 @@ psql_completion(const char *text, int start, int end) /* ALTER PUBLICATION <name> ADD */ else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD")) COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE"); + else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD", "TABLE")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables, NULL); /* ALTER PUBLICATION <name> DROP */I am not sure about this one- is that change even related to the
Col-Filter patch or is this some unrelated bugfix?
Yeah, seems unrelated - possibly from a rebase or something. Removed.
~~~
29. src/include/catalog/pg_publication.h
@@ -86,6 +86,7 @@ typedef struct Publication
typedef struct PublicationRelInfo
{
Relation relation;
+ List *columns;
} PublicationRelInfo;Perhaps that needs some comment. e.g. do you need to mention that a
NIL List means all columns?
I added a short comment.
~~~
30. src/include/nodes/parsenodes.h
@@ -3642,6 +3642,7 @@ typedef struct PublicationTable { NodeTag type; RangeVar *relation; /* relation to be published */ + List *columns; /* List of columns in a publication table */ } PublicationTable;That comment "List of columns in a publication table" doesn't really
say anything helpful.Perhaps it should mention that a NIL List means all table columns?
Not sure, seems fine.
~~~
31. src/test/regress/sql/publication.sql
The regression test file has an uncommon mixture of /* */ and -- style comments.
Perhaps change all the /* */ ones?
Yeah, that needs some cleanup. I haven't done anything about it yet.
~~~
32. src/test/regress/sql/publication.sql
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text, + d int generated always as (a + length(b)) stored); +ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x); -- error +ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c); -- error +ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d); -- error +ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c); -- okFor all these tests (and more) there seems not sufficient explanation
comments to say exactly what each test case is testing, e.g. *why* is
an "error" expected for some cases but "ok" for others.
Not sure. I think the error is generally obvious in the expected output.
~~~
33. src/test/regress/sql/publication.sql
"-- no dice"
(??) confusing comment.
Same as for the errors.
~~~
34. src/test/subscription/t/028_column_list.pl
I think a few more comments in this TAP file would help to make the
purpose of the tests more clear.
Yeah, the 0004 patch I shared a couple minutes ago does exactly that.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 2022-Feb-16, Tomas Vondra wrote:
Here's an updated version of the patch, rebased to current master. Parts
0002 and 0003 include various improvements based on review by me and another
one by Peter Smith [1].
Thanks for doing this!
1) partitioning with pubviaroot=true
I agree that preventing the inconsistencies from happening is probably
the best.
2) merging multiple column filters
When the table has multiple column filters (in different publications), we
need to merge them. Which works, except that FOR ALL TABLES [IN SCHEMA]
needs to be handled as "has no column filter" (and replicates everything).
Agreed.
3) partitioning with pubivaroot=false
When a partitioned table is added with (pubviaroot=false), it should not be
subject to column filter on the parent relation, which is the same behavior
used by the row filtering patch.
You mean each partition should define its own filter, or lack of filter?
That sounds reasonable.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
"Pensar que el espectro que vemos es ilusorio no lo despoja de espanto,
sólo le suma el nuevo terror de la locura" (Perelandra, C.S. Lewis)
On 2/16/22 01:33, Alvaro Herrera wrote:
On 2022-Feb-16, Tomas Vondra wrote:
Here's an updated version of the patch, rebased to current master. Parts
0002 and 0003 include various improvements based on review by me and another
one by Peter Smith [1].Thanks for doing this!
1) partitioning with pubviaroot=true
I agree that preventing the inconsistencies from happening is probably
the best.2) merging multiple column filters
When the table has multiple column filters (in different publications), we
need to merge them. Which works, except that FOR ALL TABLES [IN SCHEMA]
needs to be handled as "has no column filter" (and replicates everything).Agreed.
3) partitioning with pubivaroot=false
When a partitioned table is added with (pubviaroot=false), it should not be
subject to column filter on the parent relation, which is the same behavior
used by the row filtering patch.You mean each partition should define its own filter, or lack of filter?
That sounds reasonable.
If the partition is not published by the root, it shouldn't use the
filter defined on the root. I wonder what should happen to the filter
defined on the partition itself. I'd say
pubviaroot=false -> use filter defined on partition (if any)
pubviaroot=true -> use filter defined on root (if any)
I wonder what the row filter patch is doing - we should probably follow
the same logic, if only to keep the filtering stuff consistent.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Wed, Feb 16, 2022 at 6:09 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 2/16/22 01:33, Alvaro Herrera wrote:
3) partitioning with pubivaroot=false
When a partitioned table is added with (pubviaroot=false), it should not be
subject to column filter on the parent relation, which is the same behavior
used by the row filtering patch.You mean each partition should define its own filter, or lack of filter?
That sounds reasonable.If the partition is not published by the root, it shouldn't use the
filter defined on the root. I wonder what should happen to the filter
defined on the partition itself. I'd saypubviaroot=false -> use filter defined on partition (if any)
pubviaroot=true -> use filter defined on root (if any)
I wonder what the row filter patch is doing - we should probably follow
the same logic, if only to keep the filtering stuff consistent.
The row filter patch is doing the same and additionally, it gives an
error if the user provides a filter for a partitioned table with
pubviaroot as false.
--
With Regards,
Amit Kapila.
On Wed, Feb 16, 2022 at 5:03 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
Hi,
Here's an updated version of the patch, rebased to current master. Parts
0002 and 0003 include various improvements based on review by me and
another one by Peter Smith [1].Part 0003 reworks and significantly extends the TAP test, to exercise
various cases related to changes of replica identity etc. discussed in
this thread. Some of the tests however still fail, because the behavior
was not updated - I'll work on that once we agree what the expected
behavior is.1) partitioning with pubviaroot=true
The main set of failures is related to partitions with different replica
identities and (pubviaroot=true), some of which may be mismatching the
column list. There are multiple such test cases, depending on how the
inconsistency is introduced - it may be there from the beginning, the
column filter may be modified after adding the partitioned table to the
publication, etc.I think the expected behavior is to prohibit such cases from happening,
by cross-checking the column filter when adding the partitioned table to
publication, attaching a partition or changing a column filter.
I feel it is better to follow the way described by Peter E. here [1]/messages/by-id/ca91dc91-80ba-e954-213e-b4170a6160f5@enterprisedb.com
to handle these cases. The row filter patch is also using the same
scheme as that is what we are doing now for Updates/Deletes and it
would be really challenging and much more effort/code to deal with
everything at DDL time. I have tried to explain some of that in my
emails [2]/messages/by-id/CAA4eK1+m45Xyzx7AUY9TyFnB6CZ7_+_uooPb7WHSpp7UE=YmKg@mail.gmail.com[3]/messages/by-id/CAA4eK1+1DMkCip9SB3B0_u0Q6fGf-D3vgqQodkLfur0qkL482g@mail.gmail.com.
[1]: /messages/by-id/ca91dc91-80ba-e954-213e-b4170a6160f5@enterprisedb.com
[2]: /messages/by-id/CAA4eK1+m45Xyzx7AUY9TyFnB6CZ7_+_uooPb7WHSpp7UE=YmKg@mail.gmail.com
[3]: /messages/by-id/CAA4eK1+1DMkCip9SB3B0_u0Q6fGf-D3vgqQodkLfur0qkL482g@mail.gmail.com
--
With Regards,
Amit Kapila.
Hi,
Attached is an updated patch, addressing most of the issues reported so
far. There are various minor tweaks, but the main changes are:
1) regular regression tests, verifying (hopefully) all the various cases
of publication vs. column filters, replica identity check at various
changes and so on
2) pg_dump tests, testing column filters (alone and with row filter)
3) checks of column filter vs. publish_via_partition_root and replica
identity, following the same logic as the row-filter patch (hopefully,
it touches the same places, using the same logic, ...)
That means - with "publish_via_partition_root=false" it's not allowed to
specify column filters on partitioned tables, only for leaf partitions.
And we check column filter vs. replica identity when adding tables to
publications, or whenever we change the replica identity.
The patch is still a bit crude, I'm sure some of the places (especially
the new ones) may need cleanup/recovery. But I think it's much closer to
being committable, I think.
The first two simple patches are adding tests for the row filtering. So
this is not really part of this patch.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
0001-Verify-changing-WHERE-condition-for-a-publi-20220302.patchtext/x-patch; charset=UTF-8; name=0001-Verify-changing-WHERE-condition-for-a-publi-20220302.patchDownload
From e781f840e38701c63d8b57ff36bd520f2cced6ad Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Sat, 26 Feb 2022 17:33:09 +0100
Subject: [PATCH 1/3] Verify changing WHERE condition for a publication
Commit 52e4f0cd47 added support for row filters in logical replication,
including regression tests with multiple ALTER PUBLICATION commands,
modifying the row filter. But the tests never verified that the row
filter was actually updated in the catalog. This adds a couple \d and
\dRp commands, to verify the catalog was updated.
---
src/test/regress/expected/publication.out | 66 +++++++++++++++++++++++
src/test/regress/sql/publication.sql | 8 +++
2 files changed, 74 insertions(+)
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 3c382e520e4..227ce759486 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -395,15 +395,81 @@ LINE 1: ...ICATION testpub5 ADD TABLE testpub_rf_tbl1 WHERE (b < '2' CO...
DETAIL: User-defined collations are not allowed.
-- ok - NULLIF is allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (NULLIF(1,2) = a);
+\d+ testpub_rf_tbl1
+ Table "public.testpub_rf_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | | | plain | |
+ b | text | | | | extended | |
+Publications:
+ "testpub5" WHERE (NULLIF(1, 2) = a)
+
+\dRp+ testpub5
+ Publication testpub5
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | f | f
+Tables:
+ "public.testpub_rf_tbl1" WHERE (NULLIF(1, 2) = a)
+
-- ok - built-in operators are allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (a IS NULL);
+\d+ testpub_rf_tbl1
+ Table "public.testpub_rf_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | | | plain | |
+ b | text | | | | extended | |
+Publications:
+ "testpub5" WHERE (a IS NULL)
+
+\dRp+ testpub5
+ Publication testpub5
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | f | f
+Tables:
+ "public.testpub_rf_tbl1" WHERE (a IS NULL)
+
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE ((a > 5) IS FALSE);
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (a IS DISTINCT FROM 5);
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE ((a, a + 1) < (2, 3));
-- ok - built-in type coercions between two binary compatible datatypes are allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (b::varchar < '2');
+\d+ testpub_rf_tbl1
+ Table "public.testpub_rf_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | | | plain | |
+ b | text | | | | extended | |
+Publications:
+ "testpub5" WHERE (((b)::character varying)::text < '2'::text)
+
+\dRp+ testpub5
+ Publication testpub5
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | f | f
+Tables:
+ "public.testpub_rf_tbl1" WHERE (((b)::character varying)::text < '2'::text)
+
-- ok - immutable built-in functions are allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl4 WHERE (length(g) < 6);
+\d+ testpub_rf_tbl1
+ Table "public.testpub_rf_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | | | plain | |
+ b | text | | | | extended | |
+
+\dRp+ testpub5
+ Publication testpub5
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | f | f
+Tables:
+ "public.testpub_rf_tbl4" WHERE (length(g) < 6)
+
-- fail - user-defined types are not allowed
CREATE TYPE rf_bug_status AS ENUM ('new', 'open', 'closed');
CREATE TABLE rf_bug (id serial, description text, status rf_bug_status);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 3f04d34264a..cd7e0182716 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -203,15 +203,23 @@ CREATE COLLATION user_collation FROM "C";
ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl1 WHERE (b < '2' COLLATE user_collation);
-- ok - NULLIF is allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (NULLIF(1,2) = a);
+\d+ testpub_rf_tbl1
+\dRp+ testpub5
-- ok - built-in operators are allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (a IS NULL);
+\d+ testpub_rf_tbl1
+\dRp+ testpub5
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE ((a > 5) IS FALSE);
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (a IS DISTINCT FROM 5);
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE ((a, a + 1) < (2, 3));
-- ok - built-in type coercions between two binary compatible datatypes are allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (b::varchar < '2');
+\d+ testpub_rf_tbl1
+\dRp+ testpub5
-- ok - immutable built-in functions are allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl4 WHERE (length(g) < 6);
+\d+ testpub_rf_tbl1
+\dRp+ testpub5
-- fail - user-defined types are not allowed
CREATE TYPE rf_bug_status AS ENUM ('new', 'open', 'closed');
CREATE TABLE rf_bug (id serial, description text, status rf_bug_status);
--
2.34.1
0002-Test-publication-row-filters-in-pg_dump-tes-20220302.patchtext/x-patch; charset=UTF-8; name=0002-Test-publication-row-filters-in-pg_dump-tes-20220302.patchDownload
From 8ee67dd52a1fc08837aa85979dfc0842cc968012 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Tue, 1 Mar 2022 15:25:56 +0100
Subject: [PATCH 2/3] Test publication row filters in pg_dump tests
Commit 52e4f0cd47 added support for row filters when replicating tables,
but the commit added no pg_dump tests for this feature. So add at least
a simple test.
---
src/bin/pg_dump/t/002_pg_dump.pl | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index dd065c758fa..c3bcef8c0ec 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2407,12 +2407,12 @@ my %tests = (
},
},
- 'ALTER PUBLICATION pub1 ADD TABLE test_second_table' => {
+ 'ALTER PUBLICATION pub1 ADD TABLE test_second_table WHERE (col1 = 1)' => {
create_order => 52,
create_sql =>
- 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_second_table;',
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_second_table WHERE (col1 = 1);',
regexp => qr/^
- \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_second_table;\E
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_second_table WHERE ((col1 = 1));\E
/xm,
like => { %full_runs, section_post_data => 1, },
unlike => { exclude_dump_test_schema => 1, },
--
2.34.1
0003-Allow-specifying-column-filters-for-logical-20220302.patchtext/x-patch; charset=UTF-8; name=0003-Allow-specifying-column-filters-for-logical-20220302.patchDownload
From 223a3c0cbbc24814c2a207314e3d5cd8257b9fba Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Wed, 23 Feb 2022 21:15:18 +0100
Subject: [PATCH 3/3] Allow specifying column filters for logical replication
This allows specifying an optional column list when adding a table to
logical replication. Columns not included on this list are not sent to
the subscriber. The filter is specified as a list of columns after the
table name, enclosed in parentheses.
For UPDATE/DELETE publications, the column filter needs to cover all
REPLICA IDENTITY columns. For INSERT publications, the column list is
arbitrary and may omit some REPLICA IDENTITY columns. Furthermore, if
the table uses REPLICA IDENTITY FULL, column filter is not allowed.
The column filter can contain only simple column references. Complex
expressions, function calls etc. are not allowed. This restriction could
be relaxed in the future.
During the initial table synchronization, only columns specified in the
column filter are copied to the subscriber. If the subscription has
several publications, containing the same table with different column
filters, columns specified in any of the filters will be copied. This
means all columns are replicated if the table has no column filter at
all (which is treated as column filter with all columns), of when of the
publications is defined as FOR ALL TABLES (possibly IN SCHEMA for the
schema of the table).
For partitioned tables, publish_via_partition_root determines whether
the row filter for the root or leaf relation will be used. If the
parameter is 'false' (the default), the filter defined for the leaf
relation is used. Otherwise, the column filter for the root partition
will be used.
Psql commands \dRp+ and \d <table-name> now display any column filters.
Author: Rahila Syed <rahilasyed90@gmail.com>
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 13 +
doc/src/sgml/protocol.sgml | 4 +-
doc/src/sgml/ref/alter_publication.sgml | 23 +-
doc/src/sgml/ref/create_publication.sgml | 17 +-
src/backend/catalog/pg_publication.c | 397 +++++++++-
src/backend/commands/publicationcmds.c | 181 ++++-
src/backend/commands/tablecmds.c | 209 ++++-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 60 +-
src/backend/replication/logical/proto.c | 55 +-
src/backend/replication/logical/tablesync.c | 271 ++++++-
src/backend/replication/pgoutput/pgoutput.c | 121 ++-
src/bin/pg_dump/pg_dump.c | 47 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 60 ++
src/bin/psql/describe.c | 40 +-
src/include/catalog/pg_publication.h | 6 +
src/include/catalog/pg_publication_rel.h | 1 +
src/include/nodes/parsenodes.h | 4 +-
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 212 +++++
src/test/regress/sql/publication.sql | 140 ++++
src/test/subscription/t/029_column_list.pl | 836 ++++++++++++++++++++
24 files changed, 2629 insertions(+), 77 deletions(-)
create mode 100644 src/test/subscription/t/029_column_list.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 83987a99045..2b61f42b71d 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6263,6 +6263,19 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to schema
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>prattrs</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+ </para>
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all columns are published.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 1c5ab008791..91541cd8cf7 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7005,7 +7005,9 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column (except
+ generated columns and other columns that don't appear in the column
+ filter list, for tables that have one):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 32b75f6c78e..aa6827c977b 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -25,12 +25,13 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD <replace
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ALTER TABLE <replaceable class="parameter">table_name</replaceable> SET COLUMNS { ( <replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -64,6 +65,12 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
command retain their previous settings.
</para>
+ <para>
+ The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows changing
+ the set of columns that are included in the publication. If a column list
+ is specified, it must include the replica identity columns.
+ </para>
+
<para>
The remaining variants change the owner and the name of the publication.
</para>
@@ -112,6 +119,10 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
+
If the optional <literal>WHERE</literal> clause is specified, rows for
which the <replaceable class="parameter">expression</replaceable>
evaluates to false or null will not be published. Note that parentheses
@@ -172,9 +183,15 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
</para>
<para>
- Add some tables to the publication:
+ Add tables to the publication:
+<programlisting>
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS (user_id, firstname, lastname);
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 4979b9b646d..fb2d013393b 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -86,6 +86,13 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<literal>TRUNCATE</literal> commands.
</para>
+ <para>
+ When a column list is specified, only the named columns are replicated.
+ If no column list is specified, all columns of the table are replicated
+ through this publication, including any columns added later. If a column
+ list is specified, it must include the replica identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
@@ -327,6 +334,14 @@ CREATE PUBLICATION production_publication FOR TABLE users, departments, ALL TABL
<structname>sales</structname>:
<programlisting>
CREATE PUBLICATION sales_publication FOR ALL TABLES IN SCHEMA marketing, sales;
+</programlisting></para>
+
+ <para>
+ Create a publication that publishes all changes for table <structname>users</structname>,
+ but replicates only columns <structname>user_id</structname> and
+ <structname>firstname</structname>:
+<programlisting>
+CREATE PUBLICATION users_filtered FOR TABLE users (user_id, firstname);
</programlisting></para>
</refsect1>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 25998fbb39b..840f0a0eb24 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,12 +45,22 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+static void check_publication_columns(Publication *pub, Relation targetrel,
+ Bitmapset *columns);
+static void publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs,
+ Bitmapset **attset);
+
/*
- * Check if relation can be in given publication and throws appropriate
- * error if not.
+ * Check if relation can be in given publication and that the column
+ * filter is sensible, and throws appropriate error if not.
+ *
+ * columns is the bitmapset of attribute numbers included in the column list,
+ * or NULL if no column list was specified (i.e. all columns are replicated)
*/
static void
-check_publication_add_relation(Relation targetrel)
+check_publication_add_relation(Publication *pub, Relation targetrel,
+ Bitmapset *columns)
{
/* Must be a regular or partitioned table */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
@@ -82,6 +92,103 @@ check_publication_add_relation(Relation targetrel)
errmsg("cannot add relation \"%s\" to publication",
RelationGetRelationName(targetrel)),
errdetail("This operation is not supported for unlogged tables.")));
+
+ /*
+ * Ensure the column filter is compatible with the replica identity and the
+ * actions the publication is replicating.
+ */
+ check_publication_columns(pub, targetrel, columns);
+}
+
+/*
+ * Enforce that the column list can only leave out columns that don't
+ * need to be sent as part of replica identity.
+ *
+ * If the relation uses REPLICA IDENTITY FULL, we can't allow any column
+ * list even if it lists all columns of the relation - it'd cause issues
+ * if a column is added later. The column would become part of a replica
+ * identity, violating the rule that the column list includes the whole
+ * replica identity. We could add the column to the column list too, of
+ * course, but it seems rather useles - the column list would always
+ * include all columns, i.e. as if there's no column filter.
+ *
+ * In other cases, the columns in the REPLICA IDENTITY cannot be left out,
+ * except when the publication replicates only inserts. So we check that
+ * for UPDATE/DELETE the replica identity is a subset of the column filter.
+ */
+static void
+check_publication_columns(Publication *pub, Relation targetrel, Bitmapset *columns)
+{
+ List *relids;
+ Oid relid;
+ ListCell *lc;
+
+ /*
+ * If there is no column list, we treat it as if the list contains all columns. In
+ * which case there's nothing to check so we're done.
+ */
+ if (!columns)
+ return;
+
+ relid = RelationGetRelid(targetrel);
+ relids = find_all_inheritors(relid, NoLock, NULL);
+
+ /* We only care about replica identity on leaf partitions. */
+ foreach(lc, relids)
+ {
+ Oid partOid = lfirst_oid(lc);
+
+ /* XXX Do we need to lock this? */
+ Relation partrel = relation_open(partOid, AccessShareLock);
+
+ /*
+ * ignore non-leaf relations
+ *
+ * XXX Should we consider what replica identity is set for the relation?
+ */
+ if (partrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ goto cleanup;
+
+ /* With REPLICA IDENTITY FULL no column filter is allowed. */
+ if (partrel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)), /* XXX maybe report the partition? */
+ errdetail("Cannot specify column list on relations with REPLICA IDENTITY FULL."));
+
+ /* When replicating UPDATE/DELETE, the whole replica identity has to be sent. */
+ if (pub->pubactions.pubupdate || pub->pubactions.pubdelete)
+ {
+ Bitmapset *idattrs;
+ int x;
+
+ idattrs = RelationGetIndexAttrBitmap(partrel,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * Attnums in the bitmap returned by RelationGetIndexAttrBitmap are
+ * offset (to handle system columns the usual way), while column filter
+ * does not use offset, so we can't do bms_is_subset(). Instead, we have
+ * to loop over the idattrs and check all of them are in the filter.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ if (!bms_is_member(x + FirstLowInvalidHeapAttributeNumber, columns))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(targetrel)), /* XXX maybe report the partition? */
+ errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
+ }
+
+ bms_free(idattrs);
+ }
+
+cleanup:
+ relation_close(partrel, AccessShareLock);
+ }
}
/*
@@ -328,6 +435,9 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
Oid relid = RelationGetRelid(targetrel);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ Bitmapset *attset = NULL;
+ AttrNumber *attarray;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -353,7 +463,18 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
RelationGetRelationName(targetrel), pub->name)));
}
- check_publication_add_relation(targetrel);
+ /*
+ * Translate column names to attnums and check the column list is valid.
+ * We build both a bitmap and array of attnums - array is stored in the
+ * catalog, while the bitmap is more convenient for checking.
+ */
+ publication_translate_columns(pri->relation, pri->columns,
+ &natts, &attarray, &attset);
+
+ check_publication_add_relation(pub, targetrel, attset);
+
+ /* Won't need the bitmapset anymore. */
+ bms_free(attset);
/* Form a tuple. */
memset(values, 0, sizeof(values));
@@ -367,6 +488,16 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ if (pri->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+
/* Add qualifications, if available */
if (pri->whereClause != NULL)
values[Anum_pg_publication_rel_prqual - 1] = CStringGetTextDatum(nodeToString(pri->whereClause));
@@ -382,6 +513,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
/* Register dependencies as needed */
ObjectAddressSet(myself, PublicationRelRelationId, pubreloid);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ pfree(attarray);
+
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -415,6 +554,166 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
return myself;
}
+/*
+ * Update the column list for a relation in a publication.
+ */
+void
+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns)
+{
+ Bitmapset *attset;
+ AttrNumber *attarray;
+ HeapTuple copytup;
+ int natts;
+ bool nulls[Natts_pg_publication_rel];
+ bool replaces[Natts_pg_publication_rel];
+ Datum values[Natts_pg_publication_rel];
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ replaces[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ deleteDependencyRecordsForClass(PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid,
+ RelationRelationId,
+ DEPENDENCY_AUTO);
+
+ if (columns == NULL)
+ {
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+ }
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ int2vector *prattrs;
+ Publication *pub;
+
+ pub = GetPublication(((Form_pg_publication_rel) GETSTRUCT(pubreltup))->prpubid);
+
+ publication_translate_columns(targetrel, columns,
+ &natts, &attarray, &attset);
+
+ /*
+ * Make sure the column list checks out.
+ *
+ * XXX this should occur at caller in publicationcmds.c, not here.
+ * XXX How come this does not check replica identity? Should this prevent
+ * replica identity full, just like check_publication_add_relation?
+ */
+ check_publication_columns(pub, targetrel, attset);
+ bms_free(attset);
+
+ /* XXX "pub" is leaked here */
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+
+ /* Add dependencies on the new list of columns */
+ ObjectAddressSet(myself, PublicationRelRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid);
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId,
+ RelationGetRelid(targetrel), attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ }
+
+ copytup = heap_modify_tuple(pubreltup, RelationGetDescr(pubrel),
+ values, nulls, replaces);
+
+ CatalogTupleUpdate(pubrel, &pubreltup->t_self, copytup);
+
+ heap_freetuple(copytup);
+}
+
+/*
+ * qsort comparator for attnums
+ *
+ * XXX We already have compare_int16, so maybe let's share that, somehow?
+ */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list (no system or generated attributes,
+ * no duplicates). Additional checks with replica identity are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ *
+ * XXX Should this detect duplicate columns?
+ */
+static void
+publication_translate_columns(Relation targetrel, List *columns, int *natts,
+ AttrNumber **attrs, Bitmapset **attset)
+{
+ AttrNumber *attarray;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference generated column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /* Be tidy, so that the catalog representation is always sorted */
+ qsort(attarray, n, sizeof(AttrNumber), compare_int16);
+
+ *natts = n;
+ *attrs = attarray;
+ *attset = set;
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -522,6 +821,96 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull)
+ continue;
+
+ pubs = lappend_oid(pubs,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+/* FIXME maybe these two routines should be in lsyscache.c */
+/* Return the set of actions that the given publication includes */
+void
+GetActionsInPublication(Oid pubid, PublicationActions *actions)
+{
+ HeapTuple pub;
+ Form_pg_publication pubForm;
+
+ pub = SearchSysCache1(PUBLICATIONOID,
+ ObjectIdGetDatum(pubid));
+ if (!HeapTupleIsValid(pub))
+ elog(ERROR, "cache lookup failed for publication %u", pubid);
+
+ pubForm = (Form_pg_publication) GETSTRUCT(pub);
+ actions->pubinsert = pubForm->pubinsert;
+ actions->pubupdate = pubForm->pubupdate;
+ actions->pubdelete = pubForm->pubdelete;
+ actions->pubtruncate = pubForm->pubtruncate;
+
+ ReleaseSysCache(pub);
+}
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 16b8661a1b7..d5fad6c0f79 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -608,6 +608,45 @@ TransformPubWhereClauses(List *tables, const char *queryString,
}
}
+
+/*
+ * Transform the publication column lists expression for all the relations
+ * in the list.
+ *
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow filters on the leaf relations. So maybe rename it?
+ */
+static void
+TransformPubColumnList(List *tables, const char *queryString,
+ bool pubviaroot)
+{
+ ListCell *lc;
+
+ foreach(lc, tables)
+ {
+ PublicationRelInfo *pri = (PublicationRelInfo *) lfirst(lc);
+
+ if (pri->columns == NIL)
+ continue;
+
+ /*
+ * If the publication doesn't publish changes via the root partitioned
+ * table, the partition's column filter will be used. So disallow using
+ * the column list on partitioned table in this case.
+ */
+ if (!pubviaroot &&
+ pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot use publication column list for relation \"%s\"",
+ RelationGetRelationName(pri->relation)),
+ errdetail("column list cannot be used for a partitioned table when %s is false.",
+ "publish_via_partition_root")));
+ }
+}
+
/*
* Create new publication.
*/
@@ -724,6 +763,9 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
TransformPubWhereClauses(rels, pstate->p_sourcetext,
publish_via_partition_root);
+ TransformPubColumnList(rels, pstate->p_sourcetext,
+ publish_via_partition_root);
+
PublicationAddTables(puboid, rels, true, NULL);
CloseTableList(rels);
}
@@ -754,6 +796,46 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
return myself;
}
+/*
+ * Change the column list of a relation in a publication
+ */
+static void
+PublicationSetColumns(AlterPublicationStmt *stmt,
+ Form_pg_publication pubform, PublicationTable *table)
+{
+ Relation rel,
+ urel;
+ HeapTuple tup;
+ ObjectAddress obj,
+ secondary;
+
+ rel = table_open(PublicationRelRelationId, RowExclusiveLock);
+ urel = table_openrv(table->relation, ShareUpdateExclusiveLock);
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(RelationGetRelid(urel)),
+ ObjectIdGetDatum(pubform->oid));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errmsg("relation \"%s\" is not already in publication \"%s\"",
+ table->relation->relname,
+ NameStr(pubform->pubname)));
+
+ publication_set_table_columns(rel, tup, urel, table->columns);
+
+ ObjectAddressSet(obj, PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->oid);
+ ObjectAddressSet(secondary, RelationRelationId, RelationGetRelid(urel));
+ EventTriggerCollectSimpleCommand(obj, secondary, (Node *) stmt);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+ table_close(urel, NoLock);
+
+ InvokeObjectPostAlterHook(PublicationRelationId, pubform->oid, 0);
+}
+
/*
* Change options of a publication.
*/
@@ -838,6 +920,16 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
}
}
+ /*
+ * FIXME check pubactions vs. replica identity, to ensure the replica
+ * identity is included in the column filter. Only do this for update
+ * and delete publications. See check_publication_columns.
+ *
+ * XXX This is needed because publish_via_partition_root may change,
+ * in which case the row filters may be invalid (e.g. with pvpr=false
+ * there must be no filter on partitioned tables).
+ */
+
/* Everything ok, form a new tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -975,10 +1067,20 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
PublicationAddTables(pubid, rels, false, stmt);
}
else if (stmt->action == AP_DropObjects)
PublicationDropTables(pubid, rels, false);
+ else if (stmt->action == AP_SetColumns)
+ {
+ Assert(schemaidlist == NIL);
+ Assert(list_length(tables) == 1);
+
+ PublicationSetColumns(stmt, pubform,
+ linitial_node(PublicationTable, tables));
+ }
else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
@@ -991,6 +1093,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
/*
* To recreate the relation list for the publication, look for
* existing relations that do not need to be dropped.
@@ -1002,32 +1106,85 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
PublicationRelInfo *oldrel;
bool found = false;
HeapTuple rftuple;
- bool rfisnull = true;
Node *oldrelwhereclause = NULL;
+ Bitmapset *oldcolumns = NULL;
/* look up the cache for the old relmap */
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(oldrelid),
ObjectIdGetDatum(pubid));
+ /*
+ * See if the existing relation currently has a WHERE clause or a
+ * column filter. We need to compare those too.
+ */
if (HeapTupleIsValid(rftuple))
{
+ bool isnull = true;
Datum whereClauseDatum;
+ Datum columnListDatum;
+ /* Load the WHERE clause for this table. */
whereClauseDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
Anum_pg_publication_rel_prqual,
- &rfisnull);
- if (!rfisnull)
+ &isnull);
+ if (!isnull)
oldrelwhereclause = stringToNode(TextDatumGetCString(whereClauseDatum));
+ /* Transform the int2vector column list to a bitmap. */
+ columnListDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ /*
+ * XXX Maybe make this a separate function. We do this on
+ * multiple places.
+ */
+ if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(columnListDatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ for (int i = 0; i < nelems; i++)
+ oldcolumns = bms_add_member(oldcolumns, elems[i]);
+ }
+
ReleaseSysCache(rftuple);
}
foreach(newlc, rels)
{
PublicationRelInfo *newpubrel;
+ Oid newrelid;
+ Bitmapset *newcolumns = NULL;
newpubrel = (PublicationRelInfo *) lfirst(newlc);
+ newrelid = RelationGetRelid(newpubrel->relation);
+
+ /*
+ * If the new publication has column filter, transform it to
+ * a bitmap too.
+ */
+ if (newpubrel->columns)
+ {
+ ListCell *lc;
+
+ foreach(lc, newpubrel->columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(newrelid, colname);
+
+ /* no checks needed here, that happens elsewhere */
+
+ newcolumns = bms_add_member(newcolumns, attnum);
+ }
+ }
/*
* Check if any of the new set of relations matches with the
@@ -1037,7 +1194,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
*/
if (RelationGetRelid(newpubrel->relation) == oldrelid)
{
- if (equal(oldrelwhereclause, newpubrel->whereClause))
+ if (equal(oldrelwhereclause, newpubrel->whereClause) &&
+ bms_equal(oldcolumns, newcolumns))
{
found = true;
break;
@@ -1056,6 +1214,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
{
oldrel = palloc(sizeof(PublicationRelInfo));
oldrel->whereClause = NULL;
+ oldrel->columns = NIL;
oldrel->relation = table_open(oldrelid,
ShareUpdateExclusiveLock);
delrels = lappend(delrels, oldrel);
@@ -1117,7 +1276,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
}
else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* AP_SetObjects */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -1140,6 +1299,10 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
*/
PublicationAddSchemas(pubform->oid, schemaidlist, true, stmt);
}
+ else
+ {
+ /* Nothing to do for AP_SetColumns */
+ }
}
/*
@@ -1443,6 +1606,7 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
pub_rel->whereClause = t->whereClause;
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
@@ -1497,6 +1661,8 @@ OpenTableList(List *tables)
pub_rel->relation = rel;
/* child inherits WHERE clause from parent */
pub_rel->whereClause = t->whereClause;
+ /* child inherits column list from parent */
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
@@ -1610,6 +1776,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 3e83f375b55..5546d3d4387 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -625,6 +625,8 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static char GetAttributeCompression(Oid atttypid, char *compression);
+static void check_replica_identity(List *relations, char relreplident,
+ Relation rel, Bitmapset *cols);
/* ----------------------------------------------------------------
@@ -8365,6 +8367,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
bool missing_ok, LOCKMODE lockmode,
ObjectAddresses *addrs)
{
+ Oid relid = RelationGetRelid(rel);
HeapTuple tuple;
Form_pg_attribute targetatt;
AttrNumber attnum;
@@ -8384,7 +8387,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/*
* get the number of the attribute
*/
- tuple = SearchSysCacheAttName(RelationGetRelid(rel), colName);
+ tuple = SearchSysCacheAttName(relid, colName);
if (!HeapTupleIsValid(tuple))
{
if (!missing_ok)
@@ -8438,13 +8441,42 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * Also, if the column is used in the column list of a publication,
+ * disallow the drop if the DROP is RESTRICT. We don't do anything if the
+ * DROP is CASCADE, which means that the dependency mechanism will remove
+ * the relation from the publication.
+ */
+ if (behavior == DROP_RESTRICT)
+ {
+ List *pubs;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(relid);
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(relid, pubid);
+
+ if (list_member_oid(published_cols, attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot drop column \"%s\" because it is part of publication \"%s\"",
+ colName, get_publication_name(pubid, false)),
+ errhint("Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication."));
+ }
+ }
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
* use find_all_inheritors to do it in one pass.
*/
children =
- find_inheritance_children(RelationGetRelid(rel), lockmode);
+ find_inheritance_children(relid, lockmode);
if (children)
{
@@ -8532,7 +8564,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Add object to delete */
object.classId = RelationRelationId;
- object.objectId = RelationGetRelid(rel);
+ object.objectId = relid;
object.objectSubId = attnum;
add_exact_object_address(&object, addrs);
@@ -15864,7 +15896,28 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
Oid indexOid;
Relation indexRel;
int key;
+ Bitmapset *indexed_cols = NULL;
+ List *ancestors = NIL;
+
+ /*
+ * Check that the new partition is compatible with all publications the
+ * parent relation (to which we're attaching) is part of. Each ancestor
+ * may be added to multiple publications - we need to check all of them,
+ * not just the top-most, because the top-most one might be removed later.
+ */
+ ancestors = get_partition_ancestors(RelationGetRelid(rel));
+
+ /*
+ * Include the rel itself too, because it may already have publications.
+ */
+ ancestors = lappend_oid(ancestors, RelationGetRelid(rel));
+ /*
+ * Check that the replica identity and column filters are compatible.
+ */
+ check_replica_identity(ancestors, stmt->identity_type, rel, NULL);
+
+ /* cross-check the column lists and replica identity (if defined) */
if (stmt->identity_type == REPLICA_IDENTITY_DEFAULT)
{
relation_mark_replica_identity(rel, stmt->identity_type, InvalidOid, true);
@@ -15961,8 +16014,17 @@ ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKMODE lockmode
errmsg("index \"%s\" cannot be used as replica identity because column \"%s\" is nullable",
RelationGetRelationName(indexRel),
NameStr(attr->attname))));
+
+ /*
+ * Collect columns used, in case we have any publications that we need
+ * to vet. Offset by FirstLowInvalidHeapAttributeNumber, because that's
+ * what check_replica_identity expects.
+ */
+ indexed_cols = bms_add_member(indexed_cols, attno - FirstLowInvalidHeapAttributeNumber);
}
+ check_replica_identity(ancestors, REPLICA_IDENTITY_INDEX, rel, indexed_cols);
+
/* This index is suitable for use as a replica identity. Mark it. */
relation_mark_replica_identity(rel, stmt->identity_type, indexOid, true);
@@ -17623,6 +17685,7 @@ ATExecAttachPartition(List **wqueue, Relation rel, PartitionCmd *cmd,
Oid defaultPartOid;
List *partBoundConstraint;
ParseState *pstate = make_parsestate(NULL);
+ List *ancestors = NIL;
pstate->p_sourcetext = context->queryString;
@@ -17784,6 +17847,27 @@ ATExecAttachPartition(List **wqueue, Relation rel, PartitionCmd *cmd,
trigger_name, RelationGetRelationName(attachrel)),
errdetail("ROW triggers with transition tables are not supported on partitions")));
+ /*
+ * Check that the new partition is compatible with all publications the
+ * parent relation (to which we're attaching) is part of. Each ancestor
+ * may be added to multiple publications - we need to check all of them,
+ * not just the top-most, because the top-most one might be removed later.
+ */
+ ancestors = get_partition_ancestors(RelationGetRelid(rel));
+
+ /*
+ * Include the rel to which we attach the new partition, so that we have
+ * all ancestors of the attachrel.
+ */
+ ancestors = lappend_oid(ancestors, RelationGetRelid(rel));
+
+ /*
+ * Check the replica identity of the new partition is compatible with
+ * all publications.
+ */
+ check_replica_identity(ancestors, attachrel->rd_rel->relreplident,
+ attachrel, NULL);
+
/*
* Check that the new partition's bound is valid and does not overlap any
* of existing partitions of the parent - note that it does not return on
@@ -19239,3 +19323,122 @@ GetAttributeCompression(Oid atttypid, char *compression)
return cmethod;
}
+
+/*
+ * - relations - list of OIDs for relations for which we need to inspect the
+ * publications (this includes ancestors and possibly the relation itself)
+ *
+ * - rel - the relation for which we verify the replica identity
+ *
+ * XXX This has a bit of a problem, becase for ALTER TABLE ... REPLICA
+ * IDENTITY it's called before the relation has it assigned. So we get the
+ * relreplident and indexed columns separately. Maybe we could just call
+ * it a bit later?
+ */
+static void
+check_replica_identity(List *relations, char relreplident, Relation rel,
+ Bitmapset *cols)
+{
+ ListCell *lc;
+
+ /*
+ * Check that the new partition is compatible with all publications the
+ * ancestors are members of. Each ancestor may be added to multiple
+ * publications - we need to check all of them, not just the top-most,
+ * because the top-most one might be removed later.
+ */
+ foreach(lc, relations)
+ {
+ ListCell *lc2;
+ Oid ancestor = lfirst_oid(lc);
+
+ /* only publications with a column filter */
+ List *pubids = GetRelationColumnPartialPublications(ancestor);
+
+ /* no publications with column filter for this parent */
+ if (!pubids)
+ continue;
+
+ /* With REPLICA IDENTITY FULL no column filter is allowed. */
+ if (relreplident == REPLICA_IDENTITY_FULL)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(rel)), /* XXX maybe report the ancestor? */
+ errdetail("Cannot specify column list on relations with REPLICA IDENTITY FULL."));
+
+ /* check the column filter vs. replica identity for each partition */
+ foreach (lc2, pubids)
+ {
+ ListCell *lc3;
+ Oid pubid = lfirst_oid(lc2);
+ List *columns = GetRelationColumnListInPublication(ancestor, pubid);
+ Bitmapset *colset = NULL;
+ Bitmapset *idattrs;
+ Publication *pub;
+
+ pub = GetPublication(pubid);
+
+ /*
+ * When not replicating UPDATE/DELETE, we're done. Otherwise check the
+ * whole replica identity is included in the column filter.
+ */
+ if (!pub->pubactions.pubupdate && !pub->pubactions.pubdelete)
+ continue;
+
+ /* Get replica identity attributes. */
+ if ((relreplident == REPLICA_IDENTITY_INDEX) && cols)
+ idattrs = cols;
+ else
+ idattrs = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * This can happen, in which case the replica identity is inherited
+ * from the ancestor, which means it's OK for the publication.
+ */
+ if (!idattrs)
+ continue;
+
+ /*
+ * Build bitmap from the filter columns, for easy comparison with the
+ * replica identity bitmap. Offset by FirstLowInvalidHeapAttributeNumber
+ * for each comparison with idattrs.
+ *
+ * XXX Do we need to map attnums between the partitions? For now
+ * just lookup name in the ancestor (per the column list), and then
+ * lookup the attnum in the child. But maybe that's not needed?
+ */
+ foreach (lc3, columns)
+ {
+ AttrNumber attnum;
+ char *attname;
+
+ attnum = lfirst_oid(lc3);
+ attname = get_attname(ancestor, attnum, false);
+
+ /* reverse lookup for the current relation */
+ attnum = get_attnum(RelationGetRelid(rel), attname);
+
+ colset = bms_add_member(colset, attnum - FirstLowInvalidHeapAttributeNumber);
+ }
+
+ /*
+ * Attnums in the bitmap returned by RelationGetIndexAttrBitmap are
+ * offset (to handle system columns the usual way), while column filter
+ * does not use offset, so we can't do bms_is_subset(). Instead, we have
+ * to loop over the idattrs and check all of them are in the filter.
+ */
+ if (!bms_is_subset(idattrs, colset))
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("invalid column list for publishing relation \"%s\"",
+ RelationGetRelationName(rel)), /* XXX maybe report the ancestor? */
+ errdetail("All columns in REPLICA IDENTITY must be present in the column list."));
+ }
+
+ bms_free(colset);
+ bms_free(idattrs);
+ }
+ }
+}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index d4f8455a2bd..a504437873f 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4850,6 +4850,7 @@ _copyPublicationTable(const PublicationTable *from)
COPY_NODE_FIELD(relation);
COPY_NODE_FIELD(whereClause);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f1002afe7a0..4fc16ce04e3 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2322,6 +2322,7 @@ _equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
COMPARE_NODE_FIELD(whereClause);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a03b33b53bd..25c9b29afdd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9751,13 +9751,14 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr OptWhereClause
+ TABLE relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9772,11 +9773,12 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId OptWhereClause
+ | ColId opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- if ($2)
+ /* FIXME this is a bit cumbersome */
+ if ($2 || $3)
{
/*
* The OptWhereClause must be stored here but it is
@@ -9786,7 +9788,8 @@ PublicationObjSpec:
*/
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVar(NULL, $1, @1);
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
else
{
@@ -9794,23 +9797,25 @@ PublicationObjSpec:
}
$$->location = @1;
}
- | ColId indirection OptWhereClause
+ | ColId indirection opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr OptWhereClause
+ | extended_relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
| CURRENT_SCHEMA
{
@@ -9836,6 +9841,10 @@ pub_obj_list: PublicationObjSpec
*
* ALTER PUBLICATION name SET pub_obj [, ...]
*
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ *
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
* pub_obj is one of:
*
* TABLE table_name [, ...]
@@ -9869,6 +9878,32 @@ AlterPublicationStmt:
n->action = AP_SetObjects;
$$ = (Node *)n;
}
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS '(' columnList ')'
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = $10;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS ALL
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = NIL;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
| ALTER PUBLICATION name DROP pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
@@ -17488,6 +17523,13 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
errmsg("WHERE clause not allowed for schema"),
parser_errposition(pubobj->location));
+ /* Column filter is not allowed on a schema object */
+ if (pubobj->pubtable && pubobj->pubtable->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schema"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index c9b0eeefd7e..ccca326ad6a 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,10 +29,11 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary);
+ bool binary, Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -398,7 +399,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +411,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -443,7 +444,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -464,11 +465,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -537,7 +538,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, NULL);
}
/*
@@ -652,7 +653,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -674,7 +676,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -751,7 +753,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum *values;
@@ -763,7 +765,13 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ /* Don't count attributes that are not to be sent. */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
continue;
nliveatts++;
}
@@ -783,6 +791,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped || att->attgenerated)
continue;
+ /* Ignore attributes that are not to be sent.
+ *
+ * XXX Do we need the (columns != NULL) check? I don't think so, because
+ * such bitmap has no members.
+ */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +920,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -917,8 +933,15 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* send number of live attributes */
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ /* XXX we should have a function/macro for this check */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -937,6 +960,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ /* XXX we should have a function/macro for this check */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 1659964571c..42708dcf82e 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -112,6 +112,7 @@
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "utils/acl.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -700,20 +701,22 @@ fetch_remote_table_info(char *nspname, char *relname,
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
int natt;
ListCell *lc;
bool first;
+ bool am_partition;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -743,14 +746,225 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetBool(slot_getattr(slot, 4, &isnull));
+ Assert(!isnull);
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+
+ /*
+ * Get relation's column filter expressions.
+ *
+ * For initial synchronization, column filter can be ignored in following
+ * cases:
+ *
+ * 1) one of the subscribed publications for the table hasn't specified
+ * any column filter
+ *
+ * 2) one of the subscribed publications has puballtables set to true
+ *
+ * 3) one of the subscribed publications is declared as ALL TABLES IN
+ * SCHEMA that includes this relation
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ Oid tmpRow[] = {INT4OID};
+ StringInfoData publications;
+ bool first = true;
+ bool all_columns = false;
+
+ initStringInfo(&publications);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&publications, ", ");
+ appendStringInfoString(&publications, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ /*
+ * First, check if any of the publications FOR ALL TABLES? If yes, we
+ * should not use any column filter. It's enough to find a single such
+ * publication.
+ *
+ * XXX Maybe we could combine all three steps into a single query, but
+ * this seems cleaner / easier to understand.
+ *
+ * XXX Does this need any handling of partitions / publish_via_part_root?
+ */
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT 1\n"
+ " FROM pg_publication p\n"
+ " WHERE p.pubname IN ( %s ) AND p.puballtables LIMIT 1\n",
+ publications.data);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(tmpRow), tmpRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch publication info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+
+ if (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ all_columns = true;
+
+ ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(pubres);
+
+ /*
+ * If there's no FOR ALL TABLES publication, look for a FOR ALL TABLES
+ * IN SCHEMA publication, with schema of the remote relation. The logic
+ * is the same - such publications have no column filters.
+ *
+ * XXX Does this need any handling of partitions / publish_via_part_root?
+ */
+ if (!all_columns)
+ {
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT 1\n"
+ " FROM pg_publication p\n"
+ " JOIN pg_publication_namespace pn ON (pn.pnpubid = p.oid)\n"
+ " JOIN pg_class c ON (pn.pnnspid = c.relnamespace)\n"
+ " WHERE c.oid = %u AND p.pubname IN ( %s ) LIMIT 1",
+ lrel->remoteid,
+ publications.data);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(tmpRow), tmpRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch publication info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+
+ if (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ all_columns = true;
+
+ ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(pubres);
+ }
+
+ /*
+ * If we haven't found any FOR ALL TABLES [IN SCHEMA] publications for
+ * the table, we have to look for the column filters set for relations.
+ * First, we check if there's a publication with no column filter for
+ * the relation - which means all columns need to be replicated.
+ */
+ if (!all_columns)
+ {
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT 1\n"
+ " FROM pg_catalog.pg_publication p JOIN\n"
+ " pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n"
+ " WHERE p.pubname IN (%s) AND pr.prattrs IS NULL AND ",
+ publications.data);
+
+ /*
+ * For non-partitions, we simply join directly to the catalog. For
+ * partitions, we need to check all the ancestors, because maybe the
+ * root was not added to a publication but one of the intermediate
+ * partitions was.
+ */
+ if (!am_partition)
+ appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "prrelid IN (SELECT relid\n"
+ " FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(tmpRow), tmpRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch attribute info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+
+ if (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ all_columns = true;
+
+ ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(pubres);
+ }
+
+ /*
+ * All that
+ */
+ if (!all_columns)
+ {
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT unnest(pr.prattrs)\n"
+ " FROM pg_catalog.pg_publication p JOIN\n"
+ " pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n"
+ " WHERE p.pubname IN (%s) AND pr.prattrs IS NOT NULL AND ",
+ publications.data);
+
+ /*
+ * For non-partitions, we simply join directly to the catalog. For
+ * partitions, we need to check all the ancestors, because maybe the
+ * root was not added to a publication but one of the intermediate
+ * partitions was.
+ */
+ if (!am_partition)
+ appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "prrelid IN (SELECT relid\n"
+ " FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch attribute info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ included_cols = bms_add_member(included_cols, attnum);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(pubres);
+ }
+
+ pfree(publications.data);
+ }
+
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -778,16 +992,34 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * Store the columns as a list of names. Ignore those that are not
+ * present in the column list, if there is one.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ {
+ ExecClearTuple(slot);
+ continue;
+ }
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -821,6 +1053,9 @@ fetch_remote_table_info(char *nspname, char *relname,
*
* 3) one of the subscribed publications is declared as ALL TABLES IN
* SCHEMA that includes this relation
+ *
+ * XXX Does this actually handle puballtables and schema publications
+ * correctly?
*/
if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
{
@@ -930,8 +1165,24 @@ copy_table(Relation rel)
/* Regular table with no row filter */
if (lrel.relkind == RELKIND_RELATION && qual == NIL)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+
+ /*
+ * XXX Do we need to list the columns in all cases? Maybe we're replicating
+ * all columns?
+ */
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
+
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
+
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a0477f0..16239cc686d 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -29,6 +29,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -85,7 +86,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -164,6 +166,13 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
AttrMap *attrmap;
+
+ /*
+ * Columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this bitmap are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -603,11 +612,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -620,7 +629,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -643,13 +653,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip this attribute if it's not present in the column list */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -1224,7 +1238,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -1278,11 +1292,13 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel,
- new_slot, data->binary);
+ new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel,
- old_slot, new_slot, data->binary);
+ old_slot, new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel,
@@ -1731,6 +1747,7 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
memset(entry->exprstate, 0, sizeof(entry->exprstate));
entry->cache_expr_cxt = NULL;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->attrmap = NULL;
}
@@ -1775,6 +1792,8 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
entry->pubactions.pubinsert = false;
entry->pubactions.pubupdate = false;
entry->pubactions.pubdelete = false;
@@ -1807,13 +1826,16 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/*
* Build publication cache. We can't use one provided by relcache as
- * relcache considers all publications given relation is in, but here
- * we only need to consider ones that the subscriber requested.
+ * relcache considers all publications that the given relation is in,
+ * but here we only need to consider ones that the subscriber
+ * requested.
*/
foreach(lc, data->publications)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
+ bool all_columns = false;
if (pub->alltables)
{
@@ -1824,8 +1846,6 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
if (!publish)
{
- bool ancestor_published = false;
-
/*
* For a partition, check if any of the ancestors are
* published. If so, note down the topmost ancestor that is
@@ -1855,6 +1875,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
}
/*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
* Don't publish changes for partitioned tables, because
* publishing those of its partitions suffices, unless partition
* changes won't be published due to pubviaroot being set.
@@ -1867,6 +1890,80 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
+ /*
+ * This might be FOR ALL TABLES or FOR ALL TABLES IN SCHEMA
+ * publication, in which case there are no column lists, and
+ * we treat that as all_columns=true.
+ */
+ if (pub->alltables ||
+ list_member_oid(schemaPubids, pub->oid))
+ {
+ all_columns = true;
+ bms_free(entry->columns);
+ entry->columns = NULL;
+ }
+
+ /*
+ * Obtain columns published by this publication, and add them
+ * to the list for this rel. Note that if at least one
+ * publication has an empty column list, that means to publish
+ * everything; so if we saw a publication that includes all
+ * columns, skip this.
+ *
+ * FIXME This fails to consider column filters defined in
+ * FOR ALL TABLES and FOR ALL TABLES IN SCHEMA publications.
+ * We need to check those too.
+ */
+ if (!all_columns)
+ {
+ HeapTuple pub_rel_tuple;
+
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ Datum pub_rel_cols;
+ bool isnull;
+
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP,
+ pub_rel_tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull)
+ {
+ /*
+ * If we see a publication with no column filter, it
+ * means we need to publish all columns, so reset the
+ * list and ignore further ones.
+ */
+ all_columns = true;
+ bms_free(entry->columns);
+ entry->columns = NULL;
+ }
+ else
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(pub_rel_cols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
+ entry->columns = bms_add_member(entry->columns,
+ elems[i]);
+ MemoryContextSwitchTo(oldctx);
+ }
+
+ ReleaseSysCache(pub_rel_tuple);
+ }
+ }
+
rel_publications = lappend(rel_publications, pub);
}
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e69dcf8a484..f208c7a6c59 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4075,6 +4075,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_prpubid;
int i_prrelid;
int i_prrelqual;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4088,12 +4089,20 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
if (fout->remoteVersion >= 150000)
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual "
- "FROM pg_catalog.pg_publication_rel");
+ "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual, "
+ "(CASE\n"
+ " WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT array_agg(attname)\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) prattrs "
+ "FROM pg_catalog.pg_publication_rel pr");
else
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "NULL AS prrelqual "
+ "NULL AS prrelqual, NULL AS prattrs "
"FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
@@ -4104,6 +4113,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
i_prrelqual = PQfnumber(res, "prrelqual");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4149,6 +4159,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
else
pubrinfo[j].pubrelqual = pg_strdup(PQgetvalue(res, i, i_prrelqual));
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[j].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4223,10 +4255,13 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+
if (pubrinfo->pubrelqual)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 997a3b60719..680b07dcd52 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -632,6 +632,7 @@ typedef struct _PublicationRelInfo
PublicationInfo *publication;
TableInfo *pubtable;
char *pubrelqual;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index c3bcef8c0ec..1fedcdb442f 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2418,6 +2418,28 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'ALTER PUBLICATION pub1 ADD TABLE test_sixth_table (col3, col2)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_sixth_table (col3, col2);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_sixth_table (col2, col3);\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'ALTER PUBLICATION pub1 ADD TABLE test_seventh_table (col3, col2) WHERE (col1 = 1)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_seventh_table (col3, col2) WHERE (col1 = 1);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_seventh_table (col2, col3) WHERE ((col1 = 1));\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'ALTER PUBLICATION pub3 ADD ALL TABLES IN SCHEMA dump_test' => {
create_order => 51,
create_sql =>
@@ -2743,6 +2765,44 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'CREATE TABLE test_sixth_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_sixth_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_sixth_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'CREATE TABLE test_seventh_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_seventh_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_seventh_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'CREATE TABLE test_table_identity' => {
create_order => 3,
create_sql => 'CREATE TABLE dump_test.test_table_identity (
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index e3382933d98..fb18cb82d9f 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2880,6 +2880,7 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_namespace pn ON p.oid = pn.pnpubid\n"
" JOIN pg_catalog.pg_class pc ON pc.relnamespace = pn.pnnspid\n"
@@ -2887,6 +2888,12 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , pg_get_expr(pr.prqual, c.oid)\n"
+ " , (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT string_agg(attname, ', ')\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) "
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
" JOIN pg_catalog.pg_class c ON c.oid = pr.prrelid\n"
@@ -2894,6 +2901,7 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2904,12 +2912,14 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
"WHERE pr.prrelid = '%s'\n"
"UNION ALL\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2931,6 +2941,11 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, " \"%s\"",
PQgetvalue(result, i, 0));
+ /* column filter (if any) */
+ if (!PQgetisnull(result, i, 2))
+ appendPQExpBuffer(&buf, " (%s)",
+ PQgetvalue(result, i, 2));
+
/* row filter (if any) */
if (!PQgetisnull(result, i, 1))
appendPQExpBuffer(&buf, " WHERE %s",
@@ -5867,7 +5882,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5884,15 +5899,19 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (as_schema)
+ printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
+ else
{
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
+
+ if (!PQgetisnull(res, i, 3))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 3));
+
if (!PQgetisnull(res, i, 2))
appendPQExpBuffer(buf, " WHERE %s", PQgetvalue(res, i, 2));
}
- else
- printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
}
@@ -6021,11 +6040,22 @@ describePublications(const char *pattern)
printfPQExpBuffer(&buf,
"SELECT n.nspname, c.relname");
if (pset.sversion >= 150000)
+ {
appendPQExpBufferStr(&buf,
", pg_get_expr(pr.prqual, c.oid)");
+ appendPQExpBufferStr(&buf,
+ ", (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string("
+ " ARRAY(SELECT attname\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END)");
+ }
else
appendPQExpBufferStr(&buf,
- ", NULL");
+ ", NULL, NULL");
appendPQExpBuffer(&buf,
"\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index ba72e62e614..0190b91c091 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -100,6 +100,7 @@ typedef struct PublicationRelInfo
{
Relation relation;
Node *whereClause;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -123,8 +124,11 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
+extern void GetActionsInPublication(Oid pubid, PublicationActions *actions);
extern List *GetPublicationSchemas(Oid pubid);
extern List *GetSchemaPublications(Oid schemaid);
extern List *GetSchemaPublicationRelations(Oid schemaid,
@@ -142,6 +146,8 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *pri
bool if_not_exists);
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern void publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index 0dd0f425db9..4feb581899e 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -34,6 +34,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
#ifdef CATALOG_VARLEN /* variable-length fields start here */
pg_node_tree prqual; /* qualifications */
+ int2vector prattrs; /* columns to replicate */
#endif
} FormData_pg_publication_rel;
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1617702d9d6..1375a173e3b 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3652,6 +3652,7 @@ typedef struct PublicationTable
NodeTag type;
RangeVar *relation; /* relation to be published */
Node *whereClause; /* qualifications */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
@@ -3688,7 +3689,8 @@ typedef enum AlterPublicationAction
{
AP_AddObjects, /* add objects to publication */
AP_DropObjects, /* remove objects from publication */
- AP_SetObjects /* set list of objects */
+ AP_SetObjects, /* set list of objects */
+ AP_SetColumns /* change list of columns for a table */
} AlterPublicationAction;
typedef struct AlterPublicationStmt
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4d2c881644a..a771ab8ff33 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -209,12 +209,12 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary);
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -231,7 +231,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 227ce759486..09e86e3049e 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -657,6 +657,209 @@ DROP TABLE rf_tbl_abcd_pk;
DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- test for column filters
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+-- error: replica identity "a" not included in the column filter
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ERROR: cannot reference generated column "d" in publication column list
+-- error: system attributes "ctid" not allowed in column filter
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+ERROR: cannot reference system column "ctid" in publication column list
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+ERROR: cannot drop column "c" because it is part of publication "testpub_fortable"
+HINT: Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication.
+-- ok: for insert-only publication, the filter is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key; -- nope
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: change the replica identity to "b", and then filter (a, c) fails
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ERROR: invalid column list for publishing relation "testpub_tbl5"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+ Publication testpub_table_ins
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | t | f
+Tables:
+ "public.testpub_tbl5" (a)
+
+-- with REPLICA IDENTITY FULL, column filters are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+ERROR: invalid column list for publishing relation "testpub_tbl6"
+DETAIL: Cannot specify column list on relations with REPLICA IDENTITY FULL.
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+ALTER PUBLICATION testpub_fortable ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c);
+ERROR: invalid column list for publishing relation "testpub_tbl6"
+DETAIL: Cannot specify column list on relations with REPLICA IDENTITY FULL.
+-- make sure changing the column filter is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: update the column filter
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, c)
+
+-- column filter for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: column filter covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_filter FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+-- failure: column filter does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, c);
+ERROR: invalid column list for publishing relation "testpub_tbl8"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, c);
+ERROR: invalid column list for publishing relation "testpub_tbl8"
+DETAIL: Cannot specify column list on relations with REPLICA IDENTITY FULL.
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+-- failure: replica identity full can't be used with a column filter
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ERROR: invalid column list for publishing relation "testpub_tbl8_1"
+DETAIL: Cannot specify column list on relations with REPLICA IDENTITY FULL.
+-- failure: replica identity has to be covered by the column filter
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ERROR: invalid column list for publishing relation "testpub_tbl8_1"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+DROP TABLE testpub_tbl8;
+-- column filter for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: attaching first partition works, because (a) is in column filter
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column filter
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+ERROR: invalid column list for publishing relation "testpub_tbl8_1"
+DETAIL: All columns in REPLICA IDENTITY must be present in the column list.
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column filter on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+ERROR: invalid column list for publishing relation "testpub_tbl8_0"
+DETAIL: Cannot specify column list on relations with REPLICA IDENTITY FULL.
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_filter;
+-- ======================================================
+-- Test combination of column and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+ Publication testpub_both_filters
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | t | t | t | f
+Tables:
+ "public.testpub_tbl_both_filters" (a, c) WHERE (c <> 1)
+
+\d+ testpub_tbl_both_filters
+ Table "public.testpub_tbl_both_filters"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+ c | integer | | not null | | plain | |
+Indexes:
+ "testpub_tbl_both_filters_pkey" PRIMARY KEY, btree (a, c) REPLICA IDENTITY
+Publications:
+ "testpub_both_filters" (a, c) WHERE (c <> 1)
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -1102,6 +1305,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schema
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index cd7e0182716..6cd4d1ae1b3 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -378,6 +378,142 @@ DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- test for column filters
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+-- error: replica identity "a" not included in the column filter
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+-- error: system attributes "ctid" not allowed in column filter
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+-- ok: for insert-only publication, the filter is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key; -- nope
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+
+-- error: change the replica identity to "b", and then filter (a, c) fails
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+
+-- with REPLICA IDENTITY FULL, column filters are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+ALTER PUBLICATION testpub_fortable ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c);
+
+-- make sure changing the column filter is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: update the column filter
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+
+-- column filter for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: column filter covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_filter FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+
+-- failure: column filter does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, c);
+
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, c);
+
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+-- failure: replica identity full can't be used with a column filter
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+-- failure: replica identity has to be covered by the column filter
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+DROP TABLE testpub_tbl8;
+
+-- column filter for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: attaching first partition works, because (a) is in column filter
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column filter
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column filter on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_filter;
+-- ======================================================
+
+-- Test combination of column and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+\d+ testpub_tbl_both_filters
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -619,6 +755,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/029_column_list.pl b/src/test/subscription/t/029_column_list.pl
new file mode 100644
index 00000000000..ec2c8a789ad
--- /dev/null
+++ b/src/test/subscription/t/029_column_list.pl
@@ -0,0 +1,836 @@
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test partial-column publication of tables
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 26;
+
+# setup
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+sub wait_for_subscription_sync
+{
+ my ($node) = @_;
+
+ # Also wait for initial table sync to finish
+ my $synced_query = "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+
+ $node->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+}
+
+# setup tables on both nodes
+
+# tab1: simple 1:1 replication
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+# tab2: replication from regular to table with fewer columns
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)
+));
+
+# tab3: simple 1:1 replication with weird column names
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "B" varchar, "c'" int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "c'" int)
+));
+
+# test_part: partitioned tables, with partitioning (including multi-level
+# partitioning, and fewer columns on the subscriber)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5);
+));
+
+# tab4: table with user-defined enum types
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, d text);
+));
+
+
+# TEST: create publication and subscription for some of the tables with
+# column lists
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub1
+ FOR TABLE tab1 (a, "B"), tab3 ("a'", "c'"), test_part (a, b), tab4 (a, b, d)
+ WITH (publish_via_partition_root = 'true');
+));
+
+# check that we got the right prattrs values for the publication in the
+# pg_publication_rel catalog (order by relname, to get stable ordering)
+my $result = $node_publisher->safe_psql('postgres', qq(
+ SELECT relname, prattrs
+ FROM pg_publication_rel pb JOIN pg_class pc ON(pb.prrelid = pc.oid)
+ ORDER BY relname
+));
+
+is($result, qq(tab1|1 2
+tab3|1 3
+tab4|1 2 4
+test_part|1 2), 'publication relation updated');
+
+# create subscription for the publication, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+# TEST: insert data into the tables, and see we got replication of just
+# the filtered columns
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (1, 2, 3);
+ INSERT INTO tab1 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (1, 2, 3);
+ INSERT INTO tab3 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (1, 'red', 3, 'oh my');
+ INSERT INTO tab4 VALUES (2, 'blue', 4, 'hello');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (1, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (2, 'bcd', '2021-07-03 11:12:13');
+ INSERT INTO test_part VALUES (4, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (5, 'bcd', '2021-07-03 11:12:13');
+));
+
+# wait for catchup before checking the subscriber
+$node_publisher->wait_for_catchup('sub1');
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|
+4|5|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab3");
+is($result, qq(1|3
+4|6), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4");
+is($result, qq(1|red|oh my
+2|blue|hello), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part");
+is($result, qq(1|abc
+2|bcd
+4|abc
+5|bcd), 'insert on column test_part.c columns is not replicated');
+
+# TEST: do some updated on some of the tables, both on columns included
+# in the column list and other
+
+# tab1: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET "B" = 2 * "B" where a = 1));
+
+# tab1: update of non-replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET c = 2*c where a = 4));
+
+# tab3: update of non-replicated
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "B" = "B" || ' updated' where "a'" = 4));
+
+# tab3: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "c'" = 2 * "c'" where "a'" = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'blue', c = c * 2, d = d || ' updated' where a = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'red', c = c * 2, d = d || ' updated' where a = 2));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab1 ORDER BY a));
+is($result,
+qq(1|4|
+4|5|), 'only update on column tab1.b is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result,
+qq(1|6
+4|6), 'only update on column tab3.c is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab4 ORDER BY a));
+
+is($result, qq(1|blue|oh my updated
+2|red|hello updated), 'update on column tab4.c is not replicated');
+
+
+# TEST: add table with a column list, insert data, replicate
+
+# insert some data before adding it to the publication
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (1, 'abc', 3);
+));
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2 (a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION");
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (2, 'def', 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|def), 'insert on column tab2.c is not replicated');
+
+$node_publisher->safe_psql('postgres', qq(
+ UPDATE tab2 SET c = 5 where a = 1;
+ UPDATE tab2 SET b = 'xyz' where a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|xyz), 'update on column tab2.c is not replicated');
+
+
+# TEST: add a table to two publications with different column lists, and
+# create a single subscription replicating both publications
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub3
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->wait_for_catchup('sub1');
+
+# TEST: insert data and make sure all the columns (union of the columns lists)
+# were replicated
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (1, 11, 111, 1111)");
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (2, 22, 222, 2222)");
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111
+2|22|2222),
+ 'overlapping publications with overlapping column lists');
+
+# and finally, set the column filter to ALL for one of the publications,
+# which means replicating all columns (removing the column filter), but
+# first add the missing column to the table on subscriber
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub3 ALTER TABLE tab5 SET COLUMNS ALL;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ ALTER TABLE tab5 ADD COLUMN c INT;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (3, 33, 333, 3333)");
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111|
+2|22|2222|
+3|33|3333|333),
+ 'overlapping publications with overlapping column lists');
+
+# TEST: create a table with a column filter, then change the replica
+# identity by replacing a primary key (but use a different column in
+# the column filter)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub4 FOR TABLE tab6 (a, b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (1, 22, 333, 4444);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 1;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ "1|44||", 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is still covered by the column list, though)
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+# we need to do the same thing on the subscriber
+# XXX What would happen if this happens before the publisher ALTER? Or
+# interleaved, somehow? But that seems unrelated to column lists.
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (2, 55, 666, 8888);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|44||
+2|110||),
+ 'replication with the modified primary key');
+
+
+# TEST: create a table with a column filter, then change the replica
+# identity by replacing a primary key with a key on multiple columns
+# (all of them covered by the column filter)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub5 FOR TABLE tab7 (a, b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub5
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (1, 22, 333, 4444);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 1;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ "1|44||", 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is not covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ ALTER TABLE tab7 ADD PRIMARY KEY (a, b);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (2, 55, 666, 8888);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|44||
+2|110||),
+ 'replication with the modified primary key');
+
+# now switch the primary key again to another columns not covered by the
+# column filter, but also generate writes between the drop and creation
+# of the new constraint
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ INSERT INTO tab7 VALUES (3, 33, 999, 7777);
+ -- update/delete is not allowed for tables without RI
+ ALTER TABLE tab7 ADD PRIMARY KEY (b, a);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 3;
+ DELETE FROM tab7 WHERE a = 1;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(2|110||
+3|66||),
+ 'replication with the modified primary key');
+
+
+# TEST: partitioned tables (with publish_via_partition_root = false)
+# and replica identity. The (leaf) partitions may have different RI, so
+# we need to check the partition RI (with respect to the column filter)
+# while attaching the partition.
+
+# First, let's create a partitioned table with two partitions, each with
+# a different RI, but a column filter not covering all those RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (2);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+));
+
+# do the same thing on the subscriber (with the opposite column order)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (b int, a int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (2);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+));
+
+# create a publication replicating just the column "a", which is not enough
+# for the second partition
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub6 FOR TABLE test_part_a (b, a) WITH (publish_via_partition_root = true);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_1 (a);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_2 (b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub6
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_a VALUES (1, 3);
+ INSERT INTO test_part_a VALUES (2, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT a, b FROM test_part_a ORDER BY a, b"),
+ qq(1|3
+2|4),
+ 'partitions with different replica identities not replicated correctly');
+
+# This time start with a column filter covering RI for all partitions, but
+# then update the column filter to not cover column "b" (needed by the
+# second partition)
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,3);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (2,4);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,3);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (2,4);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub7 FOR TABLE test_part_b (a, b) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub7
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_b VALUES (1, 1);
+ INSERT INTO test_part_b VALUES (2, 2);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_b ORDER BY a, b"),
+ qq(1|1
+2|2),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: This time start with a column filter covering RI for all partitions,
+# but then update RI for one of the partitions to not be covered by the
+# column filter anymore.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+));
+
+# create a publication replicating data through partition root, with a column
+# filter on the root, and then add the partitions one by one with separate
+# column filters (but those are not applied)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a,c);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SUBSCRIPTION sub1;
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub8;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||5
+2|4|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# create a publication not replicating data through partition root, without
+# a column filter on the root, and then add the partitions one by one with
+# separate column filters
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub8;
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ TRUNCATE test_part_c;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ TRUNCATE test_part_c;
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||
+2|4|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: Start with a single partition, with RI compatible with the column
+# filter, and then attach a partition with incompatible RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+));
+
+# do the same thing on the subscriber (in fact, create both partitions right
+# away, no need to delay that)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ CREATE TABLE test_part_d_2 PARTITION OF test_part_d FOR VALUES IN (2,4);
+ ALTER TABLE test_part_d_2 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_2 REPLICA IDENTITY USING INDEX test_part_d_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub9 FOR TABLE test_part_d (a) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub9
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_d VALUES (1, 1);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_d ORDER BY a, b"),
+ qq(1|),
+ 'partitions with different replica identities not replicated correctly');
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column filters. So with column filters (a,b) and (a,c) we
+# should replicate (a,b,c).
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_1 VALUES (1, 2, 3);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_1"),
+ qq(1|2|3),
+ 'a mix of publications should use a union of column filter');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column filters. If any of the publications is FOR ALL
+# TABLES, we should replicate all columns.
+
+# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ test_part, test_part_a, test_part_b, test_part_c, test_part_d;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_3, pub_mix_4;
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_2 VALUES (1, 2, 3);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_2"),
+ qq(1|2|3),
+ 'a mix of publications should use a union of column filter');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column filters. If any of the publications is FOR ALL
+# TABLES IN SCHEMA, we should replicate all columns.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_5, pub_mix_6;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_3 VALUES (1, 2, 3);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_3"),
+ qq(1|2|3),
+ 'a mix of publications should use a union of column filter');
+
+
+# TEST: Check handling of publish_via_partition_root - if a partition is
+# published through partition root, we should only apply the column filter
+# defined for the whole table (not the partitions) - both during the initial
+# sync and when replicating changes. This is what we do for row filters.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+
+ CREATE PUBLICATION pub_root_true FOR TABLE test_root (a) WITH (publish_via_partition_root = true);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_root_true;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_root VALUES (1, 2, 3);
+ INSERT INTO test_root VALUES (10, 20, 30);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_root ORDER BY a, b, c"),
+ qq(1||
+10||),
+ 'publication via partition root applies column filter');
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
+
+done_testing();
--
2.34.1
I applied this patch in my branch with CI hacks to show code coverage on
cirrus.
https://api.cirrus-ci.com/v1/artifact/task/6186186539532288/coverage/coverage/00-index.html
Eyeballing it looks good. But GetActionsInPublication() isn't being hit at
all?
I think the queries in pg_dump should be written with the common portions of
the query outside the conditional.
--
Justin
On Wed, Mar 2, 2022 at 5:43 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
Attached is an updated patch, addressing most of the issues reported so
far. There are various minor tweaks, but the main changes are:
...
3) checks of column filter vs. publish_via_partition_root and replica
identity, following the same logic as the row-filter patch (hopefully,
it touches the same places, using the same logic, ...)That means - with "publish_via_partition_root=false" it's not allowed to
specify column filters on partitioned tables, only for leaf partitions.And we check column filter vs. replica identity when adding tables to
publications, or whenever we change the replica identity.
This handling is different from row filter work and I see problems
with it. The column list validation w.r.t primary key (default replica
identity) is missing. The handling of column list vs. partitions has
multiple problems: (a) In attach partition, the patch is just checking
ancestors for RI validation but what if the table being attached has
further subpartitions; (b) I think the current locking also seems to
have problems because it is quite possible that while it validates the
ancestors here, concurrently someone changes the column list. I think
it won't be enough to just change the locking mode because with the
current patch strategy during attach, we will be first taking locks
for child tables of current partition and then parent tables which can
pose deadlock hazards.
The columns list validation also needs to be done when we change
publication action.
There could be more similar problems which I might have missed. For
some of these (except for concurrency issues), my colleague Shi-San
has done testing and the results are below [1]Test-1: The patch doesn't check when the primary key changes.. I feel we should do RI
vs. column list handling similar to row filter work (at one place) to
avoid all such hazards and possibly similar handling at various
places, there is a good chance that we will miss some places or make
mistakes that are not easy to catch. Do let me know if you think it
makes sense for me or one of the people who work on row filter patch
to try this (make the handling of RI checks similar to row filter
work) and then we can see if that turns out to be a simple way to deal
with all these problems?
Some other miscellaneous comments:
=============================
*
In get_rel_sync_entry(), the handling for partitioned tables doesn't
seem to be correct. It can publish a different set of columns based on
the order of publications specified in the subscription.
For example:
----
create table parent (a int, b int, c int) partition by range (a);
create table test_part1 (like parent);
alter table parent attach partition test_part1 for values from (1) to (10);
create publication pub for table parent(a) with (PUBLISH_VIA_PARTITION_ROOT);
create publication pub2 for table test_part1(b);
---
Now, depending on the order of publications in the list while defining
subscription, the column list will change
----
create subscription sub connection 'port=10000 dbname=postgres'
publication pub, pub2;
For the above, column list will be: (a)
create subscription sub connection 'port=10000 dbname=postgres'
publication pub2, pub;
For this one, the column list will be: (a, b)
----
To avoid this, the column list should be computed based on the final
publish_as_relid as we are doing for the row filter.
*
Fetching column filter info in tablesync.c is quite expensive. It
seems to be using four round-trips to get the complete info whereas
for row-filter we use just one round trip. I think we should try to
get both row filter and column filter info in just one round trip.
[1]: Test-1: The patch doesn't check when the primary key changes.
Test-1:
The patch doesn't check when the primary key changes.
e.g.
-- publisher --
create table tbl(a int primary key, b int);
create publication pub for table tbl(a);
alter table tbl drop CONSTRAINT tbl_pkey;
alter table tbl add primary key (b);
insert into tbl values (1,1);
-- subscriber --
create table tbl(a int, b int);
create subscription sub connection 'port=5432 dbname=postgres' publication pub;
update tbl set b=1 where a=1;
alter table tbl add primary key (b);
-- publisher --
delete from tbl;
Column "b" is part of replica identity, but it is filtered, which
caused an error on the subscriber side.
ERROR: publisher did not send replica identity column expected by the
logical replication target relation "public.tbl"
CONTEXT: processing remote data during "DELETE" for replication
target relation "public.tbl" in transaction 724 at 2022-03-04
11:46:16.330892+08
Test-2: Partitioned table RI w.r.t column list.
2.1
Using "create table ... partition of".
e.g.
-- publisher --
create table parent (a int, b int) partition by range (a);
create publication pub for table parent(a)
with(publish_via_partition_root=true);
create table child partition of parent (primary key (a,b)) default;
insert into parent values (1,1);
-- subscriber --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create subscription sub connection 'port=5432 dbname=postgres'
publication pub; update child set b=1 where a=1;
alter table parent add primary key (a,b);
-- publisher --
delete from parent;
Column "b" is part of replica identity in the child table, but it is
filtered, which caused an error on the subscriber side.
ERROR: publisher did not send replica identity column expected by the
logical replication target relation "public.parent"
CONTEXT: processing remote data during "DELETE" for replication
target relation "public.parent" in transaction 723 at 2022-03-04
15:15:39.776949+08
2.2
It is likely that a table to be attached also has a partition.
e.g.
-- publisher --
create table t1 (a int, b int) partition by range (a);
create publication pub for table t1(b) with(publish_via_partition_root=true);
create table t2 (a int, b int) partition by range (a);
create table t3 (a int primary key, b int);
alter table t2 attach partition t3 default;
alter table t1 attach partition t2 default;
insert into t1 values (1,1);
-- subscriber --
create table t1 (a int, b int) partition by range (a);
create table t2 (a int, b int) partition by range (a);
create table t3 (a int, b int);
alter table t2 attach partition t3 default;
alter table t1 attach partition t2 default;
create subscription sub connection 'port=5432 dbname=postgres' publication pub;
update t1 set a=1 where b=1;
alter table t1 add primary key (a);
-- publisher --
delete from t1;
Column "a" is part of replica identity in table t3, but t3's ancestor
t1 is published with column "a" filtered, which caused an error on the
subscriber side.
ERROR: publisher did not send replica identity column expected by the
logical replication target relation "public.t1"
CONTEXT: processing remote data during "DELETE" for replication
target relation "public.t1" in transaction 726 at 2022-03-04
14:40:29.297392+08
3.
Using "alter publication pub set(publish='...'); "
e.g.
-- publisher --
create table tbl(a int primary key, b int); create publication pub for
table tbl(b) with(publish='insert'); insert into tbl values (1,1);
-- subscriber --
create table tbl(a int, b int);
create subscription sub connection 'port=5432 dbname=postgres' publication pub;
-- publisher --
alter publication pub set(publish='insert,update');
-- subscriber --
update tbl set a=1 where b=1;
alter table tbl add primary key (b);
-- publisher --
update tbl set a=2 where a=1;
Updates are replicated, and the column "a" is part of replica
identity, but it is filtered, which caused an error on the subscriber
side.
ERROR: publisher did not send replica identity column expected by the
logical replication target relation "public.tbl"
CONTEXT: processing remote data during "UPDATE" for replication
target relation "public.tbl" in transaction 723 at 2022-03-04
11:56:33.905843+08
--
With Regards,
Amit Kapila.
On 3/4/22 11:42, Amit Kapila wrote:
On Wed, Mar 2, 2022 at 5:43 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Attached is an updated patch, addressing most of the issues reported so
far. There are various minor tweaks, but the main changes are:...
3) checks of column filter vs. publish_via_partition_root and replica
identity, following the same logic as the row-filter patch (hopefully,
it touches the same places, using the same logic, ...)That means - with "publish_via_partition_root=false" it's not allowed to
specify column filters on partitioned tables, only for leaf partitions.And we check column filter vs. replica identity when adding tables to
publications, or whenever we change the replica identity.This handling is different from row filter work and I see problems
with it.
By different, I assume you mean I tried to enfoce the rules in ALTER
PUBLICATION and other ALTER commands, instead of when modifying the
data? OK, I reworked this to do the same thing as the row filtering patch.
The column list validation w.r.t primary key (default replica
identity) is missing. The handling of column list vs. partitions has
multiple problems: (a) In attach partition, the patch is just checking
ancestors for RI validation but what if the table being attached has
further subpartitions; (b) I think the current locking also seems to
have problems because it is quite possible that while it validates the
ancestors here, concurrently someone changes the column list. I think
it won't be enough to just change the locking mode because with the
current patch strategy during attach, we will be first taking locks
for child tables of current partition and then parent tables which can
pose deadlock hazards.The columns list validation also needs to be done when we change
publication action.
I believe those issues should be solved by adopting the same approach as
the row-filtering patch, right?
There could be more similar problems which I might have missed. For
some of these (except for concurrency issues), my colleague Shi-San
has done testing and the results are below [1]. I feel we should do RI
vs. column list handling similar to row filter work (at one place) to
avoid all such hazards and possibly similar handling at various
places, there is a good chance that we will miss some places or make
mistakes that are not easy to catch.
I agree if both patches use the same approach, that would reduce the
risk of missing the handling in one place, etc.
Do let me know if you think it makes sense for me or one of the
people who work on row filter patch to try this (make the handling of
RI checks similar to row filter work) and then we can see if that
turns out to be a simple way to deal with all these problems?
If someone who is more familiar with the design conclusions from the row
filtering patch, that would be immensely useful. Especially now, when I
reworked it to the same approach as the row filtering patch.
Some other miscellaneous comments:
=============================
*
In get_rel_sync_entry(), the handling for partitioned tables doesn't
seem to be correct. It can publish a different set of columns based on
the order of publications specified in the subscription.For example:
----
create table parent (a int, b int, c int) partition by range (a);
create table test_part1 (like parent);
alter table parent attach partition test_part1 for values from (1) to (10);create publication pub for table parent(a) with (PUBLISH_VIA_PARTITION_ROOT);
create publication pub2 for table test_part1(b);
---Now, depending on the order of publications in the list while defining
subscription, the column list will change
----
create subscription sub connection 'port=10000 dbname=postgres'
publication pub, pub2;For the above, column list will be: (a)
create subscription sub connection 'port=10000 dbname=postgres'
publication pub2, pub;For this one, the column list will be: (a, b)
----To avoid this, the column list should be computed based on the final
publish_as_relid as we are doing for the row filter.
Hmm, yeah. That seems like a genuine problem - it should not depend on
the order of publications in the subscription, I guess.
But is it an issue in the patch? Isn't that a pre-existing issue? AFAICS
the problem is that we initialize publish_as_relid=relid before the loop
over publications, and then just update it. So the first iteration
starts with relid, but the second iteration ends with whatever value is
set by the first iteration (e.g. the root).
So with the example you posted, we start with
publish_as_relid = relid = test_part1
but then if the first publication is pubviaroot=true, we update it to
parent. And in the second iteration, we fail to find the column filter,
because "parent" (publish_as_relid) is not part of the pub2.
If we do it in the other order, we leave the publish_as_relid value as
is (and find the filter), and then update it in the second iteration
(and find the column filter too).
Now, this can be resolved by re-calculating the publish_as_relid from
scratch in each iteration (start with relid, then maybe update it). But
that's just half the story - the issue is there even without column
filters. Consider this example:
create table t (a int, b int, c int) partition by range (a);
create table t_1 partition of t for values from (1) to (10)
partition by range (a);
create table t_2 partition of t_1 for values from (1) to (10);
create publication pub1 for table t(a)
with (PUBLISH_VIA_PARTITION_ROOT);
create publication pub2 for table t_1(a)
with (PUBLISH_VIA_PARTITION_ROOT);
Now, is you change subscribe to "pub1, pub2" and "pub2, pub1", we'll end
up with different publish_as_relid values (t or t_1). Which seems like
the same ambiguity issue.
*
Fetching column filter info in tablesync.c is quite expensive. It
seems to be using four round-trips to get the complete info whereas
for row-filter we use just one round trip. I think we should try to
get both row filter and column filter info in just one round trip.
Maybe, but I really don't think this is an issue. The sync happens only
very rarely, and the rest of the sync (starting workers, copying data)
is likely way more expensive than this.
[1] -
Test-1:
The patch doesn't check when the primary key changes.e.g.
-- publisher --
create table tbl(a int primary key, b int);
create publication pub for table tbl(a);
alter table tbl drop CONSTRAINT tbl_pkey;
alter table tbl add primary key (b);
insert into tbl values (1,1);-- subscriber --
create table tbl(a int, b int);
create subscription sub connection 'port=5432 dbname=postgres' publication pub;
update tbl set b=1 where a=1;
alter table tbl add primary key (b);-- publisher --
delete from tbl;Column "b" is part of replica identity, but it is filtered, which
caused an error on the subscriber side.ERROR: publisher did not send replica identity column expected by the
logical replication target relation "public.tbl"
CONTEXT: processing remote data during "DELETE" for replication
target relation "public.tbl" in transaction 724 at 2022-03-04
11:46:16.330892+08Test-2: Partitioned table RI w.r.t column list.
2.1
Using "create table ... partition of".e.g.
-- publisher --
create table parent (a int, b int) partition by range (a);
create publication pub for table parent(a)
with(publish_via_partition_root=true);
create table child partition of parent (primary key (a,b)) default;
insert into parent values (1,1);-- subscriber --
create table parent (a int, b int) partition by range (a);
create table child partition of parent default;
create subscription sub connection 'port=5432 dbname=postgres'
publication pub; update child set b=1 where a=1;
alter table parent add primary key (a,b);-- publisher --
delete from parent;Column "b" is part of replica identity in the child table, but it is
filtered, which caused an error on the subscriber side.ERROR: publisher did not send replica identity column expected by the
logical replication target relation "public.parent"
CONTEXT: processing remote data during "DELETE" for replication
target relation "public.parent" in transaction 723 at 2022-03-04
15:15:39.776949+082.2
It is likely that a table to be attached also has a partition.e.g.
-- publisher --
create table t1 (a int, b int) partition by range (a);
create publication pub for table t1(b) with(publish_via_partition_root=true);
create table t2 (a int, b int) partition by range (a);
create table t3 (a int primary key, b int);
alter table t2 attach partition t3 default;
alter table t1 attach partition t2 default;
insert into t1 values (1,1);-- subscriber --
create table t1 (a int, b int) partition by range (a);
create table t2 (a int, b int) partition by range (a);
create table t3 (a int, b int);
alter table t2 attach partition t3 default;
alter table t1 attach partition t2 default;
create subscription sub connection 'port=5432 dbname=postgres' publication pub;
update t1 set a=1 where b=1;
alter table t1 add primary key (a);-- publisher --
delete from t1;Column "a" is part of replica identity in table t3, but t3's ancestor
t1 is published with column "a" filtered, which caused an error on the
subscriber side.ERROR: publisher did not send replica identity column expected by the
logical replication target relation "public.t1"
CONTEXT: processing remote data during "DELETE" for replication
target relation "public.t1" in transaction 726 at 2022-03-04
14:40:29.297392+083.
Using "alter publication pub set(publish='...'); "e.g.
-- publisher --
create table tbl(a int primary key, b int); create publication pub for
table tbl(b) with(publish='insert'); insert into tbl values (1,1);-- subscriber --
create table tbl(a int, b int);
create subscription sub connection 'port=5432 dbname=postgres' publication pub;-- publisher --
alter publication pub set(publish='insert,update');-- subscriber --
update tbl set a=1 where b=1;
alter table tbl add primary key (b);-- publisher --
update tbl set a=2 where a=1;Updates are replicated, and the column "a" is part of replica
identity, but it is filtered, which caused an error on the subscriber
side.ERROR: publisher did not send replica identity column expected by the
logical replication target relation "public.tbl"
CONTEXT: processing remote data during "UPDATE" for replication
target relation "public.tbl" in transaction 723 at 2022-03-04
11:56:33.905843+08
AFAICS these issues should be resolved by the adoption of the row-filter
approach (i.e. it should fail the same way as for row filter).
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
0001-Verify-changing-WHERE-condition-for-a-publi-20220307.patchtext/x-patch; charset=UTF-8; name=0001-Verify-changing-WHERE-condition-for-a-publi-20220307.patchDownload
From 362723188a36b7e0147df7404fe64aa2e4f48bfb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Sat, 26 Feb 2022 17:33:09 +0100
Subject: [PATCH 1/3] Verify changing WHERE condition for a publication
Commit 52e4f0cd47 added support for row filters in logical replication,
including regression tests with multiple ALTER PUBLICATION commands,
modifying the row filter. But the tests never verified that the row
filter was actually updated in the catalog. This adds a couple \d and
\dRp commands, to verify the catalog was updated.
---
src/test/regress/expected/publication.out | 66 +++++++++++++++++++++++
src/test/regress/sql/publication.sql | 8 +++
2 files changed, 74 insertions(+)
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 4e191c120ac..6d16600aaea 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -417,15 +417,81 @@ LINE 1: ...ICATION testpub5 ADD TABLE testpub_rf_tbl1 WHERE (b < '2' CO...
DETAIL: User-defined collations are not allowed.
-- ok - NULLIF is allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (NULLIF(1,2) = a);
+\d+ testpub_rf_tbl1
+ Table "public.testpub_rf_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | | | plain | |
+ b | text | | | | extended | |
+Publications:
+ "testpub5" WHERE (NULLIF(1, 2) = a)
+
+\dRp+ testpub5
+ Publication testpub5
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | f | f
+Tables:
+ "public.testpub_rf_tbl1" WHERE (NULLIF(1, 2) = a)
+
-- ok - built-in operators are allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (a IS NULL);
+\d+ testpub_rf_tbl1
+ Table "public.testpub_rf_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | | | plain | |
+ b | text | | | | extended | |
+Publications:
+ "testpub5" WHERE (a IS NULL)
+
+\dRp+ testpub5
+ Publication testpub5
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | f | f
+Tables:
+ "public.testpub_rf_tbl1" WHERE (a IS NULL)
+
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE ((a > 5) IS FALSE);
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (a IS DISTINCT FROM 5);
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE ((a, a + 1) < (2, 3));
-- ok - built-in type coercions between two binary compatible datatypes are allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (b::varchar < '2');
+\d+ testpub_rf_tbl1
+ Table "public.testpub_rf_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | | | plain | |
+ b | text | | | | extended | |
+Publications:
+ "testpub5" WHERE (((b)::character varying)::text < '2'::text)
+
+\dRp+ testpub5
+ Publication testpub5
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | f | f
+Tables:
+ "public.testpub_rf_tbl1" WHERE (((b)::character varying)::text < '2'::text)
+
-- ok - immutable built-in functions are allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl4 WHERE (length(g) < 6);
+\d+ testpub_rf_tbl1
+ Table "public.testpub_rf_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | | | plain | |
+ b | text | | | | extended | |
+
+\dRp+ testpub5
+ Publication testpub5
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | f | f
+Tables:
+ "public.testpub_rf_tbl4" WHERE (length(g) < 6)
+
-- fail - user-defined types are not allowed
CREATE TYPE rf_bug_status AS ENUM ('new', 'open', 'closed');
CREATE TABLE rf_bug (id serial, description text, status rf_bug_status);
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 5457c56b33f..c135a601a30 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -206,15 +206,23 @@ CREATE COLLATION user_collation FROM "C";
ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl1 WHERE (b < '2' COLLATE user_collation);
-- ok - NULLIF is allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (NULLIF(1,2) = a);
+\d+ testpub_rf_tbl1
+\dRp+ testpub5
-- ok - built-in operators are allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (a IS NULL);
+\d+ testpub_rf_tbl1
+\dRp+ testpub5
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE ((a > 5) IS FALSE);
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (a IS DISTINCT FROM 5);
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE ((a, a + 1) < (2, 3));
-- ok - built-in type coercions between two binary compatible datatypes are allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl1 WHERE (b::varchar < '2');
+\d+ testpub_rf_tbl1
+\dRp+ testpub5
-- ok - immutable built-in functions are allowed
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl4 WHERE (length(g) < 6);
+\d+ testpub_rf_tbl1
+\dRp+ testpub5
-- fail - user-defined types are not allowed
CREATE TYPE rf_bug_status AS ENUM ('new', 'open', 'closed');
CREATE TABLE rf_bug (id serial, description text, status rf_bug_status);
--
2.34.1
0002-Test-publication-row-filters-in-pg_dump-tes-20220307.patchtext/x-patch; charset=UTF-8; name=0002-Test-publication-row-filters-in-pg_dump-tes-20220307.patchDownload
From fd95ce9ffdf421ebc9ecd02c541c5d2f0c43cd97 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Tue, 1 Mar 2022 15:25:56 +0100
Subject: [PATCH 2/3] Test publication row filters in pg_dump tests
Commit 52e4f0cd47 added support for row filters when replicating tables,
but the commit added no pg_dump tests for this feature. So add at least
a simple test.
---
src/bin/pg_dump/t/002_pg_dump.pl | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 3e55ff26f82..ae8c86a6e88 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2417,12 +2417,12 @@ my %tests = (
},
},
- 'ALTER PUBLICATION pub1 ADD TABLE test_second_table' => {
+ 'ALTER PUBLICATION pub1 ADD TABLE test_second_table WHERE (col1 = 1)' => {
create_order => 52,
create_sql =>
- 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_second_table;',
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_second_table WHERE (col1 = 1);',
regexp => qr/^
- \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_second_table;\E
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_second_table WHERE ((col1 = 1));\E
/xm,
like => { %full_runs, section_post_data => 1, },
unlike => { exclude_dump_test_schema => 1, },
--
2.34.1
0003-Allow-specifying-column-filters-for-logical-20220307.patchtext/x-patch; charset=UTF-8; name=0003-Allow-specifying-column-filters-for-logical-20220307.patchDownload
From 9c9d904c721d4d89ea475ea959f3dc96f16fa48e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Wed, 23 Feb 2022 21:15:18 +0100
Subject: [PATCH 3/3] Allow specifying column filters for logical replication
This allows specifying an optional column list when adding a table to
logical replication. Columns not included on this list are not sent to
the subscriber. The filter is specified as a list of columns after the
table name, enclosed in parentheses.
For UPDATE/DELETE publications, the column filter needs to cover all
REPLICA IDENTITY columns. For INSERT publications, the column list is
arbitrary and may omit some REPLICA IDENTITY columns. Furthermore, if
the table uses REPLICA IDENTITY FULL, column filter is not allowed.
The column filter can contain only simple column references. Complex
expressions, function calls etc. are not allowed. This restriction could
be relaxed in the future.
During the initial table synchronization, only columns specified in the
column filter are copied to the subscriber. If the subscription has
several publications, containing the same table with different column
filters, columns specified in any of the filters will be copied. This
means all columns are replicated if the table has no column filter at
all (which is treated as column filter with all columns), of when of the
publications is defined as FOR ALL TABLES (possibly IN SCHEMA for the
schema of the table).
For partitioned tables, publish_via_partition_root determines whether
the row filter for the root or leaf relation will be used. If the
parameter is 'false' (the default), the filter defined for the leaf
relation is used. Otherwise, the column filter for the root partition
will be used.
Psql commands \dRp+ and \d <table-name> now display any column filters.
Author: Rahila Syed <rahilasyed90@gmail.com>
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 13 +
doc/src/sgml/protocol.sgml | 4 +-
doc/src/sgml/ref/alter_publication.sgml | 23 +-
doc/src/sgml/ref/create_publication.sgml | 17 +-
src/backend/catalog/pg_publication.c | 260 ++++++
src/backend/commands/publicationcmds.c | 362 ++++++++-
src/backend/commands/tablecmds.c | 36 +-
src/backend/executor/execReplication.c | 19 +-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 60 +-
src/backend/replication/logical/proto.c | 55 +-
src/backend/replication/logical/tablesync.c | 271 ++++++-
src/backend/replication/pgoutput/pgoutput.c | 123 ++-
src/backend/utils/cache/relcache.c | 32 +
src/bin/pg_dump/pg_dump.c | 47 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 60 ++
src/bin/psql/describe.c | 40 +-
src/include/catalog/pg_publication.h | 14 +
src/include/catalog/pg_publication_rel.h | 1 +
src/include/commands/publicationcmds.h | 2 +
src/include/nodes/parsenodes.h | 4 +-
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 375 +++++++++
src/test/regress/sql/publication.sql | 290 +++++++
src/test/subscription/t/029_column_list.pl | 836 ++++++++++++++++++++
27 files changed, 2872 insertions(+), 81 deletions(-)
create mode 100644 src/test/subscription/t/029_column_list.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 83987a99045..2b61f42b71d 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6263,6 +6263,19 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to schema
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>prattrs</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+ </para>
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all columns are published.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index c51c4254a70..496593201b9 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7005,7 +7005,9 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column (except
+ generated columns and other columns that don't appear in the column
+ filter list, for tables that have one):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 32b75f6c78e..aa6827c977b 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -25,12 +25,13 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD <replace
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ALTER TABLE <replaceable class="parameter">table_name</replaceable> SET COLUMNS { ( <replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -64,6 +65,12 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
command retain their previous settings.
</para>
+ <para>
+ The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows changing
+ the set of columns that are included in the publication. If a column list
+ is specified, it must include the replica identity columns.
+ </para>
+
<para>
The remaining variants change the owner and the name of the publication.
</para>
@@ -112,6 +119,10 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
+
If the optional <literal>WHERE</literal> clause is specified, rows for
which the <replaceable class="parameter">expression</replaceable>
evaluates to false or null will not be published. Note that parentheses
@@ -172,9 +183,15 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
</para>
<para>
- Add some tables to the publication:
+ Add tables to the publication:
+<programlisting>
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS (user_id, firstname, lastname);
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 4979b9b646d..fb2d013393b 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -86,6 +86,13 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<literal>TRUNCATE</literal> commands.
</para>
+ <para>
+ When a column list is specified, only the named columns are replicated.
+ If no column list is specified, all columns of the table are replicated
+ through this publication, including any columns added later. If a column
+ list is specified, it must include the replica identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
@@ -327,6 +334,14 @@ CREATE PUBLICATION production_publication FOR TABLE users, departments, ALL TABL
<structname>sales</structname>:
<programlisting>
CREATE PUBLICATION sales_publication FOR ALL TABLES IN SCHEMA marketing, sales;
+</programlisting></para>
+
+ <para>
+ Create a publication that publishes all changes for table <structname>users</structname>,
+ but replicates only columns <structname>user_id</structname> and
+ <structname>firstname</structname>:
+<programlisting>
+CREATE PUBLICATION users_filtered FOR TABLE users (user_id, firstname);
</programlisting></para>
</refsect1>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 25998fbb39b..4dab96265f8 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,6 +45,9 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+static void publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs);
+
/*
* Check if relation can be in given publication and throws appropriate
* error if not.
@@ -328,6 +331,9 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
Oid relid = RelationGetRelid(targetrel);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ AttrNumber *attarray;
+ int natts = 0;
+ int i;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -355,6 +361,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
check_publication_add_relation(targetrel);
+ /*
+ * Translate column names to attnums and check the column list is valid.
+ * We also deconstruct the bitmap into an array of attnums, for storing
+ * in the catalog.
+ */
+ publication_translate_columns(pri->relation, pri->columns,
+ &natts, &attarray);
+
/* Form a tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -367,6 +381,17 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ /* Add column filter, if available */
+ if (pri->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+
/* Add qualifications, if available */
if (pri->whereClause != NULL)
values[Anum_pg_publication_rel_prqual - 1] = CStringGetTextDatum(nodeToString(pri->whereClause));
@@ -382,6 +407,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
/* Register dependencies as needed */
ObjectAddressSet(myself, PublicationRelRelationId, pubreloid);
+ /* Add dependency on the columns, if any are listed */
+ for (i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ pfree(attarray);
+
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -415,6 +448,155 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
return myself;
}
+/*
+ * Update the column list for a relation in a publication.
+ */
+void
+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns)
+{
+ AttrNumber *attarray;
+ HeapTuple copytup;
+ int natts;
+ bool nulls[Natts_pg_publication_rel];
+ bool replaces[Natts_pg_publication_rel];
+ Datum values[Natts_pg_publication_rel];
+ int i;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ replaces[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ deleteDependencyRecordsForClass(PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid,
+ RelationRelationId,
+ DEPENDENCY_AUTO);
+
+ if (columns == NULL)
+ {
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+ }
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ int2vector *prattrs;
+ Form_pg_publication_rel pubrel;
+
+ publication_translate_columns(targetrel, columns, &natts, &attarray);
+
+ /* XXX "pub" is leaked here */
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+
+ /* Add dependencies on the new list of columns */
+ pubrel = (Form_pg_publication_rel) GETSTRUCT(pubreltup);
+ ObjectAddressSet(myself, PublicationRelRelationId, pubrel->oid);
+
+ for (i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId,
+ RelationGetRelid(targetrel), attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ }
+
+ copytup = heap_modify_tuple(pubreltup, RelationGetDescr(pubrel),
+ values, nulls, replaces);
+
+ CatalogTupleUpdate(pubrel, &pubreltup->t_self, copytup);
+
+ heap_freetuple(copytup);
+}
+
+/*
+ * qsort comparator for attnums
+ *
+ * XXX We already have compare_int16, so maybe let's share that, somehow?
+ */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list (no system or generated attributes,
+ * no duplicates). Additional checks with replica identity are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ *
+ * XXX Should this detect duplicate columns?
+ */
+static void
+publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs)
+{
+ AttrNumber *attarray;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference generated column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /* Be tidy, so that the catalog representation is always sorted */
+ qsort(attarray, n, sizeof(AttrNumber), compare_int16);
+
+ *natts = n;
+ *attrs = attarray;
+
+ bms_free(set);
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -522,6 +704,84 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+ int i;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+ Form_pg_publication_rel pubrel;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ /* no column filter for this publications/relation */
+ if (isnull)
+ continue;
+
+ pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
+
+ pubs = lappend_oid(pubs, pubrel->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+ int i;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 16b8661a1b7..fa1462ae546 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -367,6 +367,123 @@ contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
return result;
}
+/*
+ * Check if all columns referenced in the column filter are part of the
+ * REPLICA IDENTITY index or not.
+ *
+ * Returns true if any invalid column is found.
+ */
+bool
+contain_invalid_cfcolumn(Oid pubid, Relation relation, List *ancestors,
+ bool pubviaroot)
+{
+ HeapTuple cftuple;
+ Oid relid = RelationGetRelid(relation);
+ Oid publish_as_relid = RelationGetRelid(relation);
+ bool result = false;
+ Datum cfdatum;
+ bool cfisnull;
+
+ /*
+ * For a partition, if pubviaroot is true, find the topmost ancestor that
+ * is published via this publication as we need to use its column filter
+ * for the changes.
+ *
+ * Note that even though the column filter used is for an ancestor, the
+ * REPLICA IDENTITY used will be for the actual child table.
+ */
+ if (pubviaroot && relation->rd_rel->relispartition)
+ {
+ publish_as_relid = GetTopMostAncestorInPublication(pubid, ancestors);
+
+ if (!OidIsValid(publish_as_relid))
+ publish_as_relid = relid;
+ }
+
+ cftuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(cftuple))
+ return false;
+
+ cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
+ Anum_pg_publication_rel_prattrs,
+ &cfisnull);
+
+ if (!cfisnull)
+ {
+ int x;
+ Bitmapset *idattrs;
+ Bitmapset *columns = NULL;
+
+ /* With REPLICA IDENTITY FULL, no column filter is allowed. */
+ if (relation->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ result = true;
+
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(cfdatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ for (int i = 0; i < nelems; i++)
+ columns = bms_add_member(columns, elems[i]);
+ }
+
+ /* Remember columns that are part of the REPLICA IDENTITY */
+ idattrs = RelationGetIndexAttrBitmap(relation,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * Attnums in the bitmap returned by RelationGetIndexAttrBitmap are
+ * offset (to handle system columns the usual way), while column filter
+ * does not use offset, so we can't do bms_is_subset(). Instead, we have
+ * to loop over the idattrs and check all of them are in the filter.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ AttrNumber attnum = (x + FirstLowInvalidHeapAttributeNumber);
+
+ /*
+ * If pubviaroot is true, we are validating the row filter of the
+ * parent table, but the bitmap contains the replica identity
+ * information of the child table. So, get the column number of the
+ * child table as parent and child column order could be different.
+ */
+ if (pubviaroot)
+ {
+ /* attnum is for child table */
+ char *colname = get_attname(relid, attnum, false);
+
+ /*
+ * Determine the matching attnum in parent (because the column
+ * filter is defined on the parent).
+ */
+ attnum = get_attnum(publish_as_relid, colname);
+ }
+
+ if (!bms_is_member(attnum, columns))
+ {
+ result = true;
+ break;
+ }
+ }
+
+ bms_free(idattrs);
+ bms_free(columns);
+ }
+
+ ReleaseSysCache(cftuple);
+
+ return result;
+}
+
/* check_functions_in_node callback */
static bool
contain_mutable_or_user_functions_checker(Oid func_id, void *context)
@@ -608,6 +725,45 @@ TransformPubWhereClauses(List *tables, const char *queryString,
}
}
+
+/*
+ * Transform the publication column lists expression for all the relations
+ * in the list.
+ *
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow filters on the leaf relations. So maybe rename it?
+ */
+static void
+TransformPubColumnList(List *tables, const char *queryString,
+ bool pubviaroot)
+{
+ ListCell *lc;
+
+ foreach(lc, tables)
+ {
+ PublicationRelInfo *pri = (PublicationRelInfo *) lfirst(lc);
+
+ if (pri->columns == NIL)
+ continue;
+
+ /*
+ * If the publication doesn't publish changes via the root partitioned
+ * table, the partition's column filter will be used. So disallow using
+ * the column list on partitioned table in this case.
+ */
+ if (!pubviaroot &&
+ pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot use publication column list for relation \"%s\"",
+ RelationGetRelationName(pri->relation)),
+ errdetail("column list cannot be used for a partitioned table when %s is false.",
+ "publish_via_partition_root")));
+ }
+}
+
/*
* Create new publication.
*/
@@ -724,6 +880,9 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
TransformPubWhereClauses(rels, pstate->p_sourcetext,
publish_via_partition_root);
+ TransformPubColumnList(rels, pstate->p_sourcetext,
+ publish_via_partition_root);
+
PublicationAddTables(puboid, rels, true, NULL);
CloseTableList(rels);
}
@@ -754,6 +913,52 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
return myself;
}
+/*
+ * Change the column list of a relation in a publication
+ */
+static void
+PublicationSetColumns(AlterPublicationStmt *stmt,
+ Form_pg_publication pubform, PublicationTable *table)
+{
+ Relation rel,
+ urel;
+ HeapTuple tup;
+ ObjectAddress obj,
+ secondary;
+
+ rel = table_open(PublicationRelRelationId, RowExclusiveLock);
+ urel = table_openrv(table->relation, ShareUpdateExclusiveLock);
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(RelationGetRelid(urel)),
+ ObjectIdGetDatum(pubform->oid));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errmsg("relation \"%s\" is not already in publication \"%s\"",
+ table->relation->relname,
+ NameStr(pubform->pubname)));
+
+ publication_set_table_columns(rel, tup, urel, table->columns);
+
+ ObjectAddressSet(obj, PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->oid);
+ ObjectAddressSet(secondary, RelationRelationId, RelationGetRelid(urel));
+ EventTriggerCollectSimpleCommand(obj, secondary, (Node *) stmt);
+
+ ReleaseSysCache(tup);
+
+ /*
+ * Invalidate relcache for this relation, to force rebuilding the
+ * publication description.
+ */
+ CacheInvalidateRelcache(urel);
+
+ table_close(rel, RowExclusiveLock);
+ table_close(urel, NoLock);
+
+ InvokeObjectPostAlterHook(PublicationRelationId, pubform->oid, 0);
+}
+
/*
* Change options of a publication.
*/
@@ -783,8 +988,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* If the publication doesn't publish changes via the root partitioned
- * table, the partition's row filter will be used. So disallow using WHERE
- * clause on partitioned table in this case.
+ * table, the partition's row and column filter will be used. So disallow
+ * using WHERE clause and column filters on partitioned table in this case.
*/
if (!pubform->puballtables && publish_via_partition_root_given &&
!publish_via_partition_root)
@@ -792,7 +997,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* Lock the publication so nobody else can do anything with it. This
* prevents concurrent alter to add partitioned table(s) with WHERE
- * clause(s) which we don't allow when not publishing via root.
+ * clause(s) and/or column filters which we don't allow when not
+ * publishing via root.
*/
LockDatabaseObject(PublicationRelationId, pubform->oid, 0,
AccessShareLock);
@@ -804,13 +1010,21 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
HeapTuple rftuple;
Oid relid = lfirst_oid(lc);
+ bool has_column_filter;
+ bool has_row_filter;
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubform->oid));
+ has_row_filter
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL);
+
+ has_column_filter
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prattrs, NULL);
+
if (HeapTupleIsValid(rftuple) &&
- !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL))
+ (has_row_filter || has_column_filter))
{
HeapTuple tuple;
@@ -819,7 +1033,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
Form_pg_class relform = (Form_pg_class) GETSTRUCT(tuple);
- if (relform->relkind == RELKIND_PARTITIONED_TABLE)
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_row_filter)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot set %s for publication \"%s\"",
@@ -830,6 +1045,18 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
NameStr(relform->relname),
"publish_via_partition_root")));
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_column_filter)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot set %s for publication \"%s\"",
+ "publish_via_partition_root = false",
+ stmt->pubname),
+ errdetail("The publication contains a column filter for a partitioned table \"%s\" "
+ "which is not allowed when %s is false.",
+ NameStr(relform->relname),
+ "publish_via_partition_root")));
+
ReleaseSysCache(tuple);
}
@@ -838,6 +1065,16 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
}
}
+ /*
+ * FIXME check pubactions vs. replica identity, to ensure the replica
+ * identity is included in the column filter. Only do this for update
+ * and delete publications. See check_publication_columns.
+ *
+ * XXX This is needed because publish_via_partition_root may change,
+ * in which case the row filters may be invalid (e.g. with pvpr=false
+ * there must be no filter on partitioned tables).
+ */
+
/* Everything ok, form a new tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -975,10 +1212,20 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
PublicationAddTables(pubid, rels, false, stmt);
}
else if (stmt->action == AP_DropObjects)
PublicationDropTables(pubid, rels, false);
+ else if (stmt->action == AP_SetColumns)
+ {
+ Assert(schemaidlist == NIL);
+ Assert(list_length(tables) == 1);
+
+ PublicationSetColumns(stmt, pubform,
+ linitial_node(PublicationTable, tables));
+ }
else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
@@ -991,6 +1238,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
/*
* To recreate the relation list for the publication, look for
* existing relations that do not need to be dropped.
@@ -1002,32 +1251,85 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
PublicationRelInfo *oldrel;
bool found = false;
HeapTuple rftuple;
- bool rfisnull = true;
Node *oldrelwhereclause = NULL;
+ Bitmapset *oldcolumns = NULL;
/* look up the cache for the old relmap */
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(oldrelid),
ObjectIdGetDatum(pubid));
+ /*
+ * See if the existing relation currently has a WHERE clause or a
+ * column filter. We need to compare those too.
+ */
if (HeapTupleIsValid(rftuple))
{
+ bool isnull = true;
Datum whereClauseDatum;
+ Datum columnListDatum;
+ /* Load the WHERE clause for this table. */
whereClauseDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
Anum_pg_publication_rel_prqual,
- &rfisnull);
- if (!rfisnull)
+ &isnull);
+ if (!isnull)
oldrelwhereclause = stringToNode(TextDatumGetCString(whereClauseDatum));
+ /* Transform the int2vector column list to a bitmap. */
+ columnListDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ /*
+ * XXX Maybe make this a separate function. We do this on
+ * multiple places.
+ */
+ if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(columnListDatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ for (int i = 0; i < nelems; i++)
+ oldcolumns = bms_add_member(oldcolumns, elems[i]);
+ }
+
ReleaseSysCache(rftuple);
}
foreach(newlc, rels)
{
PublicationRelInfo *newpubrel;
+ Oid newrelid;
+ Bitmapset *newcolumns = NULL;
newpubrel = (PublicationRelInfo *) lfirst(newlc);
+ newrelid = RelationGetRelid(newpubrel->relation);
+
+ /*
+ * If the new publication has column filter, transform it to
+ * a bitmap too.
+ */
+ if (newpubrel->columns)
+ {
+ ListCell *lc;
+
+ foreach(lc, newpubrel->columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(newrelid, colname);
+
+ /* no checks needed here, that happens elsewhere */
+
+ newcolumns = bms_add_member(newcolumns, attnum);
+ }
+ }
/*
* Check if any of the new set of relations matches with the
@@ -1037,7 +1339,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
*/
if (RelationGetRelid(newpubrel->relation) == oldrelid)
{
- if (equal(oldrelwhereclause, newpubrel->whereClause))
+ if (equal(oldrelwhereclause, newpubrel->whereClause) &&
+ bms_equal(oldcolumns, newcolumns))
{
found = true;
break;
@@ -1056,6 +1359,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
{
oldrel = palloc(sizeof(PublicationRelInfo));
oldrel->whereClause = NULL;
+ oldrel->columns = NIL;
oldrel->relation = table_open(oldrelid,
ShareUpdateExclusiveLock);
delrels = lappend(delrels, oldrel);
@@ -1117,7 +1421,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
}
else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* AP_SetObjects */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -1140,6 +1444,10 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
*/
PublicationAddSchemas(pubform->oid, schemaidlist, true, stmt);
}
+ else
+ {
+ /* Nothing to do for AP_SetColumns */
+ }
}
/*
@@ -1402,6 +1710,7 @@ OpenTableList(List *tables)
List *rels = NIL;
ListCell *lc;
List *relids_with_rf = NIL;
+ List *relids_with_cf = NIL;
/*
* Open, share-lock, and check all the explicitly-specified relations
@@ -1436,6 +1745,13 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /* Disallow duplicate tables if there are any with column filters. */
+ if (t->columns || list_member_oid(relids_with_cf, myrelid))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column filters for table \"%s\"",
+ RelationGetRelationName(rel))));
+
table_close(rel, ShareUpdateExclusiveLock);
continue;
}
@@ -1443,12 +1759,16 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
pub_rel->whereClause = t->whereClause;
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, myrelid);
+ if (t->columns)
+ relids_with_cf = lappend_oid(relids_with_cf, myrelid);
+
/*
* Add children of this rel, if requested, so that they too are added
* to the publication. A partitioned table can't have any inheritance
@@ -1488,6 +1808,18 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /*
+ * We don't allow to specify column filter for both parent
+ * and child table at the same time as it is not very
+ * clear which one should be given preference.
+ */
+ if (childrelid != myrelid &&
+ (t->columns || list_member_oid(relids_with_cf, childrelid)))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column fiters for table \"%s\"",
+ RelationGetRelationName(rel))));
+
continue;
}
@@ -1497,11 +1829,16 @@ OpenTableList(List *tables)
pub_rel->relation = rel;
/* child inherits WHERE clause from parent */
pub_rel->whereClause = t->whereClause;
+ /* child inherits column list from parent */
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, childrelid);
+
+ if (t->whereClause)
+ relids_with_cf = lappend_oid(relids_with_cf, childrelid);
}
}
}
@@ -1610,6 +1947,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index dc5872f988c..a9fd0f0c895 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -8365,6 +8365,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
bool missing_ok, LOCKMODE lockmode,
ObjectAddresses *addrs)
{
+ Oid relid = RelationGetRelid(rel);
HeapTuple tuple;
Form_pg_attribute targetatt;
AttrNumber attnum;
@@ -8384,7 +8385,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/*
* get the number of the attribute
*/
- tuple = SearchSysCacheAttName(RelationGetRelid(rel), colName);
+ tuple = SearchSysCacheAttName(relid, colName);
if (!HeapTupleIsValid(tuple))
{
if (!missing_ok)
@@ -8438,13 +8439,42 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * Also, if the column is used in the column list of a publication,
+ * disallow the drop if the DROP is RESTRICT. We don't do anything if the
+ * DROP is CASCADE, which means that the dependency mechanism will remove
+ * the relation from the publication.
+ */
+ if (behavior == DROP_RESTRICT)
+ {
+ List *pubs;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(relid);
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(relid, pubid);
+
+ if (list_member_oid(published_cols, attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot drop column \"%s\" because it is part of publication \"%s\"",
+ colName, get_publication_name(pubid, false)),
+ errhint("Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication."));
+ }
+ }
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
* use find_all_inheritors to do it in one pass.
*/
children =
- find_inheritance_children(RelationGetRelid(rel), lockmode);
+ find_inheritance_children(relid, lockmode);
if (children)
{
@@ -8532,7 +8562,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Add object to delete */
object.classId = RelationRelationId;
- object.objectId = RelationGetRelid(rel);
+ object.objectId = relid;
object.objectSubId = attnum;
add_exact_object_address(&object, addrs);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index de106d767d1..d87be2d4775 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -573,9 +573,6 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
if (cmd != CMD_UPDATE && cmd != CMD_DELETE)
return;
- if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
- return;
-
/*
* It is only safe to execute UPDATE/DELETE when all columns, referenced
* in the row filters from publications which the relation is in, are
@@ -595,17 +592,33 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
errmsg("cannot update table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_UPDATE && !pubdesc.cf_valid_for_update)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot update table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column filter used by the publication does not cover the replica identity.")));
else if (cmd == CMD_DELETE && !pubdesc.rf_valid_for_delete)
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
errmsg("cannot delete from table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_DELETE && !pubdesc.cf_valid_for_delete)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot delete from table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column filter used by the publication does not cover the replica identity.")));
/* If relation has replica identity we are always good. */
if (OidIsValid(RelationGetReplicaIndex(rel)))
return;
+ /* REPLICA IDENTITY FULL is also good for UPDATE/DELETE. */
+ if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ return;
+
/*
* This is either UPDATE OR DELETE and there is no replica identity.
*
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index d4f8455a2bd..a504437873f 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4850,6 +4850,7 @@ _copyPublicationTable(const PublicationTable *from)
COPY_NODE_FIELD(relation);
COPY_NODE_FIELD(whereClause);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f1002afe7a0..4fc16ce04e3 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2322,6 +2322,7 @@ _equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
COMPARE_NODE_FIELD(whereClause);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a03b33b53bd..25c9b29afdd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9751,13 +9751,14 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr OptWhereClause
+ TABLE relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9772,11 +9773,12 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId OptWhereClause
+ | ColId opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- if ($2)
+ /* FIXME this is a bit cumbersome */
+ if ($2 || $3)
{
/*
* The OptWhereClause must be stored here but it is
@@ -9786,7 +9788,8 @@ PublicationObjSpec:
*/
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVar(NULL, $1, @1);
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
else
{
@@ -9794,23 +9797,25 @@ PublicationObjSpec:
}
$$->location = @1;
}
- | ColId indirection OptWhereClause
+ | ColId indirection opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr OptWhereClause
+ | extended_relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
| CURRENT_SCHEMA
{
@@ -9836,6 +9841,10 @@ pub_obj_list: PublicationObjSpec
*
* ALTER PUBLICATION name SET pub_obj [, ...]
*
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ *
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
* pub_obj is one of:
*
* TABLE table_name [, ...]
@@ -9869,6 +9878,32 @@ AlterPublicationStmt:
n->action = AP_SetObjects;
$$ = (Node *)n;
}
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS '(' columnList ')'
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = $10;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS ALL
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = NIL;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
| ALTER PUBLICATION name DROP pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
@@ -17488,6 +17523,13 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
errmsg("WHERE clause not allowed for schema"),
parser_errposition(pubobj->location));
+ /* Column filter is not allowed on a schema object */
+ if (pubobj->pubtable && pubobj->pubtable->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schema"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index c9b0eeefd7e..1e8785ff9a5 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,16 +29,21 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary);
+ bool binary, Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
static void logicalrep_write_namespace(StringInfo out, Oid nspid);
static const char *logicalrep_read_namespace(StringInfo in);
+#define ColumnInFilter(columns, attnum) \
+ (((columns) == NULL) || (bms_is_member((attnum), (columns))))
+
+
/*
* Write BEGIN to the output stream.
*/
@@ -398,7 +403,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +415,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -443,7 +448,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -464,11 +469,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -537,7 +542,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, NULL);
}
/*
@@ -652,7 +657,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -674,7 +680,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -751,7 +757,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum *values;
@@ -763,7 +769,13 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ /* skip columns not included in the column filter */
+ if (!ColumnInFilter(columns, att->attnum))
continue;
nliveatts++;
}
@@ -783,6 +795,10 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped || att->attgenerated)
continue;
+ /* skip columns not included in the column filter */
+ if (!ColumnInFilter(columns, att->attnum))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +920,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -917,8 +933,15 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* send number of live attributes */
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ /* skip columns not included in the column filter */
+ if (!ColumnInFilter(columns, att->attnum))
continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -937,6 +960,10 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ /* skip columns not included in the column filter */
+ if (!ColumnInFilter(columns, att->attnum))
+ continue;
+
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 1659964571c..42708dcf82e 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -112,6 +112,7 @@
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "utils/acl.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -700,20 +701,22 @@ fetch_remote_table_info(char *nspname, char *relname,
WalRcvExecResult *res;
StringInfoData cmd;
TupleTableSlot *slot;
- Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid tableRow[] = {OIDOID, CHAROID, CHAROID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
int natt;
ListCell *lc;
bool first;
+ bool am_partition;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
/* First fetch Oid and replica identity. */
initStringInfo(&cmd);
- appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind"
+ appendStringInfo(&cmd, "SELECT c.oid, c.relreplident, c.relkind, c.relispartition"
" FROM pg_catalog.pg_class c"
" INNER JOIN pg_catalog.pg_namespace n"
" ON (c.relnamespace = n.oid)"
@@ -743,14 +746,225 @@ fetch_remote_table_info(char *nspname, char *relname,
Assert(!isnull);
lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
+ am_partition = DatumGetBool(slot_getattr(slot, 4, &isnull));
+ Assert(!isnull);
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+
+ /*
+ * Get relation's column filter expressions.
+ *
+ * For initial synchronization, column filter can be ignored in following
+ * cases:
+ *
+ * 1) one of the subscribed publications for the table hasn't specified
+ * any column filter
+ *
+ * 2) one of the subscribed publications has puballtables set to true
+ *
+ * 3) one of the subscribed publications is declared as ALL TABLES IN
+ * SCHEMA that includes this relation
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ Oid tmpRow[] = {INT4OID};
+ StringInfoData publications;
+ bool first = true;
+ bool all_columns = false;
+
+ initStringInfo(&publications);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&publications, ", ");
+ appendStringInfoString(&publications, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ /*
+ * First, check if any of the publications FOR ALL TABLES? If yes, we
+ * should not use any column filter. It's enough to find a single such
+ * publication.
+ *
+ * XXX Maybe we could combine all three steps into a single query, but
+ * this seems cleaner / easier to understand.
+ *
+ * XXX Does this need any handling of partitions / publish_via_part_root?
+ */
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT 1\n"
+ " FROM pg_publication p\n"
+ " WHERE p.pubname IN ( %s ) AND p.puballtables LIMIT 1\n",
+ publications.data);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(tmpRow), tmpRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch publication info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+
+ if (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ all_columns = true;
+
+ ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(pubres);
+
+ /*
+ * If there's no FOR ALL TABLES publication, look for a FOR ALL TABLES
+ * IN SCHEMA publication, with schema of the remote relation. The logic
+ * is the same - such publications have no column filters.
+ *
+ * XXX Does this need any handling of partitions / publish_via_part_root?
+ */
+ if (!all_columns)
+ {
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT 1\n"
+ " FROM pg_publication p\n"
+ " JOIN pg_publication_namespace pn ON (pn.pnpubid = p.oid)\n"
+ " JOIN pg_class c ON (pn.pnnspid = c.relnamespace)\n"
+ " WHERE c.oid = %u AND p.pubname IN ( %s ) LIMIT 1",
+ lrel->remoteid,
+ publications.data);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(tmpRow), tmpRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch publication info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+
+ if (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ all_columns = true;
+
+ ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(pubres);
+ }
+
+ /*
+ * If we haven't found any FOR ALL TABLES [IN SCHEMA] publications for
+ * the table, we have to look for the column filters set for relations.
+ * First, we check if there's a publication with no column filter for
+ * the relation - which means all columns need to be replicated.
+ */
+ if (!all_columns)
+ {
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT 1\n"
+ " FROM pg_catalog.pg_publication p JOIN\n"
+ " pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n"
+ " WHERE p.pubname IN (%s) AND pr.prattrs IS NULL AND ",
+ publications.data);
+
+ /*
+ * For non-partitions, we simply join directly to the catalog. For
+ * partitions, we need to check all the ancestors, because maybe the
+ * root was not added to a publication but one of the intermediate
+ * partitions was.
+ */
+ if (!am_partition)
+ appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "prrelid IN (SELECT relid\n"
+ " FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(tmpRow), tmpRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch attribute info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+
+ if (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ all_columns = true;
+
+ ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(pubres);
+ }
+
+ /*
+ * All that
+ */
+ if (!all_columns)
+ {
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT unnest(pr.prattrs)\n"
+ " FROM pg_catalog.pg_publication p JOIN\n"
+ " pg_catalog.pg_publication_rel pr ON (p.oid = pr.prpubid)\n"
+ " WHERE p.pubname IN (%s) AND pr.prattrs IS NOT NULL AND ",
+ publications.data);
+
+ /*
+ * For non-partitions, we simply join directly to the catalog. For
+ * partitions, we need to check all the ancestors, because maybe the
+ * root was not added to a publication but one of the intermediate
+ * partitions was.
+ */
+ if (!am_partition)
+ appendStringInfo(&cmd, "prrelid = %u", lrel->remoteid);
+ else
+ appendStringInfo(&cmd,
+ "prrelid IN (SELECT relid\n"
+ " FROM pg_catalog.pg_partition_tree(pg_catalog.pg_partition_root(%u)))",
+ lrel->remoteid);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch attribute info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ included_cols = bms_add_member(included_cols, attnum);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(pubres);
+ }
+
+ pfree(publications.data);
+ }
+
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -778,16 +992,34 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * Store the columns as a list of names. Ignore those that are not
+ * present in the column list, if there is one.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ {
+ ExecClearTuple(slot);
+ continue;
+ }
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -821,6 +1053,9 @@ fetch_remote_table_info(char *nspname, char *relname,
*
* 3) one of the subscribed publications is declared as ALL TABLES IN
* SCHEMA that includes this relation
+ *
+ * XXX Does this actually handle puballtables and schema publications
+ * correctly?
*/
if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
{
@@ -930,8 +1165,24 @@ copy_table(Relation rel)
/* Regular table with no row filter */
if (lrel.relkind == RELKIND_RELATION && qual == NIL)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+
+ /*
+ * XXX Do we need to list the columns in all cases? Maybe we're replicating
+ * all columns?
+ */
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
+
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
+
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a0477f0..07cdfc1d8c0 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -29,6 +29,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -85,7 +86,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -93,6 +95,8 @@ static void send_repl_origin(LogicalDecodingContext *ctx,
/*
* Only 3 publication actions are used for row filtering ("insert", "update",
* "delete"). See RelationSyncEntry.exprstate[].
+ *
+ * FIXME Do we need something similar for column filters?
*/
enum RowFilterPubAction
{
@@ -164,6 +168,13 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
AttrMap *attrmap;
+
+ /*
+ * Columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this bitmap are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -603,11 +614,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -620,7 +631,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -643,13 +655,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip this attribute if it's not present in the column list */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -1224,7 +1240,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -1278,11 +1294,13 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel,
- new_slot, data->binary);
+ new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel,
- old_slot, new_slot, data->binary);
+ old_slot, new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel,
@@ -1731,6 +1749,7 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
memset(entry->exprstate, 0, sizeof(entry->exprstate));
entry->cache_expr_cxt = NULL;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->attrmap = NULL;
}
@@ -1775,6 +1794,8 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
entry->pubactions.pubinsert = false;
entry->pubactions.pubupdate = false;
entry->pubactions.pubdelete = false;
@@ -1807,13 +1828,16 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/*
* Build publication cache. We can't use one provided by relcache as
- * relcache considers all publications given relation is in, but here
- * we only need to consider ones that the subscriber requested.
+ * relcache considers all publications that the given relation is in,
+ * but here we only need to consider ones that the subscriber
+ * requested.
*/
foreach(lc, data->publications)
{
Publication *pub = lfirst(lc);
bool publish = false;
+ bool ancestor_published = false;
+ bool all_columns = false;
if (pub->alltables)
{
@@ -1824,8 +1848,6 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
if (!publish)
{
- bool ancestor_published = false;
-
/*
* For a partition, check if any of the ancestors are
* published. If so, note down the topmost ancestor that is
@@ -1855,6 +1877,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
}
/*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
* Don't publish changes for partitioned tables, because
* publishing those of its partitions suffices, unless partition
* changes won't be published due to pubviaroot being set.
@@ -1867,6 +1892,80 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
+ /*
+ * This might be FOR ALL TABLES or FOR ALL TABLES IN SCHEMA
+ * publication, in which case there are no column lists, and
+ * we treat that as all_columns=true.
+ */
+ if (pub->alltables ||
+ list_member_oid(schemaPubids, pub->oid))
+ {
+ all_columns = true;
+ bms_free(entry->columns);
+ entry->columns = NULL;
+ }
+
+ /*
+ * Obtain columns published by this publication, and add them
+ * to the list for this rel. Note that if at least one
+ * publication has an empty column list, that means to publish
+ * everything; so if we saw a publication that includes all
+ * columns, skip this.
+ *
+ * FIXME This fails to consider column filters defined in
+ * FOR ALL TABLES and FOR ALL TABLES IN SCHEMA publications.
+ * We need to check those too.
+ */
+ if (!all_columns)
+ {
+ HeapTuple pub_rel_tuple;
+
+ pub_rel_tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(pub_rel_tuple))
+ {
+ Datum pub_rel_cols;
+ bool isnull;
+
+ pub_rel_cols = SysCacheGetAttr(PUBLICATIONRELMAP,
+ pub_rel_tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+ if (isnull)
+ {
+ /*
+ * If we see a publication with no column filter, it
+ * means we need to publish all columns, so reset the
+ * list and ignore further ones.
+ */
+ all_columns = true;
+ bms_free(entry->columns);
+ entry->columns = NULL;
+ }
+ else
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(pub_rel_cols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
+ entry->columns = bms_add_member(entry->columns,
+ elems[i]);
+ MemoryContextSwitchTo(oldctx);
+ }
+
+ ReleaseSysCache(pub_rel_tuple);
+ }
+ }
+
rel_publications = lappend(rel_publications, pub);
}
}
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index fccffce5729..82e595396e3 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -5553,6 +5553,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cf_valid_for_update = true;
+ pubdesc->cf_valid_for_delete = true;
return;
}
@@ -5565,6 +5567,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cf_valid_for_update = true;
+ pubdesc->cf_valid_for_delete = true;
/* Fetch the publication membership info. */
puboids = GetRelationPublications(relid);
@@ -5625,6 +5629,24 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->rf_valid_for_delete = false;
}
+ /*
+ * Check if all columns referenced in the column filter are part of
+ * the REPLICA IDENTITY index or not.
+ *
+ * If the publication is FOR ALL TABLES then it means the table has no
+ * column filters and we can skip the validation.
+ */
+ if (!pubform->puballtables &&
+ (pubform->pubupdate || pubform->pubdelete) &&
+ contain_invalid_cfcolumn(pubid, relation, ancestors,
+ pubform->pubviaroot))
+ {
+ if (pubform->pubupdate)
+ pubdesc->cf_valid_for_update = false;
+ if (pubform->pubdelete)
+ pubdesc->cf_valid_for_delete = false;
+ }
+
ReleaseSysCache(tup);
/*
@@ -5636,6 +5658,16 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
!pubdesc->rf_valid_for_update && !pubdesc->rf_valid_for_delete)
break;
+
+ /*
+ * If we know everything is replicated and the column filter is invalid
+ * for update and delete, there is no point to check for other
+ * publications.
+ */
+ if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
+ pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
+ !pubdesc->cf_valid_for_update && !pubdesc->cf_valid_for_delete)
+ break;
}
if (relation->rd_pubdesc)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e69dcf8a484..f208c7a6c59 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4075,6 +4075,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_prpubid;
int i_prrelid;
int i_prrelqual;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4088,12 +4089,20 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
if (fout->remoteVersion >= 150000)
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual "
- "FROM pg_catalog.pg_publication_rel");
+ "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual, "
+ "(CASE\n"
+ " WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT array_agg(attname)\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) prattrs "
+ "FROM pg_catalog.pg_publication_rel pr");
else
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "NULL AS prrelqual "
+ "NULL AS prrelqual, NULL AS prattrs "
"FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
@@ -4104,6 +4113,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
i_prrelqual = PQfnumber(res, "prrelqual");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4149,6 +4159,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
else
pubrinfo[j].pubrelqual = pg_strdup(PQgetvalue(res, i, i_prrelqual));
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[j].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4223,10 +4255,13 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+
if (pubrinfo->pubrelqual)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 997a3b60719..680b07dcd52 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -632,6 +632,7 @@ typedef struct _PublicationRelInfo
PublicationInfo *publication;
TableInfo *pubtable;
char *pubrelqual;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index ae8c86a6e88..d4d92465c49 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2428,6 +2428,28 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'ALTER PUBLICATION pub1 ADD TABLE test_sixth_table (col3, col2)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_sixth_table (col3, col2);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_sixth_table (col2, col3);\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'ALTER PUBLICATION pub1 ADD TABLE test_seventh_table (col3, col2) WHERE (col1 = 1)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_seventh_table (col3, col2) WHERE (col1 = 1);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_seventh_table (col2, col3) WHERE ((col1 = 1));\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'ALTER PUBLICATION pub3 ADD ALL TABLES IN SCHEMA dump_test' => {
create_order => 51,
create_sql =>
@@ -2778,6 +2800,44 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'CREATE TABLE test_sixth_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_sixth_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_sixth_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'CREATE TABLE test_seventh_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_seventh_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_seventh_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'CREATE TABLE test_table_identity' => {
create_order => 3,
create_sql => 'CREATE TABLE dump_test.test_table_identity (
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index e3382933d98..fb18cb82d9f 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2880,6 +2880,7 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_namespace pn ON p.oid = pn.pnpubid\n"
" JOIN pg_catalog.pg_class pc ON pc.relnamespace = pn.pnnspid\n"
@@ -2887,6 +2888,12 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , pg_get_expr(pr.prqual, c.oid)\n"
+ " , (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT string_agg(attname, ', ')\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) "
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
" JOIN pg_catalog.pg_class c ON c.oid = pr.prrelid\n"
@@ -2894,6 +2901,7 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2904,12 +2912,14 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
"WHERE pr.prrelid = '%s'\n"
"UNION ALL\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2931,6 +2941,11 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, " \"%s\"",
PQgetvalue(result, i, 0));
+ /* column filter (if any) */
+ if (!PQgetisnull(result, i, 2))
+ appendPQExpBuffer(&buf, " (%s)",
+ PQgetvalue(result, i, 2));
+
/* row filter (if any) */
if (!PQgetisnull(result, i, 1))
appendPQExpBuffer(&buf, " WHERE %s",
@@ -5867,7 +5882,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5884,15 +5899,19 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (as_schema)
+ printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
+ else
{
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
+
+ if (!PQgetisnull(res, i, 3))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 3));
+
if (!PQgetisnull(res, i, 2))
appendPQExpBuffer(buf, " WHERE %s", PQgetvalue(res, i, 2));
}
- else
- printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
}
@@ -6021,11 +6040,22 @@ describePublications(const char *pattern)
printfPQExpBuffer(&buf,
"SELECT n.nspname, c.relname");
if (pset.sversion >= 150000)
+ {
appendPQExpBufferStr(&buf,
", pg_get_expr(pr.prqual, c.oid)");
+ appendPQExpBufferStr(&buf,
+ ", (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string("
+ " ARRAY(SELECT attname\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END)");
+ }
else
appendPQExpBufferStr(&buf,
- ", NULL");
+ ", NULL, NULL");
appendPQExpBuffer(&buf,
"\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index ba72e62e614..b58f85ede27 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -85,6 +85,14 @@ typedef struct PublicationDesc
*/
bool rf_valid_for_update;
bool rf_valid_for_delete;
+
+ /*
+ * true if the columns referenced in column filters which are used for UPDATE
+ * or DELETE are part of the replica identity or the publication actions
+ * do not include UPDATE or DELETE.
+ */
+ bool cf_valid_for_update;
+ bool cf_valid_for_delete;
} PublicationDesc;
typedef struct Publication
@@ -100,6 +108,7 @@ typedef struct PublicationRelInfo
{
Relation relation;
Node *whereClause;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -123,8 +132,11 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
+extern void GetActionsInPublication(Oid pubid, PublicationActions *actions);
extern List *GetPublicationSchemas(Oid pubid);
extern List *GetSchemaPublications(Oid schemaid);
extern List *GetSchemaPublicationRelations(Oid schemaid,
@@ -142,6 +154,8 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *pri
bool if_not_exists);
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern void publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index 0dd0f425db9..4feb581899e 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -34,6 +34,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
#ifdef CATALOG_VARLEN /* variable-length fields start here */
pg_node_tree prqual; /* qualifications */
+ int2vector prattrs; /* columns to replicate */
#endif
} FormData_pg_publication_rel;
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index 7813cbcb6bb..08d14ca7245 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -33,5 +33,7 @@ extern void AlterPublicationOwner_oid(Oid pubid, Oid newOwnerId);
extern void InvalidatePublicationRels(List *relids);
extern bool contain_invalid_rfcolumn(Oid pubid, Relation relation,
List *ancestors, bool pubviaroot);
+extern bool contain_invalid_cfcolumn(Oid pubid, Relation relation,
+ List *ancestors, bool pubviaroot);
#endif /* PUBLICATIONCMDS_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1617702d9d6..1375a173e3b 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3652,6 +3652,7 @@ typedef struct PublicationTable
NodeTag type;
RangeVar *relation; /* relation to be published */
Node *whereClause; /* qualifications */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
@@ -3688,7 +3689,8 @@ typedef enum AlterPublicationAction
{
AP_AddObjects, /* add objects to publication */
AP_DropObjects, /* remove objects from publication */
- AP_SetObjects /* set list of objects */
+ AP_SetObjects, /* set list of objects */
+ AP_SetColumns /* change list of columns for a table */
} AlterPublicationAction;
typedef struct AlterPublicationStmt
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4d2c881644a..a771ab8ff33 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -209,12 +209,12 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary);
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -231,7 +231,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 6d16600aaea..152f19fb42b 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -679,6 +679,372 @@ DROP TABLE rf_tbl_abcd_pk;
DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column filters
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+ERROR: conflicting or redundant column filters for table "testpub_tbl1"
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+ERROR: conflicting or redundant column filters for table "testpub_tbl1"
+RESET client_min_messages;
+-- test for column filters
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+-- error: replica identity "a" not included in the column filter
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ERROR: cannot reference generated column "d" in publication column list
+-- error: system attributes "ctid" not allowed in column filter
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+ERROR: cannot reference system column "ctid" in publication column list
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+ERROR: cannot drop column "c" because it is part of publication "testpub_fortable"
+HINT: Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication.
+-- ok: for insert-only publication, the filter is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column filter (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: change the replica identity to "b", and then filter (a, c) fails
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+-- error: replica identity (b,c) is not covered by column filter (a,c)
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+ Publication testpub_table_ins
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | t | f
+Tables:
+ "public.testpub_tbl5" (a)
+
+-- with REPLICA IDENTITY FULL, column filters are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ERROR: cannot update table "testpub_tbl6"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+ALTER PUBLICATION testpub_fortable ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ERROR: cannot update table "testpub_tbl6"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+-- make sure changing the column filter is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: update the column filter
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, c)
+
+-- column filter for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: column filter covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_filter FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+-- failure: column filter does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+-- failure: replica identity full can't be used with a column filter
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+-- failure: replica identity has to be covered by the column filter
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl8;
+-- column filter for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: attaching first partition works, because (a) is in column filter
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column filter
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column filter on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_0"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_filter;
+-- ======================================================
+-- Test combination of column and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+ Publication testpub_both_filters
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | t | t | t | f
+Tables:
+ "public.testpub_tbl_both_filters" (a, c) WHERE (c <> 1)
+
+\d+ testpub_tbl_both_filters
+ Table "public.testpub_tbl_both_filters"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+ c | integer | | not null | | plain | |
+Indexes:
+ "testpub_tbl_both_filters_pkey" PRIMARY KEY, btree (a, c) REPLICA IDENTITY
+Publications:
+ "testpub_both_filters" (a, c) WHERE (c <> 1)
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+-- More column filter tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column filter
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column filter
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column filter works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column filter is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column filter is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column filters are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column filters are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column filters are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column filter "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column filter "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column filter "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column filter "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+-- Tests for partitioned table
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+ERROR: cannot use publication column list for relation "rf_tbl_abcd_part_pk"
+DETAIL: column list cannot be used for a partitioned table when publish_via_partition_root is false.
+-- ok - can use row filter for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any row filter is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+ERROR: cannot set publish_via_partition_root = false for publication "testpub6"
+DETAIL: The publication contains a column filter for a partitioned table "rf_tbl_abcd_part_pk" which is not allowed when publish_via_partition_root is false.
+-- Now change the root filter to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have row filter for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column filter used by the publication does not cover the replica identity.
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -1124,6 +1490,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schema
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index c135a601a30..2203dc238d2 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -381,6 +381,292 @@ DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column filters
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+RESET client_min_messages;
+
+-- test for column filters
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+-- error: replica identity "a" not included in the column filter
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+-- error: system attributes "ctid" not allowed in column filter
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+-- ok: for insert-only publication, the filter is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column filter (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+
+-- error: change the replica identity to "b", and then filter (a, c) fails
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+-- error: replica identity (b,c) is not covered by column filter (a,c)
+UPDATE testpub_tbl5 SET a = 1;
+
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+
+-- with REPLICA IDENTITY FULL, column filters are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+
+ALTER PUBLICATION testpub_fortable ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+
+-- make sure changing the column filter is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: update the column filter
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+
+-- column filter for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: column filter covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_filter FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: column filter does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+
+-- failure: replica identity full can't be used with a column filter
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: replica identity has to be covered by the column filter
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl8;
+
+-- column filter for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: attaching first partition works, because (a) is in column filter
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column filter
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column filter on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_filter;
+-- ======================================================
+
+-- Test combination of column and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+\d+ testpub_tbl_both_filters
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+
+-- More column filter tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column filter
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column filter
+UPDATE rf_tbl_abcd_pk SET a = 1;
+
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column filter works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column filter is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column filter is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column filters are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column filters are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column filters are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column filter "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column filter "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column filter "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column filter "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Tests for partitioned table
+
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - can use row filter for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any row filter is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- Now change the root filter to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have row filter for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
+
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -622,6 +908,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/029_column_list.pl b/src/test/subscription/t/029_column_list.pl
new file mode 100644
index 00000000000..ec2c8a789ad
--- /dev/null
+++ b/src/test/subscription/t/029_column_list.pl
@@ -0,0 +1,836 @@
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test partial-column publication of tables
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 26;
+
+# setup
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+sub wait_for_subscription_sync
+{
+ my ($node) = @_;
+
+ # Also wait for initial table sync to finish
+ my $synced_query = "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+
+ $node->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+}
+
+# setup tables on both nodes
+
+# tab1: simple 1:1 replication
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+# tab2: replication from regular to table with fewer columns
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)
+));
+
+# tab3: simple 1:1 replication with weird column names
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "B" varchar, "c'" int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "c'" int)
+));
+
+# test_part: partitioned tables, with partitioning (including multi-level
+# partitioning, and fewer columns on the subscriber)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (4,5,6) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (4,5);
+));
+
+# tab4: table with user-defined enum types
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, d text);
+));
+
+
+# TEST: create publication and subscription for some of the tables with
+# column lists
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub1
+ FOR TABLE tab1 (a, "B"), tab3 ("a'", "c'"), test_part (a, b), tab4 (a, b, d)
+ WITH (publish_via_partition_root = 'true');
+));
+
+# check that we got the right prattrs values for the publication in the
+# pg_publication_rel catalog (order by relname, to get stable ordering)
+my $result = $node_publisher->safe_psql('postgres', qq(
+ SELECT relname, prattrs
+ FROM pg_publication_rel pb JOIN pg_class pc ON(pb.prrelid = pc.oid)
+ ORDER BY relname
+));
+
+is($result, qq(tab1|1 2
+tab3|1 3
+tab4|1 2 4
+test_part|1 2), 'publication relation updated');
+
+# create subscription for the publication, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+# TEST: insert data into the tables, and see we got replication of just
+# the filtered columns
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (1, 2, 3);
+ INSERT INTO tab1 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (1, 2, 3);
+ INSERT INTO tab3 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (1, 'red', 3, 'oh my');
+ INSERT INTO tab4 VALUES (2, 'blue', 4, 'hello');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (1, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (2, 'bcd', '2021-07-03 11:12:13');
+ INSERT INTO test_part VALUES (4, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (5, 'bcd', '2021-07-03 11:12:13');
+));
+
+# wait for catchup before checking the subscriber
+$node_publisher->wait_for_catchup('sub1');
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1");
+is($result, qq(1|2|
+4|5|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab3");
+is($result, qq(1|3
+4|6), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4");
+is($result, qq(1|red|oh my
+2|blue|hello), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part");
+is($result, qq(1|abc
+2|bcd
+4|abc
+5|bcd), 'insert on column test_part.c columns is not replicated');
+
+# TEST: do some updated on some of the tables, both on columns included
+# in the column list and other
+
+# tab1: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET "B" = 2 * "B" where a = 1));
+
+# tab1: update of non-replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET c = 2*c where a = 4));
+
+# tab3: update of non-replicated
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "B" = "B" || ' updated' where "a'" = 4));
+
+# tab3: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "c'" = 2 * "c'" where "a'" = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'blue', c = c * 2, d = d || ' updated' where a = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'red', c = c * 2, d = d || ' updated' where a = 2));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab1 ORDER BY a));
+is($result,
+qq(1|4|
+4|5|), 'only update on column tab1.b is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result,
+qq(1|6
+4|6), 'only update on column tab3.c is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab4 ORDER BY a));
+
+is($result, qq(1|blue|oh my updated
+2|red|hello updated), 'update on column tab4.c is not replicated');
+
+
+# TEST: add table with a column list, insert data, replicate
+
+# insert some data before adding it to the publication
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (1, 'abc', 3);
+));
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2 (a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION");
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (2, 'def', 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|def), 'insert on column tab2.c is not replicated');
+
+$node_publisher->safe_psql('postgres', qq(
+ UPDATE tab2 SET c = 5 where a = 1;
+ UPDATE tab2 SET b = 'xyz' where a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|xyz), 'update on column tab2.c is not replicated');
+
+
+# TEST: add a table to two publications with different column lists, and
+# create a single subscription replicating both publications
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub3
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->wait_for_catchup('sub1');
+
+# TEST: insert data and make sure all the columns (union of the columns lists)
+# were replicated
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (1, 11, 111, 1111)");
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (2, 22, 222, 2222)");
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111
+2|22|2222),
+ 'overlapping publications with overlapping column lists');
+
+# and finally, set the column filter to ALL for one of the publications,
+# which means replicating all columns (removing the column filter), but
+# first add the missing column to the table on subscriber
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub3 ALTER TABLE tab5 SET COLUMNS ALL;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ ALTER TABLE tab5 ADD COLUMN c INT;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', "INSERT INTO tab5 VALUES (3, 33, 333, 3333)");
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111|
+2|22|2222|
+3|33|3333|333),
+ 'overlapping publications with overlapping column lists');
+
+# TEST: create a table with a column filter, then change the replica
+# identity by replacing a primary key (but use a different column in
+# the column filter)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub4 FOR TABLE tab6 (a, b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (1, 22, 333, 4444);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 1;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ "1|44||", 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is still covered by the column list, though)
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+# we need to do the same thing on the subscriber
+# XXX What would happen if this happens before the publisher ALTER? Or
+# interleaved, somehow? But that seems unrelated to column lists.
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (2, 55, 666, 8888);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|44||
+2|110||),
+ 'replication with the modified primary key');
+
+
+# TEST: create a table with a column filter, then change the replica
+# identity by replacing a primary key with a key on multiple columns
+# (all of them covered by the column filter)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub5 FOR TABLE tab7 (a, b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub5
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (1, 22, 333, 4444);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 1;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ "1|44||", 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is not covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ ALTER TABLE tab7 ADD PRIMARY KEY (a, b);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (2, 55, 666, 8888);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|44||
+2|110||),
+ 'replication with the modified primary key');
+
+# now switch the primary key again to another columns not covered by the
+# column filter, but also generate writes between the drop and creation
+# of the new constraint
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ INSERT INTO tab7 VALUES (3, 33, 999, 7777);
+ -- update/delete is not allowed for tables without RI
+ ALTER TABLE tab7 ADD PRIMARY KEY (b, a);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4 WHERE a = 3;
+ DELETE FROM tab7 WHERE a = 1;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(2|110||
+3|66||),
+ 'replication with the modified primary key');
+
+
+# TEST: partitioned tables (with publish_via_partition_root = false)
+# and replica identity. The (leaf) partitions may have different RI, so
+# we need to check the partition RI (with respect to the column filter)
+# while attaching the partition.
+
+# First, let's create a partitioned table with two partitions, each with
+# a different RI, but a column filter not covering all those RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (2);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+));
+
+# do the same thing on the subscriber (with the opposite column order)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (b int, a int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (2);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+));
+
+# create a publication replicating just the column "a", which is not enough
+# for the second partition
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub6 FOR TABLE test_part_a (b, a) WITH (publish_via_partition_root = true);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_1 (a);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_2 (b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub6
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_a VALUES (1, 3);
+ INSERT INTO test_part_a VALUES (2, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT a, b FROM test_part_a ORDER BY a, b"),
+ qq(1|3
+2|4),
+ 'partitions with different replica identities not replicated correctly');
+
+# This time start with a column filter covering RI for all partitions, but
+# then update the column filter to not cover column "b" (needed by the
+# second partition)
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,3);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (2,4);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,3);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (2,4);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub7 FOR TABLE test_part_b (a, b) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub7
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_b VALUES (1, 1);
+ INSERT INTO test_part_b VALUES (2, 2);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_b ORDER BY a, b"),
+ qq(1|1
+2|2),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: This time start with a column filter covering RI for all partitions,
+# but then update RI for one of the partitions to not be covered by the
+# column filter anymore.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+));
+
+# create a publication replicating data through partition root, with a column
+# filter on the root, and then add the partitions one by one with separate
+# column filters (but those are not applied)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a,c);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SUBSCRIPTION sub1;
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub8;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||5
+2|4|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# create a publication not replicating data through partition root, without
+# a column filter on the root, and then add the partitions one by one with
+# separate column filters
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub8;
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ TRUNCATE test_part_c;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ TRUNCATE test_part_c;
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||
+2|4|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: Start with a single partition, with RI compatible with the column
+# filter, and then attach a partition with incompatible RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+));
+
+# do the same thing on the subscriber (in fact, create both partitions right
+# away, no need to delay that)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ CREATE TABLE test_part_d_2 PARTITION OF test_part_d FOR VALUES IN (2,4);
+ ALTER TABLE test_part_d_2 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_2 REPLICA IDENTITY USING INDEX test_part_d_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub9 FOR TABLE test_part_d (a) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub9
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_d VALUES (1, 1);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_d ORDER BY a, b"),
+ qq(1|),
+ 'partitions with different replica identities not replicated correctly');
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column filters. So with column filters (a,b) and (a,c) we
+# should replicate (a,b,c).
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_1 VALUES (1, 2, 3);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_1"),
+ qq(1|2|3),
+ 'a mix of publications should use a union of column filter');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column filters. If any of the publications is FOR ALL
+# TABLES, we should replicate all columns.
+
+# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ test_part, test_part_a, test_part_b, test_part_c, test_part_d;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_3, pub_mix_4;
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_2 VALUES (1, 2, 3);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_2"),
+ qq(1|2|3),
+ 'a mix of publications should use a union of column filter');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column filters. If any of the publications is FOR ALL
+# TABLES IN SCHEMA, we should replicate all columns.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_5, pub_mix_6;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_3 VALUES (1, 2, 3);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_3"),
+ qq(1|2|3),
+ 'a mix of publications should use a union of column filter');
+
+
+# TEST: Check handling of publish_via_partition_root - if a partition is
+# published through partition root, we should only apply the column filter
+# defined for the whole table (not the partitions) - both during the initial
+# sync and when replicating changes. This is what we do for row filters.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+
+ CREATE PUBLICATION pub_root_true FOR TABLE test_root (a) WITH (publish_via_partition_root = true);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_root_true;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_root VALUES (1, 2, 3);
+ INSERT INTO test_root VALUES (10, 20, 30);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_root ORDER BY a, b, c"),
+ qq(1||
+10||),
+ 'publication via partition root applies column filter');
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
+
+done_testing();
--
2.34.1
On 07.03.22 16:18, Tomas Vondra wrote:
AFAICS these issues should be resolved by the adoption of the row-filter
approach (i.e. it should fail the same way as for row filter).
The first two patches (additional testing for row filtering feature)
look okay to me.
Attached is a fixup patch for your main feature patch (the third one).
It's a bit of code and documentation cleanup, but mainly I removed the
term "column filter" from the patch. Half the code was using "column
list" or similar and half the code "column filter", which was confusing.
Also, there seemed to be a bit of copy-and-pasting from row-filter
code going on, with some code comments not quite sensible, so I rewrote
some of them. Also some code used "rf" and "cf" symbols which were a
bit hard to tell apart. A few more letters can increase readability.
Note in publicationcmds.c OpenTableList() the wrong if condition was used.
I'm still confused about the intended replica identity handling. This
patch still checks whether the column list contains the replica identity
at DDL time. And then it also checks at execution time. I thought the
latest understanding was that the DDL-time checking would be removed. I
think it's basically useless now, since as the test cases show, you can
subvert those checks by altering the replica identity later.
Attachments:
0001-fixup-Allow-specifying-column-filters-for-logical-re.patchtext/plain; charset=UTF-8; name=0001-fixup-Allow-specifying-column-filters-for-logical-re.patchDownload
From d0e9df4674389cda9f891f5678f476d35095c618 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Tue, 8 Mar 2022 16:23:01 +0100
Subject: [PATCH] fixup! Allow specifying column filters for logical
replication
---
doc/src/sgml/catalogs.sgml | 4 +-
doc/src/sgml/protocol.sgml | 5 +-
doc/src/sgml/ref/alter_publication.sgml | 4 +
src/backend/catalog/pg_publication.c | 20 ++-
src/backend/commands/publicationcmds.c | 80 ++++++------
src/backend/executor/execReplication.c | 8 +-
src/backend/replication/logical/proto.c | 21 +--
src/backend/replication/logical/tablesync.c | 16 +--
src/backend/replication/pgoutput/pgoutput.c | 2 +-
src/backend/utils/cache/relcache.c | 25 ++--
src/bin/psql/describe.c | 2 +-
src/include/catalog/pg_publication.h | 7 +-
src/include/commands/publicationcmds.h | 4 +-
src/test/regress/expected/publication.out | 134 ++++++++++----------
src/test/regress/sql/publication.sql | 90 ++++++-------
src/test/subscription/t/029_column_list.pl | 50 ++++----
16 files changed, 235 insertions(+), 237 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 2b61f42b71..c043da37ae 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -4392,7 +4392,7 @@ <title><structname>pg_index</structname> Columns</title>
</para>
<para>
This is an array of <structfield>indnatts</structfield> values that
- indicate which table columns this index indexes. For example a value
+ indicate which table columns this index indexes. For example, a value
of <literal>1 3</literal> would mean that the first and the third table
columns make up the index entries. Key columns come before non-key
(included) columns. A zero in this array indicates that the
@@ -6271,7 +6271,7 @@ <title><structname>pg_publication_namespace</structname> Columns</title>
</para>
<para>
This is an array of values that indicates which table columns are
- part of the publication. For example a value of <literal>1 3</literal>
+ part of the publication. For example, a value of <literal>1 3</literal>
would mean that the first and the third table columns are published.
A null value indicates that all columns are published.
</para></entry>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 496593201b..6f4d76ef7f 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7005,9 +7005,8 @@ <title>Logical Replication Message Formats</title>
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except
- generated columns and other columns that don't appear in the column
- filter list, for tables that have one):
+ Next, the following message part appears for each column included in
+ the publication (except generated columns):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index aa6827c977..470d50a244 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -119,10 +119,14 @@ <title>Parameters</title>
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ </para>
+ <para>
Optionally, a column list can be specified. See <xref
linkend="sql-createpublication"/> for details.
+ </para>
+ <para>
If the optional <literal>WHERE</literal> clause is specified, rows for
which the <replaceable class="parameter">expression</replaceable>
evaluates to false or null will not be published. Note that parentheses
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 4dab96265f..3275a7c8b9 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -333,7 +333,6 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
Publication *pub = GetPublication(pubid);
AttrNumber *attarray;
int natts = 0;
- int i;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -381,7 +380,7 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
- /* Add column filter, if available */
+ /* Add column list, if available */
if (pri->columns)
{
int2vector *prattrs;
@@ -408,7 +407,7 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
ObjectAddressSet(myself, PublicationRelRelationId, pubreloid);
/* Add dependency on the columns, if any are listed */
- for (i = 0; i < natts; i++)
+ for (int i = 0; i < natts; i++)
{
ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -461,7 +460,6 @@ publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
bool nulls[Natts_pg_publication_rel];
bool replaces[Natts_pg_publication_rel];
Datum values[Natts_pg_publication_rel];
- int i;
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -480,14 +478,14 @@ publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
}
else
{
- ObjectAddress myself,
+ ObjectAddress myself,
referenced;
int2vector *prattrs;
Form_pg_publication_rel pubrel;
publication_translate_columns(targetrel, columns, &natts, &attarray);
- /* XXX "pub" is leaked here */
+ /* XXX "pub" is leaked here ??? */
prattrs = buildint2vector(attarray, natts);
values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
@@ -496,7 +494,7 @@ publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
pubrel = (Form_pg_publication_rel) GETSTRUCT(pubreltup);
ObjectAddressSet(myself, PublicationRelRelationId, pubrel->oid);
- for (i = 0; i < natts; i++)
+ for (int i = 0; i < natts; i++)
{
ObjectAddressSubSet(referenced, RelationRelationId,
RelationGetRelid(targetrel), attarray[i]);
@@ -713,11 +711,10 @@ GetRelationColumnPartialPublications(Oid relid)
{
CatCList *pubrellist;
List *pubs = NIL;
- int i;
pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
ObjectIdGetDatum(relid));
- for (i = 0; i < pubrellist->n_members; i++)
+ for (int i = 0; i < pubrellist->n_members; i++)
{
HeapTuple tup = &pubrellist->members[i]->tuple;
bool isnull;
@@ -727,7 +724,7 @@ GetRelationColumnPartialPublications(Oid relid)
Anum_pg_publication_rel_prattrs,
&isnull);
- /* no column filter for this publications/relation */
+ /* no column list for this publications/relation */
if (isnull)
continue;
@@ -756,7 +753,6 @@ GetRelationColumnListInPublication(Oid relid, Oid pubid)
int nelems;
int16 *elems;
List *attnos = NIL;
- int i;
tup = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(relid),
@@ -774,7 +770,7 @@ GetRelationColumnListInPublication(Oid relid, Oid pubid)
nelems = ARR_DIMS(arr)[0];
elems = (int16 *) ARR_DATA_PTR(arr);
- for (i = 0; i < nelems; i++)
+ for (int i = 0; i < nelems; i++)
attnos = lappend_oid(attnos, elems[i]);
ReleaseSysCache(tup);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index fa1462ae54..b32ec27555 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -296,7 +296,7 @@ contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
* Returns true if any invalid column is found.
*/
bool
-contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
+pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
bool pubviaroot)
{
HeapTuple rftuple;
@@ -368,28 +368,28 @@ contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
}
/*
- * Check if all columns referenced in the column filter are part of the
+ * Check if all columns referenced in the column list are part of the
* REPLICA IDENTITY index or not.
*
* Returns true if any invalid column is found.
*/
bool
-contain_invalid_cfcolumn(Oid pubid, Relation relation, List *ancestors,
+pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
bool pubviaroot)
{
- HeapTuple cftuple;
+ HeapTuple tuple;
Oid relid = RelationGetRelid(relation);
Oid publish_as_relid = RelationGetRelid(relation);
bool result = false;
- Datum cfdatum;
- bool cfisnull;
+ Datum datum;
+ bool isnull;
/*
* For a partition, if pubviaroot is true, find the topmost ancestor that
- * is published via this publication as we need to use its column filter
+ * is published via this publication as we need to use its column list
* for the changes.
*
- * Note that even though the column filter used is for an ancestor, the
+ * Note that even though the column list used is for an ancestor, the
* REPLICA IDENTITY used will be for the actual child table.
*/
if (pubviaroot && relation->rd_rel->relispartition)
@@ -400,24 +400,24 @@ contain_invalid_cfcolumn(Oid pubid, Relation relation, List *ancestors,
publish_as_relid = relid;
}
- cftuple = SearchSysCache2(PUBLICATIONRELMAP,
+ tuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(publish_as_relid),
ObjectIdGetDatum(pubid));
- if (!HeapTupleIsValid(cftuple))
+ if (!HeapTupleIsValid(tuple))
return false;
- cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
+ datum = SysCacheGetAttr(PUBLICATIONRELMAP, tuple,
Anum_pg_publication_rel_prattrs,
- &cfisnull);
+ &isnull);
- if (!cfisnull)
+ if (!isnull)
{
int x;
Bitmapset *idattrs;
Bitmapset *columns = NULL;
- /* With REPLICA IDENTITY FULL, no column filter is allowed. */
+ /* With REPLICA IDENTITY FULL, no column list is allowed. */
if (relation->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
result = true;
@@ -426,7 +426,7 @@ contain_invalid_cfcolumn(Oid pubid, Relation relation, List *ancestors,
int nelems;
int16 *elems;
- arr = DatumGetArrayTypeP(cfdatum);
+ arr = DatumGetArrayTypeP(datum);
nelems = ARR_DIMS(arr)[0];
elems = (int16 *) ARR_DATA_PTR(arr);
@@ -441,7 +441,7 @@ contain_invalid_cfcolumn(Oid pubid, Relation relation, List *ancestors,
/*
* Attnums in the bitmap returned by RelationGetIndexAttrBitmap are
- * offset (to handle system columns the usual way), while column filter
+ * offset (to handle system columns the usual way), while column list
* does not use offset, so we can't do bms_is_subset(). Instead, we have
* to loop over the idattrs and check all of them are in the filter.
*/
@@ -479,7 +479,7 @@ contain_invalid_cfcolumn(Oid pubid, Relation relation, List *ancestors,
bms_free(columns);
}
- ReleaseSysCache(cftuple);
+ ReleaseSysCache(tuple);
return result;
}
@@ -733,7 +733,7 @@ TransformPubWhereClauses(List *tables, const char *queryString,
* XXX The name is a bit misleading, because we don't really transform
* anything here - we merely check the column list is compatible with the
* definition of the publication (with publish_via_partition_root=false)
- * we only allow filters on the leaf relations. So maybe rename it?
+ * we only allow column lists on the leaf relations. So maybe rename it?
*/
static void
TransformPubColumnList(List *tables, const char *queryString,
@@ -750,7 +750,7 @@ TransformPubColumnList(List *tables, const char *queryString,
/*
* If the publication doesn't publish changes via the root partitioned
- * table, the partition's column filter will be used. So disallow using
+ * table, the partition's column list will be used. So disallow using
* the column list on partitioned table in this case.
*/
if (!pubviaroot &&
@@ -988,8 +988,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* If the publication doesn't publish changes via the root partitioned
- * table, the partition's row and column filter will be used. So disallow
- * using WHERE clause and column filters on partitioned table in this case.
+ * table, the partition's row filter and column list will be used. So disallow
+ * using WHERE clause and column lists on partitioned table in this case.
*/
if (!pubform->puballtables && publish_via_partition_root_given &&
!publish_via_partition_root)
@@ -997,7 +997,7 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* Lock the publication so nobody else can do anything with it. This
* prevents concurrent alter to add partitioned table(s) with WHERE
- * clause(s) and/or column filters which we don't allow when not
+ * clause(s) and/or column lists which we don't allow when not
* publishing via root.
*/
LockDatabaseObject(PublicationRelationId, pubform->oid, 0,
@@ -1010,7 +1010,7 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
HeapTuple rftuple;
Oid relid = lfirst_oid(lc);
- bool has_column_filter;
+ bool has_column_list;
bool has_row_filter;
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
@@ -1020,11 +1020,11 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
has_row_filter
= !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL);
- has_column_filter
+ has_column_list
= !heap_attisnull(rftuple, Anum_pg_publication_rel_prattrs, NULL);
if (HeapTupleIsValid(rftuple) &&
- (has_row_filter || has_column_filter))
+ (has_row_filter || has_column_list))
{
HeapTuple tuple;
@@ -1046,13 +1046,13 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
"publish_via_partition_root")));
if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
- has_column_filter)
+ has_column_list)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot set %s for publication \"%s\"",
"publish_via_partition_root = false",
stmt->pubname),
- errdetail("The publication contains a column filter for a partitioned table \"%s\" "
+ errdetail("The publication contains a column list for a partitioned table \"%s\" "
"which is not allowed when %s is false.",
NameStr(relform->relname),
"publish_via_partition_root")));
@@ -1067,7 +1067,7 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* FIXME check pubactions vs. replica identity, to ensure the replica
- * identity is included in the column filter. Only do this for update
+ * identity is included in the column list. Only do this for update
* and delete publications. See check_publication_columns.
*
* XXX This is needed because publish_via_partition_root may change,
@@ -1261,7 +1261,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
/*
* See if the existing relation currently has a WHERE clause or a
- * column filter. We need to compare those too.
+ * column list. We need to compare those too.
*/
if (HeapTupleIsValid(rftuple))
{
@@ -1313,7 +1313,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
newrelid = RelationGetRelid(newpubrel->relation);
/*
- * If the new publication has column filter, transform it to
+ * If the new publication has column list, transform it to
* a bitmap too.
*/
if (newpubrel->columns)
@@ -1710,7 +1710,7 @@ OpenTableList(List *tables)
List *rels = NIL;
ListCell *lc;
List *relids_with_rf = NIL;
- List *relids_with_cf = NIL;
+ List *relids_with_collist = NIL;
/*
* Open, share-lock, and check all the explicitly-specified relations
@@ -1745,11 +1745,11 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
- /* Disallow duplicate tables if there are any with column filters. */
- if (t->columns || list_member_oid(relids_with_cf, myrelid))
+ /* Disallow duplicate tables if there are any with column lists. */
+ if (t->columns || list_member_oid(relids_with_collist, myrelid))
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_OBJECT),
- errmsg("conflicting or redundant column filters for table \"%s\"",
+ errmsg("conflicting or redundant column lists for table \"%s\"",
RelationGetRelationName(rel))));
table_close(rel, ShareUpdateExclusiveLock);
@@ -1767,7 +1767,7 @@ OpenTableList(List *tables)
relids_with_rf = lappend_oid(relids_with_rf, myrelid);
if (t->columns)
- relids_with_cf = lappend_oid(relids_with_cf, myrelid);
+ relids_with_collist = lappend_oid(relids_with_collist, myrelid);
/*
* Add children of this rel, if requested, so that they too are added
@@ -1809,15 +1809,15 @@ OpenTableList(List *tables)
RelationGetRelationName(rel))));
/*
- * We don't allow to specify column filter for both parent
+ * We don't allow to specify column list for both parent
* and child table at the same time as it is not very
* clear which one should be given preference.
*/
if (childrelid != myrelid &&
- (t->columns || list_member_oid(relids_with_cf, childrelid)))
+ (t->columns || list_member_oid(relids_with_collist, childrelid)))
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_OBJECT),
- errmsg("conflicting or redundant column fiters for table \"%s\"",
+ errmsg("conflicting or redundant column lists for table \"%s\"",
RelationGetRelationName(rel))));
continue;
@@ -1837,8 +1837,8 @@ OpenTableList(List *tables)
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, childrelid);
- if (t->whereClause)
- relids_with_cf = lappend_oid(relids_with_cf, childrelid);
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, childrelid);
}
}
}
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 1d8d4af341..3e282ed99a 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -592,24 +592,24 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
errmsg("cannot update table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
- else if (cmd == CMD_UPDATE && !pubdesc.cf_valid_for_update)
+ else if (cmd == CMD_UPDATE && !pubdesc.cols_valid_for_update)
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
errmsg("cannot update table \"%s\"",
RelationGetRelationName(rel)),
- errdetail("Column filter used by the publication does not cover the replica identity.")));
+ errdetail("Column list used by the publication does not cover the replica identity.")));
else if (cmd == CMD_DELETE && !pubdesc.rf_valid_for_delete)
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
errmsg("cannot delete from table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
- else if (cmd == CMD_DELETE && !pubdesc.cf_valid_for_delete)
+ else if (cmd == CMD_DELETE && !pubdesc.cols_valid_for_delete)
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
errmsg("cannot delete from table \"%s\"",
RelationGetRelationName(rel)),
- errdetail("Column filter used by the publication does not cover the replica identity.")));
+ errdetail("Column list used by the publication does not cover the replica identity.")));
/* If relation has replica identity we are always good. */
if (OidIsValid(RelationGetReplicaIndex(rel)))
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 1e8785ff9a..816d461acd 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -40,8 +40,12 @@ static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
static void logicalrep_write_namespace(StringInfo out, Oid nspid);
static const char *logicalrep_read_namespace(StringInfo in);
-#define ColumnInFilter(columns, attnum) \
- (((columns) == NULL) || (bms_is_member((attnum), (columns))))
+
+static bool
+column_in_set(int attnum, Bitmapset *columns)
+{
+ return (columns == NULL || bms_is_member(attnum, columns));
+}
/*
@@ -774,9 +778,9 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped || att->attgenerated)
continue;
- /* skip columns not included in the column filter */
- if (!ColumnInFilter(columns, att->attnum))
+ if (!column_in_set(att->attnum, columns))
continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -795,8 +799,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped || att->attgenerated)
continue;
- /* skip columns not included in the column filter */
- if (!ColumnInFilter(columns, att->attnum))
+ if (!column_in_set(att->attnum, columns))
continue;
if (isnull[i])
@@ -938,8 +941,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
if (att->attisdropped || att->attgenerated)
continue;
- /* skip columns not included in the column filter */
- if (!ColumnInFilter(columns, att->attnum))
+ if (!column_in_set(att->attnum, columns))
continue;
nliveatts++;
@@ -960,8 +962,7 @@ logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
if (att->attisdropped || att->attgenerated)
continue;
- /* skip columns not included in the column filter */
- if (!ColumnInFilter(columns, att->attnum))
+ if (!column_in_set(att->attnum, columns))
continue;
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 42708dcf82..d4d504fe02 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -754,13 +754,13 @@ fetch_remote_table_info(char *nspname, char *relname,
/*
- * Get relation's column filter expressions.
+ * Get column lists for each relation.
*
- * For initial synchronization, column filter can be ignored in following
+ * For initial synchronization, column lists can be ignored in following
* cases:
*
* 1) one of the subscribed publications for the table hasn't specified
- * any column filter
+ * any column list
*
* 2) one of the subscribed publications has puballtables set to true
*
@@ -788,7 +788,7 @@ fetch_remote_table_info(char *nspname, char *relname,
/*
* First, check if any of the publications FOR ALL TABLES? If yes, we
- * should not use any column filter. It's enough to find a single such
+ * should not use any column list. It's enough to find a single such
* publication.
*
* XXX Maybe we could combine all three steps into a single query, but
@@ -823,7 +823,7 @@ fetch_remote_table_info(char *nspname, char *relname,
/*
* If there's no FOR ALL TABLES publication, look for a FOR ALL TABLES
* IN SCHEMA publication, with schema of the remote relation. The logic
- * is the same - such publications have no column filters.
+ * is the same - such publications have no column lists.
*
* XXX Does this need any handling of partitions / publish_via_part_root?
*/
@@ -859,8 +859,8 @@ fetch_remote_table_info(char *nspname, char *relname,
/*
* If we haven't found any FOR ALL TABLES [IN SCHEMA] publications for
- * the table, we have to look for the column filters set for relations.
- * First, we check if there's a publication with no column filter for
+ * the table, we have to look for the column lists for relations.
+ * First, we check if there's a publication with no column list for
* the relation - which means all columns need to be replicated.
*/
if (!all_columns)
@@ -906,7 +906,7 @@ fetch_remote_table_info(char *nspname, char *relname,
}
/*
- * All that
+ * All that FIXME
*/
if (!all_columns)
{
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 07cdfc1d8c..b4203788b0 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1936,7 +1936,7 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
if (isnull)
{
/*
- * If we see a publication with no column filter, it
+ * If we see a publication with no column list, it
* means we need to publish all columns, so reset the
* list and ignore further ones.
*/
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 82e595396e..a2da72f0d4 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -5553,8 +5553,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
- pubdesc->cf_valid_for_update = true;
- pubdesc->cf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
return;
}
@@ -5567,8 +5567,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
- pubdesc->cf_valid_for_update = true;
- pubdesc->cf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
/* Fetch the publication membership info. */
puboids = GetRelationPublications(relid);
@@ -5620,7 +5620,7 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
*/
if (!pubform->puballtables &&
(pubform->pubupdate || pubform->pubdelete) &&
- contain_invalid_rfcolumn(pubid, relation, ancestors,
+ pub_rf_contains_invalid_column(pubid, relation, ancestors,
pubform->pubviaroot))
{
if (pubform->pubupdate)
@@ -5630,21 +5630,20 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
}
/*
- * Check if all columns referenced in the column filter are part of
- * the REPLICA IDENTITY index or not.
+ * Check if all columns are part of the REPLICA IDENTITY index or not.
*
* If the publication is FOR ALL TABLES then it means the table has no
- * column filters and we can skip the validation.
+ * column list and we can skip the validation.
*/
if (!pubform->puballtables &&
(pubform->pubupdate || pubform->pubdelete) &&
- contain_invalid_cfcolumn(pubid, relation, ancestors,
+ pub_collist_contains_invalid_column(pubid, relation, ancestors,
pubform->pubviaroot))
{
if (pubform->pubupdate)
- pubdesc->cf_valid_for_update = false;
+ pubdesc->cols_valid_for_update = false;
if (pubform->pubdelete)
- pubdesc->cf_valid_for_delete = false;
+ pubdesc->cols_valid_for_delete = false;
}
ReleaseSysCache(tup);
@@ -5660,13 +5659,13 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
break;
/*
- * If we know everything is replicated and the column filter is invalid
+ * If we know everything is replicated and the column list is invalid
* for update and delete, there is no point to check for other
* publications.
*/
if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
- !pubdesc->cf_valid_for_update && !pubdesc->cf_valid_for_delete)
+ !pubdesc->cols_valid_for_update && !pubdesc->cols_valid_for_delete)
break;
}
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index fb18cb82d9..e462ccfb74 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2941,7 +2941,7 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, " \"%s\"",
PQgetvalue(result, i, 0));
- /* column filter (if any) */
+ /* column list (if any) */
if (!PQgetisnull(result, i, 2))
appendPQExpBuffer(&buf, " (%s)",
PQgetvalue(result, i, 2));
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index b58f85ede2..a06742a620 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -87,12 +87,11 @@ typedef struct PublicationDesc
bool rf_valid_for_delete;
/*
- * true if the columns referenced in column filters which are used for UPDATE
- * or DELETE are part of the replica identity or the publication actions
+ * true if the columns are part of the replica identity or the publication actions
* do not include UPDATE or DELETE.
*/
- bool cf_valid_for_update;
- bool cf_valid_for_delete;
+ bool cols_valid_for_update;
+ bool cols_valid_for_delete;
} PublicationDesc;
typedef struct Publication
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index 08d14ca724..ae87caf089 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -31,9 +31,9 @@ extern void RemovePublicationSchemaById(Oid psoid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
extern void AlterPublicationOwner_oid(Oid pubid, Oid newOwnerId);
extern void InvalidatePublicationRels(List *relids);
-extern bool contain_invalid_rfcolumn(Oid pubid, Relation relation,
+extern bool pub_rf_contains_invalid_column(Oid pubid, Relation relation,
List *ancestors, bool pubviaroot);
-extern bool contain_invalid_cfcolumn(Oid pubid, Relation relation,
+extern bool pub_collist_contains_invalid_column(Oid pubid, Relation relation,
List *ancestors, bool pubviaroot);
#endif /* PUBLICATIONCMDS_H */
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 152f19fb42..80202f84ff 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -679,14 +679,14 @@ DROP TABLE rf_tbl_abcd_pk;
DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
--- fail - duplicate tables are not allowed if that table has any column filters
+-- fail - duplicate tables are not allowed if that table has any column lists
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
-ERROR: conflicting or redundant column filters for table "testpub_tbl1"
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
-ERROR: conflicting or redundant column filters for table "testpub_tbl1"
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
RESET client_min_messages;
--- test for column filters
+-- test for column lists
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
@@ -696,16 +696,16 @@ CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
-- error: column "x" does not exist
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
ERROR: column "x" of relation "testpub_tbl5" does not exist
--- error: replica identity "a" not included in the column filter
+-- error: replica identity "a" not included in the column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
-DETAIL: Column filter used by the publication does not cover the replica identity.
+DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: generated column "d" can't be in list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
ERROR: cannot reference generated column "d" in publication column list
--- error: system attributes "ctid" not allowed in column filter
+-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
ERROR: cannot reference system column "ctid" in publication column list
-- ok
@@ -713,24 +713,24 @@ ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
ERROR: cannot drop column "c" because it is part of publication "testpub_fortable"
HINT: Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication.
--- ok: for insert-only publication, the filter is arbitrary
+-- ok: for insert-only publication, the column list is arbitrary
ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
/* not all replica identities are good enough */
CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
--- error: replica identity (b,c) is covered by column filter (a, c)
+-- error: replica identity (b,c) is covered by column list (a, c)
UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
-DETAIL: Column filter used by the publication does not cover the replica identity.
+DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: change the replica identity to "b", and then filter (a, c) fails
+-- error: change the replica identity to "b", and then column list (a, c) fails
ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
--- error: replica identity (b,c) is not covered by column filter (a,c)
+-- error: replica identity (b,c) is not covered by column list (a, c)
UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
-DETAIL: Column filter used by the publication does not cover the replica identity.
+DETAIL: Column list used by the publication does not cover the replica identity.
/* But if upd/del are not published, it works OK */
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
@@ -744,21 +744,21 @@ ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
Tables:
"public.testpub_tbl5" (a)
--- with REPLICA IDENTITY FULL, column filters are not allowed
+-- with REPLICA IDENTITY FULL, column lists are not allowed
CREATE TABLE testpub_tbl6 (a int, b text, c text);
ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
UPDATE testpub_tbl6 SET a = 1;
ERROR: cannot update table "testpub_tbl6"
-DETAIL: Column filter used by the publication does not cover the replica identity.
+DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
UPDATE testpub_tbl6 SET a = 1;
ALTER PUBLICATION testpub_fortable ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c);
UPDATE testpub_tbl6 SET a = 1;
ERROR: cannot update table "testpub_tbl6"
-DETAIL: Column filter used by the publication does not cover the replica identity.
--- make sure changing the column filter is updated in SET TABLE
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- make sure changing the column list is updated in SET TABLE
CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
\d+ testpub_tbl7
@@ -787,7 +787,7 @@ Indexes:
Publications:
"testpub_fortable" (a, b)
--- ok: update the column filter
+-- ok: update the column list
ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
\d+ testpub_tbl7
Table "public.testpub_tbl7"
@@ -801,7 +801,7 @@ Indexes:
Publications:
"testpub_fortable" (a, c)
--- column filter for partitioned tables has to cover replica identities for
+-- column list for partitioned tables has to cover replica identities for
-- all child relations
CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
-- first partition has replica identity "a"
@@ -812,48 +812,48 @@ ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
--- ok: column filter covers both "a" and "b"
+-- ok: column list covers both "a" and "b"
SET client_min_messages = 'ERROR';
-CREATE PUBLICATION testpub_col_filter FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
RESET client_min_messages;
-- ok: the same thing, but try plain ADD TABLE
-ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
-ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
UPDATE testpub_tbl8 SET a = 1;
--- failure: column filter does not cover replica identity for the second partition
-ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
-ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, c);
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
UPDATE testpub_tbl8 SET a = 1;
ERROR: cannot update table "testpub_tbl8_1"
-DETAIL: Column filter used by the publication does not cover the replica identity.
-ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
-- failure: one of the partitions has REPLICA IDENTITY FULL
ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
-ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, c);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
UPDATE testpub_tbl8 SET a = 1;
ERROR: cannot update table "testpub_tbl8_1"
-DETAIL: Column filter used by the publication does not cover the replica identity.
-ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
-- add table and then try changing replica identity
ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
-ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
--- failure: replica identity full can't be used with a column filter
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- failure: replica identity full can't be used with a column list
ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
UPDATE testpub_tbl8 SET a = 1;
ERROR: cannot update table "testpub_tbl8_1"
-DETAIL: Column filter used by the publication does not cover the replica identity.
--- failure: replica identity has to be covered by the column filter
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: replica identity has to be covered by the column list
ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
UPDATE testpub_tbl8 SET a = 1;
ERROR: cannot update table "testpub_tbl8_1"
-DETAIL: Column filter used by the publication does not cover the replica identity.
+DETAIL: Column list used by the publication does not cover the replica identity.
DROP TABLE testpub_tbl8;
--- column filter for partitioned tables has to cover replica identities for
+-- column list for partitioned tables has to cover replica identities for
-- all child relations
CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
-ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
-- first partition has replica identity "a"
CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
@@ -862,23 +862,23 @@ ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
--- ok: attaching first partition works, because (a) is in column filter
+-- ok: attaching first partition works, because (a) is in column list
ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
--- failure: second partition has replica identity (c), which si not in column filter
+-- failure: second partition has replica identity (c), which si not in column list
ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
UPDATE testpub_tbl8 SET a = 1;
ERROR: cannot update table "testpub_tbl8_1"
-DETAIL: Column filter used by the publication does not cover the replica identity.
+DETAIL: Column list used by the publication does not cover the replica identity.
-- failure: changing replica identity to FULL for partition fails, because
--- of the column filter on the parent
+-- of the column list on the parent
ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
UPDATE testpub_tbl8 SET a = 1;
ERROR: cannot update table "testpub_tbl8_0"
-DETAIL: Column filter used by the publication does not cover the replica identity.
+DETAIL: Column list used by the publication does not cover the replica identity.
DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
-DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_filter;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
-- ======================================================
--- Test combination of column and row filter
+-- Test combination of column list and row filter
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_both_filters;
RESET client_min_messages;
@@ -908,7 +908,7 @@ Publications:
DROP TABLE testpub_tbl_both_filters;
DROP PUBLICATION testpub_both_filters;
-- ======================================================
--- More column filter tests for validating column references
+-- More column list tests for validating column references
CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
@@ -925,18 +925,18 @@ ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
-- ok - (a,b,c) coverts all PK cols
UPDATE rf_tbl_abcd_pk SET a = 1;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
--- fail - "b" is missing from the column filter
+-- fail - "b" is missing from the column list
UPDATE rf_tbl_abcd_pk SET a = 1;
ERROR: cannot update table "rf_tbl_abcd_pk"
-DETAIL: Column filter used by the publication does not cover the replica identity.
+DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
--- fail - "a" is missing from the column filter
+-- fail - "a" is missing from the column list
UPDATE rf_tbl_abcd_pk SET a = 1;
ERROR: cannot update table "rf_tbl_abcd_pk"
-DETAIL: Column filter used by the publication does not cover the replica identity.
+DETAIL: Column list used by the publication does not cover the replica identity.
-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
--- ok - there's no replica identity, so any column filter works
+-- ok - there's no replica identity, so any column list works
-- note: it fails anyway, just a bit later because UPDATE requires RI
UPDATE rf_tbl_abcd_nopk SET a = 1;
ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
@@ -945,32 +945,32 @@ HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
--- fail - with REPLICA IDENTITY FULL no column filter is allowed
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
UPDATE rf_tbl_abcd_pk SET a = 1;
ERROR: cannot update table "rf_tbl_abcd_pk"
-DETAIL: Column filter used by the publication does not cover the replica identity.
+DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
--- fail - with REPLICA IDENTITY FULL no column filter is allowed
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
UPDATE rf_tbl_abcd_nopk SET a = 1;
ERROR: cannot update table "rf_tbl_abcd_nopk"
-DETAIL: Column filter used by the publication does not cover the replica identity.
+DETAIL: Column list used by the publication does not cover the replica identity.
-- Case 3. REPLICA IDENTITY NOTHING
ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
--- ok - REPLICA IDENTITY NOTHING means all column filters are valid
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
-- it still fails later because without RI we can't replicate updates
UPDATE rf_tbl_abcd_pk SET a = 1;
ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
--- ok - REPLICA IDENTITY NOTHING means all column filters are valid
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
-- it still fails later because without RI we can't replicate updates
UPDATE rf_tbl_abcd_pk SET a = 1;
ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
--- ok - REPLICA IDENTITY NOTHING means all column filters are valid
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
-- it still fails later because without RI we can't replicate updates
UPDATE rf_tbl_abcd_nopk SET a = 1;
ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
@@ -983,20 +983,20 @@ ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
--- fail - column filter "a" does not cover the REPLICA IDENTITY INDEX on "c"
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
UPDATE rf_tbl_abcd_pk SET a = 1;
ERROR: cannot update table "rf_tbl_abcd_pk"
-DETAIL: Column filter used by the publication does not cover the replica identity.
+DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
--- ok - column filter "c" does cover the REPLICA IDENTITY INDEX on "c"
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
UPDATE rf_tbl_abcd_pk SET a = 1;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
--- fail - column filter "a" does not cover the REPLICA IDENTITY INDEX on "c"
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
UPDATE rf_tbl_abcd_nopk SET a = 1;
ERROR: cannot update table "rf_tbl_abcd_nopk"
-DETAIL: Column filter used by the publication does not cover the replica identity.
+DETAIL: Column list used by the publication does not cover the replica identity.
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
--- ok - column filter "c" does cover the REPLICA IDENTITY INDEX on "c"
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
UPDATE rf_tbl_abcd_nopk SET a = 1;
-- Tests for partitioned table
-- set PUBLISH_VIA_PARTITION_ROOT to false and test row filter for partitioned
@@ -1021,7 +1021,7 @@ UPDATE rf_tbl_abcd_part_pk SET a = 1;
-- used for partitioned table
ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
ERROR: cannot set publish_via_partition_root = false for publication "testpub6"
-DETAIL: The publication contains a column filter for a partitioned table "rf_tbl_abcd_part_pk" which is not allowed when publish_via_partition_root is false.
+DETAIL: The publication contains a column list for a partitioned table "rf_tbl_abcd_part_pk" which is not allowed when publish_via_partition_root is false.
-- Now change the root filter to use a column "b"
-- (which is not in the replica identity)
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
@@ -1030,7 +1030,7 @@ ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
-- fail - "b" is not in REPLICA IDENTITY INDEX
UPDATE rf_tbl_abcd_part_pk SET a = 1;
ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
-DETAIL: Column filter used by the publication does not cover the replica identity.
+DETAIL: Column list used by the publication does not cover the replica identity.
-- set PUBLISH_VIA_PARTITION_ROOT to true
-- can use row filter for partitioned table
ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
@@ -1039,7 +1039,7 @@ ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
-- fail - "b" is not in REPLICA IDENTITY INDEX
UPDATE rf_tbl_abcd_part_pk SET a = 1;
ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
-DETAIL: Column filter used by the publication does not cover the replica identity.
+DETAIL: Column list used by the publication does not cover the replica identity.
DROP PUBLICATION testpub6;
DROP TABLE rf_tbl_abcd_pk;
DROP TABLE rf_tbl_abcd_nopk;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 2203dc238d..32a810b2d2 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -381,13 +381,13 @@ CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
--- fail - duplicate tables are not allowed if that table has any column filters
+-- fail - duplicate tables are not allowed if that table has any column lists
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
RESET client_min_messages;
--- test for column filters
+-- test for column lists
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
@@ -396,32 +396,32 @@ CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
d int generated always as (a + length(b)) stored);
-- error: column "x" does not exist
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
--- error: replica identity "a" not included in the column filter
+-- error: replica identity "a" not included in the column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
-- error: generated column "d" can't be in list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
--- error: system attributes "ctid" not allowed in column filter
+-- error: system attributes "ctid" not allowed in column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
-- ok
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
--- ok: for insert-only publication, the filter is arbitrary
+-- ok: for insert-only publication, the column list is arbitrary
ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
/* not all replica identities are good enough */
CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
--- error: replica identity (b,c) is covered by column filter (a, c)
+-- error: replica identity (b,c) is covered by column list (a, c)
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
--- error: change the replica identity to "b", and then filter (a, c) fails
+-- error: change the replica identity to "b", and then column list (a, c) fails
ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
--- error: replica identity (b,c) is not covered by column filter (a,c)
+-- error: replica identity (b,c) is not covered by column list (a, c)
UPDATE testpub_tbl5 SET a = 1;
/* But if upd/del are not published, it works OK */
@@ -431,7 +431,7 @@ CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
\dRp+ testpub_table_ins
--- with REPLICA IDENTITY FULL, column filters are not allowed
+-- with REPLICA IDENTITY FULL, column lists are not allowed
CREATE TABLE testpub_tbl6 (a int, b text, c text);
ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
@@ -445,18 +445,18 @@ CREATE TABLE testpub_tbl6 (a int, b text, c text);
ALTER PUBLICATION testpub_fortable ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c);
UPDATE testpub_tbl6 SET a = 1;
--- make sure changing the column filter is updated in SET TABLE
+-- make sure changing the column list is updated in SET TABLE
CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
\d+ testpub_tbl7
-- ok: we'll skip this table
ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
\d+ testpub_tbl7
--- ok: update the column filter
+-- ok: update the column list
ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
\d+ testpub_tbl7
--- column filter for partitioned tables has to cover replica identities for
+-- column list for partitioned tables has to cover replica identities for
-- all child relations
CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
-- first partition has replica identity "a"
@@ -468,37 +468,37 @@ CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2
ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
--- ok: column filter covers both "a" and "b"
+-- ok: column list covers both "a" and "b"
SET client_min_messages = 'ERROR';
-CREATE PUBLICATION testpub_col_filter FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
RESET client_min_messages;
-- ok: the same thing, but try plain ADD TABLE
-ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
-ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
UPDATE testpub_tbl8 SET a = 1;
--- failure: column filter does not cover replica identity for the second partition
-ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
-ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, c);
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
UPDATE testpub_tbl8 SET a = 1;
-ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
-- failure: one of the partitions has REPLICA IDENTITY FULL
ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
-ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, c);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
UPDATE testpub_tbl8 SET a = 1;
-ALTER PUBLICATION testpub_col_filter DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
-- add table and then try changing replica identity
ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
-ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
--- failure: replica identity full can't be used with a column filter
+-- failure: replica identity full can't be used with a column list
ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
UPDATE testpub_tbl8 SET a = 1;
--- failure: replica identity has to be covered by the column filter
+-- failure: replica identity has to be covered by the column list
ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
@@ -506,10 +506,10 @@ CREATE PUBLICATION testpub_col_filter FOR TABLE testpub_tbl8 (a, b) WITH (publis
DROP TABLE testpub_tbl8;
--- column filter for partitioned tables has to cover replica identities for
+-- column list for partitioned tables has to cover replica identities for
-- all child relations
CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
-ALTER PUBLICATION testpub_col_filter ADD TABLE testpub_tbl8 (a, b);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
-- first partition has replica identity "a"
CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
@@ -519,22 +519,22 @@ CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
--- ok: attaching first partition works, because (a) is in column filter
+-- ok: attaching first partition works, because (a) is in column list
ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
--- failure: second partition has replica identity (c), which si not in column filter
+-- failure: second partition has replica identity (c), which si not in column list
ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
UPDATE testpub_tbl8 SET a = 1;
-- failure: changing replica identity to FULL for partition fails, because
--- of the column filter on the parent
+-- of the column list on the parent
ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
UPDATE testpub_tbl8 SET a = 1;
DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
-DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_filter;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
-- ======================================================
--- Test combination of column and row filter
+-- Test combination of column list and row filter
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_both_filters;
RESET client_min_messages;
@@ -548,7 +548,7 @@ CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
DROP PUBLICATION testpub_both_filters;
-- ======================================================
--- More column filter tests for validating column references
+-- More column list tests for validating column references
CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
@@ -567,15 +567,15 @@ CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
-- ok - (a,b,c) coverts all PK cols
UPDATE rf_tbl_abcd_pk SET a = 1;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
--- fail - "b" is missing from the column filter
+-- fail - "b" is missing from the column list
UPDATE rf_tbl_abcd_pk SET a = 1;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
--- fail - "a" is missing from the column filter
+-- fail - "a" is missing from the column list
UPDATE rf_tbl_abcd_pk SET a = 1;
-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
--- ok - there's no replica identity, so any column filter works
+-- ok - there's no replica identity, so any column list works
-- note: it fails anyway, just a bit later because UPDATE requires RI
UPDATE rf_tbl_abcd_nopk SET a = 1;
@@ -583,25 +583,25 @@ CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
--- fail - with REPLICA IDENTITY FULL no column filter is allowed
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
UPDATE rf_tbl_abcd_pk SET a = 1;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
--- fail - with REPLICA IDENTITY FULL no column filter is allowed
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
UPDATE rf_tbl_abcd_nopk SET a = 1;
-- Case 3. REPLICA IDENTITY NOTHING
ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
--- ok - REPLICA IDENTITY NOTHING means all column filters are valid
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
-- it still fails later because without RI we can't replicate updates
UPDATE rf_tbl_abcd_pk SET a = 1;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
--- ok - REPLICA IDENTITY NOTHING means all column filters are valid
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
-- it still fails later because without RI we can't replicate updates
UPDATE rf_tbl_abcd_pk SET a = 1;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
--- ok - REPLICA IDENTITY NOTHING means all column filters are valid
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
-- it still fails later because without RI we can't replicate updates
UPDATE rf_tbl_abcd_nopk SET a = 1;
@@ -613,16 +613,16 @@ CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
--- fail - column filter "a" does not cover the REPLICA IDENTITY INDEX on "c"
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
UPDATE rf_tbl_abcd_pk SET a = 1;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
--- ok - column filter "c" does cover the REPLICA IDENTITY INDEX on "c"
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
UPDATE rf_tbl_abcd_pk SET a = 1;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
--- fail - column filter "a" does not cover the REPLICA IDENTITY INDEX on "c"
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
UPDATE rf_tbl_abcd_nopk SET a = 1;
ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
--- ok - column filter "c" does cover the REPLICA IDENTITY INDEX on "c"
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
UPDATE rf_tbl_abcd_nopk SET a = 1;
-- Tests for partitioned table
diff --git a/src/test/subscription/t/029_column_list.pl b/src/test/subscription/t/029_column_list.pl
index ec2c8a789a..e5ab5d5731 100644
--- a/src/test/subscription/t/029_column_list.pl
+++ b/src/test/subscription/t/029_column_list.pl
@@ -5,7 +5,7 @@
use warnings;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
-use Test::More tests => 26;
+use Test::More;
# setup
@@ -290,8 +290,8 @@ sub wait_for_subscription_sync
2|22|2222),
'overlapping publications with overlapping column lists');
-# and finally, set the column filter to ALL for one of the publications,
-# which means replicating all columns (removing the column filter), but
+# and finally, set the column list to ALL for one of the publications,
+# which means replicating all columns (removing the column list), but
# first add the missing column to the table on subscriber
$node_publisher->safe_psql('postgres', qq(
ALTER PUBLICATION pub3 ALTER TABLE tab5 SET COLUMNS ALL;
@@ -314,9 +314,9 @@ sub wait_for_subscription_sync
3|33|3333|333),
'overlapping publications with overlapping column lists');
-# TEST: create a table with a column filter, then change the replica
+# TEST: create a table with a column list, then change the replica
# identity by replacing a primary key (but use a different column in
-# the column filter)
+# the column list)
$node_publisher->safe_psql('postgres', qq(
CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
CREATE PUBLICATION pub4 FOR TABLE tab6 (a, b);
@@ -377,9 +377,9 @@ sub wait_for_subscription_sync
'replication with the modified primary key');
-# TEST: create a table with a column filter, then change the replica
+# TEST: create a table with a column list, then change the replica
# identity by replacing a primary key with a key on multiple columns
-# (all of them covered by the column filter)
+# (all of them covered by the column list)
$node_publisher->safe_psql('postgres', qq(
CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
CREATE PUBLICATION pub5 FOR TABLE tab7 (a, b);
@@ -425,7 +425,7 @@ sub wait_for_subscription_sync
'replication with the modified primary key');
# now switch the primary key again to another columns not covered by the
-# column filter, but also generate writes between the drop and creation
+# column list, but also generate writes between the drop and creation
# of the new constraint
$node_publisher->safe_psql('postgres', qq(
@@ -450,11 +450,11 @@ sub wait_for_subscription_sync
# TEST: partitioned tables (with publish_via_partition_root = false)
# and replica identity. The (leaf) partitions may have different RI, so
-# we need to check the partition RI (with respect to the column filter)
+# we need to check the partition RI (with respect to the column list)
# while attaching the partition.
# First, let's create a partitioned table with two partitions, each with
-# a different RI, but a column filter not covering all those RI.
+# a different RI, but a column list not covering all those RI.
$node_publisher->safe_psql('postgres', qq(
CREATE TABLE test_part_a (a int, b int, c int) PARTITION BY LIST (a);
@@ -508,8 +508,8 @@ sub wait_for_subscription_sync
2|4),
'partitions with different replica identities not replicated correctly');
-# This time start with a column filter covering RI for all partitions, but
-# then update the column filter to not cover column "b" (needed by the
+# This time start with a column list covering RI for all partitions, but
+# then update the column list to not cover column "b" (needed by the
# second partition)
$node_publisher->safe_psql('postgres', qq(
@@ -563,9 +563,9 @@ sub wait_for_subscription_sync
'partitions with different replica identities not replicated correctly');
-# TEST: This time start with a column filter covering RI for all partitions,
+# TEST: This time start with a column list covering RI for all partitions,
# but then update RI for one of the partitions to not be covered by the
-# column filter anymore.
+# column list anymore.
$node_publisher->safe_psql('postgres', qq(
CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
@@ -594,7 +594,7 @@ sub wait_for_subscription_sync
# create a publication replicating data through partition root, with a column
# filter on the root, and then add the partitions one by one with separate
-# column filters (but those are not applied)
+# column lists (but those are not applied)
$node_publisher->safe_psql('postgres', qq(
CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a,c);
@@ -623,8 +623,8 @@ sub wait_for_subscription_sync
# create a publication not replicating data through partition root, without
-# a column filter on the root, and then add the partitions one by one with
-# separate column filters
+# a column list on the root, and then add the partitions one by one with
+# separate column lists
$node_publisher->safe_psql('postgres', qq(
DROP PUBLICATION pub8;
CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
@@ -703,7 +703,7 @@ sub wait_for_subscription_sync
'partitions with different replica identities not replicated correctly');
# TEST: With a table included in multiple publications, we should use a
-# union of the column filters. So with column filters (a,b) and (a,c) we
+# union of the column lists. So with column lists (a,b) and (a,c) we
# should replicate (a,b,c).
$node_publisher->safe_psql('postgres', qq(
@@ -727,11 +727,11 @@ sub wait_for_subscription_sync
is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_1"),
qq(1|2|3),
- 'a mix of publications should use a union of column filter');
+ 'a mix of publications should use a union of column list');
# TEST: With a table included in multiple publications, we should use a
-# union of the column filters. If any of the publications is FOR ALL
+# union of the column lists. If any of the publications is FOR ALL
# TABLES, we should replicate all columns.
# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
@@ -762,11 +762,11 @@ sub wait_for_subscription_sync
is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_2"),
qq(1|2|3),
- 'a mix of publications should use a union of column filter');
+ 'a mix of publications should use a union of column list');
# TEST: With a table included in multiple publications, we should use a
-# union of the column filters. If any of the publications is FOR ALL
+# union of the column lists. If any of the publications is FOR ALL
# TABLES IN SCHEMA, we should replicate all columns.
$node_publisher->safe_psql('postgres', qq(
@@ -790,11 +790,11 @@ sub wait_for_subscription_sync
is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_3"),
qq(1|2|3),
- 'a mix of publications should use a union of column filter');
+ 'a mix of publications should use a union of column list');
# TEST: Check handling of publish_via_partition_root - if a partition is
-# published through partition root, we should only apply the column filter
+# published through partition root, we should only apply the column list
# defined for the whole table (not the partitions) - both during the initial
# sync and when replicating changes. This is what we do for row filters.
@@ -828,7 +828,7 @@ sub wait_for_subscription_sync
is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_root ORDER BY a, b, c"),
qq(1||
10||),
- 'publication via partition root applies column filter');
+ 'publication via partition root applies column list');
$node_subscriber->stop('fast');
$node_publisher->stop('fast');
--
2.35.1
On Mon, Mar 7, 2022 at 8:48 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 3/4/22 11:42, Amit Kapila wrote:
On Wed, Mar 2, 2022 at 5:43 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Attached is an updated patch, addressing most of the issues reported so
far. There are various minor tweaks, but the main changes are:...
3) checks of column filter vs. publish_via_partition_root and replica
identity, following the same logic as the row-filter patch (hopefully,
it touches the same places, using the same logic, ...)That means - with "publish_via_partition_root=false" it's not allowed to
specify column filters on partitioned tables, only for leaf partitions.And we check column filter vs. replica identity when adding tables to
publications, or whenever we change the replica identity.This handling is different from row filter work and I see problems
with it.By different, I assume you mean I tried to enfoce the rules in ALTER
PUBLICATION and other ALTER commands, instead of when modifying the
data?
Yes.
OK, I reworked this to do the same thing as the row filtering patch.
Thanks, I'll check this.
The column list validation w.r.t primary key (default replica
identity) is missing. The handling of column list vs. partitions has
multiple problems: (a) In attach partition, the patch is just checking
ancestors for RI validation but what if the table being attached has
further subpartitions; (b) I think the current locking also seems to
have problems because it is quite possible that while it validates the
ancestors here, concurrently someone changes the column list. I think
it won't be enough to just change the locking mode because with the
current patch strategy during attach, we will be first taking locks
for child tables of current partition and then parent tables which can
pose deadlock hazards.The columns list validation also needs to be done when we change
publication action.
I believe those issues should be solved by adopting the same approach as
the row-filtering patch, right?
Right.
Some other miscellaneous comments:
=============================
*
In get_rel_sync_entry(), the handling for partitioned tables doesn't
seem to be correct. It can publish a different set of columns based on
the order of publications specified in the subscription.For example:
----
create table parent (a int, b int, c int) partition by range (a);
create table test_part1 (like parent);
alter table parent attach partition test_part1 for values from (1) to (10);create publication pub for table parent(a) with (PUBLISH_VIA_PARTITION_ROOT);
create publication pub2 for table test_part1(b);
---Now, depending on the order of publications in the list while defining
subscription, the column list will change
----
create subscription sub connection 'port=10000 dbname=postgres'
publication pub, pub2;For the above, column list will be: (a)
create subscription sub connection 'port=10000 dbname=postgres'
publication pub2, pub;For this one, the column list will be: (a, b)
----To avoid this, the column list should be computed based on the final
publish_as_relid as we are doing for the row filter.Hmm, yeah. That seems like a genuine problem - it should not depend on
the order of publications in the subscription, I guess.But is it an issue in the patch? Isn't that a pre-existing issue? AFAICS
the problem is that we initialize publish_as_relid=relid before the loop
over publications, and then just update it. So the first iteration
starts with relid, but the second iteration ends with whatever value is
set by the first iteration (e.g. the root).So with the example you posted, we start with
publish_as_relid = relid = test_part1
but then if the first publication is pubviaroot=true, we update it to
parent. And in the second iteration, we fail to find the column filter,
because "parent" (publish_as_relid) is not part of the pub2.If we do it in the other order, we leave the publish_as_relid value as
is (and find the filter), and then update it in the second iteration
(and find the column filter too).Now, this can be resolved by re-calculating the publish_as_relid from
scratch in each iteration (start with relid, then maybe update it). But
that's just half the story - the issue is there even without column
filters. Consider this example:create table t (a int, b int, c int) partition by range (a);
create table t_1 partition of t for values from (1) to (10)
partition by range (a);create table t_2 partition of t_1 for values from (1) to (10);
create publication pub1 for table t(a)
with (PUBLISH_VIA_PARTITION_ROOT);create publication pub2 for table t_1(a)
with (PUBLISH_VIA_PARTITION_ROOT);Now, is you change subscribe to "pub1, pub2" and "pub2, pub1", we'll end
up with different publish_as_relid values (t or t_1). Which seems like
the same ambiguity issue.
I think we should fix this existing problem by always using the
top-most table as publish_as_relid. Basically, we can check, if the
existing publish_as_relid is an ancestor of a new rel that is going to
replace it then we shouldn't replace it. However, I think even if we
fix the existing problem, we will still have the order problem for the
column filter patch, and to avoid that instead of fetching column
filters in the publication loop, we should use the final
publish_as_relid. I think it will have another problem as well if we
don't use final publish_as_relid which is that sometimes when we
should not use any filter (say when pubviaroot is true and that
publication has root partitioned table which has no filter) as per our
rule of filters for a partitioned table, it can still use some filter
from the non-root table.
*
Fetching column filter info in tablesync.c is quite expensive. It
seems to be using four round-trips to get the complete info whereas
for row-filter we use just one round trip. I think we should try to
get both row filter and column filter info in just one round trip.Maybe, but I really don't think this is an issue.
I am not sure but it might matter for small tables. Leaving aside the
performance issue, I think the current way will get the wrong column
list in many cases: (a) The ALL TABLES IN SCHEMA case handling won't
work for partitioned tables when the partitioned table is part of one
schema and partition table is part of another schema. (b) The handling
of partition tables in other cases will fetch incorrect lists as it
tries to fetch the column list of all the partitions in the hierarchy.
One of my colleagues has even tested these cases both for column
filters and row filters and we find the behavior of row filter is okay
whereas for column filter it uses the wrong column list. We will share
the tests and results with you in a later email. We are trying to
unify the column filter queries with row filter to make their behavior
the same and will share the findings once it is done. I hope if we are
able to achieve this that we will reduce the chances of bugs in this
area.
Note: I think the first two patches for tests are not required after
commit ceb57afd3c.
--
With Regards,
Amit Kapila.
On Wednesday, March 9, 2022 6:04 PM Amit Kapila <amit.kapila16@gmail.com>
On Mon, Mar 7, 2022 at 8:48 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 3/4/22 11:42, Amit Kapila wrote:
*
Fetching column filter info in tablesync.c is quite expensive. It
seems to be using four round-trips to get the complete info whereas
for row-filter we use just one round trip. I think we should try to
get both row filter and column filter info in just one round trip.Maybe, but I really don't think this is an issue.
I am not sure but it might matter for small tables. Leaving aside the
performance issue, I think the current way will get the wrong column list in
many cases: (a) The ALL TABLES IN SCHEMA case handling won't work for
partitioned tables when the partitioned table is part of one schema and
partition table is part of another schema. (b) The handling of partition tables in
other cases will fetch incorrect lists as it tries to fetch the column list of all the
partitions in the hierarchy.One of my colleagues has even tested these cases both for column filters and
row filters and we find the behavior of row filter is okay whereas for column
filter it uses the wrong column list. We will share the tests and results with you
in a later email. We are trying to unify the column filter queries with row filter to
make their behavior the same and will share the findings once it is done. I hope
if we are able to achieve this that we will reduce the chances of bugs in this area.Note: I think the first two patches for tests are not required after commit
ceb57afd3c.
Hi,
Here are some tests and results about the table sync query of
column filter patch and row filter.
1) multiple publications which publish schema of parent table and partition.
----pub
create schema s1;
create table s1.t (a int, b int, c int) partition by range (a);
create table t_1 partition of s1.t for values from (1) to (10);
create publication pub1 for all tables in schema s1;
create publication pub2 for table t_1(b);
----sub
- prepare tables
CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub1, pub2;
When doing table sync for 't_1', the column list will be (b). I think it should
be no filter because table t_1 is also published via ALL TABLES IN SCHMEA
publication.
For Row Filter, it will use no filter for this case.
2) one publication publishes both parent and child
----pub
create table t (a int, b int, c int) partition by range (a);
create table t_1 partition of t for values from (1) to (10)
partition by range (a);
create table t_2 partition of t_1 for values from (1) to (10);
create publication pub2 for table t_1(a), t_2
with (PUBLISH_VIA_PARTITION_ROOT);
----sub
- prepare tables
CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;
When doing table sync for table 't_1', it has no column list. I think the
expected column list is (a).
For Row Filter, it will use the row filter of the top most parent table(t_1) in
this case.
3) one publication publishes both parent and child
----pub
create table t (a int, b int, c int) partition by range (a);
create table t_1 partition of t for values from (1) to (10)
partition by range (a);
create table t_2 partition of t_1 for values from (1) to (10);
create publication pub2 for table t_1(a), t_2(b)
with (PUBLISH_VIA_PARTITION_ROOT);
----sub
- prepare tables
CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;
When doing table sync for table 't_1', the column list would be (a, b). I think
the expected column list is (a).
For Row Filter, it will use the row filter of the top most parent table(t_1) in
this case.
Best regards,
Hou zj
On 3/9/22 11:03, Amit Kapila wrote:
...
Hmm, yeah. That seems like a genuine problem - it should not depend on
the order of publications in the subscription, I guess.But is it an issue in the patch? Isn't that a pre-existing issue? AFAICS
the problem is that we initialize publish_as_relid=relid before the loop
over publications, and then just update it. So the first iteration
starts with relid, but the second iteration ends with whatever value is
set by the first iteration (e.g. the root).So with the example you posted, we start with
publish_as_relid = relid = test_part1
but then if the first publication is pubviaroot=true, we update it to
parent. And in the second iteration, we fail to find the column filter,
because "parent" (publish_as_relid) is not part of the pub2.If we do it in the other order, we leave the publish_as_relid value as
is (and find the filter), and then update it in the second iteration
(and find the column filter too).Now, this can be resolved by re-calculating the publish_as_relid from
scratch in each iteration (start with relid, then maybe update it). But
that's just half the story - the issue is there even without column
filters. Consider this example:create table t (a int, b int, c int) partition by range (a);
create table t_1 partition of t for values from (1) to (10)
partition by range (a);create table t_2 partition of t_1 for values from (1) to (10);
create publication pub1 for table t(a)
with (PUBLISH_VIA_PARTITION_ROOT);create publication pub2 for table t_1(a)
with (PUBLISH_VIA_PARTITION_ROOT);Now, is you change subscribe to "pub1, pub2" and "pub2, pub1", we'll end
up with different publish_as_relid values (t or t_1). Which seems like
the same ambiguity issue.I think we should fix this existing problem by always using the
top-most table as publish_as_relid. Basically, we can check, if the
existing publish_as_relid is an ancestor of a new rel that is going to
replace it then we shouldn't replace it.
Right, using the topmost relid from all publications seems like the
correct solution.
However, I think even if we
fix the existing problem, we will still have the order problem for the
column filter patch, and to avoid that instead of fetching column
filters in the publication loop, we should use the final
publish_as_relid. I think it will have another problem as well if we
don't use final publish_as_relid which is that sometimes when we
should not use any filter (say when pubviaroot is true and that
publication has root partitioned table which has no filter) as per our
rule of filters for a partitioned table, it can still use some filter
from the non-root table.
Yeah, the current behavior is just a consequence of how we determine
publish_as_relid now. If we rework that, we should first determine the
relid and then fetch the filter only for that single rel.
*
Fetching column filter info in tablesync.c is quite expensive. It
seems to be using four round-trips to get the complete info whereas
for row-filter we use just one round trip. I think we should try to
get both row filter and column filter info in just one round trip.Maybe, but I really don't think this is an issue.
I am not sure but it might matter for small tables. Leaving aside the
performance issue, I think the current way will get the wrong column
list in many cases: (a) The ALL TABLES IN SCHEMA case handling won't
work for partitioned tables when the partitioned table is part of one
schema and partition table is part of another schema. (b) The handling
of partition tables in other cases will fetch incorrect lists as it
tries to fetch the column list of all the partitions in the hierarchy.One of my colleagues has even tested these cases both for column
filters and row filters and we find the behavior of row filter is okay
whereas for column filter it uses the wrong column list. We will share
the tests and results with you in a later email. We are trying to
unify the column filter queries with row filter to make their behavior
the same and will share the findings once it is done. I hope if we are
able to achieve this that we will reduce the chances of bugs in this
area.
OK, I'll take a look at that email.
Note: I think the first two patches for tests are not required after
commit ceb57afd3c.
Right. Will remove.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Wed, Mar 9, 2022 at 3:33 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Mar 7, 2022 at 8:48 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:OK, I reworked this to do the same thing as the row filtering patch.
Thanks, I'll check this.
Some assorted comments:
=====================
1. We don't need to send a column list for the old tuple in case of an
update (similar to delete). It is not required to apply a column
filter for those cases because we ensure that RI must be part of the
column list for updates and deletes.
2.
+ /*
+ * Check if all columns referenced in the column filter are part of
+ * the REPLICA IDENTITY index or not.
I think this comment is reverse. The rule we follow here is that
attributes that are part of RI must be there in a specified column
list. This is used at two places in the patch.
3. get_rel_sync_entry()
{
/* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
...
}
Similar to the row filter, I think we need to use
entry->cache_expr_cxt to allocate this. There are other usages of
CacheMemoryContext in this part of the code but I think those need to
be also changed and we can do that as a separate patch. If we do the
suggested change then we don't need to separately free columns.
4. I think we don't need the DDL changes in AtExecDropColumn. Instead,
we can change the dependency of columns to NORMAL during publication
commands.
5. There is a reference to check_publication_columns but that function
is removed from the patch.
6.
/*
* If we know everything is replicated and the row filter is invalid
* for update and delete, there is no point to check for other
* publications.
*/
if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
!pubdesc->rf_valid_for_update && !pubdesc->rf_valid_for_delete)
break;
/*
* If we know everything is replicated and the column filter is invalid
* for update and delete, there is no point to check for other
* publications.
*/
if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
!pubdesc->cf_valid_for_update && !pubdesc->cf_valid_for_delete)
break;
Can we combine these two checks?
I feel this patch needs a more thorough review.
--
With Regards,
Amit Kapila.
On 3/9/22 11:12, houzj.fnst@fujitsu.com wrote:
Hi,
Here are some tests and results about the table sync query of
column filter patch and row filter.1) multiple publications which publish schema of parent table and partition.
----pub
create schema s1;
create table s1.t (a int, b int, c int) partition by range (a);
create table t_1 partition of s1.t for values from (1) to (10);
create publication pub1 for all tables in schema s1;
create publication pub2 for table t_1(b);----sub
- prepare tables
CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub1, pub2;When doing table sync for 't_1', the column list will be (b). I think it should
be no filter because table t_1 is also published via ALL TABLES IN SCHMEA
publication.For Row Filter, it will use no filter for this case.
2) one publication publishes both parent and child
----pub
create table t (a int, b int, c int) partition by range (a);
create table t_1 partition of t for values from (1) to (10)
partition by range (a);
create table t_2 partition of t_1 for values from (1) to (10);create publication pub2 for table t_1(a), t_2
with (PUBLISH_VIA_PARTITION_ROOT);----sub
- prepare tables
CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;When doing table sync for table 't_1', it has no column list. I think the
expected column list is (a).For Row Filter, it will use the row filter of the top most parent table(t_1) in
this case.3) one publication publishes both parent and child
----pub
create table t (a int, b int, c int) partition by range (a);
create table t_1 partition of t for values from (1) to (10)
partition by range (a);
create table t_2 partition of t_1 for values from (1) to (10);create publication pub2 for table t_1(a), t_2(b)
with (PUBLISH_VIA_PARTITION_ROOT);----sub
- prepare tables
CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;When doing table sync for table 't_1', the column list would be (a, b). I think
the expected column list is (a).For Row Filter, it will use the row filter of the top most parent table(t_1) in
this case.
Attached is an updated patch version, addressing all of those issues.
0001 is a bugfix, reworking how we calculate publish_as_relid. The old
approach was unstable with multiple publications, giving different
results depending on order of the publications. This should be
backpatched into PG13 where publish_via_partition_root was introduced, I
think.
0002 is the main patch, merging the changes proposed by Peter and fixing
the issues reported here. In most cases this means adopting the code
used for row filters, and perhaps simplifying it a bit.
But I also tried to implement a row-filter test for 0001, and I'm not
sure I understand the behavior I observe. Consider this:
-- a chain of 3 partitions (on both publisher and subscriber)
CREATE TABLE test_part_rf (a int primary key, b int, c int)
PARTITION BY LIST (a);
CREATE TABLE test_part_rf_1
PARTITION OF test_part_rf FOR VALUES IN (1,2,3,4,5)
PARTITION BY LIST (a);
CREATE TABLE test_part_rf_2
PARTITION OF test_part_rf_1 FOR VALUES IN (1,2,3,4,5);
-- initial data
INSERT INTO test_part_rf VALUES (1, 5, 100);
INSERT INTO test_part_rf VALUES (2, 15, 200);
-- two publications, each adding a different partition
CREATE PUBLICATION test_pub_part_1 FOR TABLE test_part_rf_1
WHERE (b < 10) WITH (publish_via_partition_root);
CREATE PUBLICATION test_pub_part_2 FOR TABLE test_part_rf_2
WHERE (b > 10) WITH (publish_via_partition_root);
-- now create the subscription (also try opposite ordering)
CREATE SUBSCRIPTION test_part_sub CONNECTION '...'
PUBLICATION test_pub_part_1, test_pub_part_2;
-- wait for sync
-- inert some more data
INSERT INTO test_part_rf VALUES (3, 6, 300);
INSERT INTO test_part_rf VALUES (4, 16, 400);
-- wait for catchup
Now, based on the discussion here, my expectation is that we'll use the
row filter from the top-most ancestor in any publication, which in this
case is test_part_rf_1. Hence the filter should be (b < 10).
So I'd expect these rows to be replicated:
1,5,100
3,6,300
But that's not what I get, unfortunately. I get different results,
depending on the order of publications:
1) test_pub_part_1, test_pub_part_2
1|5|100
2|15|200
3|6|300
4|16|400
2) test_pub_part_2, test_pub_part_1
3|6|300
4|16|400
That seems pretty bizarre, because it either means we're not enforcing
any filter or some strange combination of filters (notice that for (2)
we skip/replicate rows matching either filter).
I have to be missing something important, but this seems confusing.
There's a patch adding a simple test case to 028_row_filter.sql (named
.txt, so as not to confuse cfbot).
FWIW I'm not convinced applying just the filters (both row and column)
is the right approach. It might be OK for a single publication, but with
multiple publications not so much. If you list multiple publications for
a subscription, it seems natural to expect a union of all the data, a
bit as if there were multiple subscriptions. But what you actually get
is some subset, depending on what other relations the other publications
include.
Of course, this only happens if the publications include different
ancestors. If all include the same ancestor, everything works fine and
you get the "union" of data.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
row-filter-test.txttext/plain; charset=UTF-8; name=row-filter-test.txtDownload
diff --git a/src/test/subscription/t/028_row_filter.pl b/src/test/subscription/t/028_row_filter.pl
index 89bb364e9da..333895e081b 100644
--- a/src/test/subscription/t/028_row_filter.pl
+++ b/src/test/subscription/t/028_row_filter.pl
@@ -691,6 +691,75 @@ is($result, qq(t|1), 'check replicated rows to tab_rowfilter_toast');
# Testcase end: FOR TABLE with row filter publications
# ======================================================
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_rf (a int primary key, b int, c int) PARTITION BY LIST (a);
+ CREATE TABLE test_part_rf_1 PARTITION OF test_part_rf FOR VALUES IN (1,2,3,4,5) PARTITION BY LIST (a);
+ CREATE TABLE test_part_rf_2 PARTITION OF test_part_rf_1 FOR VALUES IN (1,2,3,4,5);
+
+ CREATE PUBLICATION test_pub_part_1 FOR TABLE test_part_rf_1 WHERE (b < 10) WITH (publish_via_partition_root);
+ CREATE PUBLICATION test_pub_part_2 FOR TABLE test_part_rf_2 WHERE (b > 10) WITH (publish_via_partition_root);
+
+ INSERT INTO test_part_rf VALUES (1, 5, 100);
+ INSERT INTO test_part_rf VALUES (2, 15, 200);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_rf (a int primary key, b int, c int) PARTITION BY LIST (a);
+ CREATE TABLE test_part_rf_1 PARTITION OF test_part_rf FOR VALUES IN (1,2,3,4,5) PARTITION BY LIST (a);
+ CREATE TABLE test_part_rf_2 PARTITION OF test_part_rf_1 FOR VALUES IN (1,2,3,4,5);
+
+ CREATE SUBSCRIPTION test_part_sub CONNECTION '$publisher_connstr' PUBLICATION test_pub_part_1, test_pub_part_2;
+));
+
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_rf VALUES (3, 6, 300);
+ INSERT INTO test_part_rf VALUES (4, 16, 400);
+));
+
+$node_publisher->wait_for_catchup('test_part_sub');
+
+$result = $node_subscriber->safe_psql('postgres', qq(SELECT * FROM test_part_rf ORDER BY a));
+is($result,
+ qq(1|5|100
+3|6|300), 'check replicated rows with multiple row filters');
+
+
+$node_publisher->safe_psql('postgres',
+ qq(TRUNCATE test_part_rf));
+
+$node_publisher->wait_for_catchup('test_part_sub');
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SUBSCRIPTION test_part_sub));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_rf VALUES (1, 5, 100);
+ INSERT INTO test_part_rf VALUES (2, 15, 200);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION test_part_sub CONNECTION '$publisher_connstr' PUBLICATION test_pub_part_2, test_pub_part_1;
+));
+
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+$node_publisher->safe_psql('postgres', qq(
+ TRUNCATE test_part_rf;
+ INSERT INTO test_part_rf VALUES (3, 6, 300);
+ INSERT INTO test_part_rf VALUES (4, 16, 400);
+));
+
+$node_publisher->wait_for_catchup('test_part_sub');
+
+$result = $node_subscriber->safe_psql('postgres', qq(SELECT * FROM test_part_rf ORDER BY a));
+is($result,
+ qq(1|5|100
+3|6|300), 'check replicated rows with multiple row filters');
+
$node_subscriber->stop('fast');
$node_publisher->stop('fast');
0001-fixup-publish_as_relid-20220310.patchtext/x-patch; charset=UTF-8; name=0001-fixup-publish_as_relid-20220310.patchDownload
From 5070a26fb54b956557f4ba3a44b796ab4c232c93 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Wed, 9 Mar 2022 18:10:56 +0100
Subject: [PATCH 1/2] fixup: publish_as_relid
---
src/backend/replication/pgoutput/pgoutput.c | 32 +++++++++++++++++++--
1 file changed, 30 insertions(+), 2 deletions(-)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a0477f0..dbac2690b7f 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1815,11 +1815,17 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
Publication *pub = lfirst(lc);
bool publish = false;
+ /*
+ * Under what relid should we publish changes in this publication?
+ * We'll use the top-most relid across all publications.
+ */
+ Oid pub_relid = relid;
+
if (pub->alltables)
{
publish = true;
if (pub->pubviaroot && am_partition)
- publish_as_relid = llast_oid(get_partition_ancestors(relid));
+ pub_relid = llast_oid(get_partition_ancestors(relid));
}
if (!publish)
@@ -1844,7 +1850,7 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
{
ancestor_published = true;
if (pub->pubviaroot)
- publish_as_relid = ancestor;
+ pub_relid = ancestor;
}
}
@@ -1862,12 +1868,34 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ List *ancestors;
+
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
rel_publications = lappend(rel_publications, pub);
+
+ /*
+ * We want to publish the changes as the top-most ancestor
+ * across all publications. So we fetch all ancestors of the
+ * relid calculated for this publication, and check if the
+ * already calculated value is in the list. If yes, we can
+ * ignore the new value (as it's a child). Otherwise the new
+ * value is an ancestor, so we keep it.
+ */
+ ancestors = get_partition_ancestors(pub_relid);
+
+ /*
+ * The new pub_relid is a child of the current publish_as_relid
+ * value, so we can ignore it.
+ */
+ if (list_member_oid(ancestors, publish_as_relid))
+ continue;
+
+ /* The new value is an ancestor, so let's keep it. */
+ publish_as_relid = pub_relid;
}
}
--
2.34.1
0002-Allow-specifying-column-filters-for-logical-20220310.patchtext/x-patch; charset=UTF-8; name=0002-Allow-specifying-column-filters-for-logical-20220310.patchDownload
From 19a580e1d8e5772cd18b933cc500bba36d54fffc Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Thu, 10 Mar 2022 17:31:57 +0100
Subject: [PATCH 2/2] Allow specifying column filters for logical replication
This allows specifying an optional column list when adding a table to
logical replication. Columns not included on this list are not sent to
the subscriber. The filter is specified as a list of columns after the
table name, enclosed in parentheses.
For UPDATE/DELETE publications, the column filter needs to cover all
REPLICA IDENTITY columns. For INSERT publications, the column list is
arbitrary and may omit some REPLICA IDENTITY columns. Furthermore, if
the table uses REPLICA IDENTITY FULL, column filter is not allowed.
The column filter can contain only simple column references. Complex
expressions, function calls etc. are not allowed. This restriction could
be relaxed in the future.
During the initial table synchronization, only columns specified in the
column filter are copied to the subscriber. If the subscription has
several publications, containing the same table with different column
filters, columns specified in any of the filters will be copied. This
means all columns are replicated if the table has no column filter at
all (which is treated as column filter with all columns), of when of the
publications is defined as FOR ALL TABLES (possibly IN SCHEMA for the
schema of the table).
For partitioned tables, publish_via_partition_root determines whether
the row filter for the root or leaf relation will be used. If the
parameter is 'false' (the default), the filter defined for the leaf
relation is used. Otherwise, the column filter for the root partition
will be used.
Psql commands \dRp+ and \d <table-name> now display any column filters.
Author: Rahila Syed <rahilasyed90@gmail.com>
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 15 +-
doc/src/sgml/protocol.sgml | 3 +-
doc/src/sgml/ref/alter_publication.sgml | 27 +-
doc/src/sgml/ref/create_publication.sgml | 17 +-
src/backend/catalog/pg_publication.c | 256 +++++
src/backend/commands/publicationcmds.c | 364 +++++-
src/backend/commands/tablecmds.c | 36 +-
src/backend/executor/execReplication.c | 19 +-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 60 +-
src/backend/replication/logical/proto.c | 56 +-
src/backend/replication/logical/tablesync.c | 184 ++-
src/backend/replication/pgoutput/pgoutput.c | 156 ++-
src/backend/utils/cache/relcache.c | 33 +-
src/bin/pg_dump/pg_dump.c | 47 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 60 +
src/bin/psql/describe.c | 40 +-
src/include/catalog/pg_publication.h | 13 +
src/include/catalog/pg_publication_rel.h | 1 +
src/include/commands/publicationcmds.h | 4 +-
src/include/nodes/parsenodes.h | 4 +-
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 375 +++++++
src/test/regress/sql/publication.sql | 290 +++++
src/test/subscription/t/029_column_list.pl | 1124 +++++++++++++++++++
27 files changed, 3112 insertions(+), 81 deletions(-)
create mode 100644 src/test/subscription/t/029_column_list.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 83987a99045..c043da37aee 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -4392,7 +4392,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para>
<para>
This is an array of <structfield>indnatts</structfield> values that
- indicate which table columns this index indexes. For example a value
+ indicate which table columns this index indexes. For example, a value
of <literal>1 3</literal> would mean that the first and the third table
columns make up the index entries. Key columns come before non-key
(included) columns. A zero in this array indicates that the
@@ -6263,6 +6263,19 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to schema
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>prattrs</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+ </para>
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example, a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all columns are published.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 0695bcd423e..92cd0f9c9f7 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7006,7 +7006,8 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column included in
+ the publication (except generated columns):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 32b75f6c78e..470d50a2447 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -25,12 +25,13 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD <replace
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ALTER TABLE <replaceable class="parameter">table_name</replaceable> SET COLUMNS { ( <replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -64,6 +65,12 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
command retain their previous settings.
</para>
+ <para>
+ The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows changing
+ the set of columns that are included in the publication. If a column list
+ is specified, it must include the replica identity columns.
+ </para>
+
<para>
The remaining variants change the owner and the name of the publication.
</para>
@@ -112,6 +119,14 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ </para>
+
+ <para>
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
If the optional <literal>WHERE</literal> clause is specified, rows for
which the <replaceable class="parameter">expression</replaceable>
evaluates to false or null will not be published. Note that parentheses
@@ -172,9 +187,15 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
</para>
<para>
- Add some tables to the publication:
+ Add tables to the publication:
+<programlisting>
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS (user_id, firstname, lastname);
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 4979b9b646d..fb2d013393b 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -86,6 +86,13 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<literal>TRUNCATE</literal> commands.
</para>
+ <para>
+ When a column list is specified, only the named columns are replicated.
+ If no column list is specified, all columns of the table are replicated
+ through this publication, including any columns added later. If a column
+ list is specified, it must include the replica identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
@@ -327,6 +334,14 @@ CREATE PUBLICATION production_publication FOR TABLE users, departments, ALL TABL
<structname>sales</structname>:
<programlisting>
CREATE PUBLICATION sales_publication FOR ALL TABLES IN SCHEMA marketing, sales;
+</programlisting></para>
+
+ <para>
+ Create a publication that publishes all changes for table <structname>users</structname>,
+ but replicates only columns <structname>user_id</structname> and
+ <structname>firstname</structname>:
+<programlisting>
+CREATE PUBLICATION users_filtered FOR TABLE users (user_id, firstname);
</programlisting></para>
</refsect1>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 25998fbb39b..3275a7c8b9c 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,6 +45,9 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+static void publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs);
+
/*
* Check if relation can be in given publication and throws appropriate
* error if not.
@@ -328,6 +331,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
Oid relid = RelationGetRelid(targetrel);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ AttrNumber *attarray;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -355,6 +360,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
check_publication_add_relation(targetrel);
+ /*
+ * Translate column names to attnums and check the column list is valid.
+ * We also deconstruct the bitmap into an array of attnums, for storing
+ * in the catalog.
+ */
+ publication_translate_columns(pri->relation, pri->columns,
+ &natts, &attarray);
+
/* Form a tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -367,6 +380,17 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ /* Add column list, if available */
+ if (pri->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+
/* Add qualifications, if available */
if (pri->whereClause != NULL)
values[Anum_pg_publication_rel_prqual - 1] = CStringGetTextDatum(nodeToString(pri->whereClause));
@@ -382,6 +406,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
/* Register dependencies as needed */
ObjectAddressSet(myself, PublicationRelRelationId, pubreloid);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ pfree(attarray);
+
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -415,6 +447,154 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
return myself;
}
+/*
+ * Update the column list for a relation in a publication.
+ */
+void
+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns)
+{
+ AttrNumber *attarray;
+ HeapTuple copytup;
+ int natts;
+ bool nulls[Natts_pg_publication_rel];
+ bool replaces[Natts_pg_publication_rel];
+ Datum values[Natts_pg_publication_rel];
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ replaces[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ deleteDependencyRecordsForClass(PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid,
+ RelationRelationId,
+ DEPENDENCY_AUTO);
+
+ if (columns == NULL)
+ {
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+ }
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ int2vector *prattrs;
+ Form_pg_publication_rel pubrel;
+
+ publication_translate_columns(targetrel, columns, &natts, &attarray);
+
+ /* XXX "pub" is leaked here ??? */
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+
+ /* Add dependencies on the new list of columns */
+ pubrel = (Form_pg_publication_rel) GETSTRUCT(pubreltup);
+ ObjectAddressSet(myself, PublicationRelRelationId, pubrel->oid);
+
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId,
+ RelationGetRelid(targetrel), attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ }
+
+ copytup = heap_modify_tuple(pubreltup, RelationGetDescr(pubrel),
+ values, nulls, replaces);
+
+ CatalogTupleUpdate(pubrel, &pubreltup->t_self, copytup);
+
+ heap_freetuple(copytup);
+}
+
+/*
+ * qsort comparator for attnums
+ *
+ * XXX We already have compare_int16, so maybe let's share that, somehow?
+ */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list (no system or generated attributes,
+ * no duplicates). Additional checks with replica identity are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ *
+ * XXX Should this detect duplicate columns?
+ */
+static void
+publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs)
+{
+ AttrNumber *attarray;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference generated column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /* Be tidy, so that the catalog representation is always sorted */
+ qsort(attarray, n, sizeof(AttrNumber), compare_int16);
+
+ *natts = n;
+ *attrs = attarray;
+
+ bms_free(set);
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -522,6 +702,82 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+ Form_pg_publication_rel pubrel;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ /* no column list for this publications/relation */
+ if (isnull)
+ continue;
+
+ pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
+
+ pubs = lappend_oid(pubs, pubrel->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 16b8661a1b7..b32ec275557 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -296,7 +296,7 @@ contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
* Returns true if any invalid column is found.
*/
bool
-contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
+pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
bool pubviaroot)
{
HeapTuple rftuple;
@@ -367,6 +367,123 @@ contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
return result;
}
+/*
+ * Check if all columns referenced in the column list are part of the
+ * REPLICA IDENTITY index or not.
+ *
+ * Returns true if any invalid column is found.
+ */
+bool
+pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
+ bool pubviaroot)
+{
+ HeapTuple tuple;
+ Oid relid = RelationGetRelid(relation);
+ Oid publish_as_relid = RelationGetRelid(relation);
+ bool result = false;
+ Datum datum;
+ bool isnull;
+
+ /*
+ * For a partition, if pubviaroot is true, find the topmost ancestor that
+ * is published via this publication as we need to use its column list
+ * for the changes.
+ *
+ * Note that even though the column list used is for an ancestor, the
+ * REPLICA IDENTITY used will be for the actual child table.
+ */
+ if (pubviaroot && relation->rd_rel->relispartition)
+ {
+ publish_as_relid = GetTopMostAncestorInPublication(pubid, ancestors);
+
+ if (!OidIsValid(publish_as_relid))
+ publish_as_relid = relid;
+ }
+
+ tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tuple))
+ return false;
+
+ datum = SysCacheGetAttr(PUBLICATIONRELMAP, tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ if (!isnull)
+ {
+ int x;
+ Bitmapset *idattrs;
+ Bitmapset *columns = NULL;
+
+ /* With REPLICA IDENTITY FULL, no column list is allowed. */
+ if (relation->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ result = true;
+
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(datum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ for (int i = 0; i < nelems; i++)
+ columns = bms_add_member(columns, elems[i]);
+ }
+
+ /* Remember columns that are part of the REPLICA IDENTITY */
+ idattrs = RelationGetIndexAttrBitmap(relation,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * Attnums in the bitmap returned by RelationGetIndexAttrBitmap are
+ * offset (to handle system columns the usual way), while column list
+ * does not use offset, so we can't do bms_is_subset(). Instead, we have
+ * to loop over the idattrs and check all of them are in the filter.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ AttrNumber attnum = (x + FirstLowInvalidHeapAttributeNumber);
+
+ /*
+ * If pubviaroot is true, we are validating the row filter of the
+ * parent table, but the bitmap contains the replica identity
+ * information of the child table. So, get the column number of the
+ * child table as parent and child column order could be different.
+ */
+ if (pubviaroot)
+ {
+ /* attnum is for child table */
+ char *colname = get_attname(relid, attnum, false);
+
+ /*
+ * Determine the matching attnum in parent (because the column
+ * filter is defined on the parent).
+ */
+ attnum = get_attnum(publish_as_relid, colname);
+ }
+
+ if (!bms_is_member(attnum, columns))
+ {
+ result = true;
+ break;
+ }
+ }
+
+ bms_free(idattrs);
+ bms_free(columns);
+ }
+
+ ReleaseSysCache(tuple);
+
+ return result;
+}
+
/* check_functions_in_node callback */
static bool
contain_mutable_or_user_functions_checker(Oid func_id, void *context)
@@ -608,6 +725,45 @@ TransformPubWhereClauses(List *tables, const char *queryString,
}
}
+
+/*
+ * Transform the publication column lists expression for all the relations
+ * in the list.
+ *
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow column lists on the leaf relations. So maybe rename it?
+ */
+static void
+TransformPubColumnList(List *tables, const char *queryString,
+ bool pubviaroot)
+{
+ ListCell *lc;
+
+ foreach(lc, tables)
+ {
+ PublicationRelInfo *pri = (PublicationRelInfo *) lfirst(lc);
+
+ if (pri->columns == NIL)
+ continue;
+
+ /*
+ * If the publication doesn't publish changes via the root partitioned
+ * table, the partition's column list will be used. So disallow using
+ * the column list on partitioned table in this case.
+ */
+ if (!pubviaroot &&
+ pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot use publication column list for relation \"%s\"",
+ RelationGetRelationName(pri->relation)),
+ errdetail("column list cannot be used for a partitioned table when %s is false.",
+ "publish_via_partition_root")));
+ }
+}
+
/*
* Create new publication.
*/
@@ -724,6 +880,9 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
TransformPubWhereClauses(rels, pstate->p_sourcetext,
publish_via_partition_root);
+ TransformPubColumnList(rels, pstate->p_sourcetext,
+ publish_via_partition_root);
+
PublicationAddTables(puboid, rels, true, NULL);
CloseTableList(rels);
}
@@ -754,6 +913,52 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
return myself;
}
+/*
+ * Change the column list of a relation in a publication
+ */
+static void
+PublicationSetColumns(AlterPublicationStmt *stmt,
+ Form_pg_publication pubform, PublicationTable *table)
+{
+ Relation rel,
+ urel;
+ HeapTuple tup;
+ ObjectAddress obj,
+ secondary;
+
+ rel = table_open(PublicationRelRelationId, RowExclusiveLock);
+ urel = table_openrv(table->relation, ShareUpdateExclusiveLock);
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(RelationGetRelid(urel)),
+ ObjectIdGetDatum(pubform->oid));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errmsg("relation \"%s\" is not already in publication \"%s\"",
+ table->relation->relname,
+ NameStr(pubform->pubname)));
+
+ publication_set_table_columns(rel, tup, urel, table->columns);
+
+ ObjectAddressSet(obj, PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->oid);
+ ObjectAddressSet(secondary, RelationRelationId, RelationGetRelid(urel));
+ EventTriggerCollectSimpleCommand(obj, secondary, (Node *) stmt);
+
+ ReleaseSysCache(tup);
+
+ /*
+ * Invalidate relcache for this relation, to force rebuilding the
+ * publication description.
+ */
+ CacheInvalidateRelcache(urel);
+
+ table_close(rel, RowExclusiveLock);
+ table_close(urel, NoLock);
+
+ InvokeObjectPostAlterHook(PublicationRelationId, pubform->oid, 0);
+}
+
/*
* Change options of a publication.
*/
@@ -783,8 +988,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* If the publication doesn't publish changes via the root partitioned
- * table, the partition's row filter will be used. So disallow using WHERE
- * clause on partitioned table in this case.
+ * table, the partition's row filter and column list will be used. So disallow
+ * using WHERE clause and column lists on partitioned table in this case.
*/
if (!pubform->puballtables && publish_via_partition_root_given &&
!publish_via_partition_root)
@@ -792,7 +997,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* Lock the publication so nobody else can do anything with it. This
* prevents concurrent alter to add partitioned table(s) with WHERE
- * clause(s) which we don't allow when not publishing via root.
+ * clause(s) and/or column lists which we don't allow when not
+ * publishing via root.
*/
LockDatabaseObject(PublicationRelationId, pubform->oid, 0,
AccessShareLock);
@@ -804,13 +1010,21 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
HeapTuple rftuple;
Oid relid = lfirst_oid(lc);
+ bool has_column_list;
+ bool has_row_filter;
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubform->oid));
+ has_row_filter
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL);
+
+ has_column_list
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prattrs, NULL);
+
if (HeapTupleIsValid(rftuple) &&
- !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL))
+ (has_row_filter || has_column_list))
{
HeapTuple tuple;
@@ -819,7 +1033,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
Form_pg_class relform = (Form_pg_class) GETSTRUCT(tuple);
- if (relform->relkind == RELKIND_PARTITIONED_TABLE)
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_row_filter)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot set %s for publication \"%s\"",
@@ -830,6 +1045,18 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
NameStr(relform->relname),
"publish_via_partition_root")));
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_column_list)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot set %s for publication \"%s\"",
+ "publish_via_partition_root = false",
+ stmt->pubname),
+ errdetail("The publication contains a column list for a partitioned table \"%s\" "
+ "which is not allowed when %s is false.",
+ NameStr(relform->relname),
+ "publish_via_partition_root")));
+
ReleaseSysCache(tuple);
}
@@ -838,6 +1065,16 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
}
}
+ /*
+ * FIXME check pubactions vs. replica identity, to ensure the replica
+ * identity is included in the column list. Only do this for update
+ * and delete publications. See check_publication_columns.
+ *
+ * XXX This is needed because publish_via_partition_root may change,
+ * in which case the row filters may be invalid (e.g. with pvpr=false
+ * there must be no filter on partitioned tables).
+ */
+
/* Everything ok, form a new tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -975,10 +1212,20 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
PublicationAddTables(pubid, rels, false, stmt);
}
else if (stmt->action == AP_DropObjects)
PublicationDropTables(pubid, rels, false);
+ else if (stmt->action == AP_SetColumns)
+ {
+ Assert(schemaidlist == NIL);
+ Assert(list_length(tables) == 1);
+
+ PublicationSetColumns(stmt, pubform,
+ linitial_node(PublicationTable, tables));
+ }
else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
@@ -991,6 +1238,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
/*
* To recreate the relation list for the publication, look for
* existing relations that do not need to be dropped.
@@ -1002,32 +1251,85 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
PublicationRelInfo *oldrel;
bool found = false;
HeapTuple rftuple;
- bool rfisnull = true;
Node *oldrelwhereclause = NULL;
+ Bitmapset *oldcolumns = NULL;
/* look up the cache for the old relmap */
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(oldrelid),
ObjectIdGetDatum(pubid));
+ /*
+ * See if the existing relation currently has a WHERE clause or a
+ * column list. We need to compare those too.
+ */
if (HeapTupleIsValid(rftuple))
{
+ bool isnull = true;
Datum whereClauseDatum;
+ Datum columnListDatum;
+ /* Load the WHERE clause for this table. */
whereClauseDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
Anum_pg_publication_rel_prqual,
- &rfisnull);
- if (!rfisnull)
+ &isnull);
+ if (!isnull)
oldrelwhereclause = stringToNode(TextDatumGetCString(whereClauseDatum));
+ /* Transform the int2vector column list to a bitmap. */
+ columnListDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ /*
+ * XXX Maybe make this a separate function. We do this on
+ * multiple places.
+ */
+ if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(columnListDatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ for (int i = 0; i < nelems; i++)
+ oldcolumns = bms_add_member(oldcolumns, elems[i]);
+ }
+
ReleaseSysCache(rftuple);
}
foreach(newlc, rels)
{
PublicationRelInfo *newpubrel;
+ Oid newrelid;
+ Bitmapset *newcolumns = NULL;
newpubrel = (PublicationRelInfo *) lfirst(newlc);
+ newrelid = RelationGetRelid(newpubrel->relation);
+
+ /*
+ * If the new publication has column list, transform it to
+ * a bitmap too.
+ */
+ if (newpubrel->columns)
+ {
+ ListCell *lc;
+
+ foreach(lc, newpubrel->columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(newrelid, colname);
+
+ /* no checks needed here, that happens elsewhere */
+
+ newcolumns = bms_add_member(newcolumns, attnum);
+ }
+ }
/*
* Check if any of the new set of relations matches with the
@@ -1037,7 +1339,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
*/
if (RelationGetRelid(newpubrel->relation) == oldrelid)
{
- if (equal(oldrelwhereclause, newpubrel->whereClause))
+ if (equal(oldrelwhereclause, newpubrel->whereClause) &&
+ bms_equal(oldcolumns, newcolumns))
{
found = true;
break;
@@ -1056,6 +1359,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
{
oldrel = palloc(sizeof(PublicationRelInfo));
oldrel->whereClause = NULL;
+ oldrel->columns = NIL;
oldrel->relation = table_open(oldrelid,
ShareUpdateExclusiveLock);
delrels = lappend(delrels, oldrel);
@@ -1117,7 +1421,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
}
else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* AP_SetObjects */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -1140,6 +1444,10 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
*/
PublicationAddSchemas(pubform->oid, schemaidlist, true, stmt);
}
+ else
+ {
+ /* Nothing to do for AP_SetColumns */
+ }
}
/*
@@ -1402,6 +1710,7 @@ OpenTableList(List *tables)
List *rels = NIL;
ListCell *lc;
List *relids_with_rf = NIL;
+ List *relids_with_collist = NIL;
/*
* Open, share-lock, and check all the explicitly-specified relations
@@ -1436,6 +1745,13 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /* Disallow duplicate tables if there are any with column lists. */
+ if (t->columns || list_member_oid(relids_with_collist, myrelid))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
table_close(rel, ShareUpdateExclusiveLock);
continue;
}
@@ -1443,12 +1759,16 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
pub_rel->whereClause = t->whereClause;
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, myrelid);
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, myrelid);
+
/*
* Add children of this rel, if requested, so that they too are added
* to the publication. A partitioned table can't have any inheritance
@@ -1488,6 +1808,18 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /*
+ * We don't allow to specify column list for both parent
+ * and child table at the same time as it is not very
+ * clear which one should be given preference.
+ */
+ if (childrelid != myrelid &&
+ (t->columns || list_member_oid(relids_with_collist, childrelid)))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
continue;
}
@@ -1497,11 +1829,16 @@ OpenTableList(List *tables)
pub_rel->relation = rel;
/* child inherits WHERE clause from parent */
pub_rel->whereClause = t->whereClause;
+ /* child inherits column list from parent */
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, childrelid);
+
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, childrelid);
}
}
}
@@ -1610,6 +1947,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index dc5872f988c..a9fd0f0c895 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -8365,6 +8365,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
bool missing_ok, LOCKMODE lockmode,
ObjectAddresses *addrs)
{
+ Oid relid = RelationGetRelid(rel);
HeapTuple tuple;
Form_pg_attribute targetatt;
AttrNumber attnum;
@@ -8384,7 +8385,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/*
* get the number of the attribute
*/
- tuple = SearchSysCacheAttName(RelationGetRelid(rel), colName);
+ tuple = SearchSysCacheAttName(relid, colName);
if (!HeapTupleIsValid(tuple))
{
if (!missing_ok)
@@ -8438,13 +8439,42 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * Also, if the column is used in the column list of a publication,
+ * disallow the drop if the DROP is RESTRICT. We don't do anything if the
+ * DROP is CASCADE, which means that the dependency mechanism will remove
+ * the relation from the publication.
+ */
+ if (behavior == DROP_RESTRICT)
+ {
+ List *pubs;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(relid);
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(relid, pubid);
+
+ if (list_member_oid(published_cols, attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot drop column \"%s\" because it is part of publication \"%s\"",
+ colName, get_publication_name(pubid, false)),
+ errhint("Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication."));
+ }
+ }
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
* use find_all_inheritors to do it in one pass.
*/
children =
- find_inheritance_children(RelationGetRelid(rel), lockmode);
+ find_inheritance_children(relid, lockmode);
if (children)
{
@@ -8532,7 +8562,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Add object to delete */
object.classId = RelationRelationId;
- object.objectId = RelationGetRelid(rel);
+ object.objectId = relid;
object.objectSubId = attnum;
add_exact_object_address(&object, addrs);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 09f78f22441..3e282ed99ab 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -573,9 +573,6 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
if (cmd != CMD_UPDATE && cmd != CMD_DELETE)
return;
- if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
- return;
-
/*
* It is only safe to execute UPDATE/DELETE when all columns, referenced
* in the row filters from publications which the relation is in, are
@@ -595,17 +592,33 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
errmsg("cannot update table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_UPDATE && !pubdesc.cols_valid_for_update)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot update table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
else if (cmd == CMD_DELETE && !pubdesc.rf_valid_for_delete)
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
errmsg("cannot delete from table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_DELETE && !pubdesc.cols_valid_for_delete)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot delete from table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
/* If relation has replica identity we are always good. */
if (OidIsValid(RelationGetReplicaIndex(rel)))
return;
+ /* REPLICA IDENTITY FULL is also good for UPDATE/DELETE. */
+ if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ return;
+
/*
* This is UPDATE/DELETE and there is no replica identity.
*
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index d4f8455a2bd..a504437873f 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4850,6 +4850,7 @@ _copyPublicationTable(const PublicationTable *from)
COPY_NODE_FIELD(relation);
COPY_NODE_FIELD(whereClause);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f1002afe7a0..4fc16ce04e3 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2322,6 +2322,7 @@ _equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
COMPARE_NODE_FIELD(whereClause);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a03b33b53bd..25c9b29afdd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9751,13 +9751,14 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr OptWhereClause
+ TABLE relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9772,11 +9773,12 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId OptWhereClause
+ | ColId opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- if ($2)
+ /* FIXME this is a bit cumbersome */
+ if ($2 || $3)
{
/*
* The OptWhereClause must be stored here but it is
@@ -9786,7 +9788,8 @@ PublicationObjSpec:
*/
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVar(NULL, $1, @1);
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
else
{
@@ -9794,23 +9797,25 @@ PublicationObjSpec:
}
$$->location = @1;
}
- | ColId indirection OptWhereClause
+ | ColId indirection opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr OptWhereClause
+ | extended_relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
| CURRENT_SCHEMA
{
@@ -9836,6 +9841,10 @@ pub_obj_list: PublicationObjSpec
*
* ALTER PUBLICATION name SET pub_obj [, ...]
*
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ *
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
* pub_obj is one of:
*
* TABLE table_name [, ...]
@@ -9869,6 +9878,32 @@ AlterPublicationStmt:
n->action = AP_SetObjects;
$$ = (Node *)n;
}
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS '(' columnList ')'
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = $10;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS ALL
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = NIL;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
| ALTER PUBLICATION name DROP pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
@@ -17488,6 +17523,13 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
errmsg("WHERE clause not allowed for schema"),
parser_errposition(pubobj->location));
+ /* Column filter is not allowed on a schema object */
+ if (pubobj->pubtable && pubobj->pubtable->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schema"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index c9b0eeefd7e..816d461acd3 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,16 +29,25 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary);
+ bool binary, Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
static void logicalrep_write_namespace(StringInfo out, Oid nspid);
static const char *logicalrep_read_namespace(StringInfo in);
+
+static bool
+column_in_set(int attnum, Bitmapset *columns)
+{
+ return (columns == NULL || bms_is_member(attnum, columns));
+}
+
+
/*
* Write BEGIN to the output stream.
*/
@@ -398,7 +407,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +419,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -443,7 +452,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -464,11 +473,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -537,7 +546,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, NULL);
}
/*
@@ -652,7 +661,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -674,7 +684,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -751,7 +761,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum *values;
@@ -763,8 +773,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
continue;
+
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -783,6 +799,9 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +923,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -917,8 +936,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* send number of live attributes */
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
continue;
+
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -937,6 +962,9 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 1659964571c..6eb9fc902b7 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -112,6 +112,7 @@
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "utils/acl.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -701,12 +702,13 @@ fetch_remote_table_info(char *nspname, char *relname,
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
int natt;
ListCell *lc;
bool first;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
@@ -747,10 +749,139 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+
+ /*
+ * Get column lists for each relation.
+ *
+ * For initial synchronization, column lists can be ignored in following
+ * cases:
+ *
+ * 1) one of the subscribed publications for the table hasn't specified
+ * any column list
+ *
+ * 2) one of the subscribed publications has puballtables set to true
+ *
+ * 3) one of the subscribed publications is declared as ALL TABLES IN
+ * SCHEMA that includes this relation
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData pub_names;
+ bool first = true;
+ bool all_columns = false;
+
+ initStringInfo(&pub_names);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&pub_names, ", ");
+ appendStringInfoString(&pub_names, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ /*
+ * Check for column filters - we first check if there's any publication
+ * that has no column list for the given relation, which means we shall
+ * replicate all columns.
+ *
+ * It's easier than having to do this separately, and only then do the
+ * second query
+ */
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT 1"
+ " FROM pg_publication p"
+ " LEFT OUTER JOIN pg_publication_rel pr"
+ " ON (p.oid = pr.prpubid AND pr.prrelid = %u),"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt"
+ " WHERE gpt.relid = %u"
+ " AND p.pubname IN ( %s ) AND pr.prattrs IS NULL LIMIT 1",
+ lrel->remoteid,
+ lrel->remoteid,
+ pub_names.data);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch column list info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ if (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ all_columns = true;
+
+ ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(pubres);
+
+ if (!all_columns)
+ {
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT unnest(pr.prattrs)"
+ " FROM pg_publication p"
+ " LEFT OUTER JOIN pg_publication_rel pr"
+ " ON (p.oid = pr.prpubid AND pr.prrelid = %u),"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt"
+ " WHERE gpt.relid = %u"
+ " AND p.pubname IN ( %s ) AND pr.prattrs IS NOT NULL",
+ lrel->remoteid,
+ lrel->remoteid,
+ pub_names.data);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch column list info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ /*
+ * Multiple column list expressions for the same table will be combined
+ * by merging them. If any of the lists for this table are null, it
+ * means the whole table will be copied. In this case it is not necessary
+ * to construct a unified column list expression at all.
+ */
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ Datum cfval = slot_getattr(slot, 1, &isnull);
+
+ /* if there's no column list, we need to replicate all columns */
+ if (isnull)
+ {
+ bms_free(included_cols);
+ included_cols = NULL;
+ break;
+ }
+
+ included_cols = bms_add_member(included_cols,
+ DatumGetInt16(cfval));
+
+ ExecClearTuple(slot);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+
+ walrcv_clear_result(pubres);
+ }
+
+ pfree(pub_names.data);
+ }
+
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -778,16 +909,34 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * Store the columns as a list of names. Ignore those that are not
+ * present in the column list, if there is one.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ {
+ ExecClearTuple(slot);
+ continue;
+ }
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
+ Assert(!isnull);
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -821,6 +970,9 @@ fetch_remote_table_info(char *nspname, char *relname,
*
* 3) one of the subscribed publications is declared as ALL TABLES IN
* SCHEMA that includes this relation
+ *
+ * XXX Does this actually handle puballtables and schema publications
+ * correctly?
*/
if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
{
@@ -930,8 +1082,24 @@ copy_table(Relation rel)
/* Regular table with no row filter */
if (lrel.relkind == RELKIND_RELATION && qual == NIL)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+
+ /*
+ * XXX Do we need to list the columns in all cases? Maybe we're replicating
+ * all columns?
+ */
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
+
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
+
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index dbac2690b7f..fcdd7785a79 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -29,6 +29,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -85,7 +86,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -93,6 +95,8 @@ static void send_repl_origin(LogicalDecodingContext *ctx,
/*
* Only 3 publication actions are used for row filtering ("insert", "update",
* "delete"). See RelationSyncEntry.exprstate[].
+ *
+ * FIXME Do we need something similar for column filters?
*/
enum RowFilterPubAction
{
@@ -164,6 +168,13 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
AttrMap *attrmap;
+
+ /*
+ * Columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this bitmap are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -188,6 +199,7 @@ static EState *create_estate_for_relation(Relation rel);
static void pgoutput_row_filter_init(PGOutputData *data,
List *publications,
RelationSyncEntry *entry);
+
static bool pgoutput_row_filter_exec_expr(ExprState *state,
ExprContext *econtext);
static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
@@ -195,6 +207,11 @@ static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
RelationSyncEntry *entry,
ReorderBufferChangeType *action);
+/* column filter routines */
+static void pgoutput_column_filter_init(PGOutputData *data,
+ List *publications,
+ RelationSyncEntry *entry);
+
/*
* Specify output plugin callbacks
*/
@@ -603,11 +620,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -620,7 +637,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -643,13 +661,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip this attribute if it's not present in the column list */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -860,6 +882,108 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
}
}
+/*
+ * Initialize the column filter.
+ */
+static void
+pgoutput_column_filter_init(PGOutputData *data, List *publications,
+ RelationSyncEntry *entry)
+{
+ ListCell *lc;
+ MemoryContext oldctx;
+
+ /*
+ * Find if there are any row filters for this relation. If there are, then
+ * prepare the necessary ExprState and cache it in entry->exprstate. To
+ * build an expression state, we need to ensure the following:
+ *
+ * All the given publication-table mappings must be checked.
+ *
+ * Multiple publications might have multiple row filters for this
+ * relation. Since row filter usage depends on the DML operation, there
+ * are multiple lists (one for each operation) to which row filters will
+ * be appended.
+ *
+ * FOR ALL TABLES implies "don't use row filter expression" so it takes
+ * precedence.
+ */
+ foreach(lc, publications)
+ {
+ Publication *pub = lfirst(lc);
+ HeapTuple cftuple = NULL;
+ Datum cfdatum = 0;
+ bool pub_no_filter = false;
+
+ if (pub->alltables)
+ {
+ /*
+ * If the publication is FOR ALL TABLES then it is treated the
+ * same as if this table has no row filters (even if for other
+ * publications it does).
+ */
+ pub_no_filter = true;
+ }
+ else
+ {
+ /*
+ * Check for the presence of a row filter in this publication.
+ */
+ cftuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(entry->publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(cftuple))
+ {
+ /* Null indicates no filter. */
+ cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
+ Anum_pg_publication_rel_prattrs,
+ &pub_no_filter);
+
+ /*
+ * When no column list is defined, so publish all columns.
+ * Otherwise merge the columns to the column list.
+ */
+ if (!pub_no_filter) /* when not null */
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(cfdatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
+ entry->columns = bms_add_member(entry->columns,
+ elems[i]);
+ MemoryContextSwitchTo(oldctx);
+ }
+ }
+ else
+ {
+ pub_no_filter = true;
+ }
+ }
+
+ /* found publication with no filter, so we're done */
+ if (pub_no_filter)
+ {
+ if (cftuple)
+ ReleaseSysCache(cftuple);
+
+ bms_free(entry->columns);
+ entry->columns = NULL;
+
+ break;
+ }
+
+ ReleaseSysCache(cftuple);
+ } /* loop all subscribed publications */
+
+}
+
/*
* Initialize the slot for storing new and old tuples, and build the map that
* will be used to convert the relation's tuples into the ancestor's format.
@@ -1224,7 +1348,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -1278,11 +1402,13 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel,
- new_slot, data->binary);
+ new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel,
- old_slot, new_slot, data->binary);
+ old_slot, new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel,
@@ -1731,6 +1857,7 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
memset(entry->exprstate, 0, sizeof(entry->exprstate));
entry->cache_expr_cxt = NULL;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->attrmap = NULL;
}
@@ -1775,6 +1902,8 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
entry->pubactions.pubinsert = false;
entry->pubactions.pubupdate = false;
entry->pubactions.pubdelete = false;
@@ -1807,8 +1936,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/*
* Build publication cache. We can't use one provided by relcache as
- * relcache considers all publications given relation is in, but here
- * we only need to consider ones that the subscriber requested.
+ * relcache considers all publications that the given relation is in,
+ * but here we only need to consider ones that the subscriber
+ * requested.
*/
foreach(lc, data->publications)
{
@@ -1861,6 +1991,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
}
/*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
* Don't publish changes for partitioned tables, because
* publishing those of its partitions suffices, unless partition
* changes won't be published due to pubviaroot being set.
@@ -1913,6 +2046,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/* Initialize the row filter */
pgoutput_row_filter_init(data, rel_publications, entry);
+
+ /* Initialize the column filter */
+ pgoutput_column_filter_init(data, rel_publications, entry);
}
list_free(pubids);
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index fccffce5729..a2da72f0d48 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -5553,6 +5553,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
return;
}
@@ -5565,6 +5567,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
/* Fetch the publication membership info. */
puboids = GetRelationPublications(relid);
@@ -5616,7 +5620,7 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
*/
if (!pubform->puballtables &&
(pubform->pubupdate || pubform->pubdelete) &&
- contain_invalid_rfcolumn(pubid, relation, ancestors,
+ pub_rf_contains_invalid_column(pubid, relation, ancestors,
pubform->pubviaroot))
{
if (pubform->pubupdate)
@@ -5625,6 +5629,23 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->rf_valid_for_delete = false;
}
+ /*
+ * Check if all columns are part of the REPLICA IDENTITY index or not.
+ *
+ * If the publication is FOR ALL TABLES then it means the table has no
+ * column list and we can skip the validation.
+ */
+ if (!pubform->puballtables &&
+ (pubform->pubupdate || pubform->pubdelete) &&
+ pub_collist_contains_invalid_column(pubid, relation, ancestors,
+ pubform->pubviaroot))
+ {
+ if (pubform->pubupdate)
+ pubdesc->cols_valid_for_update = false;
+ if (pubform->pubdelete)
+ pubdesc->cols_valid_for_delete = false;
+ }
+
ReleaseSysCache(tup);
/*
@@ -5636,6 +5657,16 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
!pubdesc->rf_valid_for_update && !pubdesc->rf_valid_for_delete)
break;
+
+ /*
+ * If we know everything is replicated and the column list is invalid
+ * for update and delete, there is no point to check for other
+ * publications.
+ */
+ if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
+ pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
+ !pubdesc->cols_valid_for_update && !pubdesc->cols_valid_for_delete)
+ break;
}
if (relation->rd_pubdesc)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e69dcf8a484..f208c7a6c59 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4075,6 +4075,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_prpubid;
int i_prrelid;
int i_prrelqual;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4088,12 +4089,20 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
if (fout->remoteVersion >= 150000)
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual "
- "FROM pg_catalog.pg_publication_rel");
+ "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual, "
+ "(CASE\n"
+ " WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT array_agg(attname)\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) prattrs "
+ "FROM pg_catalog.pg_publication_rel pr");
else
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "NULL AS prrelqual "
+ "NULL AS prrelqual, NULL AS prattrs "
"FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
@@ -4104,6 +4113,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
i_prrelqual = PQfnumber(res, "prrelqual");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4149,6 +4159,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
else
pubrinfo[j].pubrelqual = pg_strdup(PQgetvalue(res, i, i_prrelqual));
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[j].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4223,10 +4255,13 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+
if (pubrinfo->pubrelqual)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 997a3b60719..680b07dcd52 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -632,6 +632,7 @@ typedef struct _PublicationRelInfo
PublicationInfo *publication;
TableInfo *pubtable;
char *pubrelqual;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 3e55ff26f82..ed57c53bcb5 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2428,6 +2428,28 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'ALTER PUBLICATION pub1 ADD TABLE test_sixth_table (col3, col2)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_sixth_table (col3, col2);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_sixth_table (col2, col3);\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'ALTER PUBLICATION pub1 ADD TABLE test_seventh_table (col3, col2) WHERE (col1 = 1)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_seventh_table (col3, col2) WHERE (col1 = 1);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_seventh_table (col2, col3) WHERE ((col1 = 1));\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'ALTER PUBLICATION pub3 ADD ALL TABLES IN SCHEMA dump_test' => {
create_order => 51,
create_sql =>
@@ -2778,6 +2800,44 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'CREATE TABLE test_sixth_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_sixth_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_sixth_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'CREATE TABLE test_seventh_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_seventh_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_seventh_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'CREATE TABLE test_table_identity' => {
create_order => 3,
create_sql => 'CREATE TABLE dump_test.test_table_identity (
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index e3382933d98..e462ccfb748 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2880,6 +2880,7 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_namespace pn ON p.oid = pn.pnpubid\n"
" JOIN pg_catalog.pg_class pc ON pc.relnamespace = pn.pnnspid\n"
@@ -2887,6 +2888,12 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , pg_get_expr(pr.prqual, c.oid)\n"
+ " , (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT string_agg(attname, ', ')\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) "
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
" JOIN pg_catalog.pg_class c ON c.oid = pr.prrelid\n"
@@ -2894,6 +2901,7 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2904,12 +2912,14 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
"WHERE pr.prrelid = '%s'\n"
"UNION ALL\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2931,6 +2941,11 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, " \"%s\"",
PQgetvalue(result, i, 0));
+ /* column list (if any) */
+ if (!PQgetisnull(result, i, 2))
+ appendPQExpBuffer(&buf, " (%s)",
+ PQgetvalue(result, i, 2));
+
/* row filter (if any) */
if (!PQgetisnull(result, i, 1))
appendPQExpBuffer(&buf, " WHERE %s",
@@ -5867,7 +5882,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5884,15 +5899,19 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (as_schema)
+ printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
+ else
{
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
+
+ if (!PQgetisnull(res, i, 3))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 3));
+
if (!PQgetisnull(res, i, 2))
appendPQExpBuffer(buf, " WHERE %s", PQgetvalue(res, i, 2));
}
- else
- printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
}
@@ -6021,11 +6040,22 @@ describePublications(const char *pattern)
printfPQExpBuffer(&buf,
"SELECT n.nspname, c.relname");
if (pset.sversion >= 150000)
+ {
appendPQExpBufferStr(&buf,
", pg_get_expr(pr.prqual, c.oid)");
+ appendPQExpBufferStr(&buf,
+ ", (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string("
+ " ARRAY(SELECT attname\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END)");
+ }
else
appendPQExpBufferStr(&buf,
- ", NULL");
+ ", NULL, NULL");
appendPQExpBuffer(&buf,
"\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index ba72e62e614..a06742a6200 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -85,6 +85,13 @@ typedef struct PublicationDesc
*/
bool rf_valid_for_update;
bool rf_valid_for_delete;
+
+ /*
+ * true if the columns are part of the replica identity or the publication actions
+ * do not include UPDATE or DELETE.
+ */
+ bool cols_valid_for_update;
+ bool cols_valid_for_delete;
} PublicationDesc;
typedef struct Publication
@@ -100,6 +107,7 @@ typedef struct PublicationRelInfo
{
Relation relation;
Node *whereClause;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -123,8 +131,11 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
+extern void GetActionsInPublication(Oid pubid, PublicationActions *actions);
extern List *GetPublicationSchemas(Oid pubid);
extern List *GetSchemaPublications(Oid schemaid);
extern List *GetSchemaPublicationRelations(Oid schemaid,
@@ -142,6 +153,8 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *pri
bool if_not_exists);
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern void publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index 0dd0f425db9..4feb581899e 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -34,6 +34,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
#ifdef CATALOG_VARLEN /* variable-length fields start here */
pg_node_tree prqual; /* qualifications */
+ int2vector prattrs; /* columns to replicate */
#endif
} FormData_pg_publication_rel;
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index 7813cbcb6bb..ae87caf089d 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -31,7 +31,9 @@ extern void RemovePublicationSchemaById(Oid psoid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
extern void AlterPublicationOwner_oid(Oid pubid, Oid newOwnerId);
extern void InvalidatePublicationRels(List *relids);
-extern bool contain_invalid_rfcolumn(Oid pubid, Relation relation,
+extern bool pub_rf_contains_invalid_column(Oid pubid, Relation relation,
+ List *ancestors, bool pubviaroot);
+extern bool pub_collist_contains_invalid_column(Oid pubid, Relation relation,
List *ancestors, bool pubviaroot);
#endif /* PUBLICATIONCMDS_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1617702d9d6..1375a173e3b 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3652,6 +3652,7 @@ typedef struct PublicationTable
NodeTag type;
RangeVar *relation; /* relation to be published */
Node *whereClause; /* qualifications */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
@@ -3688,7 +3689,8 @@ typedef enum AlterPublicationAction
{
AP_AddObjects, /* add objects to publication */
AP_DropObjects, /* remove objects from publication */
- AP_SetObjects /* set list of objects */
+ AP_SetObjects, /* set list of objects */
+ AP_SetColumns /* change list of columns for a table */
} AlterPublicationAction;
typedef struct AlterPublicationStmt
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4d2c881644a..a771ab8ff33 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -209,12 +209,12 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary);
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -231,7 +231,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 4e191c120ac..79ced2921b6 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -613,6 +613,372 @@ DROP TABLE rf_tbl_abcd_pk;
DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+RESET client_min_messages;
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ERROR: cannot reference generated column "d" in publication column list
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+ERROR: cannot reference system column "ctid" in publication column list
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+ERROR: cannot drop column "c" because it is part of publication "testpub_fortable"
+HINT: Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication.
+-- ok: for insert-only publication, the column list is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: change the replica identity to "b", and then column list (a, c) fails
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+-- error: replica identity (b,c) is not covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+ Publication testpub_table_ins
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | t | f
+Tables:
+ "public.testpub_tbl5" (a)
+
+-- with REPLICA IDENTITY FULL, column lists are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ERROR: cannot update table "testpub_tbl6"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+ALTER PUBLICATION testpub_fortable ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ERROR: cannot update table "testpub_tbl6"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- make sure changing the column list is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: update the column list
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, c)
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl8;
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_0"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+ Publication testpub_both_filters
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | t | t | t | f
+Tables:
+ "public.testpub_tbl_both_filters" (a, c) WHERE (c <> 1)
+
+\d+ testpub_tbl_both_filters
+ Table "public.testpub_tbl_both_filters"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+ c | integer | | not null | | plain | |
+Indexes:
+ "testpub_tbl_both_filters_pkey" PRIMARY KEY, btree (a, c) REPLICA IDENTITY
+Publications:
+ "testpub_both_filters" (a, c) WHERE (c <> 1)
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+-- Tests for partitioned table
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+ERROR: cannot use publication column list for relation "rf_tbl_abcd_part_pk"
+DETAIL: column list cannot be used for a partitioned table when publish_via_partition_root is false.
+-- ok - can use row filter for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any row filter is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+ERROR: cannot set publish_via_partition_root = false for publication "testpub6"
+DETAIL: The publication contains a column list for a partitioned table "rf_tbl_abcd_part_pk" which is not allowed when publish_via_partition_root is false.
+-- Now change the root filter to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have row filter for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -1058,6 +1424,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schema
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 5457c56b33f..be05ac9f763 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -373,6 +373,292 @@ DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+RESET client_min_messages;
+
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+-- ok: for insert-only publication, the column list is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+
+-- error: change the replica identity to "b", and then column list (a, c) fails
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+-- error: replica identity (b,c) is not covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+
+-- with REPLICA IDENTITY FULL, column lists are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+
+ALTER PUBLICATION testpub_fortable ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+
+-- make sure changing the column list is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: update the column list
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl8;
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+\d+ testpub_tbl_both_filters
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Tests for partitioned table
+
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - can use row filter for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any row filter is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- Now change the root filter to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have row filter for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
+
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -614,6 +900,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/029_column_list.pl b/src/test/subscription/t/029_column_list.pl
new file mode 100644
index 00000000000..5266967b3f4
--- /dev/null
+++ b/src/test/subscription/t/029_column_list.pl
@@ -0,0 +1,1124 @@
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test partial-column publication of tables
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# create publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+sub wait_for_subscription_sync
+{
+ my ($node) = @_;
+
+ # Also wait for initial table sync to finish
+ my $synced_query = "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+
+ $node->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+}
+
+# setup tables on both nodes
+
+# tab1: simple 1:1 replication
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+# tab2: replication from regular to table with fewer columns
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)
+));
+
+# tab3: simple 1:1 replication with weird column names
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "B" varchar, "c'" int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "c'" int)
+));
+
+# test_part: partitioned tables, with partitioning (including multi-level
+# partitioning, and fewer columns on the subscriber)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+# tab4: table with user-defined enum types
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, d text);
+));
+
+
+# TEST: create publication and subscription for some of the tables with
+# column lists
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub1
+ FOR TABLE tab1 (a, "B"), tab3 ("a'", "c'"), test_part (a, b), tab4 (a, b, d)
+ WITH (publish_via_partition_root = 'true');
+));
+
+# check that we got the right prattrs values for the publication in the
+# pg_publication_rel catalog (order by relname, to get stable ordering)
+my $result = $node_publisher->safe_psql('postgres', qq(
+ SELECT relname, prattrs
+ FROM pg_publication_rel pb JOIN pg_class pc ON(pb.prrelid = pc.oid)
+ ORDER BY relname
+));
+
+is($result, qq(tab1|1 2
+tab3|1 3
+tab4|1 2 4
+test_part|1 2), 'publication relation updated');
+
+# TEST: insert data into the tables, create subscription and see if sync
+# replicates the right columns
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (1, 2, 3);
+ INSERT INTO tab1 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (1, 2, 3);
+ INSERT INTO tab3 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (1, 'red', 3, 'oh my');
+ INSERT INTO tab4 VALUES (2, 'blue', 4, 'hello');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (1, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (2, 'bcd', '2021-07-03 11:12:13');
+ INSERT INTO test_part VALUES (7, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (8, 'bcd', '2021-07-03 11:12:13');
+));
+
+# create subscription for the publication, wait for sync to complete,
+# then check the sync results
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+4|5|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+4|6), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+7|abc
+8|bcd), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: now insert more data into the tables, and wait until we replicate
+# them (not by tablesync, but regular decoding and replication)
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (2, 3, 4);
+ INSERT INTO tab1 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (2, 3, 4);
+ INSERT INTO tab3 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (3, 'red', 5, 'foo');
+ INSERT INTO tab4 VALUES (4, 'blue', 6, 'bar');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (3, 'xxx', '2022-02-01 10:00:00');
+ INSERT INTO test_part VALUES (4, 'yyy', '2022-03-02 15:12:13');
+ INSERT INTO test_part VALUES (9, 'zzz', '2022-04-03 21:00:00');
+ INSERT INTO test_part VALUES (10, 'qqq', '2022-05-04 22:12:13');
+));
+
+# wait for catchup before checking the subscriber
+$node_publisher->wait_for_catchup('sub1');
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+2|3|
+4|5|
+5|6|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+2|4
+4|6
+5|7), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello
+3|red|foo
+4|blue|bar), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+3|xxx
+4|yyy
+7|abc
+8|bcd
+9|zzz
+10|qqq), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: do some updates on some of the tables, both on columns included
+# in the column list and other
+
+# tab1: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET "B" = 2 * "B" where a = 1));
+
+# tab1: update of non-replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET c = 2*c where a = 4));
+
+# tab3: update of non-replicated
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "B" = "B" || ' updated' where "a'" = 4));
+
+# tab3: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "c'" = 2 * "c'" where "a'" = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'blue', c = c * 2, d = d || ' updated' where a = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'red', c = c * 2, d = d || ' updated' where a = 2));
+
+# wait for the replication to catch up, and check the UPDATE results got
+# replicated correctly, with the right column list
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab1 ORDER BY a));
+is($result,
+qq(1|4|
+2|3|
+4|5|
+5|6|), 'only update on column tab1.b is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result,
+qq(1|6
+2|4
+4|6
+5|7), 'only update on column tab3.c is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab4 ORDER BY a));
+
+is($result, qq(1|blue|oh my updated
+2|red|hello updated
+3|red|foo
+4|blue|bar), 'update on column tab4.c is not replicated');
+
+
+# TEST: add table with a column list, insert data, replicate
+
+# insert some data before adding it to the publication
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (1, 'abc', 3);
+));
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2 (a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION");
+
+# wait for the tablesync to complete, add a bit more data and then check
+# the results of the replication
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (2, 'def', 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|def), 'insert on column tab2.c is not replicated');
+
+# do a couple updates, check the correct stuff gets replicated
+$node_publisher->safe_psql('postgres', qq(
+ UPDATE tab2 SET c = 5 where a = 1;
+ UPDATE tab2 SET b = 'xyz' where a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|xyz), 'update on column tab2.c is not replicated');
+
+
+# TEST: add a table to two publications with different column lists, and
+# create a single subscription replicating both publications
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+
+ -- insert a couple initial records
+ INSERT INTO tab5 VALUES (1, 11, 111, 1111);
+ INSERT INTO tab5 VALUES (2, 22, 222, 2222);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub3
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->wait_for_catchup('sub1');
+
+# insert data and make sure all the columns (union of the columns lists)
+# get fully replicated
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (3, 33, 333, 3333);
+ INSERT INTO tab5 VALUES (4, 44, 444, 4444);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111
+2|22|2222
+3|33|3333
+4|44|4444),
+ 'overlapping publications with overlapping column lists');
+
+# and finally, set the column list to ALL for one of the publications,
+# which means replicating all columns (removing the column list), but
+# first add the missing column to the table on subscriber
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub3 ALTER TABLE tab5 SET COLUMNS ALL;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ ALTER TABLE tab5 ADD COLUMN c INT;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (5, 55, 555, 5555);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111|
+2|22|2222|
+3|33|3333|
+4|44|4444|
+5|55|5555|555),
+ 'overlapping publications with overlapping column lists');
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key (but use a different column in
+# the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub4 FOR TABLE tab6 (a, b);
+
+ -- initial data
+ INSERT INTO tab6 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (2, 33, 444, 5555);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is still covered by the column list, though)
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+# we need to do the same thing on the subscriber
+# XXX What would happen if this happens before the publisher ALTER? Or
+# interleaved, somehow? But that seems unrelated to column lists.
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (3, 55, 666, 8888);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key with a key on multiple columns
+# (all of them covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub5 FOR TABLE tab7 (a, b);
+
+ -- some initial data
+ INSERT INTO tab7 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub5
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (2, 33, 444, 5555);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is not covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ ALTER TABLE tab7 ADD PRIMARY KEY (a, b);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (3, 55, 666, 7777);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+# now switch the primary key again to another columns not covered by the
+# column list, but also generate writes between the drop and creation
+# of the new constraint
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ INSERT INTO tab7 VALUES (4, 77, 888, 9999);
+ -- update/delete is not allowed for tables without RI
+ ALTER TABLE tab7 ADD PRIMARY KEY (b, a);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+ DELETE FROM tab7 WHERE a = 1;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(2|264||
+3|220||
+4|154||),
+ 'replication with the modified primary key');
+
+
+# TEST: partitioned tables (with publish_via_partition_root = false)
+# and replica identity. The (leaf) partitions may have different RI, so
+# we need to check the partition RI (with respect to the column list)
+# while attaching the partition.
+
+# First, let's create a partitioned table with two partitions, each with
+# a different RI, but a column list not covering all those RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+
+ -- initial data, one row in each partition
+ INSERT INTO test_part_a VALUES (1, 3);
+ INSERT INTO test_part_a VALUES (6, 4);
+));
+
+# do the same thing on the subscriber (with the opposite column order)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (b int, a int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+));
+
+# create a publication replicating just the column "a", which is not enough
+# for the second partition
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub6 FOR TABLE test_part_a (b, a) WITH (publish_via_partition_root = true);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_1 (a);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_2 (b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub6
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_a VALUES (2, 5);
+ INSERT INTO test_part_a VALUES (7, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT a, b FROM test_part_a ORDER BY a, b"),
+ qq(1|3
+2|5
+6|4
+7|6),
+ 'partitions with different replica identities not replicated correctly');
+
+# This time start with a column list covering RI for all partitions, but
+# then update the column list to not cover column "b" (needed by the
+# second partition)
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+
+ -- initial data, one row in each partitions
+ INSERT INTO test_part_b VALUES (1, 1);
+ INSERT INTO test_part_b VALUES (6, 2);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub7 FOR TABLE test_part_b (a, b) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub7
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_b VALUES (2, 3);
+ INSERT INTO test_part_b VALUES (7, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_b ORDER BY a, b"),
+ qq(1|1
+2|3
+6|2
+7|4),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: This time start with a column list covering RI for all partitions,
+# but then update RI for one of the partitions to not be covered by the
+# column list anymore.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+
+ -- initial data, one row for each partition
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+));
+
+# create a publication replicating data through partition root, with a column
+# filter on the root, and then add the partitions one by one with separate
+# column lists (but those are not applied)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a,c);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SUBSCRIPTION sub1;
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub8;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_c VALUES (3, 7, 8);
+ INSERT INTO test_part_c VALUES (4, 9, 10);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||5
+2|4|
+3||8
+4|9|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# create a publication not replicating data through partition root, without
+# a column list on the root, and then add the partitions one by one with
+# separate column lists
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub8;
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ TRUNCATE test_part_c;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ TRUNCATE test_part_c;
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||
+2|4|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: Start with a single partition, with RI compatible with the column
+# filter, and then attach a partition with incompatible RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ INSERT INTO test_part_d VALUES (1, 2);
+));
+
+# do the same thing on the subscriber (in fact, create both partitions right
+# away, no need to delay that)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ CREATE TABLE test_part_d_2 PARTITION OF test_part_d FOR VALUES IN (2,4);
+ ALTER TABLE test_part_d_2 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_2 REPLICA IDENTITY USING INDEX test_part_d_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub9 FOR TABLE test_part_d (a) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub9
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_d VALUES (3, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_d ORDER BY a, b"),
+ qq(1|
+3|),
+ 'partitions with different replica identities not replicated correctly');
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. So with column lists (a,b) and (a,c) we
+# should replicate (a,b,c).
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+
+ -- initial data
+ INSERT INTO test_mix_1 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_1 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_1 ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES, we should replicate all columns.
+
+# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ test_part, test_part_a, test_part_b, test_part_c, test_part_d;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
+
+ -- initial data
+ INSERT INTO test_mix_2 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_3, pub_mix_4;
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_2 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_2"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES IN SCHEMA, we should replicate all columns.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
+
+ -- initial data
+ INSERT INTO test_mix_3 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_5, pub_mix_6;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_3 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_3"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: Check handling of publish_via_partition_root - if a partition is
+# published through partition root, we should only apply the column list
+# defined for the whole table (not the partitions) - both during the initial
+# sync and when replicating changes. This is what we do for row filters.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+
+ CREATE PUBLICATION pub_root_true FOR TABLE test_root (a) WITH (publish_via_partition_root = true);
+
+ -- initial data
+ INSERT INTO test_root VALUES (1, 2, 3);
+ INSERT INTO test_root VALUES (10, 20, 30);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_root_true;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_root VALUES (2, 3, 4);
+ INSERT INTO test_root VALUES (11, 21, 31);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_root ORDER BY a, b, c"),
+ qq(1||
+2||
+10||
+11||),
+ 'publication via partition root applies column list');
+
+
+# TEST: Multiple publications which publish schema of parent table and
+# partition. The partition is published through two publications, once
+# through a schema (so no column list) containing the parent, and then
+# also directly (with a columns list). The expected outcome is there is
+# no column list.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub1, pub2, pub3, pub4, pub5, pub6, pub7, pub8;
+
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA s1;
+ CREATE PUBLICATION pub2 FOR TABLE t_1(b);
+
+ -- initial data
+ INSERT INTO s1.t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub1, pub2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'two publications, publishing the same relation');
+
+# Now resync the subcription, but with publications in the opposite order.
+# The result should be the same.
+
+$node_subscriber->safe_psql('postgres', qq(
+ TRUNCATE s1.t;
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub1;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (7, 8, 9);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(7|8|9),
+ 'two publications, publishing the same relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub3 FOR TABLE t_1 (a), t_2
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub3;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+# Note: The difference from the preceding test is that in this case both
+# relations have a column list defined.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub4 FOR TABLE t_1 (a), t_2 (b)
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
+
+done_testing();
--
2.34.1
On 3/9/22 10:20, Peter Eisentraut wrote:
On 07.03.22 16:18, Tomas Vondra wrote:
AFAICS these issues should be resolved by the adoption of the row-filter
approach (i.e. it should fail the same way as for row filter).The first two patches (additional testing for row filtering feature)
look okay to me.Attached is a fixup patch for your main feature patch (the third one).
It's a bit of code and documentation cleanup, but mainly I removed the
term "column filter" from the patch. Half the code was using "column
list" or similar and half the code "column filter", which was confusing.
Also, there seemed to be a bit of copy-and-pasting from row-filter code
going on, with some code comments not quite sensible, so I rewrote some
of them. Also some code used "rf" and "cf" symbols which were a bit
hard to tell apart. A few more letters can increase readability.Note in publicationcmds.c OpenTableList() the wrong if condition was used.
Thanks, I've merged these changes into the patch.
I'm still confused about the intended replica identity handling. This
patch still checks whether the column list contains the replica identity
at DDL time. And then it also checks at execution time. I thought the
latest understanding was that the DDL-time checking would be removed. I
think it's basically useless now, since as the test cases show, you can
subvert those checks by altering the replica identity later.
Are you sure? Which part of the patch does that? AFAICS we only do those
checks in CheckCmdReplicaIdentity now, but maybe I'm missing something.
Are you sure you're not looking at some older patch version?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 3/10/22 19:17, Tomas Vondra wrote:
On 3/9/22 11:12, houzj.fnst@fujitsu.com wrote:
Hi,
Here are some tests and results about the table sync query of
column filter patch and row filter.1) multiple publications which publish schema of parent table and partition.
----pub
create schema s1;
create table s1.t (a int, b int, c int) partition by range (a);
create table t_1 partition of s1.t for values from (1) to (10);
create publication pub1 for all tables in schema s1;
create publication pub2 for table t_1(b);----sub
- prepare tables
CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub1, pub2;When doing table sync for 't_1', the column list will be (b). I think it should
be no filter because table t_1 is also published via ALL TABLES IN SCHMEA
publication.For Row Filter, it will use no filter for this case.
2) one publication publishes both parent and child
----pub
create table t (a int, b int, c int) partition by range (a);
create table t_1 partition of t for values from (1) to (10)
partition by range (a);
create table t_2 partition of t_1 for values from (1) to (10);create publication pub2 for table t_1(a), t_2
with (PUBLISH_VIA_PARTITION_ROOT);----sub
- prepare tables
CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;When doing table sync for table 't_1', it has no column list. I think the
expected column list is (a).For Row Filter, it will use the row filter of the top most parent table(t_1) in
this case.3) one publication publishes both parent and child
----pub
create table t (a int, b int, c int) partition by range (a);
create table t_1 partition of t for values from (1) to (10)
partition by range (a);
create table t_2 partition of t_1 for values from (1) to (10);create publication pub2 for table t_1(a), t_2(b)
with (PUBLISH_VIA_PARTITION_ROOT);----sub
- prepare tables
CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;When doing table sync for table 't_1', the column list would be (a, b). I think
the expected column list is (a).For Row Filter, it will use the row filter of the top most parent table(t_1) in
this case.Attached is an updated patch version, addressing all of those issues.
0001 is a bugfix, reworking how we calculate publish_as_relid. The old
approach was unstable with multiple publications, giving different
results depending on order of the publications. This should be
backpatched into PG13 where publish_via_partition_root was introduced, I
think.0002 is the main patch, merging the changes proposed by Peter and fixing
the issues reported here. In most cases this means adopting the code
used for row filters, and perhaps simplifying it a bit.But I also tried to implement a row-filter test for 0001, and I'm not
sure I understand the behavior I observe. Consider this:-- a chain of 3 partitions (on both publisher and subscriber)
CREATE TABLE test_part_rf (a int primary key, b int, c int)
PARTITION BY LIST (a);CREATE TABLE test_part_rf_1
PARTITION OF test_part_rf FOR VALUES IN (1,2,3,4,5)
PARTITION BY LIST (a);CREATE TABLE test_part_rf_2
PARTITION OF test_part_rf_1 FOR VALUES IN (1,2,3,4,5);-- initial data
INSERT INTO test_part_rf VALUES (1, 5, 100);
INSERT INTO test_part_rf VALUES (2, 15, 200);-- two publications, each adding a different partition
CREATE PUBLICATION test_pub_part_1 FOR TABLE test_part_rf_1
WHERE (b < 10) WITH (publish_via_partition_root);CREATE PUBLICATION test_pub_part_2 FOR TABLE test_part_rf_2
WHERE (b > 10) WITH (publish_via_partition_root);-- now create the subscription (also try opposite ordering)
CREATE SUBSCRIPTION test_part_sub CONNECTION '...'
PUBLICATION test_pub_part_1, test_pub_part_2;-- wait for sync
-- inert some more data
INSERT INTO test_part_rf VALUES (3, 6, 300);
INSERT INTO test_part_rf VALUES (4, 16, 400);-- wait for catchup
Now, based on the discussion here, my expectation is that we'll use the
row filter from the top-most ancestor in any publication, which in this
case is test_part_rf_1. Hence the filter should be (b < 10).So I'd expect these rows to be replicated:
1,5,100
3,6,300But that's not what I get, unfortunately. I get different results,
depending on the order of publications:1) test_pub_part_1, test_pub_part_2
1|5|100
2|15|200
3|6|300
4|16|4002) test_pub_part_2, test_pub_part_1
3|6|300
4|16|400That seems pretty bizarre, because it either means we're not enforcing
any filter or some strange combination of filters (notice that for (2)
we skip/replicate rows matching either filter).I have to be missing something important, but this seems confusing.
There's a patch adding a simple test case to 028_row_filter.sql (named
.txt, so as not to confuse cfbot).
FWIW I think the reason is pretty simple - pgoutput_row_filter_init is
broken. It assumes you can just do this
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(entry->publish_as_relid),
ObjectIdGetDatum(pub->oid));
if (HeapTupleIsValid(rftuple))
{
/* Null indicates no filter. */
rfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
Anum_pg_publication_rel_prqual,
&pub_no_filter);
}
else
{
pub_no_filter = true;
}
and pub_no_filter=true means there's no filter at all. Which is
nonsense, because we're using publish_as_relid here - the publication
may not include this particular ancestor, in which case we need to just
ignore this publication.
So yeah, this needs to be reworked.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 3/10/22 04:09, Amit Kapila wrote:
On Wed, Mar 9, 2022 at 3:33 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Mar 7, 2022 at 8:48 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:OK, I reworked this to do the same thing as the row filtering patch.
Thanks, I'll check this.
Some assorted comments:
=====================
1. We don't need to send a column list for the old tuple in case of an
update (similar to delete). It is not required to apply a column
filter for those cases because we ensure that RI must be part of the
column list for updates and deletes.
I'm not sure which part of the code does this refer to?
2. + /* + * Check if all columns referenced in the column filter are part of + * the REPLICA IDENTITY index or not.I think this comment is reverse. The rule we follow here is that
attributes that are part of RI must be there in a specified column
list. This is used at two places in the patch.
Yeah, you're right. Will fix.
3. get_rel_sync_entry() { /* XXX is there a danger of memory leak here? beware */ + oldctx = MemoryContextSwitchTo(CacheMemoryContext); + for (int i = 0; i < nelems; i++) ... }Similar to the row filter, I think we need to use
entry->cache_expr_cxt to allocate this. There are other usages of
CacheMemoryContext in this part of the code but I think those need to
be also changed and we can do that as a separate patch. If we do the
suggested change then we don't need to separately free columns.
I agree a shorter-lived context would be better than CacheMemoryContext,
but "expr" seems to indicate it's for the expression only, so maybe we
should rename that. But do we really want a memory context for every
single entry?
4. I think we don't need the DDL changes in AtExecDropColumn. Instead,
we can change the dependency of columns to NORMAL during publication
commands.
I'll think about that.
5. There is a reference to check_publication_columns but that function
is removed from the patch.
Right, will fix.
6.
/*
* If we know everything is replicated and the row filter is invalid
* for update and delete, there is no point to check for other
* publications.
*/
if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
!pubdesc->rf_valid_for_update && !pubdesc->rf_valid_for_delete)
break;/*
* If we know everything is replicated and the column filter is invalid
* for update and delete, there is no point to check for other
* publications.
*/
if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
!pubdesc->cf_valid_for_update && !pubdesc->cf_valid_for_delete)
break;Can we combine these two checks?
I was worried it'd get too complex / hard to understand, but I'll think
about maybe simplifying the conditions a bit.
I feel this patch needs a more thorough review.
I won't object to more review, of course.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 3/10/22 20:10, Tomas Vondra wrote:
On 3/10/22 19:17, Tomas Vondra wrote:
On 3/9/22 11:12, houzj.fnst@fujitsu.com wrote:
Hi,
Here are some tests and results about the table sync query of
column filter patch and row filter.1) multiple publications which publish schema of parent table and partition.
----pub
create schema s1;
create table s1.t (a int, b int, c int) partition by range (a);
create table t_1 partition of s1.t for values from (1) to (10);
create publication pub1 for all tables in schema s1;
create publication pub2 for table t_1(b);----sub
- prepare tables
CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub1, pub2;When doing table sync for 't_1', the column list will be (b). I think it should
be no filter because table t_1 is also published via ALL TABLES IN SCHMEA
publication.For Row Filter, it will use no filter for this case.
2) one publication publishes both parent and child
----pub
create table t (a int, b int, c int) partition by range (a);
create table t_1 partition of t for values from (1) to (10)
partition by range (a);
create table t_2 partition of t_1 for values from (1) to (10);create publication pub2 for table t_1(a), t_2
with (PUBLISH_VIA_PARTITION_ROOT);----sub
- prepare tables
CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;When doing table sync for table 't_1', it has no column list. I think the
expected column list is (a).For Row Filter, it will use the row filter of the top most parent table(t_1) in
this case.3) one publication publishes both parent and child
----pub
create table t (a int, b int, c int) partition by range (a);
create table t_1 partition of t for values from (1) to (10)
partition by range (a);
create table t_2 partition of t_1 for values from (1) to (10);create publication pub2 for table t_1(a), t_2(b)
with (PUBLISH_VIA_PARTITION_ROOT);----sub
- prepare tables
CREATE SUBSCRIPTION sub CONNECTION 'port=10000 dbname=postgres' PUBLICATION pub2;When doing table sync for table 't_1', the column list would be (a, b). I think
the expected column list is (a).For Row Filter, it will use the row filter of the top most parent table(t_1) in
this case.Attached is an updated patch version, addressing all of those issues.
0001 is a bugfix, reworking how we calculate publish_as_relid. The old
approach was unstable with multiple publications, giving different
results depending on order of the publications. This should be
backpatched into PG13 where publish_via_partition_root was introduced, I
think.0002 is the main patch, merging the changes proposed by Peter and fixing
the issues reported here. In most cases this means adopting the code
used for row filters, and perhaps simplifying it a bit.But I also tried to implement a row-filter test for 0001, and I'm not
sure I understand the behavior I observe. Consider this:-- a chain of 3 partitions (on both publisher and subscriber)
CREATE TABLE test_part_rf (a int primary key, b int, c int)
PARTITION BY LIST (a);CREATE TABLE test_part_rf_1
PARTITION OF test_part_rf FOR VALUES IN (1,2,3,4,5)
PARTITION BY LIST (a);CREATE TABLE test_part_rf_2
PARTITION OF test_part_rf_1 FOR VALUES IN (1,2,3,4,5);-- initial data
INSERT INTO test_part_rf VALUES (1, 5, 100);
INSERT INTO test_part_rf VALUES (2, 15, 200);-- two publications, each adding a different partition
CREATE PUBLICATION test_pub_part_1 FOR TABLE test_part_rf_1
WHERE (b < 10) WITH (publish_via_partition_root);CREATE PUBLICATION test_pub_part_2 FOR TABLE test_part_rf_2
WHERE (b > 10) WITH (publish_via_partition_root);-- now create the subscription (also try opposite ordering)
CREATE SUBSCRIPTION test_part_sub CONNECTION '...'
PUBLICATION test_pub_part_1, test_pub_part_2;-- wait for sync
-- inert some more data
INSERT INTO test_part_rf VALUES (3, 6, 300);
INSERT INTO test_part_rf VALUES (4, 16, 400);-- wait for catchup
Now, based on the discussion here, my expectation is that we'll use the
row filter from the top-most ancestor in any publication, which in this
case is test_part_rf_1. Hence the filter should be (b < 10).So I'd expect these rows to be replicated:
1,5,100
3,6,300But that's not what I get, unfortunately. I get different results,
depending on the order of publications:1) test_pub_part_1, test_pub_part_2
1|5|100
2|15|200
3|6|300
4|16|4002) test_pub_part_2, test_pub_part_1
3|6|300
4|16|400That seems pretty bizarre, because it either means we're not enforcing
any filter or some strange combination of filters (notice that for (2)
we skip/replicate rows matching either filter).I have to be missing something important, but this seems confusing.
There's a patch adding a simple test case to 028_row_filter.sql (named
.txt, so as not to confuse cfbot).FWIW I think the reason is pretty simple - pgoutput_row_filter_init is
broken. It assumes you can just do thisrftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(entry->publish_as_relid),
ObjectIdGetDatum(pub->oid));if (HeapTupleIsValid(rftuple))
{
/* Null indicates no filter. */
rfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
Anum_pg_publication_rel_prqual,
&pub_no_filter);
}
else
{
pub_no_filter = true;
}and pub_no_filter=true means there's no filter at all. Which is
nonsense, because we're using publish_as_relid here - the publication
may not include this particular ancestor, in which case we need to just
ignore this publication.So yeah, this needs to be reworked.
I spent a bit of time looking at this, and I think a minor change in
get_rel_sync_entry() fixes this - it's enough to ensure rel_publications
only includes publications that actually include publish_as_relid.
But this does not address tablesync.c :-( That still copies everything,
because it decides to sync both rels (test_pub_part_1, test_pub_part_2),
with it's row filter. On older releases this would fail, because we'd
start two workers:
1) COPY public.test_part_rf_2 TO STDOUT
2) COPY (SELECT a, b, c FROM public.test_part_rf_1) TO STDOUT
And that ends up inserting date from test_part_rf_2 twice. But now we
end up doing this instead:
1) COPY (SELECT a, b, c FROM public.test_part_rf_1 WHERE (b < 10)) TO STDOUT
2) COPY (SELECT a, b, c FROM ONLY public.test_part_rf_2 WHERE (b > 10))
TO STDOUT
Which no longer conflicts, because those subsets are mutually exclusive
(due to how the filter is defined), so the sync succeeds.
But I find this really weird - I think it's reasonable to expect the
sync to produce the same result as if the data was inserted and
replicated, and this just violates that.
Shouldn't tablesync calculate a list of relations in a way that prevents
such duplicate / overlapping syncs? In any case, this sync issue looks
entirely unrelated to the column filtering patch.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
0001-fixup-publish_as_relid-20220311.patchtext/x-patch; charset=UTF-8; name=0001-fixup-publish_as_relid-20220311.patchDownload
From 0726a46f050980df2ebbca48078b4fd3b0d374f8 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Wed, 9 Mar 2022 18:10:56 +0100
Subject: [PATCH 1/3] fixup: publish_as_relid
Make sure to determine the top-most ancestor listed in any publication.
Otherwise we might end up with different values depending on the order
of publications (as listed in subscription).
---
src/backend/replication/pgoutput/pgoutput.c | 32 +++++++++++++++++++--
1 file changed, 30 insertions(+), 2 deletions(-)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a0477f0..dbac2690b7f 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1815,11 +1815,17 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
Publication *pub = lfirst(lc);
bool publish = false;
+ /*
+ * Under what relid should we publish changes in this publication?
+ * We'll use the top-most relid across all publications.
+ */
+ Oid pub_relid = relid;
+
if (pub->alltables)
{
publish = true;
if (pub->pubviaroot && am_partition)
- publish_as_relid = llast_oid(get_partition_ancestors(relid));
+ pub_relid = llast_oid(get_partition_ancestors(relid));
}
if (!publish)
@@ -1844,7 +1850,7 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
{
ancestor_published = true;
if (pub->pubviaroot)
- publish_as_relid = ancestor;
+ pub_relid = ancestor;
}
}
@@ -1862,12 +1868,34 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
if (publish &&
(relkind != RELKIND_PARTITIONED_TABLE || pub->pubviaroot))
{
+ List *ancestors;
+
entry->pubactions.pubinsert |= pub->pubactions.pubinsert;
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
rel_publications = lappend(rel_publications, pub);
+
+ /*
+ * We want to publish the changes as the top-most ancestor
+ * across all publications. So we fetch all ancestors of the
+ * relid calculated for this publication, and check if the
+ * already calculated value is in the list. If yes, we can
+ * ignore the new value (as it's a child). Otherwise the new
+ * value is an ancestor, so we keep it.
+ */
+ ancestors = get_partition_ancestors(pub_relid);
+
+ /*
+ * The new pub_relid is a child of the current publish_as_relid
+ * value, so we can ignore it.
+ */
+ if (list_member_oid(ancestors, publish_as_relid))
+ continue;
+
+ /* The new value is an ancestor, so let's keep it. */
+ publish_as_relid = pub_relid;
}
}
--
2.34.1
0002-fixup-row-filter-publications-20220311.patchtext/x-patch; charset=UTF-8; name=0002-fixup-row-filter-publications-20220311.patchDownload
From 09b3cb5368b2fd3448384c8cf8860801d7328479 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Fri, 11 Mar 2022 02:23:25 +0100
Subject: [PATCH 2/3] fixup: row-filter publications
When initializing the row filter, consider only publications that
actually include the relation (publish_as_relid). The publications may
include different ancestors, in which case the function would get
confused and conclude there's no row filter.
---
src/backend/replication/pgoutput/pgoutput.c | 16 ++++++++++++----
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index dbac2690b7f..faf7555f913 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1875,8 +1875,6 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
- rel_publications = lappend(rel_publications, pub);
-
/*
* We want to publish the changes as the top-most ancestor
* across all publications. So we fetch all ancestors of the
@@ -1894,8 +1892,18 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
if (list_member_oid(ancestors, publish_as_relid))
continue;
- /* The new value is an ancestor, so let's keep it. */
- publish_as_relid = pub_relid;
+ /*
+ * If the new value is an ancestor, discard the list of
+ * publications through which we replicate it.
+ */
+ if (publish_as_relid != pub_relid)
+ {
+ rel_publications = NIL;
+ publish_as_relid = pub_relid;
+ }
+
+ /* Track the publications. */
+ rel_publications = lappend(rel_publications, pub);
}
}
--
2.34.1
0003-Allow-specifying-column-filters-for-logical-20220311.patchtext/x-patch; charset=UTF-8; name=0003-Allow-specifying-column-filters-for-logical-20220311.patchDownload
From 66d45f6a5865c67d2377e2db2afd247cf6842fde Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Thu, 10 Mar 2022 17:31:57 +0100
Subject: [PATCH 3/3] Allow specifying column filters for logical replication
This allows specifying an optional column list when adding a table to
logical replication. Columns not included on this list are not sent to
the subscriber. The filter is specified as a list of columns after the
table name, enclosed in parentheses.
For UPDATE/DELETE publications, the column filter needs to cover all
REPLICA IDENTITY columns. For INSERT publications, the column list is
arbitrary and may omit some REPLICA IDENTITY columns. Furthermore, if
the table uses REPLICA IDENTITY FULL, column filter is not allowed.
The column filter can contain only simple column references. Complex
expressions, function calls etc. are not allowed. This restriction could
be relaxed in the future.
During the initial table synchronization, only columns specified in the
column filter are copied to the subscriber. If the subscription has
several publications, containing the same table with different column
filters, columns specified in any of the filters will be copied. This
means all columns are replicated if the table has no column filter at
all (which is treated as column filter with all columns), of when of the
publications is defined as FOR ALL TABLES (possibly IN SCHEMA for the
schema of the table).
For partitioned tables, publish_via_partition_root determines whether
the row filter for the root or leaf relation will be used. If the
parameter is 'false' (the default), the filter defined for the leaf
relation is used. Otherwise, the column filter for the root partition
will be used.
Psql commands \dRp+ and \d <table-name> now display any column filters.
Author: Rahila Syed <rahilasyed90@gmail.com>
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 15 +-
doc/src/sgml/protocol.sgml | 3 +-
doc/src/sgml/ref/alter_publication.sgml | 27 +-
doc/src/sgml/ref/create_publication.sgml | 17 +-
src/backend/catalog/pg_publication.c | 256 +++++
src/backend/commands/publicationcmds.c | 364 +++++-
src/backend/commands/tablecmds.c | 36 +-
src/backend/executor/execReplication.c | 19 +-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 60 +-
src/backend/replication/logical/proto.c | 56 +-
src/backend/replication/logical/tablesync.c | 184 ++-
src/backend/replication/pgoutput/pgoutput.c | 156 ++-
src/backend/utils/cache/relcache.c | 33 +-
src/bin/pg_dump/pg_dump.c | 47 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 60 +
src/bin/psql/describe.c | 40 +-
src/include/catalog/pg_publication.h | 13 +
src/include/catalog/pg_publication_rel.h | 1 +
src/include/commands/publicationcmds.h | 4 +-
src/include/nodes/parsenodes.h | 4 +-
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 375 +++++++
src/test/regress/sql/publication.sql | 290 +++++
src/test/subscription/t/029_column_list.pl | 1124 +++++++++++++++++++
27 files changed, 3112 insertions(+), 81 deletions(-)
create mode 100644 src/test/subscription/t/029_column_list.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 83987a99045..c043da37aee 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -4392,7 +4392,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para>
<para>
This is an array of <structfield>indnatts</structfield> values that
- indicate which table columns this index indexes. For example a value
+ indicate which table columns this index indexes. For example, a value
of <literal>1 3</literal> would mean that the first and the third table
columns make up the index entries. Key columns come before non-key
(included) columns. A zero in this array indicates that the
@@ -6263,6 +6263,19 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to schema
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>prattrs</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+ </para>
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example, a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all columns are published.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 0695bcd423e..92cd0f9c9f7 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7006,7 +7006,8 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column included in
+ the publication (except generated columns):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 32b75f6c78e..470d50a2447 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -25,12 +25,13 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD <replace
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ALTER TABLE <replaceable class="parameter">table_name</replaceable> SET COLUMNS { ( <replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -64,6 +65,12 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
command retain their previous settings.
</para>
+ <para>
+ The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows changing
+ the set of columns that are included in the publication. If a column list
+ is specified, it must include the replica identity columns.
+ </para>
+
<para>
The remaining variants change the owner and the name of the publication.
</para>
@@ -112,6 +119,14 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ </para>
+
+ <para>
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
If the optional <literal>WHERE</literal> clause is specified, rows for
which the <replaceable class="parameter">expression</replaceable>
evaluates to false or null will not be published. Note that parentheses
@@ -172,9 +187,15 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
</para>
<para>
- Add some tables to the publication:
+ Add tables to the publication:
+<programlisting>
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS (user_id, firstname, lastname);
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 4979b9b646d..fb2d013393b 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -86,6 +86,13 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<literal>TRUNCATE</literal> commands.
</para>
+ <para>
+ When a column list is specified, only the named columns are replicated.
+ If no column list is specified, all columns of the table are replicated
+ through this publication, including any columns added later. If a column
+ list is specified, it must include the replica identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
@@ -327,6 +334,14 @@ CREATE PUBLICATION production_publication FOR TABLE users, departments, ALL TABL
<structname>sales</structname>:
<programlisting>
CREATE PUBLICATION sales_publication FOR ALL TABLES IN SCHEMA marketing, sales;
+</programlisting></para>
+
+ <para>
+ Create a publication that publishes all changes for table <structname>users</structname>,
+ but replicates only columns <structname>user_id</structname> and
+ <structname>firstname</structname>:
+<programlisting>
+CREATE PUBLICATION users_filtered FOR TABLE users (user_id, firstname);
</programlisting></para>
</refsect1>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 25998fbb39b..3275a7c8b9c 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,6 +45,9 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+static void publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs);
+
/*
* Check if relation can be in given publication and throws appropriate
* error if not.
@@ -328,6 +331,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
Oid relid = RelationGetRelid(targetrel);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ AttrNumber *attarray;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -355,6 +360,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
check_publication_add_relation(targetrel);
+ /*
+ * Translate column names to attnums and check the column list is valid.
+ * We also deconstruct the bitmap into an array of attnums, for storing
+ * in the catalog.
+ */
+ publication_translate_columns(pri->relation, pri->columns,
+ &natts, &attarray);
+
/* Form a tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -367,6 +380,17 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ /* Add column list, if available */
+ if (pri->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+
/* Add qualifications, if available */
if (pri->whereClause != NULL)
values[Anum_pg_publication_rel_prqual - 1] = CStringGetTextDatum(nodeToString(pri->whereClause));
@@ -382,6 +406,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
/* Register dependencies as needed */
ObjectAddressSet(myself, PublicationRelRelationId, pubreloid);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ pfree(attarray);
+
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -415,6 +447,154 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
return myself;
}
+/*
+ * Update the column list for a relation in a publication.
+ */
+void
+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns)
+{
+ AttrNumber *attarray;
+ HeapTuple copytup;
+ int natts;
+ bool nulls[Natts_pg_publication_rel];
+ bool replaces[Natts_pg_publication_rel];
+ Datum values[Natts_pg_publication_rel];
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ replaces[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ deleteDependencyRecordsForClass(PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid,
+ RelationRelationId,
+ DEPENDENCY_AUTO);
+
+ if (columns == NULL)
+ {
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+ }
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ int2vector *prattrs;
+ Form_pg_publication_rel pubrel;
+
+ publication_translate_columns(targetrel, columns, &natts, &attarray);
+
+ /* XXX "pub" is leaked here ??? */
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+
+ /* Add dependencies on the new list of columns */
+ pubrel = (Form_pg_publication_rel) GETSTRUCT(pubreltup);
+ ObjectAddressSet(myself, PublicationRelRelationId, pubrel->oid);
+
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId,
+ RelationGetRelid(targetrel), attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ }
+
+ copytup = heap_modify_tuple(pubreltup, RelationGetDescr(pubrel),
+ values, nulls, replaces);
+
+ CatalogTupleUpdate(pubrel, &pubreltup->t_self, copytup);
+
+ heap_freetuple(copytup);
+}
+
+/*
+ * qsort comparator for attnums
+ *
+ * XXX We already have compare_int16, so maybe let's share that, somehow?
+ */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list (no system or generated attributes,
+ * no duplicates). Additional checks with replica identity are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ *
+ * XXX Should this detect duplicate columns?
+ */
+static void
+publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs)
+{
+ AttrNumber *attarray;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference generated column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /* Be tidy, so that the catalog representation is always sorted */
+ qsort(attarray, n, sizeof(AttrNumber), compare_int16);
+
+ *natts = n;
+ *attrs = attarray;
+
+ bms_free(set);
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -522,6 +702,82 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+ Form_pg_publication_rel pubrel;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ /* no column list for this publications/relation */
+ if (isnull)
+ continue;
+
+ pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
+
+ pubs = lappend_oid(pubs, pubrel->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 16b8661a1b7..b32ec275557 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -296,7 +296,7 @@ contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
* Returns true if any invalid column is found.
*/
bool
-contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
+pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
bool pubviaroot)
{
HeapTuple rftuple;
@@ -367,6 +367,123 @@ contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
return result;
}
+/*
+ * Check if all columns referenced in the column list are part of the
+ * REPLICA IDENTITY index or not.
+ *
+ * Returns true if any invalid column is found.
+ */
+bool
+pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
+ bool pubviaroot)
+{
+ HeapTuple tuple;
+ Oid relid = RelationGetRelid(relation);
+ Oid publish_as_relid = RelationGetRelid(relation);
+ bool result = false;
+ Datum datum;
+ bool isnull;
+
+ /*
+ * For a partition, if pubviaroot is true, find the topmost ancestor that
+ * is published via this publication as we need to use its column list
+ * for the changes.
+ *
+ * Note that even though the column list used is for an ancestor, the
+ * REPLICA IDENTITY used will be for the actual child table.
+ */
+ if (pubviaroot && relation->rd_rel->relispartition)
+ {
+ publish_as_relid = GetTopMostAncestorInPublication(pubid, ancestors);
+
+ if (!OidIsValid(publish_as_relid))
+ publish_as_relid = relid;
+ }
+
+ tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tuple))
+ return false;
+
+ datum = SysCacheGetAttr(PUBLICATIONRELMAP, tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ if (!isnull)
+ {
+ int x;
+ Bitmapset *idattrs;
+ Bitmapset *columns = NULL;
+
+ /* With REPLICA IDENTITY FULL, no column list is allowed. */
+ if (relation->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ result = true;
+
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(datum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ for (int i = 0; i < nelems; i++)
+ columns = bms_add_member(columns, elems[i]);
+ }
+
+ /* Remember columns that are part of the REPLICA IDENTITY */
+ idattrs = RelationGetIndexAttrBitmap(relation,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * Attnums in the bitmap returned by RelationGetIndexAttrBitmap are
+ * offset (to handle system columns the usual way), while column list
+ * does not use offset, so we can't do bms_is_subset(). Instead, we have
+ * to loop over the idattrs and check all of them are in the filter.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ AttrNumber attnum = (x + FirstLowInvalidHeapAttributeNumber);
+
+ /*
+ * If pubviaroot is true, we are validating the row filter of the
+ * parent table, but the bitmap contains the replica identity
+ * information of the child table. So, get the column number of the
+ * child table as parent and child column order could be different.
+ */
+ if (pubviaroot)
+ {
+ /* attnum is for child table */
+ char *colname = get_attname(relid, attnum, false);
+
+ /*
+ * Determine the matching attnum in parent (because the column
+ * filter is defined on the parent).
+ */
+ attnum = get_attnum(publish_as_relid, colname);
+ }
+
+ if (!bms_is_member(attnum, columns))
+ {
+ result = true;
+ break;
+ }
+ }
+
+ bms_free(idattrs);
+ bms_free(columns);
+ }
+
+ ReleaseSysCache(tuple);
+
+ return result;
+}
+
/* check_functions_in_node callback */
static bool
contain_mutable_or_user_functions_checker(Oid func_id, void *context)
@@ -608,6 +725,45 @@ TransformPubWhereClauses(List *tables, const char *queryString,
}
}
+
+/*
+ * Transform the publication column lists expression for all the relations
+ * in the list.
+ *
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow column lists on the leaf relations. So maybe rename it?
+ */
+static void
+TransformPubColumnList(List *tables, const char *queryString,
+ bool pubviaroot)
+{
+ ListCell *lc;
+
+ foreach(lc, tables)
+ {
+ PublicationRelInfo *pri = (PublicationRelInfo *) lfirst(lc);
+
+ if (pri->columns == NIL)
+ continue;
+
+ /*
+ * If the publication doesn't publish changes via the root partitioned
+ * table, the partition's column list will be used. So disallow using
+ * the column list on partitioned table in this case.
+ */
+ if (!pubviaroot &&
+ pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot use publication column list for relation \"%s\"",
+ RelationGetRelationName(pri->relation)),
+ errdetail("column list cannot be used for a partitioned table when %s is false.",
+ "publish_via_partition_root")));
+ }
+}
+
/*
* Create new publication.
*/
@@ -724,6 +880,9 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
TransformPubWhereClauses(rels, pstate->p_sourcetext,
publish_via_partition_root);
+ TransformPubColumnList(rels, pstate->p_sourcetext,
+ publish_via_partition_root);
+
PublicationAddTables(puboid, rels, true, NULL);
CloseTableList(rels);
}
@@ -754,6 +913,52 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
return myself;
}
+/*
+ * Change the column list of a relation in a publication
+ */
+static void
+PublicationSetColumns(AlterPublicationStmt *stmt,
+ Form_pg_publication pubform, PublicationTable *table)
+{
+ Relation rel,
+ urel;
+ HeapTuple tup;
+ ObjectAddress obj,
+ secondary;
+
+ rel = table_open(PublicationRelRelationId, RowExclusiveLock);
+ urel = table_openrv(table->relation, ShareUpdateExclusiveLock);
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(RelationGetRelid(urel)),
+ ObjectIdGetDatum(pubform->oid));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errmsg("relation \"%s\" is not already in publication \"%s\"",
+ table->relation->relname,
+ NameStr(pubform->pubname)));
+
+ publication_set_table_columns(rel, tup, urel, table->columns);
+
+ ObjectAddressSet(obj, PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->oid);
+ ObjectAddressSet(secondary, RelationRelationId, RelationGetRelid(urel));
+ EventTriggerCollectSimpleCommand(obj, secondary, (Node *) stmt);
+
+ ReleaseSysCache(tup);
+
+ /*
+ * Invalidate relcache for this relation, to force rebuilding the
+ * publication description.
+ */
+ CacheInvalidateRelcache(urel);
+
+ table_close(rel, RowExclusiveLock);
+ table_close(urel, NoLock);
+
+ InvokeObjectPostAlterHook(PublicationRelationId, pubform->oid, 0);
+}
+
/*
* Change options of a publication.
*/
@@ -783,8 +988,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* If the publication doesn't publish changes via the root partitioned
- * table, the partition's row filter will be used. So disallow using WHERE
- * clause on partitioned table in this case.
+ * table, the partition's row filter and column list will be used. So disallow
+ * using WHERE clause and column lists on partitioned table in this case.
*/
if (!pubform->puballtables && publish_via_partition_root_given &&
!publish_via_partition_root)
@@ -792,7 +997,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* Lock the publication so nobody else can do anything with it. This
* prevents concurrent alter to add partitioned table(s) with WHERE
- * clause(s) which we don't allow when not publishing via root.
+ * clause(s) and/or column lists which we don't allow when not
+ * publishing via root.
*/
LockDatabaseObject(PublicationRelationId, pubform->oid, 0,
AccessShareLock);
@@ -804,13 +1010,21 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
HeapTuple rftuple;
Oid relid = lfirst_oid(lc);
+ bool has_column_list;
+ bool has_row_filter;
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubform->oid));
+ has_row_filter
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL);
+
+ has_column_list
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prattrs, NULL);
+
if (HeapTupleIsValid(rftuple) &&
- !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL))
+ (has_row_filter || has_column_list))
{
HeapTuple tuple;
@@ -819,7 +1033,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
Form_pg_class relform = (Form_pg_class) GETSTRUCT(tuple);
- if (relform->relkind == RELKIND_PARTITIONED_TABLE)
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_row_filter)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot set %s for publication \"%s\"",
@@ -830,6 +1045,18 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
NameStr(relform->relname),
"publish_via_partition_root")));
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_column_list)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot set %s for publication \"%s\"",
+ "publish_via_partition_root = false",
+ stmt->pubname),
+ errdetail("The publication contains a column list for a partitioned table \"%s\" "
+ "which is not allowed when %s is false.",
+ NameStr(relform->relname),
+ "publish_via_partition_root")));
+
ReleaseSysCache(tuple);
}
@@ -838,6 +1065,16 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
}
}
+ /*
+ * FIXME check pubactions vs. replica identity, to ensure the replica
+ * identity is included in the column list. Only do this for update
+ * and delete publications. See check_publication_columns.
+ *
+ * XXX This is needed because publish_via_partition_root may change,
+ * in which case the row filters may be invalid (e.g. with pvpr=false
+ * there must be no filter on partitioned tables).
+ */
+
/* Everything ok, form a new tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -975,10 +1212,20 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
PublicationAddTables(pubid, rels, false, stmt);
}
else if (stmt->action == AP_DropObjects)
PublicationDropTables(pubid, rels, false);
+ else if (stmt->action == AP_SetColumns)
+ {
+ Assert(schemaidlist == NIL);
+ Assert(list_length(tables) == 1);
+
+ PublicationSetColumns(stmt, pubform,
+ linitial_node(PublicationTable, tables));
+ }
else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
@@ -991,6 +1238,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
/*
* To recreate the relation list for the publication, look for
* existing relations that do not need to be dropped.
@@ -1002,32 +1251,85 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
PublicationRelInfo *oldrel;
bool found = false;
HeapTuple rftuple;
- bool rfisnull = true;
Node *oldrelwhereclause = NULL;
+ Bitmapset *oldcolumns = NULL;
/* look up the cache for the old relmap */
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(oldrelid),
ObjectIdGetDatum(pubid));
+ /*
+ * See if the existing relation currently has a WHERE clause or a
+ * column list. We need to compare those too.
+ */
if (HeapTupleIsValid(rftuple))
{
+ bool isnull = true;
Datum whereClauseDatum;
+ Datum columnListDatum;
+ /* Load the WHERE clause for this table. */
whereClauseDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
Anum_pg_publication_rel_prqual,
- &rfisnull);
- if (!rfisnull)
+ &isnull);
+ if (!isnull)
oldrelwhereclause = stringToNode(TextDatumGetCString(whereClauseDatum));
+ /* Transform the int2vector column list to a bitmap. */
+ columnListDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ /*
+ * XXX Maybe make this a separate function. We do this on
+ * multiple places.
+ */
+ if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(columnListDatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ for (int i = 0; i < nelems; i++)
+ oldcolumns = bms_add_member(oldcolumns, elems[i]);
+ }
+
ReleaseSysCache(rftuple);
}
foreach(newlc, rels)
{
PublicationRelInfo *newpubrel;
+ Oid newrelid;
+ Bitmapset *newcolumns = NULL;
newpubrel = (PublicationRelInfo *) lfirst(newlc);
+ newrelid = RelationGetRelid(newpubrel->relation);
+
+ /*
+ * If the new publication has column list, transform it to
+ * a bitmap too.
+ */
+ if (newpubrel->columns)
+ {
+ ListCell *lc;
+
+ foreach(lc, newpubrel->columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(newrelid, colname);
+
+ /* no checks needed here, that happens elsewhere */
+
+ newcolumns = bms_add_member(newcolumns, attnum);
+ }
+ }
/*
* Check if any of the new set of relations matches with the
@@ -1037,7 +1339,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
*/
if (RelationGetRelid(newpubrel->relation) == oldrelid)
{
- if (equal(oldrelwhereclause, newpubrel->whereClause))
+ if (equal(oldrelwhereclause, newpubrel->whereClause) &&
+ bms_equal(oldcolumns, newcolumns))
{
found = true;
break;
@@ -1056,6 +1359,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
{
oldrel = palloc(sizeof(PublicationRelInfo));
oldrel->whereClause = NULL;
+ oldrel->columns = NIL;
oldrel->relation = table_open(oldrelid,
ShareUpdateExclusiveLock);
delrels = lappend(delrels, oldrel);
@@ -1117,7 +1421,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
}
else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* AP_SetObjects */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -1140,6 +1444,10 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
*/
PublicationAddSchemas(pubform->oid, schemaidlist, true, stmt);
}
+ else
+ {
+ /* Nothing to do for AP_SetColumns */
+ }
}
/*
@@ -1402,6 +1710,7 @@ OpenTableList(List *tables)
List *rels = NIL;
ListCell *lc;
List *relids_with_rf = NIL;
+ List *relids_with_collist = NIL;
/*
* Open, share-lock, and check all the explicitly-specified relations
@@ -1436,6 +1745,13 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /* Disallow duplicate tables if there are any with column lists. */
+ if (t->columns || list_member_oid(relids_with_collist, myrelid))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
table_close(rel, ShareUpdateExclusiveLock);
continue;
}
@@ -1443,12 +1759,16 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
pub_rel->whereClause = t->whereClause;
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, myrelid);
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, myrelid);
+
/*
* Add children of this rel, if requested, so that they too are added
* to the publication. A partitioned table can't have any inheritance
@@ -1488,6 +1808,18 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /*
+ * We don't allow to specify column list for both parent
+ * and child table at the same time as it is not very
+ * clear which one should be given preference.
+ */
+ if (childrelid != myrelid &&
+ (t->columns || list_member_oid(relids_with_collist, childrelid)))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
continue;
}
@@ -1497,11 +1829,16 @@ OpenTableList(List *tables)
pub_rel->relation = rel;
/* child inherits WHERE clause from parent */
pub_rel->whereClause = t->whereClause;
+ /* child inherits column list from parent */
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, childrelid);
+
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, childrelid);
}
}
}
@@ -1610,6 +1947,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index dc5872f988c..a9fd0f0c895 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -8365,6 +8365,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
bool missing_ok, LOCKMODE lockmode,
ObjectAddresses *addrs)
{
+ Oid relid = RelationGetRelid(rel);
HeapTuple tuple;
Form_pg_attribute targetatt;
AttrNumber attnum;
@@ -8384,7 +8385,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/*
* get the number of the attribute
*/
- tuple = SearchSysCacheAttName(RelationGetRelid(rel), colName);
+ tuple = SearchSysCacheAttName(relid, colName);
if (!HeapTupleIsValid(tuple))
{
if (!missing_ok)
@@ -8438,13 +8439,42 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * Also, if the column is used in the column list of a publication,
+ * disallow the drop if the DROP is RESTRICT. We don't do anything if the
+ * DROP is CASCADE, which means that the dependency mechanism will remove
+ * the relation from the publication.
+ */
+ if (behavior == DROP_RESTRICT)
+ {
+ List *pubs;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(relid);
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(relid, pubid);
+
+ if (list_member_oid(published_cols, attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot drop column \"%s\" because it is part of publication \"%s\"",
+ colName, get_publication_name(pubid, false)),
+ errhint("Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication."));
+ }
+ }
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
* use find_all_inheritors to do it in one pass.
*/
children =
- find_inheritance_children(RelationGetRelid(rel), lockmode);
+ find_inheritance_children(relid, lockmode);
if (children)
{
@@ -8532,7 +8562,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Add object to delete */
object.classId = RelationRelationId;
- object.objectId = RelationGetRelid(rel);
+ object.objectId = relid;
object.objectSubId = attnum;
add_exact_object_address(&object, addrs);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 09f78f22441..3e282ed99ab 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -573,9 +573,6 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
if (cmd != CMD_UPDATE && cmd != CMD_DELETE)
return;
- if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
- return;
-
/*
* It is only safe to execute UPDATE/DELETE when all columns, referenced
* in the row filters from publications which the relation is in, are
@@ -595,17 +592,33 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
errmsg("cannot update table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_UPDATE && !pubdesc.cols_valid_for_update)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot update table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
else if (cmd == CMD_DELETE && !pubdesc.rf_valid_for_delete)
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
errmsg("cannot delete from table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_DELETE && !pubdesc.cols_valid_for_delete)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot delete from table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
/* If relation has replica identity we are always good. */
if (OidIsValid(RelationGetReplicaIndex(rel)))
return;
+ /* REPLICA IDENTITY FULL is also good for UPDATE/DELETE. */
+ if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ return;
+
/*
* This is UPDATE/DELETE and there is no replica identity.
*
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index d4f8455a2bd..a504437873f 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4850,6 +4850,7 @@ _copyPublicationTable(const PublicationTable *from)
COPY_NODE_FIELD(relation);
COPY_NODE_FIELD(whereClause);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f1002afe7a0..4fc16ce04e3 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2322,6 +2322,7 @@ _equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
COMPARE_NODE_FIELD(whereClause);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a03b33b53bd..25c9b29afdd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9751,13 +9751,14 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr OptWhereClause
+ TABLE relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9772,11 +9773,12 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId OptWhereClause
+ | ColId opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- if ($2)
+ /* FIXME this is a bit cumbersome */
+ if ($2 || $3)
{
/*
* The OptWhereClause must be stored here but it is
@@ -9786,7 +9788,8 @@ PublicationObjSpec:
*/
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVar(NULL, $1, @1);
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
else
{
@@ -9794,23 +9797,25 @@ PublicationObjSpec:
}
$$->location = @1;
}
- | ColId indirection OptWhereClause
+ | ColId indirection opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr OptWhereClause
+ | extended_relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
| CURRENT_SCHEMA
{
@@ -9836,6 +9841,10 @@ pub_obj_list: PublicationObjSpec
*
* ALTER PUBLICATION name SET pub_obj [, ...]
*
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ *
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
* pub_obj is one of:
*
* TABLE table_name [, ...]
@@ -9869,6 +9878,32 @@ AlterPublicationStmt:
n->action = AP_SetObjects;
$$ = (Node *)n;
}
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS '(' columnList ')'
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = $10;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS ALL
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = NIL;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
| ALTER PUBLICATION name DROP pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
@@ -17488,6 +17523,13 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
errmsg("WHERE clause not allowed for schema"),
parser_errposition(pubobj->location));
+ /* Column filter is not allowed on a schema object */
+ if (pubobj->pubtable && pubobj->pubtable->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schema"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index c9b0eeefd7e..816d461acd3 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,16 +29,25 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary);
+ bool binary, Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
static void logicalrep_write_namespace(StringInfo out, Oid nspid);
static const char *logicalrep_read_namespace(StringInfo in);
+
+static bool
+column_in_set(int attnum, Bitmapset *columns)
+{
+ return (columns == NULL || bms_is_member(attnum, columns));
+}
+
+
/*
* Write BEGIN to the output stream.
*/
@@ -398,7 +407,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +419,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -443,7 +452,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -464,11 +473,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -537,7 +546,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, NULL);
}
/*
@@ -652,7 +661,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -674,7 +684,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -751,7 +761,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum *values;
@@ -763,8 +773,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
continue;
+
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -783,6 +799,9 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +923,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -917,8 +936,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* send number of live attributes */
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
continue;
+
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -937,6 +962,9 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 1659964571c..6eb9fc902b7 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -112,6 +112,7 @@
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "utils/acl.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -701,12 +702,13 @@ fetch_remote_table_info(char *nspname, char *relname,
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
int natt;
ListCell *lc;
bool first;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
@@ -747,10 +749,139 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+
+ /*
+ * Get column lists for each relation.
+ *
+ * For initial synchronization, column lists can be ignored in following
+ * cases:
+ *
+ * 1) one of the subscribed publications for the table hasn't specified
+ * any column list
+ *
+ * 2) one of the subscribed publications has puballtables set to true
+ *
+ * 3) one of the subscribed publications is declared as ALL TABLES IN
+ * SCHEMA that includes this relation
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData pub_names;
+ bool first = true;
+ bool all_columns = false;
+
+ initStringInfo(&pub_names);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&pub_names, ", ");
+ appendStringInfoString(&pub_names, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ /*
+ * Check for column filters - we first check if there's any publication
+ * that has no column list for the given relation, which means we shall
+ * replicate all columns.
+ *
+ * It's easier than having to do this separately, and only then do the
+ * second query
+ */
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT 1"
+ " FROM pg_publication p"
+ " LEFT OUTER JOIN pg_publication_rel pr"
+ " ON (p.oid = pr.prpubid AND pr.prrelid = %u),"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt"
+ " WHERE gpt.relid = %u"
+ " AND p.pubname IN ( %s ) AND pr.prattrs IS NULL LIMIT 1",
+ lrel->remoteid,
+ lrel->remoteid,
+ pub_names.data);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch column list info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ if (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ all_columns = true;
+
+ ExecDropSingleTupleTableSlot(slot);
+ walrcv_clear_result(pubres);
+
+ if (!all_columns)
+ {
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT unnest(pr.prattrs)"
+ " FROM pg_publication p"
+ " LEFT OUTER JOIN pg_publication_rel pr"
+ " ON (p.oid = pr.prpubid AND pr.prrelid = %u),"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt"
+ " WHERE gpt.relid = %u"
+ " AND p.pubname IN ( %s ) AND pr.prattrs IS NOT NULL",
+ lrel->remoteid,
+ lrel->remoteid,
+ pub_names.data);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch column list info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ /*
+ * Multiple column list expressions for the same table will be combined
+ * by merging them. If any of the lists for this table are null, it
+ * means the whole table will be copied. In this case it is not necessary
+ * to construct a unified column list expression at all.
+ */
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ Datum cfval = slot_getattr(slot, 1, &isnull);
+
+ /* if there's no column list, we need to replicate all columns */
+ if (isnull)
+ {
+ bms_free(included_cols);
+ included_cols = NULL;
+ break;
+ }
+
+ included_cols = bms_add_member(included_cols,
+ DatumGetInt16(cfval));
+
+ ExecClearTuple(slot);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+
+ walrcv_clear_result(pubres);
+ }
+
+ pfree(pub_names.data);
+ }
+
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -778,16 +909,34 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * Store the columns as a list of names. Ignore those that are not
+ * present in the column list, if there is one.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ {
+ ExecClearTuple(slot);
+ continue;
+ }
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
+ Assert(!isnull);
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -821,6 +970,9 @@ fetch_remote_table_info(char *nspname, char *relname,
*
* 3) one of the subscribed publications is declared as ALL TABLES IN
* SCHEMA that includes this relation
+ *
+ * XXX Does this actually handle puballtables and schema publications
+ * correctly?
*/
if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
{
@@ -930,8 +1082,24 @@ copy_table(Relation rel)
/* Regular table with no row filter */
if (lrel.relkind == RELKIND_RELATION && qual == NIL)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+
+ /*
+ * XXX Do we need to list the columns in all cases? Maybe we're replicating
+ * all columns?
+ */
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
+
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
+
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index faf7555f913..e50af5017f9 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -29,6 +29,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -85,7 +86,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -93,6 +95,8 @@ static void send_repl_origin(LogicalDecodingContext *ctx,
/*
* Only 3 publication actions are used for row filtering ("insert", "update",
* "delete"). See RelationSyncEntry.exprstate[].
+ *
+ * FIXME Do we need something similar for column filters?
*/
enum RowFilterPubAction
{
@@ -164,6 +168,13 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
AttrMap *attrmap;
+
+ /*
+ * Columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this bitmap are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -188,6 +199,7 @@ static EState *create_estate_for_relation(Relation rel);
static void pgoutput_row_filter_init(PGOutputData *data,
List *publications,
RelationSyncEntry *entry);
+
static bool pgoutput_row_filter_exec_expr(ExprState *state,
ExprContext *econtext);
static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
@@ -195,6 +207,11 @@ static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
RelationSyncEntry *entry,
ReorderBufferChangeType *action);
+/* column filter routines */
+static void pgoutput_column_filter_init(PGOutputData *data,
+ List *publications,
+ RelationSyncEntry *entry);
+
/*
* Specify output plugin callbacks
*/
@@ -603,11 +620,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -620,7 +637,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -643,13 +661,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip this attribute if it's not present in the column list */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -860,6 +882,108 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
}
}
+/*
+ * Initialize the column filter.
+ */
+static void
+pgoutput_column_filter_init(PGOutputData *data, List *publications,
+ RelationSyncEntry *entry)
+{
+ ListCell *lc;
+ MemoryContext oldctx;
+
+ /*
+ * Find if there are any row filters for this relation. If there are, then
+ * prepare the necessary ExprState and cache it in entry->exprstate. To
+ * build an expression state, we need to ensure the following:
+ *
+ * All the given publication-table mappings must be checked.
+ *
+ * Multiple publications might have multiple row filters for this
+ * relation. Since row filter usage depends on the DML operation, there
+ * are multiple lists (one for each operation) to which row filters will
+ * be appended.
+ *
+ * FOR ALL TABLES implies "don't use row filter expression" so it takes
+ * precedence.
+ */
+ foreach(lc, publications)
+ {
+ Publication *pub = lfirst(lc);
+ HeapTuple cftuple = NULL;
+ Datum cfdatum = 0;
+ bool pub_no_filter = false;
+
+ if (pub->alltables)
+ {
+ /*
+ * If the publication is FOR ALL TABLES then it is treated the
+ * same as if this table has no row filters (even if for other
+ * publications it does).
+ */
+ pub_no_filter = true;
+ }
+ else
+ {
+ /*
+ * Check for the presence of a row filter in this publication.
+ */
+ cftuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(entry->publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(cftuple))
+ {
+ /* Null indicates no filter. */
+ cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
+ Anum_pg_publication_rel_prattrs,
+ &pub_no_filter);
+
+ /*
+ * When no column list is defined, so publish all columns.
+ * Otherwise merge the columns to the column list.
+ */
+ if (!pub_no_filter) /* when not null */
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(cfdatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+ for (int i = 0; i < nelems; i++)
+ entry->columns = bms_add_member(entry->columns,
+ elems[i]);
+ MemoryContextSwitchTo(oldctx);
+ }
+ }
+ else
+ {
+ pub_no_filter = true;
+ }
+ }
+
+ /* found publication with no filter, so we're done */
+ if (pub_no_filter)
+ {
+ if (cftuple)
+ ReleaseSysCache(cftuple);
+
+ bms_free(entry->columns);
+ entry->columns = NULL;
+
+ break;
+ }
+
+ ReleaseSysCache(cftuple);
+ } /* loop all subscribed publications */
+
+}
+
/*
* Initialize the slot for storing new and old tuples, and build the map that
* will be used to convert the relation's tuples into the ancestor's format.
@@ -1224,7 +1348,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -1278,11 +1402,13 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel,
- new_slot, data->binary);
+ new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel,
- old_slot, new_slot, data->binary);
+ old_slot, new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel,
@@ -1731,6 +1857,7 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
memset(entry->exprstate, 0, sizeof(entry->exprstate));
entry->cache_expr_cxt = NULL;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->attrmap = NULL;
}
@@ -1775,6 +1902,8 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
entry->pubactions.pubinsert = false;
entry->pubactions.pubupdate = false;
entry->pubactions.pubdelete = false;
@@ -1807,8 +1936,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/*
* Build publication cache. We can't use one provided by relcache as
- * relcache considers all publications given relation is in, but here
- * we only need to consider ones that the subscriber requested.
+ * relcache considers all publications that the given relation is in,
+ * but here we only need to consider ones that the subscriber
+ * requested.
*/
foreach(lc, data->publications)
{
@@ -1861,6 +1991,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
}
/*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
* Don't publish changes for partitioned tables, because
* publishing those of its partitions suffices, unless partition
* changes won't be published due to pubviaroot being set.
@@ -1921,6 +2054,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/* Initialize the row filter */
pgoutput_row_filter_init(data, rel_publications, entry);
+
+ /* Initialize the column filter */
+ pgoutput_column_filter_init(data, rel_publications, entry);
}
list_free(pubids);
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index fccffce5729..a2da72f0d48 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -5553,6 +5553,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
return;
}
@@ -5565,6 +5567,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
/* Fetch the publication membership info. */
puboids = GetRelationPublications(relid);
@@ -5616,7 +5620,7 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
*/
if (!pubform->puballtables &&
(pubform->pubupdate || pubform->pubdelete) &&
- contain_invalid_rfcolumn(pubid, relation, ancestors,
+ pub_rf_contains_invalid_column(pubid, relation, ancestors,
pubform->pubviaroot))
{
if (pubform->pubupdate)
@@ -5625,6 +5629,23 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->rf_valid_for_delete = false;
}
+ /*
+ * Check if all columns are part of the REPLICA IDENTITY index or not.
+ *
+ * If the publication is FOR ALL TABLES then it means the table has no
+ * column list and we can skip the validation.
+ */
+ if (!pubform->puballtables &&
+ (pubform->pubupdate || pubform->pubdelete) &&
+ pub_collist_contains_invalid_column(pubid, relation, ancestors,
+ pubform->pubviaroot))
+ {
+ if (pubform->pubupdate)
+ pubdesc->cols_valid_for_update = false;
+ if (pubform->pubdelete)
+ pubdesc->cols_valid_for_delete = false;
+ }
+
ReleaseSysCache(tup);
/*
@@ -5636,6 +5657,16 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
!pubdesc->rf_valid_for_update && !pubdesc->rf_valid_for_delete)
break;
+
+ /*
+ * If we know everything is replicated and the column list is invalid
+ * for update and delete, there is no point to check for other
+ * publications.
+ */
+ if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
+ pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
+ !pubdesc->cols_valid_for_update && !pubdesc->cols_valid_for_delete)
+ break;
}
if (relation->rd_pubdesc)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e69dcf8a484..f208c7a6c59 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4075,6 +4075,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_prpubid;
int i_prrelid;
int i_prrelqual;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4088,12 +4089,20 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
if (fout->remoteVersion >= 150000)
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual "
- "FROM pg_catalog.pg_publication_rel");
+ "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual, "
+ "(CASE\n"
+ " WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT array_agg(attname)\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) prattrs "
+ "FROM pg_catalog.pg_publication_rel pr");
else
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "NULL AS prrelqual "
+ "NULL AS prrelqual, NULL AS prattrs "
"FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
@@ -4104,6 +4113,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
i_prrelqual = PQfnumber(res, "prrelqual");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4149,6 +4159,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
else
pubrinfo[j].pubrelqual = pg_strdup(PQgetvalue(res, i, i_prrelqual));
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[j].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4223,10 +4255,13 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+
if (pubrinfo->pubrelqual)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 997a3b60719..680b07dcd52 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -632,6 +632,7 @@ typedef struct _PublicationRelInfo
PublicationInfo *publication;
TableInfo *pubtable;
char *pubrelqual;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 3e55ff26f82..ed57c53bcb5 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2428,6 +2428,28 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'ALTER PUBLICATION pub1 ADD TABLE test_sixth_table (col3, col2)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_sixth_table (col3, col2);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_sixth_table (col2, col3);\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'ALTER PUBLICATION pub1 ADD TABLE test_seventh_table (col3, col2) WHERE (col1 = 1)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_seventh_table (col3, col2) WHERE (col1 = 1);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_seventh_table (col2, col3) WHERE ((col1 = 1));\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'ALTER PUBLICATION pub3 ADD ALL TABLES IN SCHEMA dump_test' => {
create_order => 51,
create_sql =>
@@ -2778,6 +2800,44 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'CREATE TABLE test_sixth_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_sixth_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_sixth_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'CREATE TABLE test_seventh_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_seventh_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_seventh_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'CREATE TABLE test_table_identity' => {
create_order => 3,
create_sql => 'CREATE TABLE dump_test.test_table_identity (
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index e3382933d98..e462ccfb748 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2880,6 +2880,7 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_namespace pn ON p.oid = pn.pnpubid\n"
" JOIN pg_catalog.pg_class pc ON pc.relnamespace = pn.pnnspid\n"
@@ -2887,6 +2888,12 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , pg_get_expr(pr.prqual, c.oid)\n"
+ " , (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT string_agg(attname, ', ')\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) "
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
" JOIN pg_catalog.pg_class c ON c.oid = pr.prrelid\n"
@@ -2894,6 +2901,7 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2904,12 +2912,14 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
"WHERE pr.prrelid = '%s'\n"
"UNION ALL\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2931,6 +2941,11 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, " \"%s\"",
PQgetvalue(result, i, 0));
+ /* column list (if any) */
+ if (!PQgetisnull(result, i, 2))
+ appendPQExpBuffer(&buf, " (%s)",
+ PQgetvalue(result, i, 2));
+
/* row filter (if any) */
if (!PQgetisnull(result, i, 1))
appendPQExpBuffer(&buf, " WHERE %s",
@@ -5867,7 +5882,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5884,15 +5899,19 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (as_schema)
+ printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
+ else
{
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
+
+ if (!PQgetisnull(res, i, 3))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 3));
+
if (!PQgetisnull(res, i, 2))
appendPQExpBuffer(buf, " WHERE %s", PQgetvalue(res, i, 2));
}
- else
- printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
}
@@ -6021,11 +6040,22 @@ describePublications(const char *pattern)
printfPQExpBuffer(&buf,
"SELECT n.nspname, c.relname");
if (pset.sversion >= 150000)
+ {
appendPQExpBufferStr(&buf,
", pg_get_expr(pr.prqual, c.oid)");
+ appendPQExpBufferStr(&buf,
+ ", (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string("
+ " ARRAY(SELECT attname\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END)");
+ }
else
appendPQExpBufferStr(&buf,
- ", NULL");
+ ", NULL, NULL");
appendPQExpBuffer(&buf,
"\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index ba72e62e614..a06742a6200 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -85,6 +85,13 @@ typedef struct PublicationDesc
*/
bool rf_valid_for_update;
bool rf_valid_for_delete;
+
+ /*
+ * true if the columns are part of the replica identity or the publication actions
+ * do not include UPDATE or DELETE.
+ */
+ bool cols_valid_for_update;
+ bool cols_valid_for_delete;
} PublicationDesc;
typedef struct Publication
@@ -100,6 +107,7 @@ typedef struct PublicationRelInfo
{
Relation relation;
Node *whereClause;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -123,8 +131,11 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
+extern void GetActionsInPublication(Oid pubid, PublicationActions *actions);
extern List *GetPublicationSchemas(Oid pubid);
extern List *GetSchemaPublications(Oid schemaid);
extern List *GetSchemaPublicationRelations(Oid schemaid,
@@ -142,6 +153,8 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *pri
bool if_not_exists);
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern void publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index 0dd0f425db9..4feb581899e 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -34,6 +34,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
#ifdef CATALOG_VARLEN /* variable-length fields start here */
pg_node_tree prqual; /* qualifications */
+ int2vector prattrs; /* columns to replicate */
#endif
} FormData_pg_publication_rel;
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index 7813cbcb6bb..ae87caf089d 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -31,7 +31,9 @@ extern void RemovePublicationSchemaById(Oid psoid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
extern void AlterPublicationOwner_oid(Oid pubid, Oid newOwnerId);
extern void InvalidatePublicationRels(List *relids);
-extern bool contain_invalid_rfcolumn(Oid pubid, Relation relation,
+extern bool pub_rf_contains_invalid_column(Oid pubid, Relation relation,
+ List *ancestors, bool pubviaroot);
+extern bool pub_collist_contains_invalid_column(Oid pubid, Relation relation,
List *ancestors, bool pubviaroot);
#endif /* PUBLICATIONCMDS_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1617702d9d6..1375a173e3b 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3652,6 +3652,7 @@ typedef struct PublicationTable
NodeTag type;
RangeVar *relation; /* relation to be published */
Node *whereClause; /* qualifications */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
@@ -3688,7 +3689,8 @@ typedef enum AlterPublicationAction
{
AP_AddObjects, /* add objects to publication */
AP_DropObjects, /* remove objects from publication */
- AP_SetObjects /* set list of objects */
+ AP_SetObjects, /* set list of objects */
+ AP_SetColumns /* change list of columns for a table */
} AlterPublicationAction;
typedef struct AlterPublicationStmt
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4d2c881644a..a771ab8ff33 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -209,12 +209,12 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary);
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -231,7 +231,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 4e191c120ac..79ced2921b6 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -613,6 +613,372 @@ DROP TABLE rf_tbl_abcd_pk;
DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+RESET client_min_messages;
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ERROR: cannot reference generated column "d" in publication column list
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+ERROR: cannot reference system column "ctid" in publication column list
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+ERROR: cannot drop column "c" because it is part of publication "testpub_fortable"
+HINT: Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication.
+-- ok: for insert-only publication, the column list is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: change the replica identity to "b", and then column list (a, c) fails
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+-- error: replica identity (b,c) is not covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+ Publication testpub_table_ins
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | t | f
+Tables:
+ "public.testpub_tbl5" (a)
+
+-- with REPLICA IDENTITY FULL, column lists are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ERROR: cannot update table "testpub_tbl6"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+ALTER PUBLICATION testpub_fortable ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ERROR: cannot update table "testpub_tbl6"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- make sure changing the column list is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: update the column list
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, c)
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl8;
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_0"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+ Publication testpub_both_filters
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | t | t | t | f
+Tables:
+ "public.testpub_tbl_both_filters" (a, c) WHERE (c <> 1)
+
+\d+ testpub_tbl_both_filters
+ Table "public.testpub_tbl_both_filters"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+ c | integer | | not null | | plain | |
+Indexes:
+ "testpub_tbl_both_filters_pkey" PRIMARY KEY, btree (a, c) REPLICA IDENTITY
+Publications:
+ "testpub_both_filters" (a, c) WHERE (c <> 1)
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+-- Tests for partitioned table
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+ERROR: cannot use publication column list for relation "rf_tbl_abcd_part_pk"
+DETAIL: column list cannot be used for a partitioned table when publish_via_partition_root is false.
+-- ok - can use row filter for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any row filter is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+ERROR: cannot set publish_via_partition_root = false for publication "testpub6"
+DETAIL: The publication contains a column list for a partitioned table "rf_tbl_abcd_part_pk" which is not allowed when publish_via_partition_root is false.
+-- Now change the root filter to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have row filter for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -1058,6 +1424,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schema
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 5457c56b33f..be05ac9f763 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -373,6 +373,292 @@ DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+RESET client_min_messages;
+
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+-- ok: for insert-only publication, the column list is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+
+-- error: change the replica identity to "b", and then column list (a, c) fails
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+-- error: replica identity (b,c) is not covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+
+-- with REPLICA IDENTITY FULL, column lists are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+
+ALTER PUBLICATION testpub_fortable ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+
+-- make sure changing the column list is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: update the column list
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl8;
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+\d+ testpub_tbl_both_filters
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Tests for partitioned table
+
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - can use row filter for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any row filter is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- Now change the root filter to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have row filter for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
+
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -614,6 +900,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/029_column_list.pl b/src/test/subscription/t/029_column_list.pl
new file mode 100644
index 00000000000..5266967b3f4
--- /dev/null
+++ b/src/test/subscription/t/029_column_list.pl
@@ -0,0 +1,1124 @@
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test partial-column publication of tables
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# create publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+sub wait_for_subscription_sync
+{
+ my ($node) = @_;
+
+ # Also wait for initial table sync to finish
+ my $synced_query = "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+
+ $node->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+}
+
+# setup tables on both nodes
+
+# tab1: simple 1:1 replication
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+# tab2: replication from regular to table with fewer columns
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)
+));
+
+# tab3: simple 1:1 replication with weird column names
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "B" varchar, "c'" int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "c'" int)
+));
+
+# test_part: partitioned tables, with partitioning (including multi-level
+# partitioning, and fewer columns on the subscriber)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+# tab4: table with user-defined enum types
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, d text);
+));
+
+
+# TEST: create publication and subscription for some of the tables with
+# column lists
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub1
+ FOR TABLE tab1 (a, "B"), tab3 ("a'", "c'"), test_part (a, b), tab4 (a, b, d)
+ WITH (publish_via_partition_root = 'true');
+));
+
+# check that we got the right prattrs values for the publication in the
+# pg_publication_rel catalog (order by relname, to get stable ordering)
+my $result = $node_publisher->safe_psql('postgres', qq(
+ SELECT relname, prattrs
+ FROM pg_publication_rel pb JOIN pg_class pc ON(pb.prrelid = pc.oid)
+ ORDER BY relname
+));
+
+is($result, qq(tab1|1 2
+tab3|1 3
+tab4|1 2 4
+test_part|1 2), 'publication relation updated');
+
+# TEST: insert data into the tables, create subscription and see if sync
+# replicates the right columns
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (1, 2, 3);
+ INSERT INTO tab1 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (1, 2, 3);
+ INSERT INTO tab3 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (1, 'red', 3, 'oh my');
+ INSERT INTO tab4 VALUES (2, 'blue', 4, 'hello');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (1, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (2, 'bcd', '2021-07-03 11:12:13');
+ INSERT INTO test_part VALUES (7, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (8, 'bcd', '2021-07-03 11:12:13');
+));
+
+# create subscription for the publication, wait for sync to complete,
+# then check the sync results
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+4|5|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+4|6), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+7|abc
+8|bcd), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: now insert more data into the tables, and wait until we replicate
+# them (not by tablesync, but regular decoding and replication)
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (2, 3, 4);
+ INSERT INTO tab1 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (2, 3, 4);
+ INSERT INTO tab3 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (3, 'red', 5, 'foo');
+ INSERT INTO tab4 VALUES (4, 'blue', 6, 'bar');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (3, 'xxx', '2022-02-01 10:00:00');
+ INSERT INTO test_part VALUES (4, 'yyy', '2022-03-02 15:12:13');
+ INSERT INTO test_part VALUES (9, 'zzz', '2022-04-03 21:00:00');
+ INSERT INTO test_part VALUES (10, 'qqq', '2022-05-04 22:12:13');
+));
+
+# wait for catchup before checking the subscriber
+$node_publisher->wait_for_catchup('sub1');
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+2|3|
+4|5|
+5|6|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+2|4
+4|6
+5|7), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello
+3|red|foo
+4|blue|bar), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+3|xxx
+4|yyy
+7|abc
+8|bcd
+9|zzz
+10|qqq), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: do some updates on some of the tables, both on columns included
+# in the column list and other
+
+# tab1: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET "B" = 2 * "B" where a = 1));
+
+# tab1: update of non-replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET c = 2*c where a = 4));
+
+# tab3: update of non-replicated
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "B" = "B" || ' updated' where "a'" = 4));
+
+# tab3: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "c'" = 2 * "c'" where "a'" = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'blue', c = c * 2, d = d || ' updated' where a = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'red', c = c * 2, d = d || ' updated' where a = 2));
+
+# wait for the replication to catch up, and check the UPDATE results got
+# replicated correctly, with the right column list
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab1 ORDER BY a));
+is($result,
+qq(1|4|
+2|3|
+4|5|
+5|6|), 'only update on column tab1.b is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result,
+qq(1|6
+2|4
+4|6
+5|7), 'only update on column tab3.c is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab4 ORDER BY a));
+
+is($result, qq(1|blue|oh my updated
+2|red|hello updated
+3|red|foo
+4|blue|bar), 'update on column tab4.c is not replicated');
+
+
+# TEST: add table with a column list, insert data, replicate
+
+# insert some data before adding it to the publication
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (1, 'abc', 3);
+));
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2 (a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION");
+
+# wait for the tablesync to complete, add a bit more data and then check
+# the results of the replication
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (2, 'def', 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|def), 'insert on column tab2.c is not replicated');
+
+# do a couple updates, check the correct stuff gets replicated
+$node_publisher->safe_psql('postgres', qq(
+ UPDATE tab2 SET c = 5 where a = 1;
+ UPDATE tab2 SET b = 'xyz' where a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|xyz), 'update on column tab2.c is not replicated');
+
+
+# TEST: add a table to two publications with different column lists, and
+# create a single subscription replicating both publications
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+
+ -- insert a couple initial records
+ INSERT INTO tab5 VALUES (1, 11, 111, 1111);
+ INSERT INTO tab5 VALUES (2, 22, 222, 2222);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub3
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->wait_for_catchup('sub1');
+
+# insert data and make sure all the columns (union of the columns lists)
+# get fully replicated
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (3, 33, 333, 3333);
+ INSERT INTO tab5 VALUES (4, 44, 444, 4444);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111
+2|22|2222
+3|33|3333
+4|44|4444),
+ 'overlapping publications with overlapping column lists');
+
+# and finally, set the column list to ALL for one of the publications,
+# which means replicating all columns (removing the column list), but
+# first add the missing column to the table on subscriber
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub3 ALTER TABLE tab5 SET COLUMNS ALL;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ ALTER TABLE tab5 ADD COLUMN c INT;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (5, 55, 555, 5555);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111|
+2|22|2222|
+3|33|3333|
+4|44|4444|
+5|55|5555|555),
+ 'overlapping publications with overlapping column lists');
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key (but use a different column in
+# the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub4 FOR TABLE tab6 (a, b);
+
+ -- initial data
+ INSERT INTO tab6 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (2, 33, 444, 5555);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is still covered by the column list, though)
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+# we need to do the same thing on the subscriber
+# XXX What would happen if this happens before the publisher ALTER? Or
+# interleaved, somehow? But that seems unrelated to column lists.
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (3, 55, 666, 8888);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key with a key on multiple columns
+# (all of them covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub5 FOR TABLE tab7 (a, b);
+
+ -- some initial data
+ INSERT INTO tab7 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub5
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (2, 33, 444, 5555);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is not covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ ALTER TABLE tab7 ADD PRIMARY KEY (a, b);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (3, 55, 666, 7777);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+# now switch the primary key again to another columns not covered by the
+# column list, but also generate writes between the drop and creation
+# of the new constraint
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ INSERT INTO tab7 VALUES (4, 77, 888, 9999);
+ -- update/delete is not allowed for tables without RI
+ ALTER TABLE tab7 ADD PRIMARY KEY (b, a);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+ DELETE FROM tab7 WHERE a = 1;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(2|264||
+3|220||
+4|154||),
+ 'replication with the modified primary key');
+
+
+# TEST: partitioned tables (with publish_via_partition_root = false)
+# and replica identity. The (leaf) partitions may have different RI, so
+# we need to check the partition RI (with respect to the column list)
+# while attaching the partition.
+
+# First, let's create a partitioned table with two partitions, each with
+# a different RI, but a column list not covering all those RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+
+ -- initial data, one row in each partition
+ INSERT INTO test_part_a VALUES (1, 3);
+ INSERT INTO test_part_a VALUES (6, 4);
+));
+
+# do the same thing on the subscriber (with the opposite column order)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (b int, a int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+));
+
+# create a publication replicating just the column "a", which is not enough
+# for the second partition
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub6 FOR TABLE test_part_a (b, a) WITH (publish_via_partition_root = true);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_1 (a);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_2 (b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub6
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_a VALUES (2, 5);
+ INSERT INTO test_part_a VALUES (7, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT a, b FROM test_part_a ORDER BY a, b"),
+ qq(1|3
+2|5
+6|4
+7|6),
+ 'partitions with different replica identities not replicated correctly');
+
+# This time start with a column list covering RI for all partitions, but
+# then update the column list to not cover column "b" (needed by the
+# second partition)
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+
+ -- initial data, one row in each partitions
+ INSERT INTO test_part_b VALUES (1, 1);
+ INSERT INTO test_part_b VALUES (6, 2);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub7 FOR TABLE test_part_b (a, b) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub7
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_b VALUES (2, 3);
+ INSERT INTO test_part_b VALUES (7, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_b ORDER BY a, b"),
+ qq(1|1
+2|3
+6|2
+7|4),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: This time start with a column list covering RI for all partitions,
+# but then update RI for one of the partitions to not be covered by the
+# column list anymore.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+
+ -- initial data, one row for each partition
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+));
+
+# create a publication replicating data through partition root, with a column
+# filter on the root, and then add the partitions one by one with separate
+# column lists (but those are not applied)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a,c);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SUBSCRIPTION sub1;
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub8;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_c VALUES (3, 7, 8);
+ INSERT INTO test_part_c VALUES (4, 9, 10);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||5
+2|4|
+3||8
+4|9|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# create a publication not replicating data through partition root, without
+# a column list on the root, and then add the partitions one by one with
+# separate column lists
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub8;
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ TRUNCATE test_part_c;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ TRUNCATE test_part_c;
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||
+2|4|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: Start with a single partition, with RI compatible with the column
+# filter, and then attach a partition with incompatible RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ INSERT INTO test_part_d VALUES (1, 2);
+));
+
+# do the same thing on the subscriber (in fact, create both partitions right
+# away, no need to delay that)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ CREATE TABLE test_part_d_2 PARTITION OF test_part_d FOR VALUES IN (2,4);
+ ALTER TABLE test_part_d_2 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_2 REPLICA IDENTITY USING INDEX test_part_d_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub9 FOR TABLE test_part_d (a) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub9
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_d VALUES (3, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_d ORDER BY a, b"),
+ qq(1|
+3|),
+ 'partitions with different replica identities not replicated correctly');
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. So with column lists (a,b) and (a,c) we
+# should replicate (a,b,c).
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+
+ -- initial data
+ INSERT INTO test_mix_1 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_1 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_1 ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES, we should replicate all columns.
+
+# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ test_part, test_part_a, test_part_b, test_part_c, test_part_d;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
+
+ -- initial data
+ INSERT INTO test_mix_2 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_3, pub_mix_4;
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_2 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_2"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES IN SCHEMA, we should replicate all columns.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
+
+ -- initial data
+ INSERT INTO test_mix_3 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_5, pub_mix_6;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_3 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_3"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: Check handling of publish_via_partition_root - if a partition is
+# published through partition root, we should only apply the column list
+# defined for the whole table (not the partitions) - both during the initial
+# sync and when replicating changes. This is what we do for row filters.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+
+ CREATE PUBLICATION pub_root_true FOR TABLE test_root (a) WITH (publish_via_partition_root = true);
+
+ -- initial data
+ INSERT INTO test_root VALUES (1, 2, 3);
+ INSERT INTO test_root VALUES (10, 20, 30);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_root_true;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_root VALUES (2, 3, 4);
+ INSERT INTO test_root VALUES (11, 21, 31);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_root ORDER BY a, b, c"),
+ qq(1||
+2||
+10||
+11||),
+ 'publication via partition root applies column list');
+
+
+# TEST: Multiple publications which publish schema of parent table and
+# partition. The partition is published through two publications, once
+# through a schema (so no column list) containing the parent, and then
+# also directly (with a columns list). The expected outcome is there is
+# no column list.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub1, pub2, pub3, pub4, pub5, pub6, pub7, pub8;
+
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA s1;
+ CREATE PUBLICATION pub2 FOR TABLE t_1(b);
+
+ -- initial data
+ INSERT INTO s1.t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub1, pub2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'two publications, publishing the same relation');
+
+# Now resync the subcription, but with publications in the opposite order.
+# The result should be the same.
+
+$node_subscriber->safe_psql('postgres', qq(
+ TRUNCATE s1.t;
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub1;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (7, 8, 9);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(7|8|9),
+ 'two publications, publishing the same relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub3 FOR TABLE t_1 (a), t_2
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub3;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+# Note: The difference from the preceding test is that in this case both
+# relations have a column list defined.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub4 FOR TABLE t_1 (a), t_2 (b)
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
+
+done_testing();
--
2.34.1
On Fri, Mar 11, 2022 at 12:44 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 3/10/22 04:09, Amit Kapila wrote:
On Wed, Mar 9, 2022 at 3:33 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Mar 7, 2022 at 8:48 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:OK, I reworked this to do the same thing as the row filtering patch.
Thanks, I'll check this.
Some assorted comments:
=====================
1. We don't need to send a column list for the old tuple in case of an
update (similar to delete). It is not required to apply a column
filter for those cases because we ensure that RI must be part of the
column list for updates and deletes.I'm not sure which part of the code does this refer to?
The below part:
@@ -464,11 +473,11 @@ logicalrep_write_update(StringInfo out,
TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
I think here instead of columns, the patch needs to send NULL as it is
already doing in logicalrep_write_delete.
2. + /* + * Check if all columns referenced in the column filter are part of + * the REPLICA IDENTITY index or not.I think this comment is reverse. The rule we follow here is that
attributes that are part of RI must be there in a specified column
list. This is used at two places in the patch.Yeah, you're right. Will fix.
3. get_rel_sync_entry() { /* XXX is there a danger of memory leak here? beware */ + oldctx = MemoryContextSwitchTo(CacheMemoryContext); + for (int i = 0; i < nelems; i++) ... }Similar to the row filter, I think we need to use
entry->cache_expr_cxt to allocate this. There are other usages of
CacheMemoryContext in this part of the code but I think those need to
be also changed and we can do that as a separate patch. If we do the
suggested change then we don't need to separately free columns.I agree a shorter-lived context would be better than CacheMemoryContext,
but "expr" seems to indicate it's for the expression only, so maybe we
should rename that.
Yeah, we can do that. How about rel_entry_cxt or something like that?
The idea is that eventually, we should move a few other things of
RelSyncEntry like attrmap where we are using CacheMemoryContext under
this context.
But do we really want a memory context for every
single entry?
Any other better idea?
--
With Regards,
Amit Kapila.
On Fri, Mar 11, 2022 at 7:26 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
But this does not address tablesync.c :-( That still copies everything,
because it decides to sync both rels (test_pub_part_1, test_pub_part_2),
with it's row filter. On older releases this would fail, because we'd
start two workers:
Yeah, this is because of the existing problem where we sync both rels
instead of one. We have fixed some similar existing problems earlier.
Hou-San has reported a similar case in another email [1]/messages/by-id/OS0PR01MB5716DC2982CC735FDE388804940B9@OS0PR01MB5716.jpnprd01.prod.outlook.com.
But I find this really weird - I think it's reasonable to expect the
sync to produce the same result as if the data was inserted and
replicated, and this just violates that.Shouldn't tablesync calculate a list of relations in a way that prevents
such duplicate / overlapping syncs?
Yes, I think it is better to fix it separately than to fix it along
with row filter or column filter work.
In any case, this sync issue looks
entirely unrelated to the column filtering patch.
Right.
[1]: /messages/by-id/OS0PR01MB5716DC2982CC735FDE388804940B9@OS0PR01MB5716.jpnprd01.prod.outlook.com
--
With Regards,
Amit Kapila.
On 3/11/22 03:46, Amit Kapila wrote:
On Fri, Mar 11, 2022 at 12:44 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 3/10/22 04:09, Amit Kapila wrote:
On Wed, Mar 9, 2022 at 3:33 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Mar 7, 2022 at 8:48 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:OK, I reworked this to do the same thing as the row filtering patch.
Thanks, I'll check this.
Some assorted comments:
=====================
1. We don't need to send a column list for the old tuple in case of an
update (similar to delete). It is not required to apply a column
filter for those cases because we ensure that RI must be part of the
column list for updates and deletes.I'm not sure which part of the code does this refer to?
The below part: @@ -464,11 +473,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel, pq_sendbyte(out, 'O'); /* old tuple follows */ else pq_sendbyte(out, 'K'); /* old key follows */ - logicalrep_write_tuple(out, rel, oldslot, binary); + logicalrep_write_tuple(out, rel, oldslot, binary, columns); }I think here instead of columns, the patch needs to send NULL as it is
already doing in logicalrep_write_delete.
Hmmm, yeah. In practice it doesn't really matter, because NULL means
"send all columns" so it actually relaxes the check. But we only send
the RI keys, which is a subset of the column filter. But will fix.
2. + /* + * Check if all columns referenced in the column filter are part of + * the REPLICA IDENTITY index or not.I think this comment is reverse. The rule we follow here is that
attributes that are part of RI must be there in a specified column
list. This is used at two places in the patch.Yeah, you're right. Will fix.
3. get_rel_sync_entry() { /* XXX is there a danger of memory leak here? beware */ + oldctx = MemoryContextSwitchTo(CacheMemoryContext); + for (int i = 0; i < nelems; i++) ... }Similar to the row filter, I think we need to use
entry->cache_expr_cxt to allocate this. There are other usages of
CacheMemoryContext in this part of the code but I think those need to
be also changed and we can do that as a separate patch. If we do the
suggested change then we don't need to separately free columns.I agree a shorter-lived context would be better than CacheMemoryContext,
but "expr" seems to indicate it's for the expression only, so maybe we
should rename that.Yeah, we can do that. How about rel_entry_cxt or something like that?
The idea is that eventually, we should move a few other things of
RelSyncEntry like attrmap where we are using CacheMemoryContext under
this context.
Yeah, rel_entry_cxt sounds fine I guess ...
But do we really want a memory context for every
single entry?Any other better idea?
No, I think you're right - it'd be hard/impossible to keep track of all
the memory allocated for expression/estate. It'd be fine for the
columns, because that's just a bitmap, but not for the expressions.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Fri, Mar 11, 2022 at 9:57 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
Hi Tomas,
Thanks for your patches.
On Mon, Mar 9, 2022 at 9:53 PM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
On Wed, Mar 9, 2022 at 6:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Mar 7, 2022 at 11:18 PM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
On Fri, Mar 4, 2022 at 6:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
Fetching column filter info in tablesync.c is quite expensive. It
seems to be using four round-trips to get the complete info whereas
for row-filter we use just one round trip. I think we should try to
get both row filter and column filter info in just one round trip.Maybe, but I really don't think this is an issue.
I am not sure but it might matter for small tables. Leaving aside the
performance issue, I think the current way will get the wrong column
list in many cases: (a) The ALL TABLES IN SCHEMA case handling won't
work for partitioned tables when the partitioned table is part of one
schema and partition table is part of another schema. (b) The handling
of partition tables in other cases will fetch incorrect lists as it
tries to fetch the column list of all the partitions in the hierarchy.One of my colleagues has even tested these cases both for column
filters and row filters and we find the behavior of row filter is okay
whereas for column filter it uses the wrong column list. We will share
the tests and results with you in a later email. We are trying to
unify the column filter queries with row filter to make their behavior
the same and will share the findings once it is done. I hope if we are
able to achieve this that we will reduce the chances of bugs in this
area.OK, I'll take a look at that email.
I tried to get both the column filters and the row filters with one SQL, but
it failed because I think the result is not easy to parse.
I noted that we use two SQLs to get column filters in the latest
patches(20220311). I think maybe we could use one SQL to get column filters to
reduce network cost. Like the SQL in the attachment.
Regards,
Wang wei
Attachments:
0001-Try-to-get-column-filters-with-one-SQL.patchapplication/octet-stream; name=0001-Try-to-get-column-filters-with-one-SQL.patchDownload
From 373a1cfb2dea0b23ca7894a5467201ce75db9ae9 Mon Sep 17 00:00:00 2001
From: wangw <wangw.fnst@fujitsu.com>
Date: Fri, 11 Mar 2022 10:08:40 +0800
Subject: [PATCH] Try to get column filters with one SQL.
---
src/backend/replication/logical/tablesync.c | 87 ++++++---------------
1 file changed, 24 insertions(+), 63 deletions(-)
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 6eb9fc902b..82c9003b54 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -782,23 +782,16 @@ fetch_remote_table_info(char *nspname, char *relname,
first = false;
}
- /*
- * Check for column filters - we first check if there's any publication
- * that has no column list for the given relation, which means we shall
- * replicate all columns.
- *
- * It's easier than having to do this separately, and only then do the
- * second query
- */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT 1"
+ "SELECT DISTINCT unnest"
" FROM pg_publication p"
" LEFT OUTER JOIN pg_publication_rel pr"
- " ON (p.oid = pr.prpubid AND pr.prrelid = %u),"
+ " ON (p.oid = pr.prpubid AND pr.prrelid = %u)"
+ " LEFT OUTER JOIN unnest(pr.prattrs) ON TRUE,"
" LATERAL pg_get_publication_tables(p.pubname) gpt"
" WHERE gpt.relid = %u"
- " AND p.pubname IN ( %s ) AND pr.prattrs IS NULL LIMIT 1",
+ " AND p.pubname IN ( %s )",
lrel->remoteid,
lrel->remoteid,
pub_names.data);
@@ -812,65 +805,33 @@ fetch_remote_table_info(char *nspname, char *relname,
errmsg("could not fetch column list info for table \"%s.%s\" from publisher: %s",
nspname, relname, pubres->err)));
+ /*
+ * Multiple column list expressions for the same table will be combined
+ * by merging them. If any of the lists for this table are null, it
+ * means the whole table will be copied. In this case it is not necessary
+ * to construct a unified column list expression at all.
+ */
slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
- if (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
- all_columns = true;
-
- ExecDropSingleTupleTableSlot(slot);
- walrcv_clear_result(pubres);
-
- if (!all_columns)
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
{
- resetStringInfo(&cmd);
- appendStringInfo(&cmd,
- "SELECT unnest(pr.prattrs)"
- " FROM pg_publication p"
- " LEFT OUTER JOIN pg_publication_rel pr"
- " ON (p.oid = pr.prpubid AND pr.prrelid = %u),"
- " LATERAL pg_get_publication_tables(p.pubname) gpt"
- " WHERE gpt.relid = %u"
- " AND p.pubname IN ( %s ) AND pr.prattrs IS NOT NULL",
- lrel->remoteid,
- lrel->remoteid,
- pub_names.data);
-
- pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
- lengthof(attrsRow), attrsRow);
-
- if (pubres->status != WALRCV_OK_TUPLES)
- ereport(ERROR,
- (errcode(ERRCODE_CONNECTION_FAILURE),
- errmsg("could not fetch column list info for table \"%s.%s\" from publisher: %s",
- nspname, relname, pubres->err)));
+ Datum cfval = slot_getattr(slot, 1, &isnull);
- /*
- * Multiple column list expressions for the same table will be combined
- * by merging them. If any of the lists for this table are null, it
- * means the whole table will be copied. In this case it is not necessary
- * to construct a unified column list expression at all.
- */
- slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
- while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ /* if there's no column list, we need to replicate all columns */
+ if (isnull)
{
- Datum cfval = slot_getattr(slot, 1, &isnull);
-
- /* if there's no column list, we need to replicate all columns */
- if (isnull)
- {
- bms_free(included_cols);
- included_cols = NULL;
- break;
- }
-
- included_cols = bms_add_member(included_cols,
- DatumGetInt16(cfval));
-
- ExecClearTuple(slot);
+ bms_free(included_cols);
+ included_cols = NULL;
+ break;
}
- ExecDropSingleTupleTableSlot(slot);
+ else
+ included_cols = bms_add_member(included_cols,
+ DatumGetInt16(cfval));
- walrcv_clear_result(pubres);
+ ExecClearTuple(slot);
}
+ ExecDropSingleTupleTableSlot(slot);
+
+ walrcv_clear_result(pubres);
pfree(pub_names.data);
}
--
2.18.4
On Fri, Mar 11, 2022 at 7:26 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 3/10/22 20:10, Tomas Vondra wrote:
FWIW I think the reason is pretty simple - pgoutput_row_filter_init is
broken. It assumes you can just do thisrftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(entry->publish_as_relid),
ObjectIdGetDatum(pub->oid));if (HeapTupleIsValid(rftuple))
{
/* Null indicates no filter. */
rfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
Anum_pg_publication_rel_prqual,
&pub_no_filter);
}
else
{
pub_no_filter = true;
}and pub_no_filter=true means there's no filter at all. Which is
nonsense, because we're using publish_as_relid here - the publication
may not include this particular ancestor, in which case we need to just
ignore this publication.So yeah, this needs to be reworked.
I spent a bit of time looking at this, and I think a minor change in
get_rel_sync_entry() fixes this - it's enough to ensure rel_publications
only includes publications that actually include publish_as_relid.
Thanks for looking into this. I think in the first patch before
calling get_partition_ancestors() we need to ensure it is a partition
(the call expects that) and pubviaroot is true. I think it would be
good if we can avoid an additional call to get_partition_ancestors()
as it could be costly. I wonder why it is not sufficient to ensure
that publish_as_relid exists after ancestor in ancestors list before
assigning the ancestor to publish_as_relid? This only needs to be done
in case of (if (!publish)). I have not tried this so I could be wrong.
--
With Regards,
Amit Kapila.
On 3/11/22 08:05, wangw.fnst@fujitsu.com wrote:
On Fri, Mar 11, 2022 at 9:57 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
Hi Tomas,
Thanks for your patches.On Mon, Mar 9, 2022 at 9:53 PM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
On Wed, Mar 9, 2022 at 6:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Mar 7, 2022 at 11:18 PM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
On Fri, Mar 4, 2022 at 6:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
Fetching column filter info in tablesync.c is quite expensive. It
seems to be using four round-trips to get the complete info whereas
for row-filter we use just one round trip. I think we should try to
get both row filter and column filter info in just one round trip.Maybe, but I really don't think this is an issue.
I am not sure but it might matter for small tables. Leaving aside the
performance issue, I think the current way will get the wrong column
list in many cases: (a) The ALL TABLES IN SCHEMA case handling won't
work for partitioned tables when the partitioned table is part of one
schema and partition table is part of another schema. (b) The handling
of partition tables in other cases will fetch incorrect lists as it
tries to fetch the column list of all the partitions in the hierarchy.One of my colleagues has even tested these cases both for column
filters and row filters and we find the behavior of row filter is okay
whereas for column filter it uses the wrong column list. We will share
the tests and results with you in a later email. We are trying to
unify the column filter queries with row filter to make their behavior
the same and will share the findings once it is done. I hope if we are
able to achieve this that we will reduce the chances of bugs in this
area.OK, I'll take a look at that email.
I tried to get both the column filters and the row filters with one SQL, but
it failed because I think the result is not easy to parse.I noted that we use two SQLs to get column filters in the latest
patches(20220311). I think maybe we could use one SQL to get column filters to
reduce network cost. Like the SQL in the attachment.
I'll take a look. But as I said before - I very much prefer SQL that is
easy to understand, and I don't think the one extra round trip is an
issue during tablesync (which is a very rare action).
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 3/11/22 10:52, Amit Kapila wrote:
On Fri, Mar 11, 2022 at 7:26 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 3/10/22 20:10, Tomas Vondra wrote:
FWIW I think the reason is pretty simple - pgoutput_row_filter_init is
broken. It assumes you can just do thisrftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(entry->publish_as_relid),
ObjectIdGetDatum(pub->oid));if (HeapTupleIsValid(rftuple))
{
/* Null indicates no filter. */
rfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
Anum_pg_publication_rel_prqual,
&pub_no_filter);
}
else
{
pub_no_filter = true;
}and pub_no_filter=true means there's no filter at all. Which is
nonsense, because we're using publish_as_relid here - the publication
may not include this particular ancestor, in which case we need to just
ignore this publication.So yeah, this needs to be reworked.
I spent a bit of time looking at this, and I think a minor change in
get_rel_sync_entry() fixes this - it's enough to ensure rel_publications
only includes publications that actually include publish_as_relid.Thanks for looking into this. I think in the first patch before
calling get_partition_ancestors() we need to ensure it is a partition
(the call expects that) and pubviaroot is true.
Does the call really require that? Also, I'm not sure why we'd need to
look at pubviaroot - that's already considered earlier when calculating
publish_as_relid, here we just need to know the relationship of the two
OIDs (if one is ancestor/child of the other).
I think it would be
good if we can avoid an additional call to get_partition_ancestors()
as it could be costly.
Maybe. OTOH we only should do this only very rarely anyway.
I wonder why it is not sufficient to ensure
that publish_as_relid exists after ancestor in ancestors list before
assigning the ancestor to publish_as_relid? This only needs to be done
in case of (if (!publish)). I have not tried this so I could be wrong.
I'm not sure what exactly are you proposing. Maybe try coding it? That's
probably faster than trying to describe what the code might do ...
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Fri, Mar 11, 2022 at 6:20 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 3/11/22 10:52, Amit Kapila wrote:
On Fri, Mar 11, 2022 at 7:26 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 3/10/22 20:10, Tomas Vondra wrote:
FWIW I think the reason is pretty simple - pgoutput_row_filter_init is
broken. It assumes you can just do thisrftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(entry->publish_as_relid),
ObjectIdGetDatum(pub->oid));if (HeapTupleIsValid(rftuple))
{
/* Null indicates no filter. */
rfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
Anum_pg_publication_rel_prqual,
&pub_no_filter);
}
else
{
pub_no_filter = true;
}and pub_no_filter=true means there's no filter at all. Which is
nonsense, because we're using publish_as_relid here - the publication
may not include this particular ancestor, in which case we need to just
ignore this publication.So yeah, this needs to be reworked.
I spent a bit of time looking at this, and I think a minor change in
get_rel_sync_entry() fixes this - it's enough to ensure rel_publications
only includes publications that actually include publish_as_relid.Thanks for looking into this. I think in the first patch before
calling get_partition_ancestors() we need to ensure it is a partition
(the call expects that) and pubviaroot is true.Does the call really require that?
There may not be any harm but I have mentioned it because (a) the
comments atop get_partition_ancestors(...it should only be called when
it is known that the relation is a partition.) indicates the same; (b)
all existing callers seems to use it only for partitions.
Also, I'm not sure why we'd need to
look at pubviaroot - that's already considered earlier when calculating
publish_as_relid, here we just need to know the relationship of the two
OIDs (if one is ancestor/child of the other).
I thought of avoiding calling get_partition_ancestors when pubviaroot
is not set. It will unnecessary check the whole hierarchy for
partitions even when it is not required. I agree that this is not a
common code path but still felt why do it needlessly?
I think it would be
good if we can avoid an additional call to get_partition_ancestors()
as it could be costly.Maybe. OTOH we only should do this only very rarely anyway.
I wonder why it is not sufficient to ensure
that publish_as_relid exists after ancestor in ancestors list before
assigning the ancestor to publish_as_relid? This only needs to be done
in case of (if (!publish)). I have not tried this so I could be wrong.I'm not sure what exactly are you proposing. Maybe try coding it? That's
probably faster than trying to describe what the code might do ...
Okay, please find attached. I have done basic testing of this, if we
agree with this approach then this will require some more testing.
--
With Regards,
Amit Kapila.
Attachments:
v1-0001-fixup-publish_as_relid.patchapplication/octet-stream; name=v1-0001-fixup-publish_as_relid.patchDownload
From 2434111eadb7fa2bd45209d3f61ad505959da02c Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Sat, 12 Mar 2022 09:23:11 +0530
Subject: [PATCH v1 1/2] fixup: publish_as_relid
Make sure to determine the top-most ancestor listed in any publication.
Otherwise we might end up with different values depending on the order
of publications (as listed in subscription).
---
src/backend/catalog/pg_publication.c | 18 ++++++++--
src/backend/commands/publicationcmds.c | 2 +-
src/backend/replication/pgoutput/pgoutput.c | 38 +++++++++++++++++++--
src/include/catalog/pg_publication.h | 3 +-
4 files changed, 54 insertions(+), 7 deletions(-)
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 25998fbb39..a9bbfce6b8 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -277,16 +277,18 @@ GetPubPartitionOptionRelations(List *result, PublicationPartOpt pub_partopt,
/*
* Returns the relid of the topmost ancestor that is published via this
- * publication if any, otherwise returns InvalidOid.
+ * publication if any and set its ancestor level to ancestor_level,
+ * otherwise returns InvalidOid.
*
* Note that the list of ancestors should be ordered such that the topmost
* ancestor is at the end of the list.
*/
Oid
-GetTopMostAncestorInPublication(Oid puboid, List *ancestors)
+GetTopMostAncestorInPublication(Oid puboid, List *ancestors, int *ancestor_level)
{
ListCell *lc;
Oid topmost_relid = InvalidOid;
+ int level = 0;
/*
* Find the "topmost" ancestor that is in this publication.
@@ -297,13 +299,25 @@ GetTopMostAncestorInPublication(Oid puboid, List *ancestors)
List *apubids = GetRelationPublications(ancestor);
List *aschemaPubids = NIL;
+ level++;
+
if (list_member_oid(apubids, puboid))
+ {
topmost_relid = ancestor;
+
+ if (ancestor_level)
+ *ancestor_level = level;
+ }
else
{
aschemaPubids = GetSchemaPublications(get_rel_namespace(ancestor));
if (list_member_oid(aschemaPubids, puboid))
+ {
topmost_relid = ancestor;
+
+ if (ancestor_level)
+ *ancestor_level = level;
+ }
}
list_free(apubids);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 16b8661a1b..a7b74dc60a 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -323,7 +323,7 @@ contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
*/
if (pubviaroot && relation->rd_rel->relispartition)
{
- publish_as_relid = GetTopMostAncestorInPublication(pubid, ancestors);
+ publish_as_relid = GetTopMostAncestorInPublication(pubid, ancestors, NULL);
if (!OidIsValid(publish_as_relid))
publish_as_relid = relid;
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a0477f..28b13adcd8 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1748,6 +1748,7 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
List *schemaPubids = GetSchemaPublications(schemaId);
ListCell *lc;
Oid publish_as_relid = relid;
+ int publish_ancestor_level = 0;
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
List *rel_publications = NIL;
@@ -1815,11 +1816,23 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
Publication *pub = lfirst(lc);
bool publish = false;
+ /*
+ * Under what relid should we publish changes in this publication?
+ * We'll use the top-most relid across all publications.
+ */
+ Oid pub_relid = relid;
+ int ancestor_level = 0;
+
if (pub->alltables)
{
publish = true;
if (pub->pubviaroot && am_partition)
- publish_as_relid = llast_oid(get_partition_ancestors(relid));
+ {
+ List *ancestors = get_partition_ancestors(relid);
+
+ pub_relid = llast_oid(ancestors);
+ ancestor_level = list_length(ancestors);
+ }
}
if (!publish)
@@ -1835,16 +1848,21 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
if (am_partition)
{
Oid ancestor;
+ int level = 0;
List *ancestors = get_partition_ancestors(relid);
ancestor = GetTopMostAncestorInPublication(pub->oid,
- ancestors);
+ ancestors,
+ &level);
if (ancestor != InvalidOid)
{
ancestor_published = true;
if (pub->pubviaroot)
- publish_as_relid = ancestor;
+ {
+ pub_relid = ancestor;
+ ancestor_level = level;
+ }
}
}
@@ -1868,6 +1886,20 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
rel_publications = lappend(rel_publications, pub);
+
+ /*
+ * We want to publish the changes as the top-most ancestor
+ * across all publications. So we need to check if the
+ * already calculated level is higher than the new one. If
+ * yes, we can ignore the new value (as it's a child).
+ * Otherwise the new value is an ancestor, so we keep it.
+ */
+ if (publish_ancestor_level > ancestor_level)
+ continue;
+
+ /* The new value is an ancestor, so let's keep it. */
+ publish_as_relid = pub_relid;
+ publish_ancestor_level = ancestor_level;
}
}
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index ba72e62e61..fe773cf9b7 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -134,7 +134,8 @@ extern List *GetAllSchemaPublicationRelations(Oid puboid,
extern List *GetPubPartitionOptionRelations(List *result,
PublicationPartOpt pub_partopt,
Oid relid);
-extern Oid GetTopMostAncestorInPublication(Oid puboid, List *ancestors);
+extern Oid GetTopMostAncestorInPublication(Oid puboid, List *ancestors,
+ int *ancestor_level);
extern bool is_publishable_relation(Relation rel);
extern bool is_schema_publication(Oid pubid);
--
2.28.0.windows.1
v1-0002-fixup-row-filter-publications.patchapplication/octet-stream; name=v1-0002-fixup-row-filter-publications.patchDownload
From f9eb1d52401ff454943c3862e2c3d81442a030f0 Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Sat, 12 Mar 2022 09:23:53 +0530
Subject: [PATCH v1 2/2] fixup: row-filter publications
When initializing the row filter, consider only publications that
actually include the relation (publish_as_relid). The publications may
include different ancestors, in which case the function would get
confused and conclude there's no row filter.
---
src/backend/replication/pgoutput/pgoutput.c | 20 +++++++++++++++-----
1 file changed, 15 insertions(+), 5 deletions(-)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 28b13adcd8..bc9abcf0e6 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1885,8 +1885,6 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
- rel_publications = lappend(rel_publications, pub);
-
/*
* We want to publish the changes as the top-most ancestor
* across all publications. So we need to check if the
@@ -1897,9 +1895,21 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
if (publish_ancestor_level > ancestor_level)
continue;
- /* The new value is an ancestor, so let's keep it. */
- publish_as_relid = pub_relid;
- publish_ancestor_level = ancestor_level;
+ /*
+ * If the new value is an ancestor, discard the list of
+ * publications through which we replicate it.
+ */
+ if (publish_as_relid != pub_relid)
+ {
+ list_free(rel_publications);
+ rel_publications = NIL;
+
+ publish_as_relid = pub_relid;
+ publish_ancestor_level = ancestor_level;
+ }
+
+ /* Track the publications. */
+ rel_publications = lappend(rel_publications, pub);
}
}
--
2.28.0.windows.1
On 3/12/22 05:30, Amit Kapila wrote:
...
Okay, please find attached. I have done basic testing of this, if we
agree with this approach then this will require some more testing.
Thanks, the proposed changes seem like a clear improvement, so I've
added them, with some minor tweaks (mostly to comments).
I've also included the memory context rename (entry_changes to the
change proposed by Wang Wei, using a single SQL command in tablesync.
And I've renamed the per-entry memory context to entry_cxt, and used it
for the column list.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
0001-fixup-publish_as_relid-20220313.patchtext/x-patch; charset=UTF-8; name=0001-fixup-publish_as_relid-20220313.patchDownload
From 49a9d1627098440a32c75727860551db3c20b733 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Wed, 9 Mar 2022 18:10:56 +0100
Subject: [PATCH 1/3] fixup: publish_as_relid
Make sure to determine the top-most ancestor listed in any publication.
Otherwise we might end up with different values depending on the order
of publications (as listed in subscription).
---
src/backend/catalog/pg_publication.c | 21 +++++++++-
src/backend/commands/publicationcmds.c | 2 +-
src/backend/replication/pgoutput/pgoutput.c | 43 +++++++++++++++++++--
src/include/catalog/pg_publication.h | 3 +-
4 files changed, 62 insertions(+), 7 deletions(-)
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 25998fbb39b..789b895db89 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -277,16 +277,21 @@ GetPubPartitionOptionRelations(List *result, PublicationPartOpt pub_partopt,
/*
* Returns the relid of the topmost ancestor that is published via this
- * publication if any, otherwise returns InvalidOid.
+ * publication if any and set its ancestor level to ancestor_level,
+ * otherwise returns InvalidOid.
+ *
+ * The ancestor_level value allows us to compare the results for multiple
+ * publications, and decide which value is higher up.
*
* Note that the list of ancestors should be ordered such that the topmost
* ancestor is at the end of the list.
*/
Oid
-GetTopMostAncestorInPublication(Oid puboid, List *ancestors)
+GetTopMostAncestorInPublication(Oid puboid, List *ancestors, int *ancestor_level)
{
ListCell *lc;
Oid topmost_relid = InvalidOid;
+ int level = 0;
/*
* Find the "topmost" ancestor that is in this publication.
@@ -297,13 +302,25 @@ GetTopMostAncestorInPublication(Oid puboid, List *ancestors)
List *apubids = GetRelationPublications(ancestor);
List *aschemaPubids = NIL;
+ level++;
+
if (list_member_oid(apubids, puboid))
+ {
topmost_relid = ancestor;
+
+ if (ancestor_level)
+ *ancestor_level = level;
+ }
else
{
aschemaPubids = GetSchemaPublications(get_rel_namespace(ancestor));
if (list_member_oid(aschemaPubids, puboid))
+ {
topmost_relid = ancestor;
+
+ if (ancestor_level)
+ *ancestor_level = level;
+ }
}
list_free(apubids);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 16b8661a1b7..a7b74dc60ad 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -323,7 +323,7 @@ contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
*/
if (pubviaroot && relation->rd_rel->relispartition)
{
- publish_as_relid = GetTopMostAncestorInPublication(pubid, ancestors);
+ publish_as_relid = GetTopMostAncestorInPublication(pubid, ancestors, NULL);
if (!OidIsValid(publish_as_relid))
publish_as_relid = relid;
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a0477f0..104432fb3a6 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1748,6 +1748,7 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
List *schemaPubids = GetSchemaPublications(schemaId);
ListCell *lc;
Oid publish_as_relid = relid;
+ int publish_ancestor_level = 0;
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
List *rel_publications = NIL;
@@ -1815,11 +1816,28 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
Publication *pub = lfirst(lc);
bool publish = false;
+ /*
+ * Under what relid should we publish changes in this publication?
+ * We'll use the top-most relid across all publications. Also track
+ * the ancestor level for this publication.
+ */
+ Oid pub_relid = relid;
+ int ancestor_level = 0;
+
+ /*
+ * If this is a FOR ALL TABLES publication, pick the partition root
+ * and set the ancestor level accordingly.
+ */
if (pub->alltables)
{
publish = true;
if (pub->pubviaroot && am_partition)
- publish_as_relid = llast_oid(get_partition_ancestors(relid));
+ {
+ List *ancestors = get_partition_ancestors(relid);
+
+ pub_relid = llast_oid(ancestors);
+ ancestor_level = list_length(ancestors);
+ }
}
if (!publish)
@@ -1835,16 +1853,21 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
if (am_partition)
{
Oid ancestor;
+ int level;
List *ancestors = get_partition_ancestors(relid);
ancestor = GetTopMostAncestorInPublication(pub->oid,
- ancestors);
+ ancestors,
+ &level);
if (ancestor != InvalidOid)
{
ancestor_published = true;
if (pub->pubviaroot)
- publish_as_relid = ancestor;
+ {
+ pub_relid = ancestor;
+ ancestor_level = level;
+ }
}
}
@@ -1868,6 +1891,20 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
rel_publications = lappend(rel_publications, pub);
+
+ /*
+ * We want to publish the changes as the top-most ancestor
+ * across all publications. So we need to check if the
+ * already calculated level is higher than the new one. If
+ * yes, we can ignore the new value (as it's a child).
+ * Otherwise the new value is an ancestor, so we keep it.
+ */
+ if (publish_ancestor_level > ancestor_level)
+ continue;
+
+ /* The new value is an ancestor, so let's keep it. */
+ publish_as_relid = pub_relid;
+ publish_ancestor_level = ancestor_level;
}
}
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index ba72e62e614..fe773cf9b7d 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -134,7 +134,8 @@ extern List *GetAllSchemaPublicationRelations(Oid puboid,
extern List *GetPubPartitionOptionRelations(List *result,
PublicationPartOpt pub_partopt,
Oid relid);
-extern Oid GetTopMostAncestorInPublication(Oid puboid, List *ancestors);
+extern Oid GetTopMostAncestorInPublication(Oid puboid, List *ancestors,
+ int *ancestor_level);
extern bool is_publishable_relation(Relation rel);
extern bool is_schema_publication(Oid pubid);
--
2.34.1
0002-fixup-row-filter-publications-20220313.patchtext/x-patch; charset=UTF-8; name=0002-fixup-row-filter-publications-20220313.patchDownload
From 5128a396c400801a317fa5043d62d559850f0548 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Fri, 11 Mar 2022 02:23:25 +0100
Subject: [PATCH 2/3] fixup: row-filter publications
When initializing the row filter, consider only publications that
actually include the relation (publish_as_relid). The publications may
include different ancestors, in which case the function would get
confused and conclude there's no row filter.
---
src/backend/replication/pgoutput/pgoutput.c | 26 +++++++++++++++++----
1 file changed, 21 insertions(+), 5 deletions(-)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 104432fb3a6..abfef4e447c 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1890,8 +1890,6 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
- rel_publications = lappend(rel_publications, pub);
-
/*
* We want to publish the changes as the top-most ancestor
* across all publications. So we need to check if the
@@ -1902,9 +1900,27 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
if (publish_ancestor_level > ancestor_level)
continue;
- /* The new value is an ancestor, so let's keep it. */
- publish_as_relid = pub_relid;
- publish_ancestor_level = ancestor_level;
+ /*
+ * If we found an ancestor higher up in the tree, discard
+ * the list of publications through which we replicate it,
+ * and use the new ancestor.
+ */
+ if (publish_ancestor_level < ancestor_level)
+ {
+ publish_as_relid = pub_relid;
+ publish_ancestor_level = ancestor_level;
+
+ /* reset the publication list for this relation */
+ rel_publications = NIL;
+ }
+ else
+ {
+ /* Same ancestor leve, has to be the same OID. */
+ Assert(publish_as_relid == pub_relid);
+ }
+
+ /* Track this publications. */
+ rel_publications = lappend(rel_publications, pub);
}
}
--
2.34.1
0003-Allow-specifying-column-filters-for-logical-20220313.patchtext/x-patch; charset=UTF-8; name=0003-Allow-specifying-column-filters-for-logical-20220313.patchDownload
From fd2b111d53c2012edaceb80ca7cb1513389577ae Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Thu, 10 Mar 2022 17:31:57 +0100
Subject: [PATCH 3/3] Allow specifying column filters for logical replication
This allows specifying an optional column list when adding a table to
logical replication. Columns not included on this list are not sent to
the subscriber. The filter is specified as a list of columns after the
table name, enclosed in parentheses.
For UPDATE/DELETE publications, the column filter needs to cover all
REPLICA IDENTITY columns. For INSERT publications, the column list is
arbitrary and may omit some REPLICA IDENTITY columns. Furthermore, if
the table uses REPLICA IDENTITY FULL, column filter is not allowed.
The column filter can contain only simple column references. Complex
expressions, function calls etc. are not allowed. This restriction could
be relaxed in the future.
During the initial table synchronization, only columns specified in the
column filter are copied to the subscriber. If the subscription has
several publications, containing the same table with different column
filters, columns specified in any of the filters will be copied. This
means all columns are replicated if the table has no column filter at
all (which is treated as column filter with all columns), of when of the
publications is defined as FOR ALL TABLES (possibly IN SCHEMA for the
schema of the table).
For partitioned tables, publish_via_partition_root determines whether
the row filter for the root or leaf relation will be used. If the
parameter is 'false' (the default), the filter defined for the leaf
relation is used. Otherwise, the column filter for the root partition
will be used.
Psql commands \dRp+ and \d <table-name> now display any column filters.
Author: Rahila Syed <rahilasyed90@gmail.com>
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 15 +-
doc/src/sgml/protocol.sgml | 3 +-
doc/src/sgml/ref/alter_publication.sgml | 27 +-
doc/src/sgml/ref/create_publication.sgml | 17 +-
src/backend/catalog/pg_publication.c | 256 +++++
src/backend/commands/publicationcmds.c | 364 +++++-
src/backend/commands/tablecmds.c | 36 +-
src/backend/executor/execReplication.c | 19 +-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 60 +-
src/backend/replication/logical/proto.c | 56 +-
src/backend/replication/logical/tablesync.c | 156 ++-
src/backend/replication/pgoutput/pgoutput.c | 203 +++-
src/backend/utils/cache/relcache.c | 33 +-
src/bin/pg_dump/pg_dump.c | 47 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 60 +
src/bin/psql/describe.c | 40 +-
src/include/catalog/pg_publication.h | 13 +
src/include/catalog/pg_publication_rel.h | 1 +
src/include/commands/publicationcmds.h | 4 +-
src/include/nodes/parsenodes.h | 4 +-
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 375 +++++++
src/test/regress/sql/publication.sql | 290 +++++
src/test/subscription/t/029_column_list.pl | 1124 +++++++++++++++++++
27 files changed, 3117 insertions(+), 95 deletions(-)
create mode 100644 src/test/subscription/t/029_column_list.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 83987a99045..c043da37aee 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -4392,7 +4392,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para>
<para>
This is an array of <structfield>indnatts</structfield> values that
- indicate which table columns this index indexes. For example a value
+ indicate which table columns this index indexes. For example, a value
of <literal>1 3</literal> would mean that the first and the third table
columns make up the index entries. Key columns come before non-key
(included) columns. A zero in this array indicates that the
@@ -6263,6 +6263,19 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to schema
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>prattrs</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+ </para>
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example, a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all columns are published.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 9178c779ba9..fb491e9ebee 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7006,7 +7006,8 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column included in
+ the publication (except generated columns):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 32b75f6c78e..470d50a2447 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -25,12 +25,13 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD <replace
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ALTER TABLE <replaceable class="parameter">table_name</replaceable> SET COLUMNS { ( <replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -64,6 +65,12 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
command retain their previous settings.
</para>
+ <para>
+ The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows changing
+ the set of columns that are included in the publication. If a column list
+ is specified, it must include the replica identity columns.
+ </para>
+
<para>
The remaining variants change the owner and the name of the publication.
</para>
@@ -112,6 +119,14 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ </para>
+
+ <para>
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
If the optional <literal>WHERE</literal> clause is specified, rows for
which the <replaceable class="parameter">expression</replaceable>
evaluates to false or null will not be published. Note that parentheses
@@ -172,9 +187,15 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
</para>
<para>
- Add some tables to the publication:
+ Add tables to the publication:
+<programlisting>
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS (user_id, firstname, lastname);
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 4979b9b646d..fb2d013393b 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -86,6 +86,13 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<literal>TRUNCATE</literal> commands.
</para>
+ <para>
+ When a column list is specified, only the named columns are replicated.
+ If no column list is specified, all columns of the table are replicated
+ through this publication, including any columns added later. If a column
+ list is specified, it must include the replica identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
@@ -327,6 +334,14 @@ CREATE PUBLICATION production_publication FOR TABLE users, departments, ALL TABL
<structname>sales</structname>:
<programlisting>
CREATE PUBLICATION sales_publication FOR ALL TABLES IN SCHEMA marketing, sales;
+</programlisting></para>
+
+ <para>
+ Create a publication that publishes all changes for table <structname>users</structname>,
+ but replicates only columns <structname>user_id</structname> and
+ <structname>firstname</structname>:
+<programlisting>
+CREATE PUBLICATION users_filtered FOR TABLE users (user_id, firstname);
</programlisting></para>
</refsect1>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 789b895db89..70a35a12a47 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,6 +45,9 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+static void publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs);
+
/*
* Check if relation can be in given publication and throws appropriate
* error if not.
@@ -345,6 +348,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
Oid relid = RelationGetRelid(targetrel);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ AttrNumber *attarray;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -372,6 +377,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
check_publication_add_relation(targetrel);
+ /*
+ * Translate column names to attnums and check the column list is valid.
+ * We also deconstruct the bitmap into an array of attnums, for storing
+ * in the catalog.
+ */
+ publication_translate_columns(pri->relation, pri->columns,
+ &natts, &attarray);
+
/* Form a tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -384,6 +397,17 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ /* Add column list, if available */
+ if (pri->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+
/* Add qualifications, if available */
if (pri->whereClause != NULL)
values[Anum_pg_publication_rel_prqual - 1] = CStringGetTextDatum(nodeToString(pri->whereClause));
@@ -399,6 +423,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
/* Register dependencies as needed */
ObjectAddressSet(myself, PublicationRelRelationId, pubreloid);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ pfree(attarray);
+
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -432,6 +464,154 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
return myself;
}
+/*
+ * Update the column list for a relation in a publication.
+ */
+void
+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns)
+{
+ AttrNumber *attarray;
+ HeapTuple copytup;
+ int natts;
+ bool nulls[Natts_pg_publication_rel];
+ bool replaces[Natts_pg_publication_rel];
+ Datum values[Natts_pg_publication_rel];
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ replaces[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ deleteDependencyRecordsForClass(PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid,
+ RelationRelationId,
+ DEPENDENCY_AUTO);
+
+ if (columns == NULL)
+ {
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+ }
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ int2vector *prattrs;
+ Form_pg_publication_rel pubrel;
+
+ publication_translate_columns(targetrel, columns, &natts, &attarray);
+
+ /* XXX "pub" is leaked here ??? */
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+
+ /* Add dependencies on the new list of columns */
+ pubrel = (Form_pg_publication_rel) GETSTRUCT(pubreltup);
+ ObjectAddressSet(myself, PublicationRelRelationId, pubrel->oid);
+
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId,
+ RelationGetRelid(targetrel), attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ }
+
+ copytup = heap_modify_tuple(pubreltup, RelationGetDescr(pubrel),
+ values, nulls, replaces);
+
+ CatalogTupleUpdate(pubrel, &pubreltup->t_self, copytup);
+
+ heap_freetuple(copytup);
+}
+
+/*
+ * qsort comparator for attnums
+ *
+ * XXX We already have compare_int16, so maybe let's share that, somehow?
+ */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list (no system or generated attributes,
+ * no duplicates). Additional checks with replica identity are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ *
+ * XXX Should this detect duplicate columns?
+ */
+static void
+publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs)
+{
+ AttrNumber *attarray;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference generated column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /* Be tidy, so that the catalog representation is always sorted */
+ qsort(attarray, n, sizeof(AttrNumber), compare_int16);
+
+ *natts = n;
+ *attrs = attarray;
+
+ bms_free(set);
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -539,6 +719,82 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+ Form_pg_publication_rel pubrel;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ /* no column list for this publications/relation */
+ if (isnull)
+ continue;
+
+ pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
+
+ pubs = lappend_oid(pubs, pubrel->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index a7b74dc60ad..b9a52fc4b0b 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -296,7 +296,7 @@ contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
* Returns true if any invalid column is found.
*/
bool
-contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
+pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
bool pubviaroot)
{
HeapTuple rftuple;
@@ -367,6 +367,123 @@ contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
return result;
}
+/*
+ * Check if all columns referenced in the column list are part of the
+ * REPLICA IDENTITY index or not.
+ *
+ * Returns true if any invalid column is found.
+ */
+bool
+pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
+ bool pubviaroot)
+{
+ HeapTuple tuple;
+ Oid relid = RelationGetRelid(relation);
+ Oid publish_as_relid = RelationGetRelid(relation);
+ bool result = false;
+ Datum datum;
+ bool isnull;
+
+ /*
+ * For a partition, if pubviaroot is true, find the topmost ancestor that
+ * is published via this publication as we need to use its column list
+ * for the changes.
+ *
+ * Note that even though the column list used is for an ancestor, the
+ * REPLICA IDENTITY used will be for the actual child table.
+ */
+ if (pubviaroot && relation->rd_rel->relispartition)
+ {
+ publish_as_relid = GetTopMostAncestorInPublication(pubid, ancestors, NULL);
+
+ if (!OidIsValid(publish_as_relid))
+ publish_as_relid = relid;
+ }
+
+ tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tuple))
+ return false;
+
+ datum = SysCacheGetAttr(PUBLICATIONRELMAP, tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ if (!isnull)
+ {
+ int x;
+ Bitmapset *idattrs;
+ Bitmapset *columns = NULL;
+
+ /* With REPLICA IDENTITY FULL, no column list is allowed. */
+ if (relation->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ result = true;
+
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(datum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ for (int i = 0; i < nelems; i++)
+ columns = bms_add_member(columns, elems[i]);
+ }
+
+ /* Remember columns that are part of the REPLICA IDENTITY */
+ idattrs = RelationGetIndexAttrBitmap(relation,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * Attnums in the bitmap returned by RelationGetIndexAttrBitmap are
+ * offset (to handle system columns the usual way), while column list
+ * does not use offset, so we can't do bms_is_subset(). Instead, we have
+ * to loop over the idattrs and check all of them are in the filter.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ AttrNumber attnum = (x + FirstLowInvalidHeapAttributeNumber);
+
+ /*
+ * If pubviaroot is true, we are validating the row filter of the
+ * parent table, but the bitmap contains the replica identity
+ * information of the child table. So, get the column number of the
+ * child table as parent and child column order could be different.
+ */
+ if (pubviaroot)
+ {
+ /* attnum is for child table */
+ char *colname = get_attname(relid, attnum, false);
+
+ /*
+ * Determine the matching attnum in parent (because the column
+ * filter is defined on the parent).
+ */
+ attnum = get_attnum(publish_as_relid, colname);
+ }
+
+ if (!bms_is_member(attnum, columns))
+ {
+ result = true;
+ break;
+ }
+ }
+
+ bms_free(idattrs);
+ bms_free(columns);
+ }
+
+ ReleaseSysCache(tuple);
+
+ return result;
+}
+
/* check_functions_in_node callback */
static bool
contain_mutable_or_user_functions_checker(Oid func_id, void *context)
@@ -608,6 +725,45 @@ TransformPubWhereClauses(List *tables, const char *queryString,
}
}
+
+/*
+ * Transform the publication column lists expression for all the relations
+ * in the list.
+ *
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow column lists on the leaf relations. So maybe rename it?
+ */
+static void
+TransformPubColumnList(List *tables, const char *queryString,
+ bool pubviaroot)
+{
+ ListCell *lc;
+
+ foreach(lc, tables)
+ {
+ PublicationRelInfo *pri = (PublicationRelInfo *) lfirst(lc);
+
+ if (pri->columns == NIL)
+ continue;
+
+ /*
+ * If the publication doesn't publish changes via the root partitioned
+ * table, the partition's column list will be used. So disallow using
+ * the column list on partitioned table in this case.
+ */
+ if (!pubviaroot &&
+ pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot use publication column list for relation \"%s\"",
+ RelationGetRelationName(pri->relation)),
+ errdetail("column list cannot be used for a partitioned table when %s is false.",
+ "publish_via_partition_root")));
+ }
+}
+
/*
* Create new publication.
*/
@@ -724,6 +880,9 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
TransformPubWhereClauses(rels, pstate->p_sourcetext,
publish_via_partition_root);
+ TransformPubColumnList(rels, pstate->p_sourcetext,
+ publish_via_partition_root);
+
PublicationAddTables(puboid, rels, true, NULL);
CloseTableList(rels);
}
@@ -754,6 +913,52 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
return myself;
}
+/*
+ * Change the column list of a relation in a publication
+ */
+static void
+PublicationSetColumns(AlterPublicationStmt *stmt,
+ Form_pg_publication pubform, PublicationTable *table)
+{
+ Relation rel,
+ urel;
+ HeapTuple tup;
+ ObjectAddress obj,
+ secondary;
+
+ rel = table_open(PublicationRelRelationId, RowExclusiveLock);
+ urel = table_openrv(table->relation, ShareUpdateExclusiveLock);
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(RelationGetRelid(urel)),
+ ObjectIdGetDatum(pubform->oid));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errmsg("relation \"%s\" is not already in publication \"%s\"",
+ table->relation->relname,
+ NameStr(pubform->pubname)));
+
+ publication_set_table_columns(rel, tup, urel, table->columns);
+
+ ObjectAddressSet(obj, PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->oid);
+ ObjectAddressSet(secondary, RelationRelationId, RelationGetRelid(urel));
+ EventTriggerCollectSimpleCommand(obj, secondary, (Node *) stmt);
+
+ ReleaseSysCache(tup);
+
+ /*
+ * Invalidate relcache for this relation, to force rebuilding the
+ * publication description.
+ */
+ CacheInvalidateRelcache(urel);
+
+ table_close(rel, RowExclusiveLock);
+ table_close(urel, NoLock);
+
+ InvokeObjectPostAlterHook(PublicationRelationId, pubform->oid, 0);
+}
+
/*
* Change options of a publication.
*/
@@ -783,8 +988,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* If the publication doesn't publish changes via the root partitioned
- * table, the partition's row filter will be used. So disallow using WHERE
- * clause on partitioned table in this case.
+ * table, the partition's row filter and column list will be used. So disallow
+ * using WHERE clause and column lists on partitioned table in this case.
*/
if (!pubform->puballtables && publish_via_partition_root_given &&
!publish_via_partition_root)
@@ -792,7 +997,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* Lock the publication so nobody else can do anything with it. This
* prevents concurrent alter to add partitioned table(s) with WHERE
- * clause(s) which we don't allow when not publishing via root.
+ * clause(s) and/or column lists which we don't allow when not
+ * publishing via root.
*/
LockDatabaseObject(PublicationRelationId, pubform->oid, 0,
AccessShareLock);
@@ -804,13 +1010,21 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
HeapTuple rftuple;
Oid relid = lfirst_oid(lc);
+ bool has_column_list;
+ bool has_row_filter;
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubform->oid));
+ has_row_filter
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL);
+
+ has_column_list
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prattrs, NULL);
+
if (HeapTupleIsValid(rftuple) &&
- !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL))
+ (has_row_filter || has_column_list))
{
HeapTuple tuple;
@@ -819,7 +1033,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
Form_pg_class relform = (Form_pg_class) GETSTRUCT(tuple);
- if (relform->relkind == RELKIND_PARTITIONED_TABLE)
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_row_filter)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot set %s for publication \"%s\"",
@@ -830,6 +1045,18 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
NameStr(relform->relname),
"publish_via_partition_root")));
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_column_list)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot set %s for publication \"%s\"",
+ "publish_via_partition_root = false",
+ stmt->pubname),
+ errdetail("The publication contains a column list for a partitioned table \"%s\" "
+ "which is not allowed when %s is false.",
+ NameStr(relform->relname),
+ "publish_via_partition_root")));
+
ReleaseSysCache(tuple);
}
@@ -838,6 +1065,16 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
}
}
+ /*
+ * FIXME check pubactions vs. replica identity, to ensure the replica
+ * identity is included in the column list. Only do this for update
+ * and delete publications. See check_publication_columns.
+ *
+ * XXX This is needed because publish_via_partition_root may change,
+ * in which case the row filters may be invalid (e.g. with pvpr=false
+ * there must be no filter on partitioned tables).
+ */
+
/* Everything ok, form a new tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -975,10 +1212,20 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
PublicationAddTables(pubid, rels, false, stmt);
}
else if (stmt->action == AP_DropObjects)
PublicationDropTables(pubid, rels, false);
+ else if (stmt->action == AP_SetColumns)
+ {
+ Assert(schemaidlist == NIL);
+ Assert(list_length(tables) == 1);
+
+ PublicationSetColumns(stmt, pubform,
+ linitial_node(PublicationTable, tables));
+ }
else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
@@ -991,6 +1238,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
/*
* To recreate the relation list for the publication, look for
* existing relations that do not need to be dropped.
@@ -1002,32 +1251,85 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
PublicationRelInfo *oldrel;
bool found = false;
HeapTuple rftuple;
- bool rfisnull = true;
Node *oldrelwhereclause = NULL;
+ Bitmapset *oldcolumns = NULL;
/* look up the cache for the old relmap */
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(oldrelid),
ObjectIdGetDatum(pubid));
+ /*
+ * See if the existing relation currently has a WHERE clause or a
+ * column list. We need to compare those too.
+ */
if (HeapTupleIsValid(rftuple))
{
+ bool isnull = true;
Datum whereClauseDatum;
+ Datum columnListDatum;
+ /* Load the WHERE clause for this table. */
whereClauseDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
Anum_pg_publication_rel_prqual,
- &rfisnull);
- if (!rfisnull)
+ &isnull);
+ if (!isnull)
oldrelwhereclause = stringToNode(TextDatumGetCString(whereClauseDatum));
+ /* Transform the int2vector column list to a bitmap. */
+ columnListDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ /*
+ * XXX Maybe make this a separate function. We do this on
+ * multiple places.
+ */
+ if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(columnListDatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ for (int i = 0; i < nelems; i++)
+ oldcolumns = bms_add_member(oldcolumns, elems[i]);
+ }
+
ReleaseSysCache(rftuple);
}
foreach(newlc, rels)
{
PublicationRelInfo *newpubrel;
+ Oid newrelid;
+ Bitmapset *newcolumns = NULL;
newpubrel = (PublicationRelInfo *) lfirst(newlc);
+ newrelid = RelationGetRelid(newpubrel->relation);
+
+ /*
+ * If the new publication has column list, transform it to
+ * a bitmap too.
+ */
+ if (newpubrel->columns)
+ {
+ ListCell *lc;
+
+ foreach(lc, newpubrel->columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(newrelid, colname);
+
+ /* no checks needed here, that happens elsewhere */
+
+ newcolumns = bms_add_member(newcolumns, attnum);
+ }
+ }
/*
* Check if any of the new set of relations matches with the
@@ -1037,7 +1339,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
*/
if (RelationGetRelid(newpubrel->relation) == oldrelid)
{
- if (equal(oldrelwhereclause, newpubrel->whereClause))
+ if (equal(oldrelwhereclause, newpubrel->whereClause) &&
+ bms_equal(oldcolumns, newcolumns))
{
found = true;
break;
@@ -1056,6 +1359,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
{
oldrel = palloc(sizeof(PublicationRelInfo));
oldrel->whereClause = NULL;
+ oldrel->columns = NIL;
oldrel->relation = table_open(oldrelid,
ShareUpdateExclusiveLock);
delrels = lappend(delrels, oldrel);
@@ -1117,7 +1421,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
}
else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* AP_SetObjects */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -1140,6 +1444,10 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
*/
PublicationAddSchemas(pubform->oid, schemaidlist, true, stmt);
}
+ else
+ {
+ /* Nothing to do for AP_SetColumns */
+ }
}
/*
@@ -1402,6 +1710,7 @@ OpenTableList(List *tables)
List *rels = NIL;
ListCell *lc;
List *relids_with_rf = NIL;
+ List *relids_with_collist = NIL;
/*
* Open, share-lock, and check all the explicitly-specified relations
@@ -1436,6 +1745,13 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /* Disallow duplicate tables if there are any with column lists. */
+ if (t->columns || list_member_oid(relids_with_collist, myrelid))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
table_close(rel, ShareUpdateExclusiveLock);
continue;
}
@@ -1443,12 +1759,16 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
pub_rel->whereClause = t->whereClause;
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, myrelid);
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, myrelid);
+
/*
* Add children of this rel, if requested, so that they too are added
* to the publication. A partitioned table can't have any inheritance
@@ -1488,6 +1808,18 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /*
+ * We don't allow to specify column list for both parent
+ * and child table at the same time as it is not very
+ * clear which one should be given preference.
+ */
+ if (childrelid != myrelid &&
+ (t->columns || list_member_oid(relids_with_collist, childrelid)))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
continue;
}
@@ -1497,11 +1829,16 @@ OpenTableList(List *tables)
pub_rel->relation = rel;
/* child inherits WHERE clause from parent */
pub_rel->whereClause = t->whereClause;
+ /* child inherits column list from parent */
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, childrelid);
+
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, childrelid);
}
}
}
@@ -1610,6 +1947,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index dc5872f988c..a9fd0f0c895 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -8365,6 +8365,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
bool missing_ok, LOCKMODE lockmode,
ObjectAddresses *addrs)
{
+ Oid relid = RelationGetRelid(rel);
HeapTuple tuple;
Form_pg_attribute targetatt;
AttrNumber attnum;
@@ -8384,7 +8385,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/*
* get the number of the attribute
*/
- tuple = SearchSysCacheAttName(RelationGetRelid(rel), colName);
+ tuple = SearchSysCacheAttName(relid, colName);
if (!HeapTupleIsValid(tuple))
{
if (!missing_ok)
@@ -8438,13 +8439,42 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * Also, if the column is used in the column list of a publication,
+ * disallow the drop if the DROP is RESTRICT. We don't do anything if the
+ * DROP is CASCADE, which means that the dependency mechanism will remove
+ * the relation from the publication.
+ */
+ if (behavior == DROP_RESTRICT)
+ {
+ List *pubs;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(relid);
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(relid, pubid);
+
+ if (list_member_oid(published_cols, attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot drop column \"%s\" because it is part of publication \"%s\"",
+ colName, get_publication_name(pubid, false)),
+ errhint("Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication."));
+ }
+ }
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
* use find_all_inheritors to do it in one pass.
*/
children =
- find_inheritance_children(RelationGetRelid(rel), lockmode);
+ find_inheritance_children(relid, lockmode);
if (children)
{
@@ -8532,7 +8562,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Add object to delete */
object.classId = RelationRelationId;
- object.objectId = RelationGetRelid(rel);
+ object.objectId = relid;
object.objectSubId = attnum;
add_exact_object_address(&object, addrs);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 09f78f22441..3e282ed99ab 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -573,9 +573,6 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
if (cmd != CMD_UPDATE && cmd != CMD_DELETE)
return;
- if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
- return;
-
/*
* It is only safe to execute UPDATE/DELETE when all columns, referenced
* in the row filters from publications which the relation is in, are
@@ -595,17 +592,33 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
errmsg("cannot update table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_UPDATE && !pubdesc.cols_valid_for_update)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot update table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
else if (cmd == CMD_DELETE && !pubdesc.rf_valid_for_delete)
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
errmsg("cannot delete from table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_DELETE && !pubdesc.cols_valid_for_delete)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot delete from table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
/* If relation has replica identity we are always good. */
if (OidIsValid(RelationGetReplicaIndex(rel)))
return;
+ /* REPLICA IDENTITY FULL is also good for UPDATE/DELETE. */
+ if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ return;
+
/*
* This is UPDATE/DELETE and there is no replica identity.
*
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index d4f8455a2bd..a504437873f 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4850,6 +4850,7 @@ _copyPublicationTable(const PublicationTable *from)
COPY_NODE_FIELD(relation);
COPY_NODE_FIELD(whereClause);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f1002afe7a0..4fc16ce04e3 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2322,6 +2322,7 @@ _equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
COMPARE_NODE_FIELD(whereClause);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a03b33b53bd..25c9b29afdd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9751,13 +9751,14 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr OptWhereClause
+ TABLE relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9772,11 +9773,12 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId OptWhereClause
+ | ColId opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- if ($2)
+ /* FIXME this is a bit cumbersome */
+ if ($2 || $3)
{
/*
* The OptWhereClause must be stored here but it is
@@ -9786,7 +9788,8 @@ PublicationObjSpec:
*/
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVar(NULL, $1, @1);
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
else
{
@@ -9794,23 +9797,25 @@ PublicationObjSpec:
}
$$->location = @1;
}
- | ColId indirection OptWhereClause
+ | ColId indirection opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr OptWhereClause
+ | extended_relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
| CURRENT_SCHEMA
{
@@ -9836,6 +9841,10 @@ pub_obj_list: PublicationObjSpec
*
* ALTER PUBLICATION name SET pub_obj [, ...]
*
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ *
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
* pub_obj is one of:
*
* TABLE table_name [, ...]
@@ -9869,6 +9878,32 @@ AlterPublicationStmt:
n->action = AP_SetObjects;
$$ = (Node *)n;
}
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS '(' columnList ')'
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = $10;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS ALL
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = NIL;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
| ALTER PUBLICATION name DROP pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
@@ -17488,6 +17523,13 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
errmsg("WHERE clause not allowed for schema"),
parser_errposition(pubobj->location));
+ /* Column filter is not allowed on a schema object */
+ if (pubobj->pubtable && pubobj->pubtable->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schema"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index c9b0eeefd7e..816d461acd3 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,16 +29,25 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary);
+ bool binary, Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
static void logicalrep_write_namespace(StringInfo out, Oid nspid);
static const char *logicalrep_read_namespace(StringInfo in);
+
+static bool
+column_in_set(int attnum, Bitmapset *columns)
+{
+ return (columns == NULL || bms_is_member(attnum, columns));
+}
+
+
/*
* Write BEGIN to the output stream.
*/
@@ -398,7 +407,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +419,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -443,7 +452,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -464,11 +473,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -537,7 +546,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, NULL);
}
/*
@@ -652,7 +661,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -674,7 +684,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -751,7 +761,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum *values;
@@ -763,8 +773,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
continue;
+
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -783,6 +799,9 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +923,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -917,8 +936,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* send number of live attributes */
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
continue;
+
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -937,6 +962,9 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 1659964571c..5a28039023b 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -112,6 +112,7 @@
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "utils/acl.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -701,12 +702,13 @@ fetch_remote_table_info(char *nspname, char *relname,
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
int natt;
ListCell *lc;
bool first;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
@@ -747,10 +749,110 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+
+ /*
+ * Get column lists for each relation.
+ *
+ * For initial synchronization, column lists can be ignored in following
+ * cases:
+ *
+ * 1) one of the subscribed publications for the table hasn't specified
+ * any column list
+ *
+ * 2) one of the subscribed publications has puballtables set to true
+ *
+ * 3) one of the subscribed publications is declared as ALL TABLES IN
+ * SCHEMA that includes this relation
+ *
+ * We need to do this before fetching info about column names and types,
+ * so that we can skip columns that should not be replicated.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData pub_names;
+ bool first = true;
+
+ initStringInfo(&pub_names);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&pub_names, ", ");
+ appendStringInfoString(&pub_names, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ /*
+ * Fetch info about column filters for the relation (from all the
+ * publications). We unnest the int2vector values, because that
+ * makes it easier to combine filters by simply adding the attnums
+ * to a new bitmap (without having to parse the int2vector data).
+ * This preserves NULL values, so that if one of the publications
+ * has no column filter, we'll know that.
+ */
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT DISTINCT unnest"
+ " FROM pg_publication p"
+ " LEFT OUTER JOIN pg_publication_rel pr"
+ " ON (p.oid = pr.prpubid AND pr.prrelid = %u)"
+ " LEFT OUTER JOIN unnest(pr.prattrs) ON TRUE,"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt"
+ " WHERE gpt.relid = %u"
+ " AND p.pubname IN ( %s )",
+ lrel->remoteid,
+ lrel->remoteid,
+ pub_names.data);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch column list info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ /*
+ * Merge the column lists (from different publications) by creating
+ * a single bitmap with all the attnums. If we find a NULL value,
+ * that means one of the publications has no column filter for the
+ * table we're syncing.
+ */
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ Datum cfval = slot_getattr(slot, 1, &isnull);
+
+ /* NULL means empty column list, so we're done. */
+ if (isnull)
+ {
+ bms_free(included_cols);
+ included_cols = NULL;
+ break;
+ }
+
+ included_cols = bms_add_member(included_cols,
+ DatumGetInt16(cfval));
+
+ ExecClearTuple(slot);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+
+ walrcv_clear_result(pubres);
+
+ pfree(pub_names.data);
+ }
+
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -778,16 +880,35 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * Store the columns as a list of names. Ignore those that are not
+ * present in the column list, if there is one.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ /* If the column is not in the column list, skip it. */
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ {
+ ExecClearTuple(slot);
+ continue;
+ }
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -821,6 +942,9 @@ fetch_remote_table_info(char *nspname, char *relname,
*
* 3) one of the subscribed publications is declared as ALL TABLES IN
* SCHEMA that includes this relation
+ *
+ * XXX Does this actually handle puballtables and schema publications
+ * correctly?
*/
if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
{
@@ -930,8 +1054,24 @@ copy_table(Relation rel)
/* Regular table with no row filter */
if (lrel.relkind == RELKIND_RELATION && qual == NIL)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+
+ /*
+ * XXX Do we need to list the columns in all cases? Maybe we're replicating
+ * all columns?
+ */
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
+
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
+
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index abfef4e447c..4fae9b7eb0a 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -29,6 +29,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -85,7 +86,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -93,6 +95,8 @@ static void send_repl_origin(LogicalDecodingContext *ctx,
/*
* Only 3 publication actions are used for row filtering ("insert", "update",
* "delete"). See RelationSyncEntry.exprstate[].
+ *
+ * FIXME Do we need something similar for column filters?
*/
enum RowFilterPubAction
{
@@ -143,9 +147,6 @@ typedef struct RelationSyncEntry
*/
ExprState *exprstate[NUM_ROWFILTER_PUBACTIONS];
EState *estate; /* executor state used for row filter */
- MemoryContext cache_expr_cxt; /* private context for exprstate and
- * estate, if any */
-
TupleTableSlot *new_slot; /* slot for storing new tuple */
TupleTableSlot *old_slot; /* slot for storing old tuple */
@@ -164,6 +165,19 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
AttrMap *attrmap;
+
+ /*
+ * Columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this bitmap are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
+
+ /*
+ * Private context to store additional data for this entry - state for
+ * the row filter expressions, column list, etc.
+ */
+ MemoryContext entry_cxt;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -188,6 +202,7 @@ static EState *create_estate_for_relation(Relation rel);
static void pgoutput_row_filter_init(PGOutputData *data,
List *publications,
RelationSyncEntry *entry);
+
static bool pgoutput_row_filter_exec_expr(ExprState *state,
ExprContext *econtext);
static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
@@ -195,6 +210,11 @@ static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
RelationSyncEntry *entry,
ReorderBufferChangeType *action);
+/* column filter routines */
+static void pgoutput_column_filter_init(PGOutputData *data,
+ List *publications,
+ RelationSyncEntry *entry);
+
/*
* Specify output plugin callbacks
*/
@@ -603,11 +623,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -620,7 +640,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -643,13 +664,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip this attribute if it's not present in the column list */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -823,21 +848,21 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
{
Relation relation = RelationIdGetRelation(entry->publish_as_relid);
- Assert(entry->cache_expr_cxt == NULL);
+ Assert(entry->entry_cxt == NULL);
- /* Create the memory context for row filters */
- entry->cache_expr_cxt = AllocSetContextCreate(data->cachectx,
- "Row filter expressions",
- ALLOCSET_DEFAULT_SIZES);
+ /* Create the memory context for entry data (row filters, ...) */
+ entry->entry_cxt = AllocSetContextCreate(data->cachectx,
+ "entry private context",
+ ALLOCSET_SMALL_SIZES);
- MemoryContextCopyAndSetIdentifier(entry->cache_expr_cxt,
+ MemoryContextCopyAndSetIdentifier(entry->entry_cxt,
RelationGetRelationName(relation));
/*
* Now all the filters for all pubactions are known. Combine them when
* their pubactions are the same.
*/
- oldctx = MemoryContextSwitchTo(entry->cache_expr_cxt);
+ oldctx = MemoryContextSwitchTo(entry->entry_cxt);
entry->estate = create_estate_for_relation(relation);
for (idx = 0; idx < NUM_ROWFILTER_PUBACTIONS; idx++)
{
@@ -860,6 +885,124 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
}
}
+/*
+ * Initialize the column filter.
+ */
+static void
+pgoutput_column_filter_init(PGOutputData *data, List *publications,
+ RelationSyncEntry *entry)
+{
+ ListCell *lc;
+ MemoryContext oldctx;
+
+ /*
+ * Find if there are any row filters for this relation. If there are, then
+ * prepare the necessary ExprState and cache it in entry->exprstate. To
+ * build an expression state, we need to ensure the following:
+ *
+ * All the given publication-table mappings must be checked.
+ *
+ * Multiple publications might have multiple row filters for this
+ * relation. Since row filter usage depends on the DML operation, there
+ * are multiple lists (one for each operation) to which row filters will
+ * be appended.
+ *
+ * FOR ALL TABLES implies "don't use row filter expression" so it takes
+ * precedence.
+ */
+ foreach(lc, publications)
+ {
+ Publication *pub = lfirst(lc);
+ HeapTuple cftuple = NULL;
+ Datum cfdatum = 0;
+ bool pub_no_filter = false;
+
+ if (pub->alltables)
+ {
+ /*
+ * If the publication is FOR ALL TABLES then it is treated the
+ * same as if this table has no row filters (even if for other
+ * publications it does).
+ */
+ pub_no_filter = true;
+ }
+ else
+ {
+ /*
+ * Check for the presence of a row filter in this publication.
+ */
+ cftuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(entry->publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(cftuple))
+ {
+ /* Null indicates no filter. */
+ cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
+ Anum_pg_publication_rel_prattrs,
+ &pub_no_filter);
+
+ /*
+ * When no column list is defined, so publish all columns.
+ * Otherwise merge the columns to the column list.
+ */
+ if (!pub_no_filter) /* when not null */
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(cfdatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /*
+ * If the memory context for the entry does not exist yet,
+ * create it now. It may already exist if the relation has
+ * a row filter.
+ */
+ if (!entry->entry_cxt)
+ {
+ Relation relation = RelationIdGetRelation(entry->publish_as_relid);
+
+ entry->entry_cxt = AllocSetContextCreate(data->cachectx,
+ "entry private context",
+ ALLOCSET_SMALL_SIZES);
+
+ MemoryContextCopyAndSetIdentifier(entry->entry_cxt,
+ RelationGetRelationName(relation));
+ }
+
+ oldctx = MemoryContextSwitchTo(entry->entry_cxt);
+ for (int i = 0; i < nelems; i++)
+ entry->columns = bms_add_member(entry->columns,
+ elems[i]);
+ MemoryContextSwitchTo(oldctx);
+ }
+ }
+ else
+ {
+ pub_no_filter = true;
+ }
+ }
+
+ /* found publication with no filter, so we're done */
+ if (pub_no_filter)
+ {
+ if (cftuple)
+ ReleaseSysCache(cftuple);
+
+ bms_free(entry->columns);
+ entry->columns = NULL;
+
+ break;
+ }
+
+ ReleaseSysCache(cftuple);
+ } /* loop all subscribed publications */
+
+}
+
/*
* Initialize the slot for storing new and old tuples, and build the map that
* will be used to convert the relation's tuples into the ancestor's format.
@@ -1224,7 +1367,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -1278,11 +1421,13 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel,
- new_slot, data->binary);
+ new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel,
- old_slot, new_slot, data->binary);
+ old_slot, new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel,
@@ -1729,8 +1874,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->new_slot = NULL;
entry->old_slot = NULL;
memset(entry->exprstate, 0, sizeof(entry->exprstate));
- entry->cache_expr_cxt = NULL;
+ entry->entry_cxt = NULL;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->attrmap = NULL;
}
@@ -1776,6 +1922,8 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
entry->pubactions.pubinsert = false;
entry->pubactions.pubupdate = false;
entry->pubactions.pubdelete = false;
@@ -1799,17 +1947,18 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/*
* Row filter cache cleanups.
*/
- if (entry->cache_expr_cxt)
- MemoryContextDelete(entry->cache_expr_cxt);
+ if (entry->entry_cxt)
+ MemoryContextDelete(entry->entry_cxt);
- entry->cache_expr_cxt = NULL;
+ entry->entry_cxt = NULL;
entry->estate = NULL;
memset(entry->exprstate, 0, sizeof(entry->exprstate));
/*
* Build publication cache. We can't use one provided by relcache as
- * relcache considers all publications given relation is in, but here
- * we only need to consider ones that the subscriber requested.
+ * relcache considers all publications that the given relation is in,
+ * but here we only need to consider ones that the subscriber
+ * requested.
*/
foreach(lc, data->publications)
{
@@ -1878,6 +2027,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
}
/*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
* Don't publish changes for partitioned tables, because
* publishing those of its partitions suffices, unless partition
* changes won't be published due to pubviaroot being set.
@@ -1938,6 +2090,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/* Initialize the row filter */
pgoutput_row_filter_init(data, rel_publications, entry);
+
+ /* Initialize the column filter */
+ pgoutput_column_filter_init(data, rel_publications, entry);
}
list_free(pubids);
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index fccffce5729..a2da72f0d48 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -5553,6 +5553,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
return;
}
@@ -5565,6 +5567,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
/* Fetch the publication membership info. */
puboids = GetRelationPublications(relid);
@@ -5616,7 +5620,7 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
*/
if (!pubform->puballtables &&
(pubform->pubupdate || pubform->pubdelete) &&
- contain_invalid_rfcolumn(pubid, relation, ancestors,
+ pub_rf_contains_invalid_column(pubid, relation, ancestors,
pubform->pubviaroot))
{
if (pubform->pubupdate)
@@ -5625,6 +5629,23 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->rf_valid_for_delete = false;
}
+ /*
+ * Check if all columns are part of the REPLICA IDENTITY index or not.
+ *
+ * If the publication is FOR ALL TABLES then it means the table has no
+ * column list and we can skip the validation.
+ */
+ if (!pubform->puballtables &&
+ (pubform->pubupdate || pubform->pubdelete) &&
+ pub_collist_contains_invalid_column(pubid, relation, ancestors,
+ pubform->pubviaroot))
+ {
+ if (pubform->pubupdate)
+ pubdesc->cols_valid_for_update = false;
+ if (pubform->pubdelete)
+ pubdesc->cols_valid_for_delete = false;
+ }
+
ReleaseSysCache(tup);
/*
@@ -5636,6 +5657,16 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
!pubdesc->rf_valid_for_update && !pubdesc->rf_valid_for_delete)
break;
+
+ /*
+ * If we know everything is replicated and the column list is invalid
+ * for update and delete, there is no point to check for other
+ * publications.
+ */
+ if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
+ pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
+ !pubdesc->cols_valid_for_update && !pubdesc->cols_valid_for_delete)
+ break;
}
if (relation->rd_pubdesc)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e69dcf8a484..f208c7a6c59 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4075,6 +4075,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_prpubid;
int i_prrelid;
int i_prrelqual;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4088,12 +4089,20 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
if (fout->remoteVersion >= 150000)
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual "
- "FROM pg_catalog.pg_publication_rel");
+ "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual, "
+ "(CASE\n"
+ " WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT array_agg(attname)\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) prattrs "
+ "FROM pg_catalog.pg_publication_rel pr");
else
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "NULL AS prrelqual "
+ "NULL AS prrelqual, NULL AS prattrs "
"FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
@@ -4104,6 +4113,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
i_prrelqual = PQfnumber(res, "prrelqual");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4149,6 +4159,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
else
pubrinfo[j].pubrelqual = pg_strdup(PQgetvalue(res, i, i_prrelqual));
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[j].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4223,10 +4255,13 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+
if (pubrinfo->pubrelqual)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 997a3b60719..680b07dcd52 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -632,6 +632,7 @@ typedef struct _PublicationRelInfo
PublicationInfo *publication;
TableInfo *pubtable;
char *pubrelqual;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 3e55ff26f82..ed57c53bcb5 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2428,6 +2428,28 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'ALTER PUBLICATION pub1 ADD TABLE test_sixth_table (col3, col2)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_sixth_table (col3, col2);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_sixth_table (col2, col3);\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'ALTER PUBLICATION pub1 ADD TABLE test_seventh_table (col3, col2) WHERE (col1 = 1)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_seventh_table (col3, col2) WHERE (col1 = 1);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_seventh_table (col2, col3) WHERE ((col1 = 1));\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'ALTER PUBLICATION pub3 ADD ALL TABLES IN SCHEMA dump_test' => {
create_order => 51,
create_sql =>
@@ -2778,6 +2800,44 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'CREATE TABLE test_sixth_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_sixth_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_sixth_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'CREATE TABLE test_seventh_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_seventh_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_seventh_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'CREATE TABLE test_table_identity' => {
create_order => 3,
create_sql => 'CREATE TABLE dump_test.test_table_identity (
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index e3382933d98..e462ccfb748 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2880,6 +2880,7 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_namespace pn ON p.oid = pn.pnpubid\n"
" JOIN pg_catalog.pg_class pc ON pc.relnamespace = pn.pnnspid\n"
@@ -2887,6 +2888,12 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , pg_get_expr(pr.prqual, c.oid)\n"
+ " , (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT string_agg(attname, ', ')\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) "
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
" JOIN pg_catalog.pg_class c ON c.oid = pr.prrelid\n"
@@ -2894,6 +2901,7 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2904,12 +2912,14 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
"WHERE pr.prrelid = '%s'\n"
"UNION ALL\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2931,6 +2941,11 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, " \"%s\"",
PQgetvalue(result, i, 0));
+ /* column list (if any) */
+ if (!PQgetisnull(result, i, 2))
+ appendPQExpBuffer(&buf, " (%s)",
+ PQgetvalue(result, i, 2));
+
/* row filter (if any) */
if (!PQgetisnull(result, i, 1))
appendPQExpBuffer(&buf, " WHERE %s",
@@ -5867,7 +5882,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5884,15 +5899,19 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (as_schema)
+ printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
+ else
{
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
+
+ if (!PQgetisnull(res, i, 3))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 3));
+
if (!PQgetisnull(res, i, 2))
appendPQExpBuffer(buf, " WHERE %s", PQgetvalue(res, i, 2));
}
- else
- printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
}
@@ -6021,11 +6040,22 @@ describePublications(const char *pattern)
printfPQExpBuffer(&buf,
"SELECT n.nspname, c.relname");
if (pset.sversion >= 150000)
+ {
appendPQExpBufferStr(&buf,
", pg_get_expr(pr.prqual, c.oid)");
+ appendPQExpBufferStr(&buf,
+ ", (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string("
+ " ARRAY(SELECT attname\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END)");
+ }
else
appendPQExpBufferStr(&buf,
- ", NULL");
+ ", NULL, NULL");
appendPQExpBuffer(&buf,
"\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index fe773cf9b7d..70e053e04f1 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -85,6 +85,13 @@ typedef struct PublicationDesc
*/
bool rf_valid_for_update;
bool rf_valid_for_delete;
+
+ /*
+ * true if the columns are part of the replica identity or the publication actions
+ * do not include UPDATE or DELETE.
+ */
+ bool cols_valid_for_update;
+ bool cols_valid_for_delete;
} PublicationDesc;
typedef struct Publication
@@ -100,6 +107,7 @@ typedef struct PublicationRelInfo
{
Relation relation;
Node *whereClause;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -123,8 +131,11 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
+extern void GetActionsInPublication(Oid pubid, PublicationActions *actions);
extern List *GetPublicationSchemas(Oid pubid);
extern List *GetSchemaPublications(Oid schemaid);
extern List *GetSchemaPublicationRelations(Oid schemaid,
@@ -143,6 +154,8 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *pri
bool if_not_exists);
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern void publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index 0dd0f425db9..4feb581899e 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -34,6 +34,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
#ifdef CATALOG_VARLEN /* variable-length fields start here */
pg_node_tree prqual; /* qualifications */
+ int2vector prattrs; /* columns to replicate */
#endif
} FormData_pg_publication_rel;
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index 7813cbcb6bb..ae87caf089d 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -31,7 +31,9 @@ extern void RemovePublicationSchemaById(Oid psoid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
extern void AlterPublicationOwner_oid(Oid pubid, Oid newOwnerId);
extern void InvalidatePublicationRels(List *relids);
-extern bool contain_invalid_rfcolumn(Oid pubid, Relation relation,
+extern bool pub_rf_contains_invalid_column(Oid pubid, Relation relation,
+ List *ancestors, bool pubviaroot);
+extern bool pub_collist_contains_invalid_column(Oid pubid, Relation relation,
List *ancestors, bool pubviaroot);
#endif /* PUBLICATIONCMDS_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1617702d9d6..1375a173e3b 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3652,6 +3652,7 @@ typedef struct PublicationTable
NodeTag type;
RangeVar *relation; /* relation to be published */
Node *whereClause; /* qualifications */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
@@ -3688,7 +3689,8 @@ typedef enum AlterPublicationAction
{
AP_AddObjects, /* add objects to publication */
AP_DropObjects, /* remove objects from publication */
- AP_SetObjects /* set list of objects */
+ AP_SetObjects, /* set list of objects */
+ AP_SetColumns /* change list of columns for a table */
} AlterPublicationAction;
typedef struct AlterPublicationStmt
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4d2c881644a..a771ab8ff33 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -209,12 +209,12 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary);
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -231,7 +231,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 4e191c120ac..79ced2921b6 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -613,6 +613,372 @@ DROP TABLE rf_tbl_abcd_pk;
DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+RESET client_min_messages;
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ERROR: cannot reference generated column "d" in publication column list
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+ERROR: cannot reference system column "ctid" in publication column list
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+ERROR: cannot drop column "c" because it is part of publication "testpub_fortable"
+HINT: Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication.
+-- ok: for insert-only publication, the column list is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: change the replica identity to "b", and then column list (a, c) fails
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+-- error: replica identity (b,c) is not covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+ Publication testpub_table_ins
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | t | f
+Tables:
+ "public.testpub_tbl5" (a)
+
+-- with REPLICA IDENTITY FULL, column lists are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ERROR: cannot update table "testpub_tbl6"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+ALTER PUBLICATION testpub_fortable ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ERROR: cannot update table "testpub_tbl6"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- make sure changing the column list is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: update the column list
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, c)
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl8;
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_0"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+ Publication testpub_both_filters
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | t | t | t | f
+Tables:
+ "public.testpub_tbl_both_filters" (a, c) WHERE (c <> 1)
+
+\d+ testpub_tbl_both_filters
+ Table "public.testpub_tbl_both_filters"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+ c | integer | | not null | | plain | |
+Indexes:
+ "testpub_tbl_both_filters_pkey" PRIMARY KEY, btree (a, c) REPLICA IDENTITY
+Publications:
+ "testpub_both_filters" (a, c) WHERE (c <> 1)
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+-- Tests for partitioned table
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+ERROR: cannot use publication column list for relation "rf_tbl_abcd_part_pk"
+DETAIL: column list cannot be used for a partitioned table when publish_via_partition_root is false.
+-- ok - can use row filter for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any row filter is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+ERROR: cannot set publish_via_partition_root = false for publication "testpub6"
+DETAIL: The publication contains a column list for a partitioned table "rf_tbl_abcd_part_pk" which is not allowed when publish_via_partition_root is false.
+-- Now change the root filter to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have row filter for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -1058,6 +1424,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schema
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 5457c56b33f..be05ac9f763 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -373,6 +373,292 @@ DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+RESET client_min_messages;
+
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+-- ok: for insert-only publication, the column list is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+
+-- error: change the replica identity to "b", and then column list (a, c) fails
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+-- error: replica identity (b,c) is not covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+
+-- with REPLICA IDENTITY FULL, column lists are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+
+ALTER PUBLICATION testpub_fortable ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+
+-- make sure changing the column list is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: update the column list
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl8;
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+\d+ testpub_tbl_both_filters
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Tests for partitioned table
+
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - can use row filter for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any row filter is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- Now change the root filter to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have row filter for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
+
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -614,6 +900,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/029_column_list.pl b/src/test/subscription/t/029_column_list.pl
new file mode 100644
index 00000000000..5266967b3f4
--- /dev/null
+++ b/src/test/subscription/t/029_column_list.pl
@@ -0,0 +1,1124 @@
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test partial-column publication of tables
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# create publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+sub wait_for_subscription_sync
+{
+ my ($node) = @_;
+
+ # Also wait for initial table sync to finish
+ my $synced_query = "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+
+ $node->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+}
+
+# setup tables on both nodes
+
+# tab1: simple 1:1 replication
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+# tab2: replication from regular to table with fewer columns
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)
+));
+
+# tab3: simple 1:1 replication with weird column names
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "B" varchar, "c'" int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "c'" int)
+));
+
+# test_part: partitioned tables, with partitioning (including multi-level
+# partitioning, and fewer columns on the subscriber)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+# tab4: table with user-defined enum types
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, d text);
+));
+
+
+# TEST: create publication and subscription for some of the tables with
+# column lists
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub1
+ FOR TABLE tab1 (a, "B"), tab3 ("a'", "c'"), test_part (a, b), tab4 (a, b, d)
+ WITH (publish_via_partition_root = 'true');
+));
+
+# check that we got the right prattrs values for the publication in the
+# pg_publication_rel catalog (order by relname, to get stable ordering)
+my $result = $node_publisher->safe_psql('postgres', qq(
+ SELECT relname, prattrs
+ FROM pg_publication_rel pb JOIN pg_class pc ON(pb.prrelid = pc.oid)
+ ORDER BY relname
+));
+
+is($result, qq(tab1|1 2
+tab3|1 3
+tab4|1 2 4
+test_part|1 2), 'publication relation updated');
+
+# TEST: insert data into the tables, create subscription and see if sync
+# replicates the right columns
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (1, 2, 3);
+ INSERT INTO tab1 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (1, 2, 3);
+ INSERT INTO tab3 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (1, 'red', 3, 'oh my');
+ INSERT INTO tab4 VALUES (2, 'blue', 4, 'hello');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (1, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (2, 'bcd', '2021-07-03 11:12:13');
+ INSERT INTO test_part VALUES (7, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (8, 'bcd', '2021-07-03 11:12:13');
+));
+
+# create subscription for the publication, wait for sync to complete,
+# then check the sync results
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+4|5|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+4|6), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+7|abc
+8|bcd), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: now insert more data into the tables, and wait until we replicate
+# them (not by tablesync, but regular decoding and replication)
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (2, 3, 4);
+ INSERT INTO tab1 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (2, 3, 4);
+ INSERT INTO tab3 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (3, 'red', 5, 'foo');
+ INSERT INTO tab4 VALUES (4, 'blue', 6, 'bar');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (3, 'xxx', '2022-02-01 10:00:00');
+ INSERT INTO test_part VALUES (4, 'yyy', '2022-03-02 15:12:13');
+ INSERT INTO test_part VALUES (9, 'zzz', '2022-04-03 21:00:00');
+ INSERT INTO test_part VALUES (10, 'qqq', '2022-05-04 22:12:13');
+));
+
+# wait for catchup before checking the subscriber
+$node_publisher->wait_for_catchup('sub1');
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+2|3|
+4|5|
+5|6|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+2|4
+4|6
+5|7), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello
+3|red|foo
+4|blue|bar), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+3|xxx
+4|yyy
+7|abc
+8|bcd
+9|zzz
+10|qqq), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: do some updates on some of the tables, both on columns included
+# in the column list and other
+
+# tab1: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET "B" = 2 * "B" where a = 1));
+
+# tab1: update of non-replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET c = 2*c where a = 4));
+
+# tab3: update of non-replicated
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "B" = "B" || ' updated' where "a'" = 4));
+
+# tab3: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "c'" = 2 * "c'" where "a'" = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'blue', c = c * 2, d = d || ' updated' where a = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'red', c = c * 2, d = d || ' updated' where a = 2));
+
+# wait for the replication to catch up, and check the UPDATE results got
+# replicated correctly, with the right column list
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab1 ORDER BY a));
+is($result,
+qq(1|4|
+2|3|
+4|5|
+5|6|), 'only update on column tab1.b is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result,
+qq(1|6
+2|4
+4|6
+5|7), 'only update on column tab3.c is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab4 ORDER BY a));
+
+is($result, qq(1|blue|oh my updated
+2|red|hello updated
+3|red|foo
+4|blue|bar), 'update on column tab4.c is not replicated');
+
+
+# TEST: add table with a column list, insert data, replicate
+
+# insert some data before adding it to the publication
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (1, 'abc', 3);
+));
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2 (a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION");
+
+# wait for the tablesync to complete, add a bit more data and then check
+# the results of the replication
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (2, 'def', 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|def), 'insert on column tab2.c is not replicated');
+
+# do a couple updates, check the correct stuff gets replicated
+$node_publisher->safe_psql('postgres', qq(
+ UPDATE tab2 SET c = 5 where a = 1;
+ UPDATE tab2 SET b = 'xyz' where a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|xyz), 'update on column tab2.c is not replicated');
+
+
+# TEST: add a table to two publications with different column lists, and
+# create a single subscription replicating both publications
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+
+ -- insert a couple initial records
+ INSERT INTO tab5 VALUES (1, 11, 111, 1111);
+ INSERT INTO tab5 VALUES (2, 22, 222, 2222);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub3
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->wait_for_catchup('sub1');
+
+# insert data and make sure all the columns (union of the columns lists)
+# get fully replicated
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (3, 33, 333, 3333);
+ INSERT INTO tab5 VALUES (4, 44, 444, 4444);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111
+2|22|2222
+3|33|3333
+4|44|4444),
+ 'overlapping publications with overlapping column lists');
+
+# and finally, set the column list to ALL for one of the publications,
+# which means replicating all columns (removing the column list), but
+# first add the missing column to the table on subscriber
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub3 ALTER TABLE tab5 SET COLUMNS ALL;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ ALTER TABLE tab5 ADD COLUMN c INT;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (5, 55, 555, 5555);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111|
+2|22|2222|
+3|33|3333|
+4|44|4444|
+5|55|5555|555),
+ 'overlapping publications with overlapping column lists');
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key (but use a different column in
+# the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub4 FOR TABLE tab6 (a, b);
+
+ -- initial data
+ INSERT INTO tab6 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (2, 33, 444, 5555);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is still covered by the column list, though)
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+# we need to do the same thing on the subscriber
+# XXX What would happen if this happens before the publisher ALTER? Or
+# interleaved, somehow? But that seems unrelated to column lists.
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (3, 55, 666, 8888);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key with a key on multiple columns
+# (all of them covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub5 FOR TABLE tab7 (a, b);
+
+ -- some initial data
+ INSERT INTO tab7 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub5
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (2, 33, 444, 5555);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is not covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ ALTER TABLE tab7 ADD PRIMARY KEY (a, b);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (3, 55, 666, 7777);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+# now switch the primary key again to another columns not covered by the
+# column list, but also generate writes between the drop and creation
+# of the new constraint
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ INSERT INTO tab7 VALUES (4, 77, 888, 9999);
+ -- update/delete is not allowed for tables without RI
+ ALTER TABLE tab7 ADD PRIMARY KEY (b, a);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+ DELETE FROM tab7 WHERE a = 1;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(2|264||
+3|220||
+4|154||),
+ 'replication with the modified primary key');
+
+
+# TEST: partitioned tables (with publish_via_partition_root = false)
+# and replica identity. The (leaf) partitions may have different RI, so
+# we need to check the partition RI (with respect to the column list)
+# while attaching the partition.
+
+# First, let's create a partitioned table with two partitions, each with
+# a different RI, but a column list not covering all those RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+
+ -- initial data, one row in each partition
+ INSERT INTO test_part_a VALUES (1, 3);
+ INSERT INTO test_part_a VALUES (6, 4);
+));
+
+# do the same thing on the subscriber (with the opposite column order)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (b int, a int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+));
+
+# create a publication replicating just the column "a", which is not enough
+# for the second partition
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub6 FOR TABLE test_part_a (b, a) WITH (publish_via_partition_root = true);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_1 (a);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_2 (b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub6
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_a VALUES (2, 5);
+ INSERT INTO test_part_a VALUES (7, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT a, b FROM test_part_a ORDER BY a, b"),
+ qq(1|3
+2|5
+6|4
+7|6),
+ 'partitions with different replica identities not replicated correctly');
+
+# This time start with a column list covering RI for all partitions, but
+# then update the column list to not cover column "b" (needed by the
+# second partition)
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+
+ -- initial data, one row in each partitions
+ INSERT INTO test_part_b VALUES (1, 1);
+ INSERT INTO test_part_b VALUES (6, 2);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub7 FOR TABLE test_part_b (a, b) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub7
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_b VALUES (2, 3);
+ INSERT INTO test_part_b VALUES (7, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_b ORDER BY a, b"),
+ qq(1|1
+2|3
+6|2
+7|4),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: This time start with a column list covering RI for all partitions,
+# but then update RI for one of the partitions to not be covered by the
+# column list anymore.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+
+ -- initial data, one row for each partition
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+));
+
+# create a publication replicating data through partition root, with a column
+# filter on the root, and then add the partitions one by one with separate
+# column lists (but those are not applied)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a,c);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SUBSCRIPTION sub1;
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub8;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_c VALUES (3, 7, 8);
+ INSERT INTO test_part_c VALUES (4, 9, 10);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||5
+2|4|
+3||8
+4|9|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# create a publication not replicating data through partition root, without
+# a column list on the root, and then add the partitions one by one with
+# separate column lists
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub8;
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ TRUNCATE test_part_c;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ TRUNCATE test_part_c;
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||
+2|4|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: Start with a single partition, with RI compatible with the column
+# filter, and then attach a partition with incompatible RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ INSERT INTO test_part_d VALUES (1, 2);
+));
+
+# do the same thing on the subscriber (in fact, create both partitions right
+# away, no need to delay that)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ CREATE TABLE test_part_d_2 PARTITION OF test_part_d FOR VALUES IN (2,4);
+ ALTER TABLE test_part_d_2 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_2 REPLICA IDENTITY USING INDEX test_part_d_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub9 FOR TABLE test_part_d (a) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub9
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_d VALUES (3, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_d ORDER BY a, b"),
+ qq(1|
+3|),
+ 'partitions with different replica identities not replicated correctly');
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. So with column lists (a,b) and (a,c) we
+# should replicate (a,b,c).
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+
+ -- initial data
+ INSERT INTO test_mix_1 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_1 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_1 ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES, we should replicate all columns.
+
+# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ test_part, test_part_a, test_part_b, test_part_c, test_part_d;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
+
+ -- initial data
+ INSERT INTO test_mix_2 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_3, pub_mix_4;
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_2 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_2"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES IN SCHEMA, we should replicate all columns.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
+
+ -- initial data
+ INSERT INTO test_mix_3 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_5, pub_mix_6;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_3 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_3"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: Check handling of publish_via_partition_root - if a partition is
+# published through partition root, we should only apply the column list
+# defined for the whole table (not the partitions) - both during the initial
+# sync and when replicating changes. This is what we do for row filters.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+
+ CREATE PUBLICATION pub_root_true FOR TABLE test_root (a) WITH (publish_via_partition_root = true);
+
+ -- initial data
+ INSERT INTO test_root VALUES (1, 2, 3);
+ INSERT INTO test_root VALUES (10, 20, 30);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_root_true;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_root VALUES (2, 3, 4);
+ INSERT INTO test_root VALUES (11, 21, 31);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_root ORDER BY a, b, c"),
+ qq(1||
+2||
+10||
+11||),
+ 'publication via partition root applies column list');
+
+
+# TEST: Multiple publications which publish schema of parent table and
+# partition. The partition is published through two publications, once
+# through a schema (so no column list) containing the parent, and then
+# also directly (with a columns list). The expected outcome is there is
+# no column list.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub1, pub2, pub3, pub4, pub5, pub6, pub7, pub8;
+
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA s1;
+ CREATE PUBLICATION pub2 FOR TABLE t_1(b);
+
+ -- initial data
+ INSERT INTO s1.t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub1, pub2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'two publications, publishing the same relation');
+
+# Now resync the subcription, but with publications in the opposite order.
+# The result should be the same.
+
+$node_subscriber->safe_psql('postgres', qq(
+ TRUNCATE s1.t;
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub1;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (7, 8, 9);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(7|8|9),
+ 'two publications, publishing the same relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub3 FOR TABLE t_1 (a), t_2
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub3;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+# Note: The difference from the preceding test is that in this case both
+# relations have a column list defined.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub4 FOR TABLE t_1 (a), t_2 (b)
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
+
+done_testing();
--
2.34.1
On Mon, Mar 14, 2022 at 2:37 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 3/12/22 05:30, Amit Kapila wrote:
...
Okay, please find attached. I have done basic testing of this, if we
agree with this approach then this will require some more testing.Thanks, the proposed changes seem like a clear improvement, so I've
added them, with some minor tweaks (mostly to comments).
One minor point: Did you intentionally remove
list_free(rel_publications) before resetting the list from the second
patch? The memory for rel_publications is allocated in
TopTransactionContext, so a large transaction touching many relations
will only free this at end of the transaction which may not be a big
deal as we don't do this every time. We free this list a few lines
down in successful case so this appears slightly odd to me but I am
fine if you think it doesn't matter.
--
With Regards,
Amit Kapila.
On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
On 3/12/22 05:30, Amit Kapila wrote:
...
Okay, please find attached. I have done basic testing of this, if we
agree with this approach then this will require some more testing.Thanks, the proposed changes seem like a clear improvement, so I've
added them, with some minor tweaks (mostly to comments).
Hi,
Thanks for updating the patches !
And sorry for the row filter bug caused by my mistake.
I looked at the two fixup patches. I am thinking would it be better if we
add one testcase for these two bugs? Maybe like the attachment.
(Attach the fixup patch to make the cfbot happy)
Best regards,
Hou zj
Attachments:
0003-fixup-row-filter-publications-20220313.patchapplication/octet-stream; name=0003-fixup-row-filter-publications-20220313.patchDownload
From 5128a396c400801a317fa5043d62d559850f0548 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Fri, 11 Mar 2022 02:23:25 +0100
Subject: [PATCH 2/3] fixup: row-filter publications
When initializing the row filter, consider only publications that
actually include the relation (publish_as_relid). The publications may
include different ancestors, in which case the function would get
confused and conclude there's no row filter.
---
src/backend/replication/pgoutput/pgoutput.c | 26 +++++++++++++++++----
1 file changed, 21 insertions(+), 5 deletions(-)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 104432fb3a6..abfef4e447c 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1890,8 +1890,6 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
- rel_publications = lappend(rel_publications, pub);
-
/*
* We want to publish the changes as the top-most ancestor
* across all publications. So we need to check if the
@@ -1902,9 +1900,27 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
if (publish_ancestor_level > ancestor_level)
continue;
- /* The new value is an ancestor, so let's keep it. */
- publish_as_relid = pub_relid;
- publish_ancestor_level = ancestor_level;
+ /*
+ * If we found an ancestor higher up in the tree, discard
+ * the list of publications through which we replicate it,
+ * and use the new ancestor.
+ */
+ if (publish_ancestor_level < ancestor_level)
+ {
+ publish_as_relid = pub_relid;
+ publish_ancestor_level = ancestor_level;
+
+ /* reset the publication list for this relation */
+ rel_publications = NIL;
+ }
+ else
+ {
+ /* Same ancestor leve, has to be the same OID. */
+ Assert(publish_as_relid == pub_relid);
+ }
+
+ /* Track this publications. */
+ rel_publications = lappend(rel_publications, pub);
}
}
--
2.34.1
0001-fixup-publish_as_relid-20220313.patchapplication/octet-stream; name=0001-fixup-publish_as_relid-20220313.patchDownload
From 49a9d1627098440a32c75727860551db3c20b733 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Wed, 9 Mar 2022 18:10:56 +0100
Subject: [PATCH 1/3] fixup: publish_as_relid
Make sure to determine the top-most ancestor listed in any publication.
Otherwise we might end up with different values depending on the order
of publications (as listed in subscription).
---
src/backend/catalog/pg_publication.c | 21 +++++++++-
src/backend/commands/publicationcmds.c | 2 +-
src/backend/replication/pgoutput/pgoutput.c | 43 +++++++++++++++++++--
src/include/catalog/pg_publication.h | 3 +-
4 files changed, 62 insertions(+), 7 deletions(-)
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 25998fbb39b..789b895db89 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -277,16 +277,21 @@ GetPubPartitionOptionRelations(List *result, PublicationPartOpt pub_partopt,
/*
* Returns the relid of the topmost ancestor that is published via this
- * publication if any, otherwise returns InvalidOid.
+ * publication if any and set its ancestor level to ancestor_level,
+ * otherwise returns InvalidOid.
+ *
+ * The ancestor_level value allows us to compare the results for multiple
+ * publications, and decide which value is higher up.
*
* Note that the list of ancestors should be ordered such that the topmost
* ancestor is at the end of the list.
*/
Oid
-GetTopMostAncestorInPublication(Oid puboid, List *ancestors)
+GetTopMostAncestorInPublication(Oid puboid, List *ancestors, int *ancestor_level)
{
ListCell *lc;
Oid topmost_relid = InvalidOid;
+ int level = 0;
/*
* Find the "topmost" ancestor that is in this publication.
@@ -297,13 +302,25 @@ GetTopMostAncestorInPublication(Oid puboid, List *ancestors)
List *apubids = GetRelationPublications(ancestor);
List *aschemaPubids = NIL;
+ level++;
+
if (list_member_oid(apubids, puboid))
+ {
topmost_relid = ancestor;
+
+ if (ancestor_level)
+ *ancestor_level = level;
+ }
else
{
aschemaPubids = GetSchemaPublications(get_rel_namespace(ancestor));
if (list_member_oid(aschemaPubids, puboid))
+ {
topmost_relid = ancestor;
+
+ if (ancestor_level)
+ *ancestor_level = level;
+ }
}
list_free(apubids);
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 16b8661a1b7..a7b74dc60ad 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -323,7 +323,7 @@ contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
*/
if (pubviaroot && relation->rd_rel->relispartition)
{
- publish_as_relid = GetTopMostAncestorInPublication(pubid, ancestors);
+ publish_as_relid = GetTopMostAncestorInPublication(pubid, ancestors, NULL);
if (!OidIsValid(publish_as_relid))
publish_as_relid = relid;
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ea57a0477f0..104432fb3a6 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1748,6 +1748,7 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
List *schemaPubids = GetSchemaPublications(schemaId);
ListCell *lc;
Oid publish_as_relid = relid;
+ int publish_ancestor_level = 0;
bool am_partition = get_rel_relispartition(relid);
char relkind = get_rel_relkind(relid);
List *rel_publications = NIL;
@@ -1815,11 +1816,28 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
Publication *pub = lfirst(lc);
bool publish = false;
+ /*
+ * Under what relid should we publish changes in this publication?
+ * We'll use the top-most relid across all publications. Also track
+ * the ancestor level for this publication.
+ */
+ Oid pub_relid = relid;
+ int ancestor_level = 0;
+
+ /*
+ * If this is a FOR ALL TABLES publication, pick the partition root
+ * and set the ancestor level accordingly.
+ */
if (pub->alltables)
{
publish = true;
if (pub->pubviaroot && am_partition)
- publish_as_relid = llast_oid(get_partition_ancestors(relid));
+ {
+ List *ancestors = get_partition_ancestors(relid);
+
+ pub_relid = llast_oid(ancestors);
+ ancestor_level = list_length(ancestors);
+ }
}
if (!publish)
@@ -1835,16 +1853,21 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
if (am_partition)
{
Oid ancestor;
+ int level;
List *ancestors = get_partition_ancestors(relid);
ancestor = GetTopMostAncestorInPublication(pub->oid,
- ancestors);
+ ancestors,
+ &level);
if (ancestor != InvalidOid)
{
ancestor_published = true;
if (pub->pubviaroot)
- publish_as_relid = ancestor;
+ {
+ pub_relid = ancestor;
+ ancestor_level = level;
+ }
}
}
@@ -1868,6 +1891,20 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
rel_publications = lappend(rel_publications, pub);
+
+ /*
+ * We want to publish the changes as the top-most ancestor
+ * across all publications. So we need to check if the
+ * already calculated level is higher than the new one. If
+ * yes, we can ignore the new value (as it's a child).
+ * Otherwise the new value is an ancestor, so we keep it.
+ */
+ if (publish_ancestor_level > ancestor_level)
+ continue;
+
+ /* The new value is an ancestor, so let's keep it. */
+ publish_as_relid = pub_relid;
+ publish_ancestor_level = ancestor_level;
}
}
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index ba72e62e614..fe773cf9b7d 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -134,7 +134,8 @@ extern List *GetAllSchemaPublicationRelations(Oid puboid,
extern List *GetPubPartitionOptionRelations(List *result,
PublicationPartOpt pub_partopt,
Oid relid);
-extern Oid GetTopMostAncestorInPublication(Oid puboid, List *ancestors);
+extern Oid GetTopMostAncestorInPublication(Oid puboid, List *ancestors,
+ int *ancestor_level);
extern bool is_publishable_relation(Relation rel);
extern bool is_schema_publication(Oid pubid);
--
2.34.1
0002-testcase-for-publish-as-relid.patchapplication/octet-stream; name=0002-testcase-for-publish-as-relid.patchDownload
From 50d6044203ac3e96ad72a3ee98be7005fc67e6d9 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <houzj.fnst@cn.fujitsu.com>
Date: Mon, 14 Mar 2022 16:55:10 +0800
Subject: [PATCH] testcase for publish as relid
---
src/test/subscription/t/013_partition.pl | 28 +++++++++++++++++++++++++++-
1 file changed, 27 insertions(+), 1 deletion(-)
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 5266471a7..afd2c15 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -410,6 +410,12 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab3_1 PARTITION OF tab3 FOR VALUES IN (0, 1, 2, 3, 5, 6)");
$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab4 (a int PRIMARY KEY) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab4_1 PARTITION OF tab4 FOR VALUES IN (0, 1) PARTITION BY LIST (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab4_1_1 PARTITION OF tab4_1 FOR VALUES IN (0, 1)");
+$node_publisher->safe_psql('postgres',
"ALTER PUBLICATION pub_all SET (publish_via_partition_root = true)");
# Note: tab3_1's parent is not in the publication, in which case its
# changes are published using own identity. For tab2, even though both parent
@@ -418,6 +424,9 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub_viaroot FOR TABLE tab2, tab2_1, tab3_1 WITH (publish_via_partition_root = true)"
);
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub_lower_level FOR TABLE tab4_1 WITH (publish_via_partition_root = true)"
+);
# prepare data for the initial sync
$node_publisher->safe_psql('postgres', "INSERT INTO tab2 VALUES (1)");
@@ -462,10 +471,16 @@ $node_subscriber2->safe_psql('postgres',
$node_subscriber2->safe_psql('postgres',
"CREATE TABLE tab3_1 (a int PRIMARY KEY, c text DEFAULT 'sub2_tab3_1', b text)"
);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab4 (a int PRIMARY KEY)"
+);
+$node_subscriber2->safe_psql('postgres',
+ "CREATE TABLE tab4_1 (a int PRIMARY KEY)"
+);
# Publication that sub2 points to now publishes via root, so must update
# subscription target relations.
$node_subscriber2->safe_psql('postgres',
- "ALTER SUBSCRIPTION sub2 REFRESH PUBLICATION");
+ "ALTER SUBSCRIPTION sub2 ADD PUBLICATION pub_lower_level");
# Wait for initial sync of all subscriptions
$node_subscriber1->poll_query_until('postgres', $synced_query)
@@ -486,6 +501,8 @@ $node_publisher->safe_psql('postgres',
"INSERT INTO tab2 VALUES (0), (3), (5)");
$node_publisher->safe_psql('postgres',
"INSERT INTO tab3 VALUES (1), (0), (3), (5)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab4 VALUES (0), (1)");
$node_publisher->wait_for_catchup('sub_viaroot');
$node_publisher->wait_for_catchup('sub2');
@@ -525,6 +542,15 @@ sub2_tab3|1
sub2_tab3|3
sub2_tab3|5), 'inserts into tab3 replicated');
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a FROM tab4 ORDER BY 1");
+is( $result, qq(0
+1), 'inserts into tab4 replicated');
+
+$result = $node_subscriber2->safe_psql('postgres',
+ "SELECT a FROM tab4_1 ORDER BY 1");
+is( $result, qq(), 'inserts into tab4_1 replicated');
+
# update (replicated as update)
$node_publisher->safe_psql('postgres', "UPDATE tab1 SET a = 6 WHERE a = 5");
$node_publisher->safe_psql('postgres', "UPDATE tab2 SET a = 6 WHERE a = 5");
--
2.7.2.windows.1
0004-testcase-for-row-filter-publication.patchapplication/octet-stream; name=0004-testcase-for-row-filter-publication.patchDownload
From b030e0334d65da51d0bdff9201d0b3c7dab0b282 Mon Sep 17 00:00:00 2001
From: "houzj.fnst" <houzj.fnst@cn.fujitsu.com>
Date: Mon, 14 Mar 2022 15:54:19 +0800
Subject: [PATCH] fix row filter publication
---
src/test/subscription/t/028_row_filter.pl | 42 ++++++++++++++++++++++++++++++-
1 file changed, 41 insertions(+), 1 deletion(-)
diff --git a/src/test/subscription/t/028_row_filter.pl b/src/test/subscription/t/028_row_filter.pl
index 89bb364..9c20f0a 100644
--- a/src/test/subscription/t/028_row_filter.pl
+++ b/src/test/subscription/t/028_row_filter.pl
@@ -237,6 +237,11 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE TABLE tab_rowfilter_child (b text) INHERITS (tab_rowfilter_inherited)"
);
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_rowfilter_viaroot_part (a int) PARTITION BY RANGE (a)");
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE tab_rowfilter_viaroot_part_1 PARTITION OF tab_rowfilter_viaroot_part FOR VALUES FROM (1) TO (20)"
+);
# setup structure on subscriber
$node_subscriber->safe_psql('postgres',
@@ -283,6 +288,11 @@ $node_subscriber->safe_psql('postgres',
$node_subscriber->safe_psql('postgres',
"CREATE TABLE tab_rowfilter_child (b text) INHERITS (tab_rowfilter_inherited)"
);
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_rowfilter_viaroot_part (a int)");
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE tab_rowfilter_viaroot_part_1 (a int)"
+);
# setup logical replication
$node_publisher->safe_psql('postgres',
@@ -329,6 +339,12 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub_inherits FOR TABLE tab_rowfilter_inherited WHERE (a > 15)"
);
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_viaroot_1 FOR TABLE tab_rowfilter_viaroot_part WHERE (a > 15) WITH (publish_via_partition_root)"
+);
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub_viaroot_2 FOR TABLE tab_rowfilter_viaroot_part_1 WHERE (a < 15) WITH (publish_via_partition_root)"
+);
#
# The following INSERTs are executed before the CREATE SUBSCRIPTION, so these
@@ -376,7 +392,7 @@ $node_publisher->safe_psql('postgres',
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub_1, tap_pub_2, tap_pub_3, tap_pub_4a, tap_pub_4b, tap_pub_5a, tap_pub_5b, tap_pub_toast, tap_pub_inherits"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub_1, tap_pub_2, tap_pub_3, tap_pub_4a, tap_pub_4b, tap_pub_5a, tap_pub_5b, tap_pub_toast, tap_pub_inherits, tap_pub_viaroot_2, tap_pub_viaroot_1"
);
$node_publisher->wait_for_catchup($appname);
@@ -534,6 +550,8 @@ $node_publisher->safe_psql('postgres',
"INSERT INTO tab_rowfilter_inherited (a) VALUES (14), (16)");
$node_publisher->safe_psql('postgres',
"INSERT INTO tab_rowfilter_child (a, b) VALUES (13, '13'), (17, '17')");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO tab_rowfilter_viaroot_part (a) VALUES (14), (15), (16)");
$node_publisher->wait_for_catchup($appname);
@@ -688,6 +706,28 @@ $result =
"SELECT a = repeat('1234567890', 200), b FROM tab_rowfilter_toast");
is($result, qq(t|1), 'check replicated rows to tab_rowfilter_toast');
+# Check expected replicated rows for tab_rowfilter_viaroot_part and
+# tab_rowfilter_viaroot_part_1
+# tab_rowfilter_viaroot_part filter is: (a > 15)
+# - INSERT (14) NO, 14 < 15
+# - INSERT (15) NO, 15 = 15
+# - INSERT (16) YES, 16 > 15
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a FROM tab_rowfilter_viaroot_part");
+is( $result, qq(16),
+ 'check replicated rows to tab_rowfilter_viaroot_part'
+);
+
+# Check there is no data in tab_rowfilter_viaroot_part_1 because rows are
+# replicated via the top most parent table tab_rowfilter_viaroot_part
+$result =
+ $node_subscriber->safe_psql('postgres',
+ "SELECT a FROM tab_rowfilter_viaroot_part_1");
+is( $result, qq(),
+ 'check replicated rows to tab_rowfilter_viaroot_part_1'
+);
+
# Testcase end: FOR TABLE with row filter publications
# ======================================================
--
2.7.2.windows.1
On 3/14/22 10:53, Amit Kapila wrote:
On Mon, Mar 14, 2022 at 2:37 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 3/12/22 05:30, Amit Kapila wrote:
...
Okay, please find attached. I have done basic testing of this, if we
agree with this approach then this will require some more testing.Thanks, the proposed changes seem like a clear improvement, so I've
added them, with some minor tweaks (mostly to comments).One minor point: Did you intentionally remove
list_free(rel_publications) before resetting the list from the second
patch? The memory for rel_publications is allocated in
TopTransactionContext, so a large transaction touching many relations
will only free this at end of the transaction which may not be a big
deal as we don't do this every time. We free this list a few lines
down in successful case so this appears slightly odd to me but I am
fine if you think it doesn't matter.
The removal was not intentional, but I don't think it's an issue exactly
because it's a tiny mount of memory and we'll release it at the end of
the transaction. Which should not take long.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 3/14/22 12:12, houzj.fnst@fujitsu.com wrote:
On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
On 3/12/22 05:30, Amit Kapila wrote:
...
Okay, please find attached. I have done basic testing of this, if we
agree with this approach then this will require some more testing.Thanks, the proposed changes seem like a clear improvement, so I've
added them, with some minor tweaks (mostly to comments).Hi,
Thanks for updating the patches !
And sorry for the row filter bug caused by my mistake.I looked at the two fixup patches. I am thinking would it be better if we
add one testcase for these two bugs? Maybe like the attachment.
Yeah, a test would be nice - I'll take a look later.
Anyway, the fix does not address tablesync, as explained in [1]/messages/by-id/822a8e40-287c-59ff-0ea9-35eb759f4fe6@enterprisedb.com. I'm not
sure what to do about it - in principle, we could calculate which
relations to sync, and then eliminate "duplicates" (i.e. relations where
we are going to sync an ancestor).
regards
[1]: /messages/by-id/822a8e40-287c-59ff-0ea9-35eb759f4fe6@enterprisedb.com
/messages/by-id/822a8e40-287c-59ff-0ea9-35eb759f4fe6@enterprisedb.com
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Mon, Mar 14, 2022 at 5:42 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 3/14/22 12:12, houzj.fnst@fujitsu.com wrote:
On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
Anyway, the fix does not address tablesync, as explained in [1]. I'm not
sure what to do about it - in principle, we could calculate which
relations to sync, and then eliminate "duplicates" (i.e. relations where
we are going to sync an ancestor).
As mentioned in my previous email [1]/messages/by-id/CAA4eK1LSb-xrvGEm3ShaRA=Mkdii2d+4vqh9DGPvVDA+D9ibYw@mail.gmail.com, this appears to be a base code
issue (even without row filter or column filter work), so it seems
better to deal with it separately. It has been reported separately as
well [2]/messages/by-id/OS0PR01MB5716DC2982CC735FDE388804940B9@OS0PR01MB5716.jpnprd01.prod.outlook.com where we found some similar issues.
[1]: /messages/by-id/CAA4eK1LSb-xrvGEm3ShaRA=Mkdii2d+4vqh9DGPvVDA+D9ibYw@mail.gmail.com
[2]: /messages/by-id/OS0PR01MB5716DC2982CC735FDE388804940B9@OS0PR01MB5716.jpnprd01.prod.outlook.com
--
With Regards,
Amit Kapila.
On 3/14/22 13:47, Amit Kapila wrote:
On Mon, Mar 14, 2022 at 5:42 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 3/14/22 12:12, houzj.fnst@fujitsu.com wrote:
On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
Anyway, the fix does not address tablesync, as explained in [1]. I'm not
sure what to do about it - in principle, we could calculate which
relations to sync, and then eliminate "duplicates" (i.e. relations where
we are going to sync an ancestor).As mentioned in my previous email [1], this appears to be a base code
issue (even without row filter or column filter work), so it seems
better to deal with it separately. It has been reported separately as
well [2] where we found some similar issues.
Right. I don't want to be waiting for that fix either, that'd block this
patch unnecessarily. If there are no other comments, I'll go ahead,
polish the existing patches a bit more and get them committed. We can
worry about this pre-existing issue later.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Mon, Mar 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
On 3/12/22 05:30, Amit Kapila wrote:
...
Okay, please find attached. I have done basic testing of this, if we
agree with this approach then this will require some more testing.Thanks, the proposed changes seem like a clear improvement, so I've
added them, with some minor tweaks (mostly to comments).I've also included the memory context rename (entry_changes to the
change proposed by Wang Wei, using a single SQL command in tablesync.And I've renamed the per-entry memory context to entry_cxt, and used it
for the column list.
Thanks for your patch.
Here are some comments for column filter main patch (0003 patch).
1. doc/src/sgml/catalogs.sgml
@@ -6263,6 +6263,19 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to schema
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>prattrs</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+ </para>
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example, a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all columns are published.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
This change was added to pg_publication_namespace view. I think it should be
added to pg_publication_rel view, right?
2. src/backend/replication/pgoutput/pgoutput.c
@@ -188,6 +202,7 @@ static EState *create_estate_for_relation(Relation rel);
static void pgoutput_row_filter_init(PGOutputData *data,
List *publications,
RelationSyncEntry *entry);
+
static bool pgoutput_row_filter_exec_expr(ExprState *state,
ExprContext *econtext);
static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
Should we remove this change?
3. src/backend/commands/publicationcmds.c
+/*
+ * Check if all columns referenced in the column list are part of the
+ * REPLICA IDENTITY index or not.
+ *
+ * Returns true if any invalid column is found.
+ */
The comment for pub_collist_contains_invalid_column() seems wrong. Should it be
"Check if all REPLICA IDENTITY columns are covered by the column list or not"?
4.
The patch doesn't allow delete and update operations if the target table uses
replica identity full and it is published with column list specified, even if
column list includes all columns in the table.
For example:
create table tbl (a int, b int, c int);
create publication pub for table tbl (a, b, c);
alter table tbl replica identity full;
postgres=# delete from tbl;
ERROR: cannot delete from table "tbl"
DETAIL: Column list used by the publication does not cover the replica identity.
Should we allow this case? I think it doesn't seem to cause harm.
5.
Maybe we need some changes for tab-complete.c.
Regards,
Shi yu
On Mon, Mar 14, 2022 at 4:42 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
On 3/12/22 05:30, Amit Kapila wrote:
...
Okay, please find attached. I have done basic testing of this, if we
agree with this approach then this will require some more testing.Thanks, the proposed changes seem like a clear improvement, so I've
added them, with some minor tweaks (mostly to comments).Hi,
Thanks for updating the patches !
And sorry for the row filter bug caused by my mistake.I looked at the two fixup patches. I am thinking would it be better if we
add one testcase for these two bugs? Maybe like the attachment.
Your tests look good to me. We might want to add some comments for
each test but I guess that can be done before committing. Tomas, it
seems you are planning to push these bug fixes, do let me know if you
want me to take care of these while you focus on the main patch? I
think the first patch needs to be backpatched till 13 and the second
one is for just HEAD.
--
With Regards,
Amit Kapila.
On Mon, Mar 14, 2022 at 7:02 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 3/14/22 13:47, Amit Kapila wrote:
On Mon, Mar 14, 2022 at 5:42 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 3/14/22 12:12, houzj.fnst@fujitsu.com wrote:
On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
Anyway, the fix does not address tablesync, as explained in [1]. I'm not
sure what to do about it - in principle, we could calculate which
relations to sync, and then eliminate "duplicates" (i.e. relations where
we are going to sync an ancestor).As mentioned in my previous email [1], this appears to be a base code
issue (even without row filter or column filter work), so it seems
better to deal with it separately. It has been reported separately as
well [2] where we found some similar issues.Right. I don't want to be waiting for that fix either, that'd block this
patch unnecessarily. If there are no other comments, I'll go ahead,
polish the existing patches a bit more and get them committed. We can
worry about this pre-existing issue later.
I think the first two patches are ready to go. I haven't read the
latest version in detail but I have in mind that we want to get this
in for PG-15.
--
With Regards,
Amit Kapila.
On 3/15/22 05:43, Amit Kapila wrote:
On Mon, Mar 14, 2022 at 4:42 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
On 3/12/22 05:30, Amit Kapila wrote:
...
Okay, please find attached. I have done basic testing of this, if we
agree with this approach then this will require some more testing.Thanks, the proposed changes seem like a clear improvement, so I've
added them, with some minor tweaks (mostly to comments).Hi,
Thanks for updating the patches !
And sorry for the row filter bug caused by my mistake.I looked at the two fixup patches. I am thinking would it be better if we
add one testcase for these two bugs? Maybe like the attachment.Your tests look good to me. We might want to add some comments for
each test but I guess that can be done before committing. Tomas, it
seems you are planning to push these bug fixes, do let me know if you
want me to take care of these while you focus on the main patch? I
think the first patch needs to be backpatched till 13 and the second
one is for just HEAD.
Yeah, I plan to push the fixes later today. I'll polish them a bit
first, and merge the tests (shared by Hou zj) into the patches etc.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Tue, Mar 15, 2022 at 7:38 AM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
On Mon, Mar 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
3. src/backend/commands/publicationcmds.c +/* + * Check if all columns referenced in the column list are part of the + * REPLICA IDENTITY index or not. + * + * Returns true if any invalid column is found. + */The comment for pub_collist_contains_invalid_column() seems wrong. Should it be
"Check if all REPLICA IDENTITY columns are covered by the column list or not"?
On similar lines, I think errdetail for below messages need to be changed.
ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot update table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the
replica identity.")));
else if (cmd == CMD_DELETE && !pubdesc.rf_valid_for_delete)
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
errmsg("cannot delete from table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not
part of the replica identity.")));
+ else if (cmd == CMD_DELETE && !pubdesc.cols_valid_for_delete)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot delete from table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the
replica identity.")));
Some assorted comments:
========================
1. As mentioned previously as well[1]/messages/by-id/CAA4eK1K5pkrPT9z5TByUPptExian5c18g6GnfNf9Cr97QdPbjw@mail.gmail.com, the change in ATExecDropColumn
is not required. Similarly, the change you seem to agree upon in
logicalrep_write_update[2]/messages/by-id/43c15aa8-aa15-ca0f-40e4-3be68d98df05@enterprisedb.com doesn't seem to be present.
2. I think the dependency handling in publication_set_table_columns()
has problems. While removing existing dependencies, it uses
PublicationRelationId as classId whereas while adding new dependencies
it uses PublicationRelRelationId as classId. This will create problems
while removing columns from table. For example,
postgres=# create table t1(c1 int, c2 int, c3 int);
CREATE TABLE
postgres=# create publication pub1 for table t1(c1, c2);
CREATE PUBLICATION
postgres=# select * from pg_depend where classid = 6106 or refclassid
= 6106 or classid = 6104;
classid | objid | objsubid | refclassid | refobjid | refobjsubid | deptype
---------+-------+----------+------------+----------+-------------+---------
6106 | 16409 | 0 | 1259 | 16405 | 1 | a
6106 | 16409 | 0 | 1259 | 16405 | 2 | a
6106 | 16409 | 0 | 6104 | 16408 | 0 | a
6106 | 16409 | 0 | 1259 | 16405 | 0 | a
(4 rows)
Till here everything is fine.
postgres=# Alter publication pub1 alter table t1 set columns(c2);
ALTER PUBLICATION
postgres=# select * from pg_depend where classid = 6106 or refclassid
= 6106 or classid = 6104;
classid | objid | objsubid | refclassid | refobjid | refobjsubid | deptype
---------+-------+----------+------------+----------+-------------+---------
6106 | 16409 | 0 | 1259 | 16405 | 1 | a
6106 | 16409 | 0 | 1259 | 16405 | 2 | a
6106 | 16409 | 0 | 6104 | 16408 | 0 | a
6106 | 16409 | 0 | 1259 | 16405 | 0 | a
6106 | 16409 | 0 | 1259 | 16405 | 2 | a
(5 rows)
Now without removing dependencies for columns 1 and 2, it added a new
dependency for column 2.
3.
@@ -930,8 +1054,24 @@ copy_table(Relation rel)
...
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
+
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
...
...
for (int i = 0; i < lrel.natts; i++)
{
appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
if (i < lrel.natts - 1)
appendStringInfoString(&cmd, ", ");
}
In the same function, we use two different styles to achieve the same
thing. I think it is better to use the same style (probably existing)
at both places for the sake of consistency.
4.
+ <para>
+ The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows changing
+ the set of columns that are included in the publication. If a column list
+ is specified, it must include the replica identity columns.
+ </para>
I think the second part holds true only for update/delete publications.
5.
+ * XXX Should this detect duplicate columns?
+ */
+static void
+publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs)
{
...
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
...
}
It seems we already detect duplicate columns in this function. So XXX
part of the comment doesn't seem to be required.
6.
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow column lists on the leaf relations. So maybe rename it?
+ */
+static void
+TransformPubColumnList(List *tables, const char *queryString,
+ bool pubviaroot)
The second parameter is not used in this function. As noted in the
comments, I also think it is better to rename this. How about
ValidatePubColumnList?
7.
+ /*
+ * FIXME check pubactions vs. replica identity, to ensure the replica
+ * identity is included in the column list. Only do this for update
+ * and delete publications. See check_publication_columns.
+ *
+ * XXX This is needed because publish_via_partition_root may change,
+ * in which case the row filters may be invalid (e.g. with pvpr=false
+ * there must be no filter on partitioned tables).
+ */
+
This entire comment doesn't seem to be required.
8.
+publication_set_table_columns()
{
...
+ /* XXX "pub" is leaked here ??? */
...
}
It is not clear what this means?
9.
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ *
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
* pub_obj is one of:
*
* TABLE table_name [, ...]
@@ -9869,6 +9878,32 @@ AlterPublicationStmt:
n->action = AP_SetObjects;
$$ = (Node *)n;
}
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS '('
columnList ')'
The comments in gram.y indicates different rules than the actual implementation.
10.
+ *
+ * FIXME Do we need something similar for column filters?
*/
enum RowFilterPubAction
I have thought about this point and it seems we don't need anything on
this front for this patch. We need the filter combining of
update/delete for row filter because if inserts have some column which
is not present in RI then during update filtering it can give an error
as the column won't be present in WAL log.
Now, the same problem won't be there for the column list/filter patch
because all the RI columns are there in the column list (for
update/delete) and we don't need to apply a column filter for old
tuples in either update or delete.
We can remove this FIXME.
11.
+ } /* loop all subscribed publications */
+
+}
No need for an empty line here.
[1]: /messages/by-id/CAA4eK1K5pkrPT9z5TByUPptExian5c18g6GnfNf9Cr97QdPbjw@mail.gmail.com
[2]: /messages/by-id/43c15aa8-aa15-ca0f-40e4-3be68d98df05@enterprisedb.com
--
With Regards,
Amit Kapila.
On 3/15/22 09:30, Tomas Vondra wrote:
On 3/15/22 05:43, Amit Kapila wrote:
On Mon, Mar 14, 2022 at 4:42 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
On 3/12/22 05:30, Amit Kapila wrote:
...
Okay, please find attached. I have done basic testing of this, if we
agree with this approach then this will require some more testing.Thanks, the proposed changes seem like a clear improvement, so I've
added them, with some minor tweaks (mostly to comments).Hi,
Thanks for updating the patches !
And sorry for the row filter bug caused by my mistake.I looked at the two fixup patches. I am thinking would it be better if we
add one testcase for these two bugs? Maybe like the attachment.Your tests look good to me. We might want to add some comments for
each test but I guess that can be done before committing. Tomas, it
seems you are planning to push these bug fixes, do let me know if you
want me to take care of these while you focus on the main patch? I
think the first patch needs to be backpatched till 13 and the second
one is for just HEAD.Yeah, I plan to push the fixes later today. I'll polish them a bit
first, and merge the tests (shared by Hou zj) into the patches etc.
I've pushed (and backpatched to 13+) the fix for the publish_as_relid
issue, including the test. I tweaked the test a bit, to check both
orderings of the publication list.
While doing that, I discovered yet ANOTHER bug in the publish_as_relid
loop, affecting 12+13. There was a break once all actions were
replicated, but skipping additional publications ignores the fact that
the publications may replicate a different (higher-up) ancestor.
I removed the break, if anyone thinks this optimization is worth it we
could still do that once we replicate the top-most ancestor.
I'll push the second fix soon.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
I notice that the publication.sql regression tests contain a number of
comments like
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
but the error doesn't actually happen, because of the way the replica
identity checking was changed. This needs to be checked again.
On 3/17/22 15:17, Peter Eisentraut wrote:
I notice that the publication.sql regression tests contain a number of
comments like+-- error: replica identity "a" not included in the column list +ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);but the error doesn't actually happen, because of the way the replica
identity checking was changed. This needs to be checked again.
But the comment describes the error for the whole block, which looks
like this:
-- error: replica identity "a" not included in the column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica
identity.
So IMHO the comment is correct.
But there was one place where it wasn't entirely clear, as the block was
split by another comment. So I tweaked it to:
-- error: change the replica identity to "b", and column list to (a, c)
-- then update fails, because (a, c) does not cover replica identity
Attached is a rebased patch, on top of the two fixes I pushed.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
0001-Allow-specifying-column-filters-for-logical-20220317.patchtext/x-patch; charset=UTF-8; name=0001-Allow-specifying-column-filters-for-logical-20220317.patchDownload
From 0b46643432e3d5839d11b75c388f5cf42ec427ba Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Thu, 17 Mar 2022 19:16:39 +0100
Subject: [PATCH] Allow specifying column filters for logical replication
This allows specifying an optional column list when adding a table to
logical replication. Columns not included on this list are not sent to
the subscriber. The filter is specified as a list of columns after the
table name, enclosed in parentheses.
For UPDATE/DELETE publications, the column filter needs to cover all
REPLICA IDENTITY columns. For INSERT publications, the column list is
arbitrary and may omit some REPLICA IDENTITY columns. Furthermore, if
the table uses REPLICA IDENTITY FULL, column filter is not allowed.
The column filter can contain only simple column references. Complex
expressions, function calls etc. are not allowed. This restriction could
be relaxed in the future.
During the initial table synchronization, only columns specified in the
column filter are copied to the subscriber. If the subscription has
several publications, containing the same table with different column
filters, columns specified in any of the filters will be copied. This
means all columns are replicated if the table has no column filter at
all (which is treated as column filter with all columns), of when of the
publications is defined as FOR ALL TABLES (possibly IN SCHEMA for the
schema of the table).
For partitioned tables, publish_via_partition_root determines whether
the row filter for the root or leaf relation will be used. If the
parameter is 'false' (the default), the filter defined for the leaf
relation is used. Otherwise, the column filter for the root partition
will be used.
Psql commands \dRp+ and \d <table-name> now display any column filters.
Author: Tomas Vondra, Rahila Syed
Reviewed-by: Peter Eisentraut, Alvaro Herrera, Vignesh C, Ibrar Ahmed,
Amit Kapila, Hou zj, Peter Smith, Wang wei, Tang, Shi yu
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 15 +-
doc/src/sgml/protocol.sgml | 3 +-
doc/src/sgml/ref/alter_publication.sgml | 27 +-
doc/src/sgml/ref/create_publication.sgml | 17 +-
src/backend/catalog/pg_publication.c | 256 +++++
src/backend/commands/publicationcmds.c | 364 +++++-
src/backend/commands/tablecmds.c | 36 +-
src/backend/executor/execReplication.c | 19 +-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 60 +-
src/backend/replication/logical/proto.c | 56 +-
src/backend/replication/logical/tablesync.c | 156 ++-
src/backend/replication/pgoutput/pgoutput.c | 203 +++-
src/backend/utils/cache/relcache.c | 33 +-
src/bin/pg_dump/pg_dump.c | 47 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 60 +
src/bin/psql/describe.c | 40 +-
src/include/catalog/pg_publication.h | 13 +
src/include/catalog/pg_publication_rel.h | 1 +
src/include/commands/publicationcmds.h | 4 +-
src/include/nodes/parsenodes.h | 4 +-
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 375 +++++++
src/test/regress/sql/publication.sql | 290 +++++
src/test/subscription/t/029_column_list.pl | 1124 +++++++++++++++++++
27 files changed, 3117 insertions(+), 95 deletions(-)
create mode 100644 src/test/subscription/t/029_column_list.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 4dc5b34d21c..89827c373bd 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -4410,7 +4410,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para>
<para>
This is an array of <structfield>indnatts</structfield> values that
- indicate which table columns this index indexes. For example a value
+ indicate which table columns this index indexes. For example, a value
of <literal>1 3</literal> would mean that the first and the third table
columns make up the index entries. Key columns come before non-key
(included) columns. A zero in this array indicates that the
@@ -6281,6 +6281,19 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to schema
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>prattrs</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+ </para>
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example, a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all columns are published.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 9178c779ba9..fb491e9ebee 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7006,7 +7006,8 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column included in
+ the publication (except generated columns):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 32b75f6c78e..470d50a2447 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -25,12 +25,13 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD <replace
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP <replaceable class="parameter">publication_object</replaceable> [, ...]
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ALTER TABLE <replaceable class="parameter">table_name</replaceable> SET COLUMNS { ( <replaceable class="parameter">name</replaceable> [, ...] ) | ALL }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -64,6 +65,12 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
command retain their previous settings.
</para>
+ <para>
+ The <literal>ALTER TABLE ... SET COLUMNS</literal> variant allows changing
+ the set of columns that are included in the publication. If a column list
+ is specified, it must include the replica identity columns.
+ </para>
+
<para>
The remaining variants change the owner and the name of the publication.
</para>
@@ -112,6 +119,14 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ </para>
+
+ <para>
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
If the optional <literal>WHERE</literal> clause is specified, rows for
which the <replaceable class="parameter">expression</replaceable>
evaluates to false or null will not be published. Note that parentheses
@@ -172,9 +187,15 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
</para>
<para>
- Add some tables to the publication:
+ Add tables to the publication:
+<programlisting>
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ALTER TABLE users SET COLUMNS (user_id, firstname, lastname);
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 4979b9b646d..fb2d013393b 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -86,6 +86,13 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<literal>TRUNCATE</literal> commands.
</para>
+ <para>
+ When a column list is specified, only the named columns are replicated.
+ If no column list is specified, all columns of the table are replicated
+ through this publication, including any columns added later. If a column
+ list is specified, it must include the replica identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
@@ -327,6 +334,14 @@ CREATE PUBLICATION production_publication FOR TABLE users, departments, ALL TABL
<structname>sales</structname>:
<programlisting>
CREATE PUBLICATION sales_publication FOR ALL TABLES IN SCHEMA marketing, sales;
+</programlisting></para>
+
+ <para>
+ Create a publication that publishes all changes for table <structname>users</structname>,
+ but replicates only columns <structname>user_id</structname> and
+ <structname>firstname</structname>:
+<programlisting>
+CREATE PUBLICATION users_filtered FOR TABLE users (user_id, firstname);
</programlisting></para>
</refsect1>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 789b895db89..70a35a12a47 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,6 +45,9 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+static void publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs);
+
/*
* Check if relation can be in given publication and throws appropriate
* error if not.
@@ -345,6 +348,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
Oid relid = RelationGetRelid(targetrel);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ AttrNumber *attarray;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -372,6 +377,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
check_publication_add_relation(targetrel);
+ /*
+ * Translate column names to attnums and check the column list is valid.
+ * We also deconstruct the bitmap into an array of attnums, for storing
+ * in the catalog.
+ */
+ publication_translate_columns(pri->relation, pri->columns,
+ &natts, &attarray);
+
/* Form a tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -384,6 +397,17 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
values[Anum_pg_publication_rel_prrelid - 1] =
ObjectIdGetDatum(relid);
+ /* Add column list, if available */
+ if (pri->columns)
+ {
+ int2vector *prattrs;
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+ }
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+
/* Add qualifications, if available */
if (pri->whereClause != NULL)
values[Anum_pg_publication_rel_prqual - 1] = CStringGetTextDatum(nodeToString(pri->whereClause));
@@ -399,6 +423,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
/* Register dependencies as needed */
ObjectAddressSet(myself, PublicationRelRelationId, pubreloid);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ pfree(attarray);
+
/* Add dependency on the publication */
ObjectAddressSet(referenced, PublicationRelationId, pubid);
recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
@@ -432,6 +464,154 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
return myself;
}
+/*
+ * Update the column list for a relation in a publication.
+ */
+void
+publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns)
+{
+ AttrNumber *attarray;
+ HeapTuple copytup;
+ int natts;
+ bool nulls[Natts_pg_publication_rel];
+ bool replaces[Natts_pg_publication_rel];
+ Datum values[Natts_pg_publication_rel];
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ replaces[Anum_pg_publication_rel_prattrs - 1] = true;
+
+ deleteDependencyRecordsForClass(PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(pubreltup))->oid,
+ RelationRelationId,
+ DEPENDENCY_AUTO);
+
+ if (columns == NULL)
+ {
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+ }
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ int2vector *prattrs;
+ Form_pg_publication_rel pubrel;
+
+ publication_translate_columns(targetrel, columns, &natts, &attarray);
+
+ /* XXX "pub" is leaked here ??? */
+
+ prattrs = buildint2vector(attarray, natts);
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(prattrs);
+
+ /* Add dependencies on the new list of columns */
+ pubrel = (Form_pg_publication_rel) GETSTRUCT(pubreltup);
+ ObjectAddressSet(myself, PublicationRelRelationId, pubrel->oid);
+
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId,
+ RelationGetRelid(targetrel), attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+ }
+
+ copytup = heap_modify_tuple(pubreltup, RelationGetDescr(pubrel),
+ values, nulls, replaces);
+
+ CatalogTupleUpdate(pubrel, &pubreltup->t_self, copytup);
+
+ heap_freetuple(copytup);
+}
+
+/*
+ * qsort comparator for attnums
+ *
+ * XXX We already have compare_int16, so maybe let's share that, somehow?
+ */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list (no system or generated attributes,
+ * no duplicates). Additional checks with replica identity are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ *
+ * XXX Should this detect duplicate columns?
+ */
+static void
+publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs)
+{
+ AttrNumber *attarray;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference generated column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /* Be tidy, so that the catalog representation is always sorted */
+ qsort(attarray, n, sizeof(AttrNumber), compare_int16);
+
+ *natts = n;
+ *attrs = attarray;
+
+ bms_free(set);
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -539,6 +719,82 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+ Form_pg_publication_rel pubrel;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ /* no column list for this publications/relation */
+ if (isnull)
+ continue;
+
+ pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
+
+ pubs = lappend_oid(pubs, pubrel->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 1aad2e769cb..0fbcc6994b5 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -296,7 +296,7 @@ contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
* Returns true if any invalid column is found.
*/
bool
-contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
+pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
bool pubviaroot)
{
HeapTuple rftuple;
@@ -368,6 +368,123 @@ contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
return result;
}
+/*
+ * Check if all columns referenced in the column list are part of the
+ * REPLICA IDENTITY index or not.
+ *
+ * Returns true if any invalid column is found.
+ */
+bool
+pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
+ bool pubviaroot)
+{
+ HeapTuple tuple;
+ Oid relid = RelationGetRelid(relation);
+ Oid publish_as_relid = RelationGetRelid(relation);
+ bool result = false;
+ Datum datum;
+ bool isnull;
+
+ /*
+ * For a partition, if pubviaroot is true, find the topmost ancestor that
+ * is published via this publication as we need to use its column list
+ * for the changes.
+ *
+ * Note that even though the column list used is for an ancestor, the
+ * REPLICA IDENTITY used will be for the actual child table.
+ */
+ if (pubviaroot && relation->rd_rel->relispartition)
+ {
+ publish_as_relid = GetTopMostAncestorInPublication(pubid, ancestors, NULL);
+
+ if (!OidIsValid(publish_as_relid))
+ publish_as_relid = relid;
+ }
+
+ tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tuple))
+ return false;
+
+ datum = SysCacheGetAttr(PUBLICATIONRELMAP, tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ if (!isnull)
+ {
+ int x;
+ Bitmapset *idattrs;
+ Bitmapset *columns = NULL;
+
+ /* With REPLICA IDENTITY FULL, no column list is allowed. */
+ if (relation->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ result = true;
+
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(datum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ for (int i = 0; i < nelems; i++)
+ columns = bms_add_member(columns, elems[i]);
+ }
+
+ /* Remember columns that are part of the REPLICA IDENTITY */
+ idattrs = RelationGetIndexAttrBitmap(relation,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * Attnums in the bitmap returned by RelationGetIndexAttrBitmap are
+ * offset (to handle system columns the usual way), while column list
+ * does not use offset, so we can't do bms_is_subset(). Instead, we have
+ * to loop over the idattrs and check all of them are in the filter.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ AttrNumber attnum = (x + FirstLowInvalidHeapAttributeNumber);
+
+ /*
+ * If pubviaroot is true, we are validating the row filter of the
+ * parent table, but the bitmap contains the replica identity
+ * information of the child table. So, get the column number of the
+ * child table as parent and child column order could be different.
+ */
+ if (pubviaroot)
+ {
+ /* attnum is for child table */
+ char *colname = get_attname(relid, attnum, false);
+
+ /*
+ * Determine the matching attnum in parent (because the column
+ * filter is defined on the parent).
+ */
+ attnum = get_attnum(publish_as_relid, colname);
+ }
+
+ if (!bms_is_member(attnum, columns))
+ {
+ result = true;
+ break;
+ }
+ }
+
+ bms_free(idattrs);
+ bms_free(columns);
+ }
+
+ ReleaseSysCache(tuple);
+
+ return result;
+}
+
/* check_functions_in_node callback */
static bool
contain_mutable_or_user_functions_checker(Oid func_id, void *context)
@@ -609,6 +726,45 @@ TransformPubWhereClauses(List *tables, const char *queryString,
}
}
+
+/*
+ * Transform the publication column lists expression for all the relations
+ * in the list.
+ *
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow column lists on the leaf relations. So maybe rename it?
+ */
+static void
+TransformPubColumnList(List *tables, const char *queryString,
+ bool pubviaroot)
+{
+ ListCell *lc;
+
+ foreach(lc, tables)
+ {
+ PublicationRelInfo *pri = (PublicationRelInfo *) lfirst(lc);
+
+ if (pri->columns == NIL)
+ continue;
+
+ /*
+ * If the publication doesn't publish changes via the root partitioned
+ * table, the partition's column list will be used. So disallow using
+ * the column list on partitioned table in this case.
+ */
+ if (!pubviaroot &&
+ pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot use publication column list for relation \"%s\"",
+ RelationGetRelationName(pri->relation)),
+ errdetail("column list cannot be used for a partitioned table when %s is false.",
+ "publish_via_partition_root")));
+ }
+}
+
/*
* Create new publication.
*/
@@ -725,6 +881,9 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
TransformPubWhereClauses(rels, pstate->p_sourcetext,
publish_via_partition_root);
+ TransformPubColumnList(rels, pstate->p_sourcetext,
+ publish_via_partition_root);
+
PublicationAddTables(puboid, rels, true, NULL);
CloseTableList(rels);
}
@@ -755,6 +914,52 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
return myself;
}
+/*
+ * Change the column list of a relation in a publication
+ */
+static void
+PublicationSetColumns(AlterPublicationStmt *stmt,
+ Form_pg_publication pubform, PublicationTable *table)
+{
+ Relation rel,
+ urel;
+ HeapTuple tup;
+ ObjectAddress obj,
+ secondary;
+
+ rel = table_open(PublicationRelRelationId, RowExclusiveLock);
+ urel = table_openrv(table->relation, ShareUpdateExclusiveLock);
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(RelationGetRelid(urel)),
+ ObjectIdGetDatum(pubform->oid));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ errmsg("relation \"%s\" is not already in publication \"%s\"",
+ table->relation->relname,
+ NameStr(pubform->pubname)));
+
+ publication_set_table_columns(rel, tup, urel, table->columns);
+
+ ObjectAddressSet(obj, PublicationRelationId,
+ ((Form_pg_publication_rel) GETSTRUCT(tup))->oid);
+ ObjectAddressSet(secondary, RelationRelationId, RelationGetRelid(urel));
+ EventTriggerCollectSimpleCommand(obj, secondary, (Node *) stmt);
+
+ ReleaseSysCache(tup);
+
+ /*
+ * Invalidate relcache for this relation, to force rebuilding the
+ * publication description.
+ */
+ CacheInvalidateRelcache(urel);
+
+ table_close(rel, RowExclusiveLock);
+ table_close(urel, NoLock);
+
+ InvokeObjectPostAlterHook(PublicationRelationId, pubform->oid, 0);
+}
+
/*
* Change options of a publication.
*/
@@ -784,8 +989,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* If the publication doesn't publish changes via the root partitioned
- * table, the partition's row filter will be used. So disallow using WHERE
- * clause on partitioned table in this case.
+ * table, the partition's row filter and column list will be used. So disallow
+ * using WHERE clause and column lists on partitioned table in this case.
*/
if (!pubform->puballtables && publish_via_partition_root_given &&
!publish_via_partition_root)
@@ -793,7 +998,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* Lock the publication so nobody else can do anything with it. This
* prevents concurrent alter to add partitioned table(s) with WHERE
- * clause(s) which we don't allow when not publishing via root.
+ * clause(s) and/or column lists which we don't allow when not
+ * publishing via root.
*/
LockDatabaseObject(PublicationRelationId, pubform->oid, 0,
AccessShareLock);
@@ -805,13 +1011,21 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
HeapTuple rftuple;
Oid relid = lfirst_oid(lc);
+ bool has_column_list;
+ bool has_row_filter;
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubform->oid));
+ has_row_filter
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL);
+
+ has_column_list
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prattrs, NULL);
+
if (HeapTupleIsValid(rftuple) &&
- !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL))
+ (has_row_filter || has_column_list))
{
HeapTuple tuple;
@@ -820,7 +1034,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
Form_pg_class relform = (Form_pg_class) GETSTRUCT(tuple);
- if (relform->relkind == RELKIND_PARTITIONED_TABLE)
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_row_filter)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot set %s for publication \"%s\"",
@@ -831,6 +1046,18 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
NameStr(relform->relname),
"publish_via_partition_root")));
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_column_list)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot set %s for publication \"%s\"",
+ "publish_via_partition_root = false",
+ stmt->pubname),
+ errdetail("The publication contains a column list for a partitioned table \"%s\" "
+ "which is not allowed when %s is false.",
+ NameStr(relform->relname),
+ "publish_via_partition_root")));
+
ReleaseSysCache(tuple);
}
@@ -839,6 +1066,16 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
}
}
+ /*
+ * FIXME check pubactions vs. replica identity, to ensure the replica
+ * identity is included in the column list. Only do this for update
+ * and delete publications. See check_publication_columns.
+ *
+ * XXX This is needed because publish_via_partition_root may change,
+ * in which case the row filters may be invalid (e.g. with pvpr=false
+ * there must be no filter on partitioned tables).
+ */
+
/* Everything ok, form a new tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -976,10 +1213,20 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
PublicationAddTables(pubid, rels, false, stmt);
}
else if (stmt->action == AP_DropObjects)
PublicationDropTables(pubid, rels, false);
+ else if (stmt->action == AP_SetColumns)
+ {
+ Assert(schemaidlist == NIL);
+ Assert(list_length(tables) == 1);
+
+ PublicationSetColumns(stmt, pubform,
+ linitial_node(PublicationTable, tables));
+ }
else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
@@ -992,6 +1239,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
/*
* To recreate the relation list for the publication, look for
* existing relations that do not need to be dropped.
@@ -1003,32 +1252,85 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
PublicationRelInfo *oldrel;
bool found = false;
HeapTuple rftuple;
- bool rfisnull = true;
Node *oldrelwhereclause = NULL;
+ Bitmapset *oldcolumns = NULL;
/* look up the cache for the old relmap */
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(oldrelid),
ObjectIdGetDatum(pubid));
+ /*
+ * See if the existing relation currently has a WHERE clause or a
+ * column list. We need to compare those too.
+ */
if (HeapTupleIsValid(rftuple))
{
+ bool isnull = true;
Datum whereClauseDatum;
+ Datum columnListDatum;
+ /* Load the WHERE clause for this table. */
whereClauseDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
Anum_pg_publication_rel_prqual,
- &rfisnull);
- if (!rfisnull)
+ &isnull);
+ if (!isnull)
oldrelwhereclause = stringToNode(TextDatumGetCString(whereClauseDatum));
+ /* Transform the int2vector column list to a bitmap. */
+ columnListDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ /*
+ * XXX Maybe make this a separate function. We do this on
+ * multiple places.
+ */
+ if (!isnull)
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(columnListDatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* XXX is there a danger of memory leak here? beware */
+ for (int i = 0; i < nelems; i++)
+ oldcolumns = bms_add_member(oldcolumns, elems[i]);
+ }
+
ReleaseSysCache(rftuple);
}
foreach(newlc, rels)
{
PublicationRelInfo *newpubrel;
+ Oid newrelid;
+ Bitmapset *newcolumns = NULL;
newpubrel = (PublicationRelInfo *) lfirst(newlc);
+ newrelid = RelationGetRelid(newpubrel->relation);
+
+ /*
+ * If the new publication has column list, transform it to
+ * a bitmap too.
+ */
+ if (newpubrel->columns)
+ {
+ ListCell *lc;
+
+ foreach(lc, newpubrel->columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(newrelid, colname);
+
+ /* no checks needed here, that happens elsewhere */
+
+ newcolumns = bms_add_member(newcolumns, attnum);
+ }
+ }
/*
* Check if any of the new set of relations matches with the
@@ -1038,7 +1340,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
*/
if (RelationGetRelid(newpubrel->relation) == oldrelid)
{
- if (equal(oldrelwhereclause, newpubrel->whereClause))
+ if (equal(oldrelwhereclause, newpubrel->whereClause) &&
+ bms_equal(oldcolumns, newcolumns))
{
found = true;
break;
@@ -1057,6 +1360,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
{
oldrel = palloc(sizeof(PublicationRelInfo));
oldrel->whereClause = NULL;
+ oldrel->columns = NIL;
oldrel->relation = table_open(oldrelid,
ShareUpdateExclusiveLock);
delrels = lappend(delrels, oldrel);
@@ -1118,7 +1422,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
}
else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* AP_SetObjects */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -1141,6 +1445,10 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
*/
PublicationAddSchemas(pubform->oid, schemaidlist, true, stmt);
}
+ else
+ {
+ /* Nothing to do for AP_SetColumns */
+ }
}
/*
@@ -1403,6 +1711,7 @@ OpenTableList(List *tables)
List *rels = NIL;
ListCell *lc;
List *relids_with_rf = NIL;
+ List *relids_with_collist = NIL;
/*
* Open, share-lock, and check all the explicitly-specified relations
@@ -1437,6 +1746,13 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /* Disallow duplicate tables if there are any with column lists. */
+ if (t->columns || list_member_oid(relids_with_collist, myrelid))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
table_close(rel, ShareUpdateExclusiveLock);
continue;
}
@@ -1444,12 +1760,16 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
pub_rel->whereClause = t->whereClause;
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, myrelid);
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, myrelid);
+
/*
* Add children of this rel, if requested, so that they too are added
* to the publication. A partitioned table can't have any inheritance
@@ -1489,6 +1809,18 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /*
+ * We don't allow to specify column list for both parent
+ * and child table at the same time as it is not very
+ * clear which one should be given preference.
+ */
+ if (childrelid != myrelid &&
+ (t->columns || list_member_oid(relids_with_collist, childrelid)))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
continue;
}
@@ -1498,11 +1830,16 @@ OpenTableList(List *tables)
pub_rel->relation = rel;
/* child inherits WHERE clause from parent */
pub_rel->whereClause = t->whereClause;
+ /* child inherits column list from parent */
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, childrelid);
+
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, childrelid);
}
}
}
@@ -1611,6 +1948,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index dc5872f988c..a9fd0f0c895 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -8365,6 +8365,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
bool missing_ok, LOCKMODE lockmode,
ObjectAddresses *addrs)
{
+ Oid relid = RelationGetRelid(rel);
HeapTuple tuple;
Form_pg_attribute targetatt;
AttrNumber attnum;
@@ -8384,7 +8385,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/*
* get the number of the attribute
*/
- tuple = SearchSysCacheAttName(RelationGetRelid(rel), colName);
+ tuple = SearchSysCacheAttName(relid, colName);
if (!HeapTupleIsValid(tuple))
{
if (!missing_ok)
@@ -8438,13 +8439,42 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
ReleaseSysCache(tuple);
+ /*
+ * Also, if the column is used in the column list of a publication,
+ * disallow the drop if the DROP is RESTRICT. We don't do anything if the
+ * DROP is CASCADE, which means that the dependency mechanism will remove
+ * the relation from the publication.
+ */
+ if (behavior == DROP_RESTRICT)
+ {
+ List *pubs;
+ ListCell *lc;
+
+ pubs = GetRelationColumnPartialPublications(relid);
+ foreach(lc, pubs)
+ {
+ Oid pubid = lfirst_oid(lc);
+ List *published_cols;
+
+ published_cols =
+ GetRelationColumnListInPublication(relid, pubid);
+
+ if (list_member_oid(published_cols, attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot drop column \"%s\" because it is part of publication \"%s\"",
+ colName, get_publication_name(pubid, false)),
+ errhint("Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication."));
+ }
+ }
+
/*
* Propagate to children as appropriate. Unlike most other ALTER
* routines, we have to do this one level of recursion at a time; we can't
* use find_all_inheritors to do it in one pass.
*/
children =
- find_inheritance_children(RelationGetRelid(rel), lockmode);
+ find_inheritance_children(relid, lockmode);
if (children)
{
@@ -8532,7 +8562,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
/* Add object to delete */
object.classId = RelationRelationId;
- object.objectId = RelationGetRelid(rel);
+ object.objectId = relid;
object.objectSubId = attnum;
add_exact_object_address(&object, addrs);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 09f78f22441..3e282ed99ab 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -573,9 +573,6 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
if (cmd != CMD_UPDATE && cmd != CMD_DELETE)
return;
- if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
- return;
-
/*
* It is only safe to execute UPDATE/DELETE when all columns, referenced
* in the row filters from publications which the relation is in, are
@@ -595,17 +592,33 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
errmsg("cannot update table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_UPDATE && !pubdesc.cols_valid_for_update)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot update table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
else if (cmd == CMD_DELETE && !pubdesc.rf_valid_for_delete)
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
errmsg("cannot delete from table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_DELETE && !pubdesc.cols_valid_for_delete)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot delete from table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
/* If relation has replica identity we are always good. */
if (OidIsValid(RelationGetReplicaIndex(rel)))
return;
+ /* REPLICA IDENTITY FULL is also good for UPDATE/DELETE. */
+ if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ return;
+
/*
* This is UPDATE/DELETE and there is no replica identity.
*
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index d4f8455a2bd..a504437873f 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4850,6 +4850,7 @@ _copyPublicationTable(const PublicationTable *from)
COPY_NODE_FIELD(relation);
COPY_NODE_FIELD(whereClause);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f1002afe7a0..4fc16ce04e3 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2322,6 +2322,7 @@ _equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
COMPARE_NODE_FIELD(whereClause);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a03b33b53bd..25c9b29afdd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9751,13 +9751,14 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr OptWhereClause
+ TABLE relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9772,11 +9773,12 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId OptWhereClause
+ | ColId opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- if ($2)
+ /* FIXME this is a bit cumbersome */
+ if ($2 || $3)
{
/*
* The OptWhereClause must be stored here but it is
@@ -9786,7 +9788,8 @@ PublicationObjSpec:
*/
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVar(NULL, $1, @1);
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
else
{
@@ -9794,23 +9797,25 @@ PublicationObjSpec:
}
$$->location = @1;
}
- | ColId indirection OptWhereClause
+ | ColId indirection opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr OptWhereClause
+ | extended_relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
| CURRENT_SCHEMA
{
@@ -9836,6 +9841,10 @@ pub_obj_list: PublicationObjSpec
*
* ALTER PUBLICATION name SET pub_obj [, ...]
*
+ * ALTER PUBLICATION name SET COLUMNS table_name (column[, ...])
+ *
+ * ALTER PUBLICATION name SET COLUMNS table_name ALL
+ *
* pub_obj is one of:
*
* TABLE table_name [, ...]
@@ -9869,6 +9878,32 @@ AlterPublicationStmt:
n->action = AP_SetObjects;
$$ = (Node *)n;
}
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS '(' columnList ')'
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = $10;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
+ | ALTER PUBLICATION name ALTER TABLE relation_expr SET COLUMNS ALL
+ {
+ AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
+ PublicationObjSpec *obj = makeNode(PublicationObjSpec);
+ obj->pubobjtype = PUBLICATIONOBJ_TABLE;
+ obj->pubtable = makeNode(PublicationTable);
+ obj->pubtable->relation = $6;
+ obj->pubtable->columns = NIL;
+ n->pubname = $3;
+ n->pubobjects = list_make1(obj);
+ n->action = AP_SetColumns;
+ $$ = (Node *) n;
+ }
| ALTER PUBLICATION name DROP pub_obj_list
{
AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
@@ -17488,6 +17523,13 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
errmsg("WHERE clause not allowed for schema"),
parser_errposition(pubobj->location));
+ /* Column filter is not allowed on a schema object */
+ if (pubobj->pubtable && pubobj->pubtable->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schema"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index c9b0eeefd7e..816d461acd3 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,16 +29,25 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary);
+ bool binary, Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
static void logicalrep_write_namespace(StringInfo out, Oid nspid);
static const char *logicalrep_read_namespace(StringInfo in);
+
+static bool
+column_in_set(int attnum, Bitmapset *columns)
+{
+ return (columns == NULL || bms_is_member(attnum, columns));
+}
+
+
/*
* Write BEGIN to the output stream.
*/
@@ -398,7 +407,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +419,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -443,7 +452,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -464,11 +473,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -537,7 +546,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, NULL);
}
/*
@@ -652,7 +661,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -674,7 +684,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -751,7 +761,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum *values;
@@ -763,8 +773,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
continue;
+
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -783,6 +799,9 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +923,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -917,8 +936,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* send number of live attributes */
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
continue;
+
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -937,6 +962,9 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_set(att->attnum, columns))
+ continue;
+
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 1659964571c..5a28039023b 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -112,6 +112,7 @@
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "utils/acl.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -701,12 +702,13 @@ fetch_remote_table_info(char *nspname, char *relname,
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
int natt;
ListCell *lc;
bool first;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
@@ -747,10 +749,110 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+
+ /*
+ * Get column lists for each relation.
+ *
+ * For initial synchronization, column lists can be ignored in following
+ * cases:
+ *
+ * 1) one of the subscribed publications for the table hasn't specified
+ * any column list
+ *
+ * 2) one of the subscribed publications has puballtables set to true
+ *
+ * 3) one of the subscribed publications is declared as ALL TABLES IN
+ * SCHEMA that includes this relation
+ *
+ * We need to do this before fetching info about column names and types,
+ * so that we can skip columns that should not be replicated.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData pub_names;
+ bool first = true;
+
+ initStringInfo(&pub_names);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&pub_names, ", ");
+ appendStringInfoString(&pub_names, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ /*
+ * Fetch info about column filters for the relation (from all the
+ * publications). We unnest the int2vector values, because that
+ * makes it easier to combine filters by simply adding the attnums
+ * to a new bitmap (without having to parse the int2vector data).
+ * This preserves NULL values, so that if one of the publications
+ * has no column filter, we'll know that.
+ */
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT DISTINCT unnest"
+ " FROM pg_publication p"
+ " LEFT OUTER JOIN pg_publication_rel pr"
+ " ON (p.oid = pr.prpubid AND pr.prrelid = %u)"
+ " LEFT OUTER JOIN unnest(pr.prattrs) ON TRUE,"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt"
+ " WHERE gpt.relid = %u"
+ " AND p.pubname IN ( %s )",
+ lrel->remoteid,
+ lrel->remoteid,
+ pub_names.data);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch column list info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ /*
+ * Merge the column lists (from different publications) by creating
+ * a single bitmap with all the attnums. If we find a NULL value,
+ * that means one of the publications has no column filter for the
+ * table we're syncing.
+ */
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ Datum cfval = slot_getattr(slot, 1, &isnull);
+
+ /* NULL means empty column list, so we're done. */
+ if (isnull)
+ {
+ bms_free(included_cols);
+ included_cols = NULL;
+ break;
+ }
+
+ included_cols = bms_add_member(included_cols,
+ DatumGetInt16(cfval));
+
+ ExecClearTuple(slot);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+
+ walrcv_clear_result(pubres);
+
+ pfree(pub_names.data);
+ }
+
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -778,16 +880,35 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * Store the columns as a list of names. Ignore those that are not
+ * present in the column list, if there is one.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ /* If the column is not in the column list, skip it. */
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ {
+ ExecClearTuple(slot);
+ continue;
+ }
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -821,6 +942,9 @@ fetch_remote_table_info(char *nspname, char *relname,
*
* 3) one of the subscribed publications is declared as ALL TABLES IN
* SCHEMA that includes this relation
+ *
+ * XXX Does this actually handle puballtables and schema publications
+ * correctly?
*/
if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
{
@@ -930,8 +1054,24 @@ copy_table(Relation rel)
/* Regular table with no row filter */
if (lrel.relkind == RELKIND_RELATION && qual == NIL)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+
+ /*
+ * XXX Do we need to list the columns in all cases? Maybe we're replicating
+ * all columns?
+ */
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
+
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
+
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 5fddab3a3d4..dcb9723ad3c 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -29,6 +29,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -85,7 +86,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -93,6 +95,8 @@ static void send_repl_origin(LogicalDecodingContext *ctx,
/*
* Only 3 publication actions are used for row filtering ("insert", "update",
* "delete"). See RelationSyncEntry.exprstate[].
+ *
+ * FIXME Do we need something similar for column filters?
*/
enum RowFilterPubAction
{
@@ -143,9 +147,6 @@ typedef struct RelationSyncEntry
*/
ExprState *exprstate[NUM_ROWFILTER_PUBACTIONS];
EState *estate; /* executor state used for row filter */
- MemoryContext cache_expr_cxt; /* private context for exprstate and
- * estate, if any */
-
TupleTableSlot *new_slot; /* slot for storing new tuple */
TupleTableSlot *old_slot; /* slot for storing old tuple */
@@ -164,6 +165,19 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
AttrMap *attrmap;
+
+ /*
+ * Columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this bitmap are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
+
+ /*
+ * Private context to store additional data for this entry - state for
+ * the row filter expressions, column list, etc.
+ */
+ MemoryContext entry_cxt;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -188,6 +202,7 @@ static EState *create_estate_for_relation(Relation rel);
static void pgoutput_row_filter_init(PGOutputData *data,
List *publications,
RelationSyncEntry *entry);
+
static bool pgoutput_row_filter_exec_expr(ExprState *state,
ExprContext *econtext);
static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
@@ -195,6 +210,11 @@ static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
RelationSyncEntry *entry,
ReorderBufferChangeType *action);
+/* column filter routines */
+static void pgoutput_column_filter_init(PGOutputData *data,
+ List *publications,
+ RelationSyncEntry *entry);
+
/*
* Specify output plugin callbacks
*/
@@ -603,11 +623,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -620,7 +640,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -643,13 +664,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip this attribute if it's not present in the column list */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -823,21 +848,21 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
{
Relation relation = RelationIdGetRelation(entry->publish_as_relid);
- Assert(entry->cache_expr_cxt == NULL);
+ Assert(entry->entry_cxt == NULL);
- /* Create the memory context for row filters */
- entry->cache_expr_cxt = AllocSetContextCreate(data->cachectx,
- "Row filter expressions",
- ALLOCSET_DEFAULT_SIZES);
+ /* Create the memory context for entry data (row filters, ...) */
+ entry->entry_cxt = AllocSetContextCreate(data->cachectx,
+ "entry private context",
+ ALLOCSET_SMALL_SIZES);
- MemoryContextCopyAndSetIdentifier(entry->cache_expr_cxt,
+ MemoryContextCopyAndSetIdentifier(entry->entry_cxt,
RelationGetRelationName(relation));
/*
* Now all the filters for all pubactions are known. Combine them when
* their pubactions are the same.
*/
- oldctx = MemoryContextSwitchTo(entry->cache_expr_cxt);
+ oldctx = MemoryContextSwitchTo(entry->entry_cxt);
entry->estate = create_estate_for_relation(relation);
for (idx = 0; idx < NUM_ROWFILTER_PUBACTIONS; idx++)
{
@@ -860,6 +885,124 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
}
}
+/*
+ * Initialize the column filter.
+ */
+static void
+pgoutput_column_filter_init(PGOutputData *data, List *publications,
+ RelationSyncEntry *entry)
+{
+ ListCell *lc;
+ MemoryContext oldctx;
+
+ /*
+ * Find if there are any row filters for this relation. If there are, then
+ * prepare the necessary ExprState and cache it in entry->exprstate. To
+ * build an expression state, we need to ensure the following:
+ *
+ * All the given publication-table mappings must be checked.
+ *
+ * Multiple publications might have multiple row filters for this
+ * relation. Since row filter usage depends on the DML operation, there
+ * are multiple lists (one for each operation) to which row filters will
+ * be appended.
+ *
+ * FOR ALL TABLES implies "don't use row filter expression" so it takes
+ * precedence.
+ */
+ foreach(lc, publications)
+ {
+ Publication *pub = lfirst(lc);
+ HeapTuple cftuple = NULL;
+ Datum cfdatum = 0;
+ bool pub_no_filter = false;
+
+ if (pub->alltables)
+ {
+ /*
+ * If the publication is FOR ALL TABLES then it is treated the
+ * same as if this table has no row filters (even if for other
+ * publications it does).
+ */
+ pub_no_filter = true;
+ }
+ else
+ {
+ /*
+ * Check for the presence of a row filter in this publication.
+ */
+ cftuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(entry->publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(cftuple))
+ {
+ /* Null indicates no filter. */
+ cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
+ Anum_pg_publication_rel_prattrs,
+ &pub_no_filter);
+
+ /*
+ * When no column list is defined, so publish all columns.
+ * Otherwise merge the columns to the column list.
+ */
+ if (!pub_no_filter) /* when not null */
+ {
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+
+ arr = DatumGetArrayTypeP(cfdatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /*
+ * If the memory context for the entry does not exist yet,
+ * create it now. It may already exist if the relation has
+ * a row filter.
+ */
+ if (!entry->entry_cxt)
+ {
+ Relation relation = RelationIdGetRelation(entry->publish_as_relid);
+
+ entry->entry_cxt = AllocSetContextCreate(data->cachectx,
+ "entry private context",
+ ALLOCSET_SMALL_SIZES);
+
+ MemoryContextCopyAndSetIdentifier(entry->entry_cxt,
+ RelationGetRelationName(relation));
+ }
+
+ oldctx = MemoryContextSwitchTo(entry->entry_cxt);
+ for (int i = 0; i < nelems; i++)
+ entry->columns = bms_add_member(entry->columns,
+ elems[i]);
+ MemoryContextSwitchTo(oldctx);
+ }
+ }
+ else
+ {
+ pub_no_filter = true;
+ }
+ }
+
+ /* found publication with no filter, so we're done */
+ if (pub_no_filter)
+ {
+ if (cftuple)
+ ReleaseSysCache(cftuple);
+
+ bms_free(entry->columns);
+ entry->columns = NULL;
+
+ break;
+ }
+
+ ReleaseSysCache(cftuple);
+ } /* loop all subscribed publications */
+
+}
+
/*
* Initialize the slot for storing new and old tuples, and build the map that
* will be used to convert the relation's tuples into the ancestor's format.
@@ -1224,7 +1367,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -1278,11 +1421,13 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel,
- new_slot, data->binary);
+ new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel,
- old_slot, new_slot, data->binary);
+ old_slot, new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel,
@@ -1729,8 +1874,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->new_slot = NULL;
entry->old_slot = NULL;
memset(entry->exprstate, 0, sizeof(entry->exprstate));
- entry->cache_expr_cxt = NULL;
+ entry->entry_cxt = NULL;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->attrmap = NULL;
}
@@ -1776,6 +1922,8 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
entry->pubactions.pubinsert = false;
entry->pubactions.pubupdate = false;
entry->pubactions.pubdelete = false;
@@ -1799,17 +1947,18 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/*
* Row filter cache cleanups.
*/
- if (entry->cache_expr_cxt)
- MemoryContextDelete(entry->cache_expr_cxt);
+ if (entry->entry_cxt)
+ MemoryContextDelete(entry->entry_cxt);
- entry->cache_expr_cxt = NULL;
+ entry->entry_cxt = NULL;
entry->estate = NULL;
memset(entry->exprstate, 0, sizeof(entry->exprstate));
/*
* Build publication cache. We can't use one provided by relcache as
- * relcache considers all publications given relation is in, but here
- * we only need to consider ones that the subscriber requested.
+ * relcache considers all publications that the given relation is in,
+ * but here we only need to consider ones that the subscriber
+ * requested.
*/
foreach(lc, data->publications)
{
@@ -1878,6 +2027,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
}
/*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
* Don't publish changes for partitioned tables, because
* publishing those of its partitions suffices, unless partition
* changes won't be published due to pubviaroot being set.
@@ -1938,6 +2090,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/* Initialize the row filter */
pgoutput_row_filter_init(data, rel_publications, entry);
+
+ /* Initialize the column filter */
+ pgoutput_column_filter_init(data, rel_publications, entry);
}
list_free(pubids);
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index fccffce5729..a2da72f0d48 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -5553,6 +5553,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
return;
}
@@ -5565,6 +5567,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
/* Fetch the publication membership info. */
puboids = GetRelationPublications(relid);
@@ -5616,7 +5620,7 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
*/
if (!pubform->puballtables &&
(pubform->pubupdate || pubform->pubdelete) &&
- contain_invalid_rfcolumn(pubid, relation, ancestors,
+ pub_rf_contains_invalid_column(pubid, relation, ancestors,
pubform->pubviaroot))
{
if (pubform->pubupdate)
@@ -5625,6 +5629,23 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->rf_valid_for_delete = false;
}
+ /*
+ * Check if all columns are part of the REPLICA IDENTITY index or not.
+ *
+ * If the publication is FOR ALL TABLES then it means the table has no
+ * column list and we can skip the validation.
+ */
+ if (!pubform->puballtables &&
+ (pubform->pubupdate || pubform->pubdelete) &&
+ pub_collist_contains_invalid_column(pubid, relation, ancestors,
+ pubform->pubviaroot))
+ {
+ if (pubform->pubupdate)
+ pubdesc->cols_valid_for_update = false;
+ if (pubform->pubdelete)
+ pubdesc->cols_valid_for_delete = false;
+ }
+
ReleaseSysCache(tup);
/*
@@ -5636,6 +5657,16 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
!pubdesc->rf_valid_for_update && !pubdesc->rf_valid_for_delete)
break;
+
+ /*
+ * If we know everything is replicated and the column list is invalid
+ * for update and delete, there is no point to check for other
+ * publications.
+ */
+ if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
+ pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
+ !pubdesc->cols_valid_for_update && !pubdesc->cols_valid_for_delete)
+ break;
}
if (relation->rd_pubdesc)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 725cd2e4ebc..be40acd3e37 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4101,6 +4101,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_prpubid;
int i_prrelid;
int i_prrelqual;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4114,12 +4115,20 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
if (fout->remoteVersion >= 150000)
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual "
- "FROM pg_catalog.pg_publication_rel");
+ "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual, "
+ "(CASE\n"
+ " WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT array_agg(attname)\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) prattrs "
+ "FROM pg_catalog.pg_publication_rel pr");
else
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "NULL AS prrelqual "
+ "NULL AS prrelqual, NULL AS prattrs "
"FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
@@ -4130,6 +4139,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
i_prrelqual = PQfnumber(res, "prrelqual");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4175,6 +4185,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
else
pubrinfo[j].pubrelqual = pg_strdup(PQgetvalue(res, i, i_prrelqual));
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[j].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4249,10 +4281,13 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+
if (pubrinfo->pubrelqual)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 772dc0cf7a2..1d21c2906f1 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -632,6 +632,7 @@ typedef struct _PublicationRelInfo
PublicationInfo *publication;
TableInfo *pubtable;
char *pubrelqual;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index fd1052e5db8..05a7e28bdcc 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2428,6 +2428,28 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'ALTER PUBLICATION pub1 ADD TABLE test_sixth_table (col3, col2)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_sixth_table (col3, col2);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_sixth_table (col2, col3);\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'ALTER PUBLICATION pub1 ADD TABLE test_seventh_table (col3, col2) WHERE (col1 = 1)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_seventh_table (col3, col2) WHERE (col1 = 1);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_seventh_table (col2, col3) WHERE ((col1 = 1));\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'ALTER PUBLICATION pub3 ADD ALL TABLES IN SCHEMA dump_test' => {
create_order => 51,
create_sql =>
@@ -2778,6 +2800,44 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'CREATE TABLE test_sixth_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_sixth_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_sixth_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'CREATE TABLE test_seventh_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_seventh_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_seventh_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'CREATE TABLE test_table_identity' => {
create_order => 3,
create_sql => 'CREATE TABLE dump_test.test_table_identity (
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 991bfc1546b..88bb75ac658 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2892,6 +2892,7 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_namespace pn ON p.oid = pn.pnpubid\n"
" JOIN pg_catalog.pg_class pc ON pc.relnamespace = pn.pnnspid\n"
@@ -2899,6 +2900,12 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , pg_get_expr(pr.prqual, c.oid)\n"
+ " , (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT string_agg(attname, ', ')\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) "
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
" JOIN pg_catalog.pg_class c ON c.oid = pr.prrelid\n"
@@ -2906,6 +2913,7 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2916,12 +2924,14 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
"WHERE pr.prrelid = '%s'\n"
"UNION ALL\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2943,6 +2953,11 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, " \"%s\"",
PQgetvalue(result, i, 0));
+ /* column list (if any) */
+ if (!PQgetisnull(result, i, 2))
+ appendPQExpBuffer(&buf, " (%s)",
+ PQgetvalue(result, i, 2));
+
/* row filter (if any) */
if (!PQgetisnull(result, i, 1))
appendPQExpBuffer(&buf, " WHERE %s",
@@ -5888,7 +5903,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5905,15 +5920,19 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (as_schema)
+ printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
+ else
{
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
+
+ if (!PQgetisnull(res, i, 3))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 3));
+
if (!PQgetisnull(res, i, 2))
appendPQExpBuffer(buf, " WHERE %s", PQgetvalue(res, i, 2));
}
- else
- printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
}
@@ -6042,11 +6061,22 @@ describePublications(const char *pattern)
printfPQExpBuffer(&buf,
"SELECT n.nspname, c.relname");
if (pset.sversion >= 150000)
+ {
appendPQExpBufferStr(&buf,
", pg_get_expr(pr.prqual, c.oid)");
+ appendPQExpBufferStr(&buf,
+ ", (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string("
+ " ARRAY(SELECT attname\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END)");
+ }
else
appendPQExpBufferStr(&buf,
- ", NULL");
+ ", NULL, NULL");
appendPQExpBuffer(&buf,
"\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index fe773cf9b7d..70e053e04f1 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -85,6 +85,13 @@ typedef struct PublicationDesc
*/
bool rf_valid_for_update;
bool rf_valid_for_delete;
+
+ /*
+ * true if the columns are part of the replica identity or the publication actions
+ * do not include UPDATE or DELETE.
+ */
+ bool cols_valid_for_update;
+ bool cols_valid_for_delete;
} PublicationDesc;
typedef struct Publication
@@ -100,6 +107,7 @@ typedef struct PublicationRelInfo
{
Relation relation;
Node *whereClause;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -123,8 +131,11 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
+extern void GetActionsInPublication(Oid pubid, PublicationActions *actions);
extern List *GetPublicationSchemas(Oid pubid);
extern List *GetSchemaPublications(Oid schemaid);
extern List *GetSchemaPublicationRelations(Oid schemaid,
@@ -143,6 +154,8 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *pri
bool if_not_exists);
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern void publication_set_table_columns(Relation pubrel, HeapTuple pubreltup,
+ Relation targetrel, List *columns);
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index 0dd0f425db9..4feb581899e 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -34,6 +34,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
#ifdef CATALOG_VARLEN /* variable-length fields start here */
pg_node_tree prqual; /* qualifications */
+ int2vector prattrs; /* columns to replicate */
#endif
} FormData_pg_publication_rel;
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index 7813cbcb6bb..ae87caf089d 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -31,7 +31,9 @@ extern void RemovePublicationSchemaById(Oid psoid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
extern void AlterPublicationOwner_oid(Oid pubid, Oid newOwnerId);
extern void InvalidatePublicationRels(List *relids);
-extern bool contain_invalid_rfcolumn(Oid pubid, Relation relation,
+extern bool pub_rf_contains_invalid_column(Oid pubid, Relation relation,
+ List *ancestors, bool pubviaroot);
+extern bool pub_collist_contains_invalid_column(Oid pubid, Relation relation,
List *ancestors, bool pubviaroot);
#endif /* PUBLICATIONCMDS_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1617702d9d6..1375a173e3b 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3652,6 +3652,7 @@ typedef struct PublicationTable
NodeTag type;
RangeVar *relation; /* relation to be published */
Node *whereClause; /* qualifications */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
@@ -3688,7 +3689,8 @@ typedef enum AlterPublicationAction
{
AP_AddObjects, /* add objects to publication */
AP_DropObjects, /* remove objects from publication */
- AP_SetObjects /* set list of objects */
+ AP_SetObjects, /* set list of objects */
+ AP_SetColumns /* change list of columns for a table */
} AlterPublicationAction;
typedef struct AlterPublicationStmt
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4d2c881644a..a771ab8ff33 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -209,12 +209,12 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary);
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -231,7 +231,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 4e191c120ac..4ec3ac5b17f 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -613,6 +613,372 @@ DROP TABLE rf_tbl_abcd_pk;
DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+RESET client_min_messages;
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ERROR: cannot reference generated column "d" in publication column list
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+ERROR: cannot reference system column "ctid" in publication column list
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+ERROR: cannot drop column "c" because it is part of publication "testpub_fortable"
+HINT: Specify CASCADE or use ALTER PUBLICATION to remove the column from the publication.
+-- ok: for insert-only publication, the column list is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: change the replica identity to "b", and column list to (a, c)
+-- then update fails, because (a, c) does not cover replica identity
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+ Publication testpub_table_ins
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | t | f
+Tables:
+ "public.testpub_tbl5" (a)
+
+-- with REPLICA IDENTITY FULL, column lists are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ERROR: cannot update table "testpub_tbl6"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+ALTER PUBLICATION testpub_fortable ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ERROR: cannot update table "testpub_tbl6"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- make sure changing the column list is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: update the column list
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, c)
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl8;
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_0"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+ Publication testpub_both_filters
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | t | t | t | f
+Tables:
+ "public.testpub_tbl_both_filters" (a, c) WHERE (c <> 1)
+
+\d+ testpub_tbl_both_filters
+ Table "public.testpub_tbl_both_filters"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+ c | integer | | not null | | plain | |
+Indexes:
+ "testpub_tbl_both_filters_pkey" PRIMARY KEY, btree (a, c) REPLICA IDENTITY
+Publications:
+ "testpub_both_filters" (a, c) WHERE (c <> 1)
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+-- Tests for partitioned table
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+ERROR: cannot use publication column list for relation "rf_tbl_abcd_part_pk"
+DETAIL: column list cannot be used for a partitioned table when publish_via_partition_root is false.
+-- ok - can use row filter for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any row filter is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+ERROR: cannot set publish_via_partition_root = false for publication "testpub6"
+DETAIL: The publication contains a column list for a partitioned table "rf_tbl_abcd_part_pk" which is not allowed when publish_via_partition_root is false.
+-- Now change the root filter to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have row filter for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -1058,6 +1424,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schema
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 5457c56b33f..b39a4953632 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -373,6 +373,292 @@ DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+RESET client_min_messages;
+
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+-- ok: for insert-only publication, the column list is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+
+-- error: change the replica identity to "b", and column list to (a, c)
+-- then update fails, because (a, c) does not cover replica identity
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+UPDATE testpub_tbl5 SET a = 1;
+
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+
+-- with REPLICA IDENTITY FULL, column lists are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+
+ALTER PUBLICATION testpub_fortable ALTER TABLE testpub_tbl6 SET COLUMNS (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+
+-- make sure changing the column list is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: update the column list
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl8;
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+\d+ testpub_tbl_both_filters
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Tests for partitioned table
+
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - can use row filter for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test row filter for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any row filter is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- Now change the root filter to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have row filter for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use row filter for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
+
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -614,6 +900,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/029_column_list.pl b/src/test/subscription/t/029_column_list.pl
new file mode 100644
index 00000000000..5266967b3f4
--- /dev/null
+++ b/src/test/subscription/t/029_column_list.pl
@@ -0,0 +1,1124 @@
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test partial-column publication of tables
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# create publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+sub wait_for_subscription_sync
+{
+ my ($node) = @_;
+
+ # Also wait for initial table sync to finish
+ my $synced_query = "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+
+ $node->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+}
+
+# setup tables on both nodes
+
+# tab1: simple 1:1 replication
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+# tab2: replication from regular to table with fewer columns
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)
+));
+
+# tab3: simple 1:1 replication with weird column names
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "B" varchar, "c'" int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "c'" int)
+));
+
+# test_part: partitioned tables, with partitioning (including multi-level
+# partitioning, and fewer columns on the subscriber)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+# tab4: table with user-defined enum types
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, d text);
+));
+
+
+# TEST: create publication and subscription for some of the tables with
+# column lists
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub1
+ FOR TABLE tab1 (a, "B"), tab3 ("a'", "c'"), test_part (a, b), tab4 (a, b, d)
+ WITH (publish_via_partition_root = 'true');
+));
+
+# check that we got the right prattrs values for the publication in the
+# pg_publication_rel catalog (order by relname, to get stable ordering)
+my $result = $node_publisher->safe_psql('postgres', qq(
+ SELECT relname, prattrs
+ FROM pg_publication_rel pb JOIN pg_class pc ON(pb.prrelid = pc.oid)
+ ORDER BY relname
+));
+
+is($result, qq(tab1|1 2
+tab3|1 3
+tab4|1 2 4
+test_part|1 2), 'publication relation updated');
+
+# TEST: insert data into the tables, create subscription and see if sync
+# replicates the right columns
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (1, 2, 3);
+ INSERT INTO tab1 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (1, 2, 3);
+ INSERT INTO tab3 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (1, 'red', 3, 'oh my');
+ INSERT INTO tab4 VALUES (2, 'blue', 4, 'hello');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (1, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (2, 'bcd', '2021-07-03 11:12:13');
+ INSERT INTO test_part VALUES (7, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (8, 'bcd', '2021-07-03 11:12:13');
+));
+
+# create subscription for the publication, wait for sync to complete,
+# then check the sync results
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+4|5|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+4|6), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+7|abc
+8|bcd), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: now insert more data into the tables, and wait until we replicate
+# them (not by tablesync, but regular decoding and replication)
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (2, 3, 4);
+ INSERT INTO tab1 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (2, 3, 4);
+ INSERT INTO tab3 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (3, 'red', 5, 'foo');
+ INSERT INTO tab4 VALUES (4, 'blue', 6, 'bar');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (3, 'xxx', '2022-02-01 10:00:00');
+ INSERT INTO test_part VALUES (4, 'yyy', '2022-03-02 15:12:13');
+ INSERT INTO test_part VALUES (9, 'zzz', '2022-04-03 21:00:00');
+ INSERT INTO test_part VALUES (10, 'qqq', '2022-05-04 22:12:13');
+));
+
+# wait for catchup before checking the subscriber
+$node_publisher->wait_for_catchup('sub1');
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+2|3|
+4|5|
+5|6|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+2|4
+4|6
+5|7), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello
+3|red|foo
+4|blue|bar), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+3|xxx
+4|yyy
+7|abc
+8|bcd
+9|zzz
+10|qqq), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: do some updates on some of the tables, both on columns included
+# in the column list and other
+
+# tab1: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET "B" = 2 * "B" where a = 1));
+
+# tab1: update of non-replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET c = 2*c where a = 4));
+
+# tab3: update of non-replicated
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "B" = "B" || ' updated' where "a'" = 4));
+
+# tab3: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "c'" = 2 * "c'" where "a'" = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'blue', c = c * 2, d = d || ' updated' where a = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'red', c = c * 2, d = d || ' updated' where a = 2));
+
+# wait for the replication to catch up, and check the UPDATE results got
+# replicated correctly, with the right column list
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab1 ORDER BY a));
+is($result,
+qq(1|4|
+2|3|
+4|5|
+5|6|), 'only update on column tab1.b is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result,
+qq(1|6
+2|4
+4|6
+5|7), 'only update on column tab3.c is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab4 ORDER BY a));
+
+is($result, qq(1|blue|oh my updated
+2|red|hello updated
+3|red|foo
+4|blue|bar), 'update on column tab4.c is not replicated');
+
+
+# TEST: add table with a column list, insert data, replicate
+
+# insert some data before adding it to the publication
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (1, 'abc', 3);
+));
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2 (a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION");
+
+# wait for the tablesync to complete, add a bit more data and then check
+# the results of the replication
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (2, 'def', 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|def), 'insert on column tab2.c is not replicated');
+
+# do a couple updates, check the correct stuff gets replicated
+$node_publisher->safe_psql('postgres', qq(
+ UPDATE tab2 SET c = 5 where a = 1;
+ UPDATE tab2 SET b = 'xyz' where a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|xyz), 'update on column tab2.c is not replicated');
+
+
+# TEST: add a table to two publications with different column lists, and
+# create a single subscription replicating both publications
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+
+ -- insert a couple initial records
+ INSERT INTO tab5 VALUES (1, 11, 111, 1111);
+ INSERT INTO tab5 VALUES (2, 22, 222, 2222);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub3
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->wait_for_catchup('sub1');
+
+# insert data and make sure all the columns (union of the columns lists)
+# get fully replicated
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (3, 33, 333, 3333);
+ INSERT INTO tab5 VALUES (4, 44, 444, 4444);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111
+2|22|2222
+3|33|3333
+4|44|4444),
+ 'overlapping publications with overlapping column lists');
+
+# and finally, set the column list to ALL for one of the publications,
+# which means replicating all columns (removing the column list), but
+# first add the missing column to the table on subscriber
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub3 ALTER TABLE tab5 SET COLUMNS ALL;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ ALTER TABLE tab5 ADD COLUMN c INT;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (5, 55, 555, 5555);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111|
+2|22|2222|
+3|33|3333|
+4|44|4444|
+5|55|5555|555),
+ 'overlapping publications with overlapping column lists');
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key (but use a different column in
+# the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub4 FOR TABLE tab6 (a, b);
+
+ -- initial data
+ INSERT INTO tab6 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (2, 33, 444, 5555);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is still covered by the column list, though)
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+# we need to do the same thing on the subscriber
+# XXX What would happen if this happens before the publisher ALTER? Or
+# interleaved, somehow? But that seems unrelated to column lists.
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (3, 55, 666, 8888);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key with a key on multiple columns
+# (all of them covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub5 FOR TABLE tab7 (a, b);
+
+ -- some initial data
+ INSERT INTO tab7 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub5
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (2, 33, 444, 5555);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is not covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ ALTER TABLE tab7 ADD PRIMARY KEY (a, b);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (3, 55, 666, 7777);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+# now switch the primary key again to another columns not covered by the
+# column list, but also generate writes between the drop and creation
+# of the new constraint
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ INSERT INTO tab7 VALUES (4, 77, 888, 9999);
+ -- update/delete is not allowed for tables without RI
+ ALTER TABLE tab7 ADD PRIMARY KEY (b, a);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+ DELETE FROM tab7 WHERE a = 1;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(2|264||
+3|220||
+4|154||),
+ 'replication with the modified primary key');
+
+
+# TEST: partitioned tables (with publish_via_partition_root = false)
+# and replica identity. The (leaf) partitions may have different RI, so
+# we need to check the partition RI (with respect to the column list)
+# while attaching the partition.
+
+# First, let's create a partitioned table with two partitions, each with
+# a different RI, but a column list not covering all those RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+
+ -- initial data, one row in each partition
+ INSERT INTO test_part_a VALUES (1, 3);
+ INSERT INTO test_part_a VALUES (6, 4);
+));
+
+# do the same thing on the subscriber (with the opposite column order)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (b int, a int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+));
+
+# create a publication replicating just the column "a", which is not enough
+# for the second partition
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub6 FOR TABLE test_part_a (b, a) WITH (publish_via_partition_root = true);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_1 (a);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_2 (b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub6
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_a VALUES (2, 5);
+ INSERT INTO test_part_a VALUES (7, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT a, b FROM test_part_a ORDER BY a, b"),
+ qq(1|3
+2|5
+6|4
+7|6),
+ 'partitions with different replica identities not replicated correctly');
+
+# This time start with a column list covering RI for all partitions, but
+# then update the column list to not cover column "b" (needed by the
+# second partition)
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+
+ -- initial data, one row in each partitions
+ INSERT INTO test_part_b VALUES (1, 1);
+ INSERT INTO test_part_b VALUES (6, 2);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub7 FOR TABLE test_part_b (a, b) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub7
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_b VALUES (2, 3);
+ INSERT INTO test_part_b VALUES (7, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_b ORDER BY a, b"),
+ qq(1|1
+2|3
+6|2
+7|4),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: This time start with a column list covering RI for all partitions,
+# but then update RI for one of the partitions to not be covered by the
+# column list anymore.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+
+ -- initial data, one row for each partition
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+));
+
+# create a publication replicating data through partition root, with a column
+# filter on the root, and then add the partitions one by one with separate
+# column lists (but those are not applied)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a,c);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SUBSCRIPTION sub1;
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub8;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_c VALUES (3, 7, 8);
+ INSERT INTO test_part_c VALUES (4, 9, 10);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||5
+2|4|
+3||8
+4|9|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# create a publication not replicating data through partition root, without
+# a column list on the root, and then add the partitions one by one with
+# separate column lists
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub8;
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ TRUNCATE test_part_c;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ TRUNCATE test_part_c;
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||
+2|4|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: Start with a single partition, with RI compatible with the column
+# filter, and then attach a partition with incompatible RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ INSERT INTO test_part_d VALUES (1, 2);
+));
+
+# do the same thing on the subscriber (in fact, create both partitions right
+# away, no need to delay that)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ CREATE TABLE test_part_d_2 PARTITION OF test_part_d FOR VALUES IN (2,4);
+ ALTER TABLE test_part_d_2 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_2 REPLICA IDENTITY USING INDEX test_part_d_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub9 FOR TABLE test_part_d (a) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub9
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_d VALUES (3, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_d ORDER BY a, b"),
+ qq(1|
+3|),
+ 'partitions with different replica identities not replicated correctly');
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. So with column lists (a,b) and (a,c) we
+# should replicate (a,b,c).
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+
+ -- initial data
+ INSERT INTO test_mix_1 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_1 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_1 ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES, we should replicate all columns.
+
+# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ test_part, test_part_a, test_part_b, test_part_c, test_part_d;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
+
+ -- initial data
+ INSERT INTO test_mix_2 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_3, pub_mix_4;
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_2 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_2"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES IN SCHEMA, we should replicate all columns.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
+
+ -- initial data
+ INSERT INTO test_mix_3 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_5, pub_mix_6;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_3 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_3"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: Check handling of publish_via_partition_root - if a partition is
+# published through partition root, we should only apply the column list
+# defined for the whole table (not the partitions) - both during the initial
+# sync and when replicating changes. This is what we do for row filters.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+
+ CREATE PUBLICATION pub_root_true FOR TABLE test_root (a) WITH (publish_via_partition_root = true);
+
+ -- initial data
+ INSERT INTO test_root VALUES (1, 2, 3);
+ INSERT INTO test_root VALUES (10, 20, 30);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_root_true;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_root VALUES (2, 3, 4);
+ INSERT INTO test_root VALUES (11, 21, 31);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_root ORDER BY a, b, c"),
+ qq(1||
+2||
+10||
+11||),
+ 'publication via partition root applies column list');
+
+
+# TEST: Multiple publications which publish schema of parent table and
+# partition. The partition is published through two publications, once
+# through a schema (so no column list) containing the parent, and then
+# also directly (with a columns list). The expected outcome is there is
+# no column list.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub1, pub2, pub3, pub4, pub5, pub6, pub7, pub8;
+
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA s1;
+ CREATE PUBLICATION pub2 FOR TABLE t_1(b);
+
+ -- initial data
+ INSERT INTO s1.t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub1, pub2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'two publications, publishing the same relation');
+
+# Now resync the subcription, but with publications in the opposite order.
+# The result should be the same.
+
+$node_subscriber->safe_psql('postgres', qq(
+ TRUNCATE s1.t;
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub1;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (7, 8, 9);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(7|8|9),
+ 'two publications, publishing the same relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub3 FOR TABLE t_1 (a), t_2
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub3;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+# Note: The difference from the preceding test is that in this case both
+# relations have a column list defined.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub4 FOR TABLE t_1 (a), t_2 (b)
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
+
+done_testing();
--
2.34.1
I pushed the second fix. Interestingly enough, wrasse failed in the
013_partition test. I don't see how that could be caused by this
particular commit, though - see the pgsql-committers thread [1]/messages/by-id/E1nUsch-0008rQ-FW@gemulon.postgresql.org.
I'd like to test & polish the main patch over the weekend, and get it
committed early next week. Unless someone thinks it's definitely not
ready for that ...
[1]: /messages/by-id/E1nUsch-0008rQ-FW@gemulon.postgresql.org
/messages/by-id/E1nUsch-0008rQ-FW@gemulon.postgresql.org
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Fri, Mar 18, 2022 at 12:47 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
I pushed the second fix. Interestingly enough, wrasse failed in the
013_partition test. I don't see how that could be caused by this
particular commit, though - see the pgsql-committers thread [1].
I have a theory about what's going on here. I think this is due to a
test added in your previous commit c91f71b9dc. The newly added test
added hangs in tablesync because there was no apply worker to set the
state to SUBREL_STATE_CATCHUP which blocked tablesync workers from
proceeding.
See below logs from pogona [1]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=pogona&dt=2022-03-17%2023%3A10%3A04.
2022-03-18 01:33:15.190 CET [2551176][client
backend][3/74:0][013_partition.pl] LOG: statement: ALTER SUBSCRIPTION
sub2 SET PUBLICATION pub_lower_level, pub_all
2022-03-18 01:33:15.354 CET [2551193][logical replication
worker][4/57:0][] LOG: logical replication apply worker for
subscription "sub2" has started
2022-03-18 01:33:15.605 CET [2551176][client
backend][:0][013_partition.pl] LOG: disconnection: session time:
0:00:00.415 user=bf database=postgres host=[local]
2022-03-18 01:33:15.607 CET [2551209][logical replication
worker][3/76:0][] LOG: logical replication table synchronization
worker for subscription "sub2", table "tab4_1" has started
2022-03-18 01:33:15.609 CET [2551211][logical replication
worker][5/11:0][] LOG: logical replication table synchronization
worker for subscription "sub2", table "tab3" has started
2022-03-18 01:33:15.617 CET [2551193][logical replication
worker][4/62:0][] LOG: logical replication apply worker for
subscription "sub2" will restart because of a parameter change
You will notice that the apply worker is never restarted after a
parameter change. The reason was that the particular subscription
reaches the limit of max_sync_workers_per_subscription after which we
don't allow to restart the apply worker. I think you might want to
increase the values of
max_sync_workers_per_subscription/max_logical_replication_workers to
make it work.
I'd like to test & polish the main patch over the weekend, and get it
committed early next week. Unless someone thinks it's definitely not
ready for that ...
I think it is in good shape but apart from cleanup, there are issues
with dependency handling which I have analyzed and reported as one of
the comments in the email [2]/messages/by-id/CAA4eK1KR+yUQquK0Bx9uO3eb5xB1e0rAD9xKf-ddm5nSf4WfNg@mail.gmail.com. I was getting some weird behavior
during my testing due to that. Apart from that still the patch has DDL
handling code in tablecmds.c which probably is not required.
Similarly, Shi-San has reported an issue with replica full in her
email [3]/messages/by-id/TYAPR01MB6315D664D926EF66DD6E91FCFD109@TYAPR01MB6315.jpnprd01.prod.outlook.com. It is up to you what to do here but it would be good if you
can once share the patch after fixing these issues so that we can
re-test/review it.
[1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=pogona&dt=2022-03-17%2023%3A10%3A04
[2]: /messages/by-id/CAA4eK1KR+yUQquK0Bx9uO3eb5xB1e0rAD9xKf-ddm5nSf4WfNg@mail.gmail.com
[3]: /messages/by-id/TYAPR01MB6315D664D926EF66DD6E91FCFD109@TYAPR01MB6315.jpnprd01.prod.outlook.com
--
With Regards,
Amit Kapila.
On 3/18/22 06:52, Amit Kapila wrote:
On Fri, Mar 18, 2022 at 12:47 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:I pushed the second fix. Interestingly enough, wrasse failed in the
013_partition test. I don't see how that could be caused by this
particular commit, though - see the pgsql-committers thread [1].I have a theory about what's going on here. I think this is due to a
test added in your previous commit c91f71b9dc. The newly added test
added hangs in tablesync because there was no apply worker to set the
state to SUBREL_STATE_CATCHUP which blocked tablesync workers from
proceeding.See below logs from pogona [1].
2022-03-18 01:33:15.190 CET [2551176][client
backend][3/74:0][013_partition.pl] LOG: statement: ALTER SUBSCRIPTION
sub2 SET PUBLICATION pub_lower_level, pub_all
2022-03-18 01:33:15.354 CET [2551193][logical replication
worker][4/57:0][] LOG: logical replication apply worker for
subscription "sub2" has started
2022-03-18 01:33:15.605 CET [2551176][client
backend][:0][013_partition.pl] LOG: disconnection: session time:
0:00:00.415 user=bf database=postgres host=[local]
2022-03-18 01:33:15.607 CET [2551209][logical replication
worker][3/76:0][] LOG: logical replication table synchronization
worker for subscription "sub2", table "tab4_1" has started
2022-03-18 01:33:15.609 CET [2551211][logical replication
worker][5/11:0][] LOG: logical replication table synchronization
worker for subscription "sub2", table "tab3" has started
2022-03-18 01:33:15.617 CET [2551193][logical replication
worker][4/62:0][] LOG: logical replication apply worker for
subscription "sub2" will restart because of a parameter changeYou will notice that the apply worker is never restarted after a
parameter change. The reason was that the particular subscription
reaches the limit of max_sync_workers_per_subscription after which we
don't allow to restart the apply worker. I think you might want to
increase the values of
max_sync_workers_per_subscription/max_logical_replication_workers to
make it work.
Hmmm. So the theory is that in most runs we manage to sync the tables
faster than starting the workers, so we don't hit the limit. But on some
machines the sync worker takes a bit longer, we hit the limit. Seems
possible, yes. Unfortunately we don't seem to log anything when we hit
the limit, so hard to say for sure :-( I suggest we add a WARNING
message to logicalrep_worker_launch or something. Not just because of
this test, it seems useful in general.
However, how come we don't retry the sync? Surely we don't just give up
forever, that'd be a pretty annoying behavior. Presumably we just end up
sleeping for a long time before restarting the sync worker, somewhere.
I'd like to test & polish the main patch over the weekend, and get it
committed early next week. Unless someone thinks it's definitely not
ready for that ...I think it is in good shape but apart from cleanup, there are issues
with dependency handling which I have analyzed and reported as one of
the comments in the email [2]. I was getting some weird behavior
during my testing due to that. Apart from that still the patch has DDL
handling code in tablecmds.c which probably is not required.
Similarly, Shi-San has reported an issue with replica full in her
email [3]. It is up to you what to do here but it would be good if you
can once share the patch after fixing these issues so that we can
re-test/review it.
Ah, thanks for reminding me - it's hard to keep track of all the issues
in threads as long as this one.
BTW do you have any opinion on the SET COLUMNS syntax? Peter Smith
proposed to get rid of it in [1] but I'm not sure that's a good idea.
Because if we ditch it, then removing the column list would look like this:
ALTER PUBLICATION pub ALTER TABLE tab;
And if we happen to add other per-table options, this would become
pretty ambiguous.
Actually, do we even want to allow resetting column lists like this? We
don't allow this for row filters, so if you want to change a row filter
you have to re-add the table, right? So maybe we should just ditch ALTER
TABLE entirely.
regards
[4]: /messages/by-id/CAHut+Ptc7Rh187eQKrxdUmUNWyfxz7OkhYAX=AW411Qwxya0LQ@mail.gmail.com
/messages/by-id/CAHut+Ptc7Rh187eQKrxdUmUNWyfxz7OkhYAX=AW411Qwxya0LQ@mail.gmail.com
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 3/18/22 15:43, Tomas Vondra wrote:
On 3/18/22 06:52, Amit Kapila wrote:
On Fri, Mar 18, 2022 at 12:47 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:I pushed the second fix. Interestingly enough, wrasse failed in the
013_partition test. I don't see how that could be caused by this
particular commit, though - see the pgsql-committers thread [1].I have a theory about what's going on here. I think this is due to a
test added in your previous commit c91f71b9dc. The newly added test
added hangs in tablesync because there was no apply worker to set the
state to SUBREL_STATE_CATCHUP which blocked tablesync workers from
proceeding.See below logs from pogona [1].
2022-03-18 01:33:15.190 CET [2551176][client
backend][3/74:0][013_partition.pl] LOG: statement: ALTER SUBSCRIPTION
sub2 SET PUBLICATION pub_lower_level, pub_all
2022-03-18 01:33:15.354 CET [2551193][logical replication
worker][4/57:0][] LOG: logical replication apply worker for
subscription "sub2" has started
2022-03-18 01:33:15.605 CET [2551176][client
backend][:0][013_partition.pl] LOG: disconnection: session time:
0:00:00.415 user=bf database=postgres host=[local]
2022-03-18 01:33:15.607 CET [2551209][logical replication
worker][3/76:0][] LOG: logical replication table synchronization
worker for subscription "sub2", table "tab4_1" has started
2022-03-18 01:33:15.609 CET [2551211][logical replication
worker][5/11:0][] LOG: logical replication table synchronization
worker for subscription "sub2", table "tab3" has started
2022-03-18 01:33:15.617 CET [2551193][logical replication
worker][4/62:0][] LOG: logical replication apply worker for
subscription "sub2" will restart because of a parameter changeYou will notice that the apply worker is never restarted after a
parameter change. The reason was that the particular subscription
reaches the limit of max_sync_workers_per_subscription after which we
don't allow to restart the apply worker. I think you might want to
increase the values of
max_sync_workers_per_subscription/max_logical_replication_workers to
make it work.Hmmm. So the theory is that in most runs we manage to sync the tables
faster than starting the workers, so we don't hit the limit. But on some
machines the sync worker takes a bit longer, we hit the limit. Seems
possible, yes. Unfortunately we don't seem to log anything when we hit
the limit, so hard to say for sure :-( I suggest we add a WARNING
message to logicalrep_worker_launch or something. Not just because of
this test, it seems useful in general.However, how come we don't retry the sync? Surely we don't just give up
forever, that'd be a pretty annoying behavior. Presumably we just end up
sleeping for a long time before restarting the sync worker, somewhere.
I tried lowering the max_sync_workers_per_subscription to 1 and making
the workers to run for a couple seconds (doing some CPU intensive
stuff), but everything still works just fine.
Looking a bit closer at the logs (from pogona and other), I doubt this
is about hitting the max_sync_workers_per_subscription limit. Notice we
start two sync workers, but neither of them ever completes. So we never
update the sync status or start syncing the remaining tables.
So the question is why those two sync workers never complete - I guess
there's some sort of lock wait (deadlock?) or infinite loop.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 3/18/22 15:43, Tomas Vondra wrote:
On 3/18/22 06:52, Amit Kapila wrote:
...
I think it is in good shape but apart from cleanup, there are issues
with dependency handling which I have analyzed and reported as one of
the comments in the email [2]. I was getting some weird behavior
during my testing due to that. Apart from that still the patch has DDL
handling code in tablecmds.c which probably is not required.
Similarly, Shi-San has reported an issue with replica full in her
email [3]. It is up to you what to do here but it would be good if you
can once share the patch after fixing these issues so that we can
re-test/review it.Ah, thanks for reminding me - it's hard to keep track of all the issues
in threads as long as this one.
Attached is an updated patch, hopefully addressing these issues.
Firstly, I've reverted the changes in tablecmds.c, instead relying on
regular dependency behavior. I've also switched from DEPENDENCY_AUTO to
DEPENDENCY_NORMAL. This makes the code simpler, and the behavior should
be the same as for row filters, which makes it more consistent.
As for the SET COLUMNS breaking behaviors, I've decided to drop this
feature entirely, for the reasons outlined earlier today. We don't have
that for row filters either, etc. This means the dependency issue simply
disappears.
Without SET COLUMNS, if you want to change the column list you have to
remove the table from the subscription, and add it back (with the new
column list). Perhaps inconvenient, but the behavior is clearly defined.
Maybe we need a more convenient way to tweak column lists, but I'd say
we should have the same thing for row filters too.
As for the issue reported by Shi-San about replica identity full and
column filters, presumably you're referring to this:
create table tbl (a int, b int, c int);
create publication pub for table tbl (a, b, c);
alter table tbl replica identity full;
postgres=# delete from tbl;
ERROR: cannot delete from table "tbl"
DETAIL: Column list used by the publication does not cover the
replica identity.
I believe not allowing column lists with REPLICA IDENTITY FULL is
expected / correct behavior. I mean, for that to work the column list
has to always include all columns anyway, so it's pretty pointless. Of
course, we might check that the column list contains everything, but
considering the list does always have to contain all columns, and it
break as soon as you add any columns, it seems reasonable (cheaper) to
just require no column lists.
I also went through the patch and made the naming more consistent. The
comments used both "column filter" and "column list" randomly, and I
think the agreement is to use "list" so I adopted that wording.
However, while looking at how pgoutput, I realized one thing - for row
filters we track them "per operation", depending on which operations are
defined for a given publication. Shouldn't we do the same thing for
column lists, really?
I mean, if there are two publications with different column lists, one
for inserts and the other one for updates, isn't it wrong to merge these
two column lists?
Also, doesn't this mean publish_as_relid should be "per operation" too?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
0001-Allow-specifying-column-lists-for-logical-r-20220318.patchtext/x-patch; charset=UTF-8; name=0001-Allow-specifying-column-lists-for-logical-r-20220318.patchDownload
From f3fb9815c8d982e7d31c952154830bc027637534 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Thu, 17 Mar 2022 19:16:39 +0100
Subject: [PATCH] Allow specifying column lists for logical replication
This allows specifying an optional column list when adding a table to
logical replication. Columns not included on this list are not sent to
the subscriber. The list is specified after the table name, enclosed
in parentheses.
For UPDATE/DELETE publications, the column list needs to cover all
REPLICA IDENTITY columns. For INSERT publications, the column list is
arbitrary and may omit some REPLICA IDENTITY columns. Furthermore, if
the table uses REPLICA IDENTITY FULL, column list is not allowed.
The column list can contain only simple column references. Complex
expressions, function calls etc. are not allowed. This restriction could
be relaxed in the future.
During the initial table synchronization, only columns specified in the
column list are copied to the subscriber. If the subscription has
several publications, containing the same table with different column
lists, columns specified in any of the lists will be copied. This
means all columns are replicated if the table has no column list at
all (which is treated as column list with all columns), of when of the
publications is defined as FOR ALL TABLES (possibly IN SCHEMA for the
schema of the table).
For partitioned tables, publish_via_partition_root determines whether
the column list for the root or leaf relation will be used. If the
parameter is 'false' (the default), the list defined for the leaf
relation is used. Otherwise, the column list for the root partition
will be used.
Psql commands \dRp+ and \d <table-name> now display any column lists.
Author: Tomas Vondra, Rahila Syed
Reviewed-by: Peter Eisentraut, Alvaro Herrera, Vignesh C, Ibrar Ahmed,
Amit Kapila, Hou zj, Peter Smith, Wang wei, Tang, Shi yu
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 15 +-
doc/src/sgml/protocol.sgml | 3 +-
doc/src/sgml/ref/alter_publication.sgml | 18 +-
doc/src/sgml/ref/create_publication.sgml | 17 +-
src/backend/catalog/pg_publication.c | 221 ++++
src/backend/commands/publicationcmds.c | 272 ++++-
src/backend/executor/execReplication.c | 19 +-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 33 +-
src/backend/replication/logical/proto.c | 61 +-
src/backend/replication/logical/tablesync.c | 156 ++-
src/backend/replication/pgoutput/pgoutput.c | 202 +++-
src/backend/utils/cache/relcache.c | 33 +-
src/bin/pg_dump/pg_dump.c | 47 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 60 +
src/bin/psql/describe.c | 40 +-
src/include/catalog/pg_publication.h | 14 +
src/include/catalog/pg_publication_rel.h | 1 +
src/include/commands/publicationcmds.h | 4 +-
src/include/nodes/parsenodes.h | 1 +
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 372 ++++++
src/test/regress/sql/publication.sql | 287 +++++
src/test/subscription/t/030_column_list.pl | 1124 +++++++++++++++++++
26 files changed, 2915 insertions(+), 94 deletions(-)
create mode 100644 src/test/subscription/t/030_column_list.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 4dc5b34d21c..89827c373bd 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -4410,7 +4410,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para>
<para>
This is an array of <structfield>indnatts</structfield> values that
- indicate which table columns this index indexes. For example a value
+ indicate which table columns this index indexes. For example, a value
of <literal>1 3</literal> would mean that the first and the third table
columns make up the index entries. Key columns come before non-key
(included) columns. A zero in this array indicates that the
@@ -6281,6 +6281,19 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to schema
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>prattrs</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+ </para>
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example, a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all columns are published.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 9178c779ba9..fb491e9ebee 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7006,7 +7006,8 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column included in
+ the publication (except generated columns):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 32b75f6c78e..9e9fc19df71 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -30,7 +30,7 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -112,6 +112,14 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ </para>
+
+ <para>
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
If the optional <literal>WHERE</literal> clause is specified, rows for
which the <replaceable class="parameter">expression</replaceable>
evaluates to false or null will not be published. Note that parentheses
@@ -174,7 +182,13 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
<para>
Add some tables to the publication:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
+<programlisting>
+ALTER PUBLICATION mypublication SET TABLE users (user_id, firstname, lastname), TABLE departments;
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 4979b9b646d..fb2d013393b 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -86,6 +86,13 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<literal>TRUNCATE</literal> commands.
</para>
+ <para>
+ When a column list is specified, only the named columns are replicated.
+ If no column list is specified, all columns of the table are replicated
+ through this publication, including any columns added later. If a column
+ list is specified, it must include the replica identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
@@ -327,6 +334,14 @@ CREATE PUBLICATION production_publication FOR TABLE users, departments, ALL TABL
<structname>sales</structname>:
<programlisting>
CREATE PUBLICATION sales_publication FOR ALL TABLES IN SCHEMA marketing, sales;
+</programlisting></para>
+
+ <para>
+ Create a publication that publishes all changes for table <structname>users</structname>,
+ but replicates only columns <structname>user_id</structname> and
+ <structname>firstname</structname>:
+<programlisting>
+CREATE PUBLICATION users_filtered FOR TABLE users (user_id, firstname);
</programlisting></para>
</refsect1>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 789b895db89..54ea8a4cccb 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,6 +45,9 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+static void publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs);
+
/*
* Check if relation can be in given publication and throws appropriate
* error if not.
@@ -345,6 +348,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
Oid relid = RelationGetRelid(targetrel);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ AttrNumber *attarray;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -372,6 +377,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
check_publication_add_relation(targetrel);
+ /*
+ * Translate column names to attnums and make sure the column list contains
+ * only allowed elements (no system or generated columns etc.). Also build
+ * an array of attnums, for storing in the catalog.
+ */
+ publication_translate_columns(pri->relation, pri->columns,
+ &natts, &attarray);
+
/* Form a tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -390,6 +403,12 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
else
nulls[Anum_pg_publication_rel_prqual - 1] = true;
+ /* Add column list, if available */
+ if (pri->columns)
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(buildint2vector(attarray, natts));
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -413,6 +432,13 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
DEPENDENCY_NORMAL, DEPENDENCY_NORMAL,
false);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL);
+ }
+
/* Close the table. */
table_close(rel, RowExclusiveLock);
@@ -432,6 +458,125 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
return myself;
}
+/* qsort comparator for attnums */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list (no system or generated attributes,
+ * no duplicates). Additional checks with replica identity are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ */
+static void
+publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs)
+{
+ AttrNumber *attarray = NULL;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
+
+ /* Bail out when no column list defined. */
+ if (!columns)
+ return;
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference generated column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /* Be tidy, so that the catalog representation is always sorted */
+ qsort(attarray, n, sizeof(AttrNumber), compare_int16);
+
+ *natts = n;
+ *attrs = attarray;
+
+ bms_free(set);
+}
+
+/*
+ * Transform the column list (represented by an array) to a bitmapset.
+ */
+Bitmapset *
+pub_collist_to_bitmapset(Bitmapset *columns, Datum pubcols, MemoryContext mcxt)
+{
+ Bitmapset *result = NULL;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ MemoryContext oldcxt;
+
+ /*
+ * If an existing bitmap was provided, use it. Otherwise just use NULL
+ * and build a new bitmap.
+ */
+ if (columns)
+ result = columns;
+
+ arr = DatumGetArrayTypeP(pubcols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* If a memory context was specified, switch to it. */
+ if (mcxt)
+ oldcxt = MemoryContextSwitchTo(mcxt);
+
+ for (int i = 0; i < nelems; i++)
+ result = bms_add_member(result, elems[i]);
+
+ if (mcxt)
+ MemoryContextSwitchTo(oldcxt);
+
+ return result;
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -539,6 +684,82 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+ Form_pg_publication_rel pubrel;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ /* no column list for this publications/relation */
+ if (isnull)
+ continue;
+
+ pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
+
+ pubs = lappend_oid(pubs, pubrel->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 1aad2e769cb..0c9993a155b 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -296,7 +296,7 @@ contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
* Returns true if any invalid column is found.
*/
bool
-contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
+pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
bool pubviaroot)
{
HeapTuple rftuple;
@@ -368,6 +368,114 @@ contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
return result;
}
+/*
+ * Check if all columns referenced in the REPLICA IDENTITY are covered by
+ * the column list.
+ *
+ * Returns true if any replica identity column is not covered by column list.
+ */
+bool
+pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
+ bool pubviaroot)
+{
+ HeapTuple tuple;
+ Oid relid = RelationGetRelid(relation);
+ Oid publish_as_relid = RelationGetRelid(relation);
+ bool result = false;
+ Datum datum;
+ bool isnull;
+
+ /*
+ * For a partition, if pubviaroot is true, find the topmost ancestor that
+ * is published via this publication as we need to use its column list
+ * for the changes.
+ *
+ * Note that even though the column list used is for an ancestor, the
+ * REPLICA IDENTITY used will be for the actual child table.
+ */
+ if (pubviaroot && relation->rd_rel->relispartition)
+ {
+ publish_as_relid = GetTopMostAncestorInPublication(pubid, ancestors, NULL);
+
+ if (!OidIsValid(publish_as_relid))
+ publish_as_relid = relid;
+ }
+
+ tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tuple))
+ return false;
+
+ datum = SysCacheGetAttr(PUBLICATIONRELMAP, tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ if (!isnull)
+ {
+ int x;
+ Bitmapset *idattrs;
+ Bitmapset *columns = NULL;
+
+ /* With REPLICA IDENTITY FULL, no column list is allowed. */
+ if (relation->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ result = true;
+
+ /* Transform the column list datum to a bitmapset. */
+ columns = pub_collist_to_bitmapset(NULL, datum, NULL);
+
+ /* Remember columns that are part of the REPLICA IDENTITY */
+ idattrs = RelationGetIndexAttrBitmap(relation,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * Attnums in the bitmap returned by RelationGetIndexAttrBitmap are
+ * offset (to handle system columns the usual way), while column list
+ * does not use offset, so we can't do bms_is_subset(). Instead, we have
+ * to loop over the idattrs and check all of them are in the list.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ AttrNumber attnum = (x + FirstLowInvalidHeapAttributeNumber);
+
+ /*
+ * If pubviaroot is true, we are validating the column list of the
+ * parent table, but the bitmap contains the replica identity
+ * information of the child table. The parent/child attnums may not
+ * match, so translate them to the parent - get the attname from
+ * the child, and look it up in the parent.
+ */
+ if (pubviaroot)
+ {
+ /* attribute name in the child table */
+ char *colname = get_attname(relid, attnum, false);
+
+ /*
+ * Determine the attnum for the attribute name in parent (we
+ * are using the column list defined on the parent).
+ */
+ attnum = get_attnum(publish_as_relid, colname);
+ }
+
+ /* replica identity column, not covered by the column list */
+ if (!bms_is_member(attnum, columns))
+ {
+ result = true;
+ break;
+ }
+ }
+
+ bms_free(idattrs);
+ bms_free(columns);
+ }
+
+ ReleaseSysCache(tuple);
+
+ return result;
+}
+
/* check_functions_in_node callback */
static bool
contain_mutable_or_user_functions_checker(Oid func_id, void *context)
@@ -609,6 +717,45 @@ TransformPubWhereClauses(List *tables, const char *queryString,
}
}
+
+/*
+ * Transform the publication column lists expression for all the relations
+ * in the list.
+ *
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow column lists on the leaf relations. So maybe rename it?
+ */
+static void
+TransformPubColumnList(List *tables, const char *queryString,
+ bool pubviaroot)
+{
+ ListCell *lc;
+
+ foreach(lc, tables)
+ {
+ PublicationRelInfo *pri = (PublicationRelInfo *) lfirst(lc);
+
+ if (pri->columns == NIL)
+ continue;
+
+ /*
+ * If the publication doesn't publish changes via the root partitioned
+ * table, the partition's column list will be used. So disallow using
+ * the column list on partitioned table in this case.
+ */
+ if (!pubviaroot &&
+ pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot use publication column list for relation \"%s\"",
+ RelationGetRelationName(pri->relation)),
+ errdetail("column list cannot be used for a partitioned table when %s is false.",
+ "publish_via_partition_root")));
+ }
+}
+
/*
* Create new publication.
*/
@@ -725,6 +872,9 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
TransformPubWhereClauses(rels, pstate->p_sourcetext,
publish_via_partition_root);
+ TransformPubColumnList(rels, pstate->p_sourcetext,
+ publish_via_partition_root);
+
PublicationAddTables(puboid, rels, true, NULL);
CloseTableList(rels);
}
@@ -784,8 +934,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* If the publication doesn't publish changes via the root partitioned
- * table, the partition's row filter will be used. So disallow using WHERE
- * clause on partitioned table in this case.
+ * table, the partition's row filter and column list will be used. So disallow
+ * using WHERE clause and column lists on partitioned table in this case.
*/
if (!pubform->puballtables && publish_via_partition_root_given &&
!publish_via_partition_root)
@@ -793,7 +943,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* Lock the publication so nobody else can do anything with it. This
* prevents concurrent alter to add partitioned table(s) with WHERE
- * clause(s) which we don't allow when not publishing via root.
+ * clause(s) and/or column lists which we don't allow when not
+ * publishing via root.
*/
LockDatabaseObject(PublicationRelationId, pubform->oid, 0,
AccessShareLock);
@@ -805,13 +956,21 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
HeapTuple rftuple;
Oid relid = lfirst_oid(lc);
+ bool has_column_list;
+ bool has_row_filter;
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubform->oid));
+ has_row_filter
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL);
+
+ has_column_list
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prattrs, NULL);
+
if (HeapTupleIsValid(rftuple) &&
- !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL))
+ (has_row_filter || has_column_list))
{
HeapTuple tuple;
@@ -820,7 +979,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
Form_pg_class relform = (Form_pg_class) GETSTRUCT(tuple);
- if (relform->relkind == RELKIND_PARTITIONED_TABLE)
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_row_filter)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot set %s for publication \"%s\"",
@@ -831,6 +991,18 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
NameStr(relform->relname),
"publish_via_partition_root")));
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_column_list)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot set %s for publication \"%s\"",
+ "publish_via_partition_root = false",
+ stmt->pubname),
+ errdetail("The publication contains a column list for a partitioned table \"%s\" "
+ "which is not allowed when %s is false.",
+ NameStr(relform->relname),
+ "publish_via_partition_root")));
+
ReleaseSysCache(tuple);
}
@@ -976,6 +1148,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
PublicationAddTables(pubid, rels, false, stmt);
}
else if (stmt->action == AP_DropObjects)
@@ -992,6 +1166,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
/*
* To recreate the relation list for the publication, look for
* existing relations that do not need to be dropped.
@@ -1003,42 +1179,79 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
PublicationRelInfo *oldrel;
bool found = false;
HeapTuple rftuple;
- bool rfisnull = true;
Node *oldrelwhereclause = NULL;
+ Bitmapset *oldcolumns = NULL;
/* look up the cache for the old relmap */
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(oldrelid),
ObjectIdGetDatum(pubid));
+ /*
+ * See if the existing relation currently has a WHERE clause or a
+ * column list. We need to compare those too.
+ */
if (HeapTupleIsValid(rftuple))
{
+ bool isnull = true;
Datum whereClauseDatum;
+ Datum columnListDatum;
+ /* Load the WHERE clause for this table. */
whereClauseDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
Anum_pg_publication_rel_prqual,
- &rfisnull);
- if (!rfisnull)
+ &isnull);
+ if (!isnull)
oldrelwhereclause = stringToNode(TextDatumGetCString(whereClauseDatum));
+ /* Transform the int2vector column list to a bitmap. */
+ columnListDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ if (!isnull)
+ oldcolumns = pub_collist_to_bitmapset(NULL, columnListDatum, NULL);
+
ReleaseSysCache(rftuple);
}
foreach(newlc, rels)
{
PublicationRelInfo *newpubrel;
+ Oid newrelid;
+ Bitmapset *newcolumns = NULL;
newpubrel = (PublicationRelInfo *) lfirst(newlc);
+ newrelid = RelationGetRelid(newpubrel->relation);
+
+ /*
+ * If the new publication has column list, transform it to
+ * a bitmap too.
+ */
+ if (newpubrel->columns)
+ {
+ ListCell *lc;
+
+ foreach(lc, newpubrel->columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(newrelid, colname);
+
+ newcolumns = bms_add_member(newcolumns, attnum);
+ }
+ }
/*
* Check if any of the new set of relations matches with the
* existing relations in the publication. Additionally, if the
* relation has an associated WHERE clause, check the WHERE
- * expressions also match. Drop the rest.
+ * expressions also match. Same for the column list. Drop the
+ * rest.
*/
if (RelationGetRelid(newpubrel->relation) == oldrelid)
{
- if (equal(oldrelwhereclause, newpubrel->whereClause))
+ if (equal(oldrelwhereclause, newpubrel->whereClause) &&
+ bms_equal(oldcolumns, newcolumns))
{
found = true;
break;
@@ -1057,6 +1270,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
{
oldrel = palloc(sizeof(PublicationRelInfo));
oldrel->whereClause = NULL;
+ oldrel->columns = NIL;
oldrel->relation = table_open(oldrelid,
ShareUpdateExclusiveLock);
delrels = lappend(delrels, oldrel);
@@ -1118,7 +1332,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
}
else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* AP_SetObjects */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -1403,6 +1617,7 @@ OpenTableList(List *tables)
List *rels = NIL;
ListCell *lc;
List *relids_with_rf = NIL;
+ List *relids_with_collist = NIL;
/*
* Open, share-lock, and check all the explicitly-specified relations
@@ -1437,6 +1652,13 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /* Disallow duplicate tables if there are any with column lists. */
+ if (t->columns || list_member_oid(relids_with_collist, myrelid))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
table_close(rel, ShareUpdateExclusiveLock);
continue;
}
@@ -1444,12 +1666,16 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
pub_rel->whereClause = t->whereClause;
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, myrelid);
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, myrelid);
+
/*
* Add children of this rel, if requested, so that they too are added
* to the publication. A partitioned table can't have any inheritance
@@ -1489,6 +1715,18 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /*
+ * We don't allow to specify column list for both parent
+ * and child table at the same time as it is not very
+ * clear which one should be given preference.
+ */
+ if (childrelid != myrelid &&
+ (t->columns || list_member_oid(relids_with_collist, childrelid)))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
continue;
}
@@ -1498,11 +1736,16 @@ OpenTableList(List *tables)
pub_rel->relation = rel;
/* child inherits WHERE clause from parent */
pub_rel->whereClause = t->whereClause;
+ /* child inherits column list from parent */
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, childrelid);
+
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, childrelid);
}
}
}
@@ -1611,6 +1854,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 09f78f22441..3e282ed99ab 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -573,9 +573,6 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
if (cmd != CMD_UPDATE && cmd != CMD_DELETE)
return;
- if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
- return;
-
/*
* It is only safe to execute UPDATE/DELETE when all columns, referenced
* in the row filters from publications which the relation is in, are
@@ -595,17 +592,33 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
errmsg("cannot update table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_UPDATE && !pubdesc.cols_valid_for_update)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot update table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
else if (cmd == CMD_DELETE && !pubdesc.rf_valid_for_delete)
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
errmsg("cannot delete from table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_DELETE && !pubdesc.cols_valid_for_delete)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot delete from table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
/* If relation has replica identity we are always good. */
if (OidIsValid(RelationGetReplicaIndex(rel)))
return;
+ /* REPLICA IDENTITY FULL is also good for UPDATE/DELETE. */
+ if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ return;
+
/*
* This is UPDATE/DELETE and there is no replica identity.
*
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index d4f8455a2bd..a504437873f 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4850,6 +4850,7 @@ _copyPublicationTable(const PublicationTable *from)
COPY_NODE_FIELD(relation);
COPY_NODE_FIELD(whereClause);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f1002afe7a0..4fc16ce04e3 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2322,6 +2322,7 @@ _equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
COMPARE_NODE_FIELD(whereClause);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a03b33b53bd..ff4573390c5 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9751,13 +9751,14 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr OptWhereClause
+ TABLE relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9772,11 +9773,15 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId OptWhereClause
+ | ColId opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- if ($2)
+ /*
+ * If either a row filter or column list is specified, create
+ * a PublicationTable object.
+ */
+ if ($2 || $3)
{
/*
* The OptWhereClause must be stored here but it is
@@ -9786,7 +9791,8 @@ PublicationObjSpec:
*/
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVar(NULL, $1, @1);
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
else
{
@@ -9794,23 +9800,25 @@ PublicationObjSpec:
}
$$->location = @1;
}
- | ColId indirection OptWhereClause
+ | ColId indirection opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr OptWhereClause
+ | extended_relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
| CURRENT_SCHEMA
{
@@ -17488,6 +17496,13 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
errmsg("WHERE clause not allowed for schema"),
parser_errposition(pubobj->location));
+ /* Column list is not allowed on a schema object */
+ if (pubobj->pubtable && pubobj->pubtable->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schema"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index c9b0eeefd7e..f9de1d16dc2 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,16 +29,30 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary);
+ bool binary, Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
static void logicalrep_write_namespace(StringInfo out, Oid nspid);
static const char *logicalrep_read_namespace(StringInfo in);
+/*
+ * Check if a column is covered by a column list.
+ *
+ * Need to be careful about NULL, which is treated as a column list covering
+ * all columns.
+ */
+static bool
+column_in_column_list(int attnum, Bitmapset *columns)
+{
+ return (columns == NULL || bms_is_member(attnum, columns));
+}
+
+
/*
* Write BEGIN to the output stream.
*/
@@ -398,7 +412,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +424,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -443,7 +457,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -464,11 +478,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -537,7 +551,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, NULL);
}
/*
@@ -652,7 +666,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -674,7 +689,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -751,7 +766,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum *values;
@@ -763,8 +778,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ if (!column_in_column_list(att->attnum, columns))
continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -783,6 +804,9 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_column_list(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +928,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -917,8 +941,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* send number of live attributes */
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
continue;
+
+ if (!column_in_column_list(att->attnum, columns))
+ continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -937,6 +967,9 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_column_list(att->attnum, columns))
+ continue;
+
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 1659964571c..caeab853e4c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -112,6 +112,7 @@
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "utils/acl.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -701,12 +702,13 @@ fetch_remote_table_info(char *nspname, char *relname,
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
int natt;
ListCell *lc;
bool first;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
@@ -747,10 +749,110 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+
+ /*
+ * Get column lists for each relation.
+ *
+ * For initial synchronization, column lists can be ignored in following
+ * cases:
+ *
+ * 1) one of the subscribed publications for the table hasn't specified
+ * any column list
+ *
+ * 2) one of the subscribed publications has puballtables set to true
+ *
+ * 3) one of the subscribed publications is declared as ALL TABLES IN
+ * SCHEMA that includes this relation
+ *
+ * We need to do this before fetching info about column names and types,
+ * so that we can skip columns that should not be replicated.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData pub_names;
+ bool first = true;
+
+ initStringInfo(&pub_names);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&pub_names, ", ");
+ appendStringInfoString(&pub_names, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ /*
+ * Fetch info about column lists for the relation (from all the
+ * publications). We unnest the int2vector values, because that
+ * makes it easier to combine lists by simply adding the attnums
+ * to a new bitmap (without having to parse the int2vector data).
+ * This preserves NULL values, so that if one of the publications
+ * has no column list, we'll know that.
+ */
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT DISTINCT unnest"
+ " FROM pg_publication p"
+ " LEFT OUTER JOIN pg_publication_rel pr"
+ " ON (p.oid = pr.prpubid AND pr.prrelid = %u)"
+ " LEFT OUTER JOIN unnest(pr.prattrs) ON TRUE,"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt"
+ " WHERE gpt.relid = %u"
+ " AND p.pubname IN ( %s )",
+ lrel->remoteid,
+ lrel->remoteid,
+ pub_names.data);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch column list info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ /*
+ * Merge the column lists (from different publications) by creating
+ * a single bitmap with all the attnums. If we find a NULL value,
+ * that means one of the publications has no column list for the
+ * table we're syncing.
+ */
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ Datum cfval = slot_getattr(slot, 1, &isnull);
+
+ /* NULL means empty column list, so we're done. */
+ if (isnull)
+ {
+ bms_free(included_cols);
+ included_cols = NULL;
+ break;
+ }
+
+ included_cols = bms_add_member(included_cols,
+ DatumGetInt16(cfval));
+
+ ExecClearTuple(slot);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+
+ walrcv_clear_result(pubres);
+
+ pfree(pub_names.data);
+ }
+
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -778,16 +880,35 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * Store the columns as a list of names. Ignore those that are not
+ * present in the column list, if there is one.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ /* If the column is not in the column list, skip it. */
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ {
+ ExecClearTuple(slot);
+ continue;
+ }
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -821,6 +942,9 @@ fetch_remote_table_info(char *nspname, char *relname,
*
* 3) one of the subscribed publications is declared as ALL TABLES IN
* SCHEMA that includes this relation
+ *
+ * XXX Does this actually handle puballtables and schema publications
+ * correctly?
*/
if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
{
@@ -930,8 +1054,24 @@ copy_table(Relation rel)
/* Regular table with no row filter */
if (lrel.relkind == RELKIND_RELATION && qual == NIL)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+
+ /*
+ * XXX Do we need to list the columns in all cases? Maybe we're replicating
+ * all columns?
+ */
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
+
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
+
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 5fddab3a3d4..f5e7610a172 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -29,6 +29,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -85,7 +86,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -143,9 +145,6 @@ typedef struct RelationSyncEntry
*/
ExprState *exprstate[NUM_ROWFILTER_PUBACTIONS];
EState *estate; /* executor state used for row filter */
- MemoryContext cache_expr_cxt; /* private context for exprstate and
- * estate, if any */
-
TupleTableSlot *new_slot; /* slot for storing new tuple */
TupleTableSlot *old_slot; /* slot for storing old tuple */
@@ -164,6 +163,19 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
AttrMap *attrmap;
+
+ /*
+ * Columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this bitmap are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
+
+ /*
+ * Private context to store additional data for this entry - state for
+ * the row filter expressions, column list, etc.
+ */
+ MemoryContext entry_cxt;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -188,6 +200,7 @@ static EState *create_estate_for_relation(Relation rel);
static void pgoutput_row_filter_init(PGOutputData *data,
List *publications,
RelationSyncEntry *entry);
+
static bool pgoutput_row_filter_exec_expr(ExprState *state,
ExprContext *econtext);
static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
@@ -195,6 +208,11 @@ static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
RelationSyncEntry *entry,
ReorderBufferChangeType *action);
+/* column list routines */
+static void pgoutput_column_list_init(PGOutputData *data,
+ List *publications,
+ RelationSyncEntry *entry);
+
/*
* Specify output plugin callbacks
*/
@@ -603,11 +621,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -620,7 +638,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -643,13 +662,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip this attribute if it's not present in the column list */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -703,6 +726,28 @@ pgoutput_row_filter_exec_expr(ExprState *state, ExprContext *econtext)
return DatumGetBool(ret);
}
+/*
+ * Make sure the per-entry memory context exists.
+ */
+static void
+pgoutput_ensure_entry_cxt(PGOutputData *data, RelationSyncEntry *entry)
+{
+ Relation relation;
+
+ /* The context may already exist, in which case bail out. */
+ if (entry->entry_cxt)
+ return;
+
+ relation = RelationIdGetRelation(entry->publish_as_relid);
+
+ entry->entry_cxt = AllocSetContextCreate(data->cachectx,
+ "entry private context",
+ ALLOCSET_SMALL_SIZES);
+
+ MemoryContextCopyAndSetIdentifier(entry->entry_cxt,
+ RelationGetRelationName(relation));
+}
+
/*
* Initialize the row filter.
*/
@@ -823,21 +868,13 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
{
Relation relation = RelationIdGetRelation(entry->publish_as_relid);
- Assert(entry->cache_expr_cxt == NULL);
-
- /* Create the memory context for row filters */
- entry->cache_expr_cxt = AllocSetContextCreate(data->cachectx,
- "Row filter expressions",
- ALLOCSET_DEFAULT_SIZES);
-
- MemoryContextCopyAndSetIdentifier(entry->cache_expr_cxt,
- RelationGetRelationName(relation));
+ pgoutput_ensure_entry_cxt(data, entry);
/*
* Now all the filters for all pubactions are known. Combine them when
* their pubactions are the same.
*/
- oldctx = MemoryContextSwitchTo(entry->cache_expr_cxt);
+ oldctx = MemoryContextSwitchTo(entry->entry_cxt);
entry->estate = create_estate_for_relation(relation);
for (idx = 0; idx < NUM_ROWFILTER_PUBACTIONS; idx++)
{
@@ -860,6 +897,105 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
}
}
+/*
+ * Initialize the column list.
+ */
+static void
+pgoutput_column_list_init(PGOutputData *data, List *publications,
+ RelationSyncEntry *entry)
+{
+ ListCell *lc;
+
+ /*
+ * Find if there are any column lists for this relation. If there are,
+ * build a bitmap merging all the column lists.
+ *
+ * All the given publication-table mappings must be checked.
+ *
+ * Multiple publications might have multiple column lists for this relation.
+ *
+ * FOR ALL TABLES and FOR ALL TABLES IN SCHEMA implies "don't use column
+ * list" so it takes precedence.
+ */
+ foreach(lc, publications)
+ {
+ Publication *pub = lfirst(lc);
+ HeapTuple cftuple = NULL;
+ Datum cfdatum = 0;
+
+ /*
+ * Assume there's no column list. Only if we find pg_publication_rel
+ * entry with a column list we'll switch it to false.
+ */
+ bool pub_no_list = true;
+
+ /*
+ * If the publication is FOR ALL TABLES then it is treated the same as if
+ * there are no column lists (even if other publications have a list).
+ */
+ if (!pub->alltables)
+ {
+ /*
+ * Check for the presence of a column list in this publication.
+ *
+ * Note: If we find no pg_publication_rel row, it's a publication
+ * defined for a whole schema, so it can't have a column list, just
+ * like a FOR ALL TABLES publication.
+ */
+ cftuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(entry->publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(cftuple))
+ {
+ /*
+ * Lookup the column list attribute.
+ *
+ * Note: We update the pub_no_list value directly, because if
+ * the value is NULL, we have no list (and vice versa).
+ */
+ cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
+ Anum_pg_publication_rel_prattrs,
+ &pub_no_list);
+
+ /*
+ * Build the column list bitmap in the per-entry context.
+ *
+ * We need to merge column lists from all publications, so we
+ * update the same bitmapset. If the column list is null, we
+ * interpret it as replicating all columns.
+ */
+ if (!pub_no_list) /* when not null */
+ {
+ pgoutput_ensure_entry_cxt(data, entry);
+
+ entry->columns = pub_collist_to_bitmapset(entry->columns,
+ cfdatum,
+ entry->entry_cxt);
+ }
+ }
+ }
+
+ /*
+ * Found a publication with no column list, so we're done. But first
+ * discard column list we might have from preceding publications.
+ */
+ if (pub_no_list)
+ {
+ if (cftuple)
+ ReleaseSysCache(cftuple);
+
+ bms_free(entry->columns);
+ entry->columns = NULL;
+
+ break;
+ }
+
+ ReleaseSysCache(cftuple);
+ } /* loop all subscribed publications */
+
+}
+
/*
* Initialize the slot for storing new and old tuples, and build the map that
* will be used to convert the relation's tuples into the ancestor's format.
@@ -1224,7 +1360,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -1278,11 +1414,13 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel,
- new_slot, data->binary);
+ new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel,
- old_slot, new_slot, data->binary);
+ old_slot, new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel,
@@ -1729,8 +1867,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->new_slot = NULL;
entry->old_slot = NULL;
memset(entry->exprstate, 0, sizeof(entry->exprstate));
- entry->cache_expr_cxt = NULL;
+ entry->entry_cxt = NULL;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->attrmap = NULL;
}
@@ -1776,6 +1915,8 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
entry->pubactions.pubinsert = false;
entry->pubactions.pubupdate = false;
entry->pubactions.pubdelete = false;
@@ -1799,17 +1940,18 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/*
* Row filter cache cleanups.
*/
- if (entry->cache_expr_cxt)
- MemoryContextDelete(entry->cache_expr_cxt);
+ if (entry->entry_cxt)
+ MemoryContextDelete(entry->entry_cxt);
- entry->cache_expr_cxt = NULL;
+ entry->entry_cxt = NULL;
entry->estate = NULL;
memset(entry->exprstate, 0, sizeof(entry->exprstate));
/*
* Build publication cache. We can't use one provided by relcache as
- * relcache considers all publications given relation is in, but here
- * we only need to consider ones that the subscriber requested.
+ * relcache considers all publications that the given relation is in,
+ * but here we only need to consider ones that the subscriber
+ * requested.
*/
foreach(lc, data->publications)
{
@@ -1878,6 +2020,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
}
/*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
* Don't publish changes for partitioned tables, because
* publishing those of its partitions suffices, unless partition
* changes won't be published due to pubviaroot being set.
@@ -1938,6 +2083,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/* Initialize the row filter */
pgoutput_row_filter_init(data, rel_publications, entry);
+
+ /* Initialize the column list */
+ pgoutput_column_list_init(data, rel_publications, entry);
}
list_free(pubids);
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index fccffce5729..a2da72f0d48 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -5553,6 +5553,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
return;
}
@@ -5565,6 +5567,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
/* Fetch the publication membership info. */
puboids = GetRelationPublications(relid);
@@ -5616,7 +5620,7 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
*/
if (!pubform->puballtables &&
(pubform->pubupdate || pubform->pubdelete) &&
- contain_invalid_rfcolumn(pubid, relation, ancestors,
+ pub_rf_contains_invalid_column(pubid, relation, ancestors,
pubform->pubviaroot))
{
if (pubform->pubupdate)
@@ -5625,6 +5629,23 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->rf_valid_for_delete = false;
}
+ /*
+ * Check if all columns are part of the REPLICA IDENTITY index or not.
+ *
+ * If the publication is FOR ALL TABLES then it means the table has no
+ * column list and we can skip the validation.
+ */
+ if (!pubform->puballtables &&
+ (pubform->pubupdate || pubform->pubdelete) &&
+ pub_collist_contains_invalid_column(pubid, relation, ancestors,
+ pubform->pubviaroot))
+ {
+ if (pubform->pubupdate)
+ pubdesc->cols_valid_for_update = false;
+ if (pubform->pubdelete)
+ pubdesc->cols_valid_for_delete = false;
+ }
+
ReleaseSysCache(tup);
/*
@@ -5636,6 +5657,16 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
!pubdesc->rf_valid_for_update && !pubdesc->rf_valid_for_delete)
break;
+
+ /*
+ * If we know everything is replicated and the column list is invalid
+ * for update and delete, there is no point to check for other
+ * publications.
+ */
+ if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
+ pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
+ !pubdesc->cols_valid_for_update && !pubdesc->cols_valid_for_delete)
+ break;
}
if (relation->rd_pubdesc)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 725cd2e4ebc..be40acd3e37 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4101,6 +4101,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_prpubid;
int i_prrelid;
int i_prrelqual;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4114,12 +4115,20 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
if (fout->remoteVersion >= 150000)
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual "
- "FROM pg_catalog.pg_publication_rel");
+ "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual, "
+ "(CASE\n"
+ " WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT array_agg(attname)\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) prattrs "
+ "FROM pg_catalog.pg_publication_rel pr");
else
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "NULL AS prrelqual "
+ "NULL AS prrelqual, NULL AS prattrs "
"FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
@@ -4130,6 +4139,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
i_prrelqual = PQfnumber(res, "prrelqual");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4175,6 +4185,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
else
pubrinfo[j].pubrelqual = pg_strdup(PQgetvalue(res, i, i_prrelqual));
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[j].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4249,10 +4281,13 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+
if (pubrinfo->pubrelqual)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 772dc0cf7a2..1d21c2906f1 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -632,6 +632,7 @@ typedef struct _PublicationRelInfo
PublicationInfo *publication;
TableInfo *pubtable;
char *pubrelqual;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index fd1052e5db8..05a7e28bdcc 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2428,6 +2428,28 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'ALTER PUBLICATION pub1 ADD TABLE test_sixth_table (col3, col2)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_sixth_table (col3, col2);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_sixth_table (col2, col3);\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'ALTER PUBLICATION pub1 ADD TABLE test_seventh_table (col3, col2) WHERE (col1 = 1)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_seventh_table (col3, col2) WHERE (col1 = 1);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_seventh_table (col2, col3) WHERE ((col1 = 1));\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'ALTER PUBLICATION pub3 ADD ALL TABLES IN SCHEMA dump_test' => {
create_order => 51,
create_sql =>
@@ -2778,6 +2800,44 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'CREATE TABLE test_sixth_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_sixth_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_sixth_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'CREATE TABLE test_seventh_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_seventh_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_seventh_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'CREATE TABLE test_table_identity' => {
create_order => 3,
create_sql => 'CREATE TABLE dump_test.test_table_identity (
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 991bfc1546b..88bb75ac658 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2892,6 +2892,7 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_namespace pn ON p.oid = pn.pnpubid\n"
" JOIN pg_catalog.pg_class pc ON pc.relnamespace = pn.pnnspid\n"
@@ -2899,6 +2900,12 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , pg_get_expr(pr.prqual, c.oid)\n"
+ " , (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT string_agg(attname, ', ')\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) "
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
" JOIN pg_catalog.pg_class c ON c.oid = pr.prrelid\n"
@@ -2906,6 +2913,7 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2916,12 +2924,14 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
"WHERE pr.prrelid = '%s'\n"
"UNION ALL\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2943,6 +2953,11 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, " \"%s\"",
PQgetvalue(result, i, 0));
+ /* column list (if any) */
+ if (!PQgetisnull(result, i, 2))
+ appendPQExpBuffer(&buf, " (%s)",
+ PQgetvalue(result, i, 2));
+
/* row filter (if any) */
if (!PQgetisnull(result, i, 1))
appendPQExpBuffer(&buf, " WHERE %s",
@@ -5888,7 +5903,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5905,15 +5920,19 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (as_schema)
+ printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
+ else
{
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
+
+ if (!PQgetisnull(res, i, 3))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 3));
+
if (!PQgetisnull(res, i, 2))
appendPQExpBuffer(buf, " WHERE %s", PQgetvalue(res, i, 2));
}
- else
- printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
}
@@ -6042,11 +6061,22 @@ describePublications(const char *pattern)
printfPQExpBuffer(&buf,
"SELECT n.nspname, c.relname");
if (pset.sversion >= 150000)
+ {
appendPQExpBufferStr(&buf,
", pg_get_expr(pr.prqual, c.oid)");
+ appendPQExpBufferStr(&buf,
+ ", (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string("
+ " ARRAY(SELECT attname\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END)");
+ }
else
appendPQExpBufferStr(&buf,
- ", NULL");
+ ", NULL, NULL");
appendPQExpBuffer(&buf,
"\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index fe773cf9b7d..a56c1102463 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -85,6 +85,13 @@ typedef struct PublicationDesc
*/
bool rf_valid_for_update;
bool rf_valid_for_delete;
+
+ /*
+ * true if the columns are part of the replica identity or the publication actions
+ * do not include UPDATE or DELETE.
+ */
+ bool cols_valid_for_update;
+ bool cols_valid_for_delete;
} PublicationDesc;
typedef struct Publication
@@ -100,6 +107,7 @@ typedef struct PublicationRelInfo
{
Relation relation;
Node *whereClause;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -123,8 +131,11 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
+extern void GetActionsInPublication(Oid pubid, PublicationActions *actions);
extern List *GetPublicationSchemas(Oid pubid);
extern List *GetSchemaPublications(Oid schemaid);
extern List *GetSchemaPublicationRelations(Oid schemaid,
@@ -144,6 +155,9 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *pri
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern Bitmapset *pub_collist_to_bitmapset(Bitmapset *columns, Datum pubcols,
+ MemoryContext mcxt);
+
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index 0dd0f425db9..4feb581899e 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -34,6 +34,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
#ifdef CATALOG_VARLEN /* variable-length fields start here */
pg_node_tree prqual; /* qualifications */
+ int2vector prattrs; /* columns to replicate */
#endif
} FormData_pg_publication_rel;
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index 7813cbcb6bb..ae87caf089d 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -31,7 +31,9 @@ extern void RemovePublicationSchemaById(Oid psoid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
extern void AlterPublicationOwner_oid(Oid pubid, Oid newOwnerId);
extern void InvalidatePublicationRels(List *relids);
-extern bool contain_invalid_rfcolumn(Oid pubid, Relation relation,
+extern bool pub_rf_contains_invalid_column(Oid pubid, Relation relation,
+ List *ancestors, bool pubviaroot);
+extern bool pub_collist_contains_invalid_column(Oid pubid, Relation relation,
List *ancestors, bool pubviaroot);
#endif /* PUBLICATIONCMDS_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1617702d9d6..b4479c7049a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3652,6 +3652,7 @@ typedef struct PublicationTable
NodeTag type;
RangeVar *relation; /* relation to be published */
Node *whereClause; /* qualifications */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4d2c881644a..a771ab8ff33 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -209,12 +209,12 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary);
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -231,7 +231,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 4e191c120ac..227b5611915 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -613,6 +613,369 @@ DROP TABLE rf_tbl_abcd_pk;
DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+RESET client_min_messages;
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ERROR: cannot reference generated column "d" in publication column list
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+ERROR: cannot reference system column "ctid" in publication column list
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+ERROR: cannot drop column c of table testpub_tbl5 because other objects depend on it
+DETAIL: publication of table testpub_tbl5 in publication testpub_fortable depends on column c of table testpub_tbl5
+HINT: Use DROP ... CASCADE to drop the dependent objects too.
+-- ok: for insert-only publication, the column list is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: change the replica identity to "b", and column list to (a, c)
+-- then update fails, because (a, c) does not cover replica identity
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+ Publication testpub_table_ins
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | t | f
+Tables:
+ "public.testpub_tbl5" (a)
+
+-- with REPLICA IDENTITY FULL, column lists are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ERROR: cannot update table "testpub_tbl6"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+-- make sure changing the column list is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: update the column list
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, c)
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl8;
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_0"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+ Publication testpub_both_filters
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | t | t | t | f
+Tables:
+ "public.testpub_tbl_both_filters" (a, c) WHERE (c <> 1)
+
+\d+ testpub_tbl_both_filters
+ Table "public.testpub_tbl_both_filters"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+ c | integer | | not null | | plain | |
+Indexes:
+ "testpub_tbl_both_filters_pkey" PRIMARY KEY, btree (a, c) REPLICA IDENTITY
+Publications:
+ "testpub_both_filters" (a, c) WHERE (c <> 1)
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+-- Tests for partitioned table
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+ERROR: cannot use publication column list for relation "rf_tbl_abcd_part_pk"
+DETAIL: column list cannot be used for a partitioned table when publish_via_partition_root is false.
+-- ok - can use column list for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any column list is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+ERROR: cannot set publish_via_partition_root = false for publication "testpub6"
+DETAIL: The publication contains a column list for a partitioned table "rf_tbl_abcd_part_pk" which is not allowed when publish_via_partition_root is false.
+-- Now change the root column list to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have column list for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -1058,6 +1421,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schema
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 5457c56b33f..aeb1b572af8 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -373,6 +373,289 @@ DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+RESET client_min_messages;
+
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+-- ok: for insert-only publication, the column list is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+
+-- error: change the replica identity to "b", and column list to (a, c)
+-- then update fails, because (a, c) does not cover replica identity
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+UPDATE testpub_tbl5 SET a = 1;
+
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+
+-- with REPLICA IDENTITY FULL, column lists are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+
+-- make sure changing the column list is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: update the column list
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl8;
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+\d+ testpub_tbl_both_filters
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Tests for partitioned table
+
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - can use column list for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any column list is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- Now change the root column list to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have column list for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
+
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -614,6 +897,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/030_column_list.pl b/src/test/subscription/t/030_column_list.pl
new file mode 100644
index 00000000000..5ceaec83cdb
--- /dev/null
+++ b/src/test/subscription/t/030_column_list.pl
@@ -0,0 +1,1124 @@
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test partial-column publication of tables
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# create publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+sub wait_for_subscription_sync
+{
+ my ($node) = @_;
+
+ # Also wait for initial table sync to finish
+ my $synced_query = "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+
+ $node->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+}
+
+# setup tables on both nodes
+
+# tab1: simple 1:1 replication
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+# tab2: replication from regular to table with fewer columns
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)
+));
+
+# tab3: simple 1:1 replication with weird column names
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "B" varchar, "c'" int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "c'" int)
+));
+
+# test_part: partitioned tables, with partitioning (including multi-level
+# partitioning, and fewer columns on the subscriber)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+# tab4: table with user-defined enum types
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, d text);
+));
+
+
+# TEST: create publication and subscription for some of the tables with
+# column lists
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub1
+ FOR TABLE tab1 (a, "B"), tab3 ("a'", "c'"), test_part (a, b), tab4 (a, b, d)
+ WITH (publish_via_partition_root = 'true');
+));
+
+# check that we got the right prattrs values for the publication in the
+# pg_publication_rel catalog (order by relname, to get stable ordering)
+my $result = $node_publisher->safe_psql('postgres', qq(
+ SELECT relname, prattrs
+ FROM pg_publication_rel pb JOIN pg_class pc ON(pb.prrelid = pc.oid)
+ ORDER BY relname
+));
+
+is($result, qq(tab1|1 2
+tab3|1 3
+tab4|1 2 4
+test_part|1 2), 'publication relation updated');
+
+# TEST: insert data into the tables, create subscription and see if sync
+# replicates the right columns
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (1, 2, 3);
+ INSERT INTO tab1 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (1, 2, 3);
+ INSERT INTO tab3 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (1, 'red', 3, 'oh my');
+ INSERT INTO tab4 VALUES (2, 'blue', 4, 'hello');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (1, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (2, 'bcd', '2021-07-03 11:12:13');
+ INSERT INTO test_part VALUES (7, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (8, 'bcd', '2021-07-03 11:12:13');
+));
+
+# create subscription for the publication, wait for sync to complete,
+# then check the sync results
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+4|5|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+4|6), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+7|abc
+8|bcd), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: now insert more data into the tables, and wait until we replicate
+# them (not by tablesync, but regular decoding and replication)
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (2, 3, 4);
+ INSERT INTO tab1 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (2, 3, 4);
+ INSERT INTO tab3 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (3, 'red', 5, 'foo');
+ INSERT INTO tab4 VALUES (4, 'blue', 6, 'bar');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (3, 'xxx', '2022-02-01 10:00:00');
+ INSERT INTO test_part VALUES (4, 'yyy', '2022-03-02 15:12:13');
+ INSERT INTO test_part VALUES (9, 'zzz', '2022-04-03 21:00:00');
+ INSERT INTO test_part VALUES (10, 'qqq', '2022-05-04 22:12:13');
+));
+
+# wait for catchup before checking the subscriber
+$node_publisher->wait_for_catchup('sub1');
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+2|3|
+4|5|
+5|6|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+2|4
+4|6
+5|7), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello
+3|red|foo
+4|blue|bar), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+3|xxx
+4|yyy
+7|abc
+8|bcd
+9|zzz
+10|qqq), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: do some updates on some of the tables, both on columns included
+# in the column list and other
+
+# tab1: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET "B" = 2 * "B" where a = 1));
+
+# tab1: update of non-replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET c = 2*c where a = 4));
+
+# tab3: update of non-replicated
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "B" = "B" || ' updated' where "a'" = 4));
+
+# tab3: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "c'" = 2 * "c'" where "a'" = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'blue', c = c * 2, d = d || ' updated' where a = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'red', c = c * 2, d = d || ' updated' where a = 2));
+
+# wait for the replication to catch up, and check the UPDATE results got
+# replicated correctly, with the right column list
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab1 ORDER BY a));
+is($result,
+qq(1|4|
+2|3|
+4|5|
+5|6|), 'only update on column tab1.b is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result,
+qq(1|6
+2|4
+4|6
+5|7), 'only update on column tab3.c is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab4 ORDER BY a));
+
+is($result, qq(1|blue|oh my updated
+2|red|hello updated
+3|red|foo
+4|blue|bar), 'update on column tab4.c is not replicated');
+
+
+# TEST: add table with a column list, insert data, replicate
+
+# insert some data before adding it to the publication
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (1, 'abc', 3);
+));
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2 (a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION");
+
+# wait for the tablesync to complete, add a bit more data and then check
+# the results of the replication
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (2, 'def', 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|def), 'insert on column tab2.c is not replicated');
+
+# do a couple updates, check the correct stuff gets replicated
+$node_publisher->safe_psql('postgres', qq(
+ UPDATE tab2 SET c = 5 where a = 1;
+ UPDATE tab2 SET b = 'xyz' where a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|xyz), 'update on column tab2.c is not replicated');
+
+
+# TEST: add a table to two publications with different column lists, and
+# create a single subscription replicating both publications
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+
+ -- insert a couple initial records
+ INSERT INTO tab5 VALUES (1, 11, 111, 1111);
+ INSERT INTO tab5 VALUES (2, 22, 222, 2222);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub3
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->wait_for_catchup('sub1');
+
+# insert data and make sure all the columns (union of the columns lists)
+# get fully replicated
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (3, 33, 333, 3333);
+ INSERT INTO tab5 VALUES (4, 44, 444, 4444);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111
+2|22|2222
+3|33|3333
+4|44|4444),
+ 'overlapping publications with overlapping column lists');
+
+# and finally, remove the column list for one of the publications, which
+# means replicating all columns (removing the column list), but first add
+# the missing column to the table on subscriber
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub3 SET TABLE tab5;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ ALTER TABLE tab5 ADD COLUMN c INT;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (5, 55, 555, 5555);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111|
+2|22|2222|
+3|33|3333|
+4|44|4444|
+5|55|5555|555),
+ 'overlapping publications with overlapping column lists');
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key (but use a different column in
+# the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub4 FOR TABLE tab6 (a, b);
+
+ -- initial data
+ INSERT INTO tab6 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (2, 33, 444, 5555);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is still covered by the column list, though)
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+# we need to do the same thing on the subscriber
+# XXX What would happen if this happens before the publisher ALTER? Or
+# interleaved, somehow? But that seems unrelated to column lists.
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (3, 55, 666, 8888);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key with a key on multiple columns
+# (all of them covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub5 FOR TABLE tab7 (a, b);
+
+ -- some initial data
+ INSERT INTO tab7 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub5
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (2, 33, 444, 5555);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is not covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ ALTER TABLE tab7 ADD PRIMARY KEY (a, b);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (3, 55, 666, 7777);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+# now switch the primary key again to another columns not covered by the
+# column list, but also generate writes between the drop and creation
+# of the new constraint
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ INSERT INTO tab7 VALUES (4, 77, 888, 9999);
+ -- update/delete is not allowed for tables without RI
+ ALTER TABLE tab7 ADD PRIMARY KEY (b, a);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+ DELETE FROM tab7 WHERE a = 1;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(2|264||
+3|220||
+4|154||),
+ 'replication with the modified primary key');
+
+
+# TEST: partitioned tables (with publish_via_partition_root = false)
+# and replica identity. The (leaf) partitions may have different RI, so
+# we need to check the partition RI (with respect to the column list)
+# while attaching the partition.
+
+# First, let's create a partitioned table with two partitions, each with
+# a different RI, but a column list not covering all those RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+
+ -- initial data, one row in each partition
+ INSERT INTO test_part_a VALUES (1, 3);
+ INSERT INTO test_part_a VALUES (6, 4);
+));
+
+# do the same thing on the subscriber (with the opposite column order)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (b int, a int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+));
+
+# create a publication replicating just the column "a", which is not enough
+# for the second partition
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub6 FOR TABLE test_part_a (b, a) WITH (publish_via_partition_root = true);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_1 (a);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_2 (b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub6
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_a VALUES (2, 5);
+ INSERT INTO test_part_a VALUES (7, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT a, b FROM test_part_a ORDER BY a, b"),
+ qq(1|3
+2|5
+6|4
+7|6),
+ 'partitions with different replica identities not replicated correctly');
+
+# This time start with a column list covering RI for all partitions, but
+# then update the column list to not cover column "b" (needed by the
+# second partition)
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+
+ -- initial data, one row in each partitions
+ INSERT INTO test_part_b VALUES (1, 1);
+ INSERT INTO test_part_b VALUES (6, 2);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub7 FOR TABLE test_part_b (a, b) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub7
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_b VALUES (2, 3);
+ INSERT INTO test_part_b VALUES (7, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_b ORDER BY a, b"),
+ qq(1|1
+2|3
+6|2
+7|4),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: This time start with a column list covering RI for all partitions,
+# but then update RI for one of the partitions to not be covered by the
+# column list anymore.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+
+ -- initial data, one row for each partition
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+));
+
+# create a publication replicating data through partition root, with a column
+# list on the root, and then add the partitions one by one with separate
+# column lists (but those are not applied)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a,c);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SUBSCRIPTION sub1;
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub8;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_c VALUES (3, 7, 8);
+ INSERT INTO test_part_c VALUES (4, 9, 10);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||5
+2|4|
+3||8
+4|9|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# create a publication not replicating data through partition root, without
+# a column list on the root, and then add the partitions one by one with
+# separate column lists
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub8;
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ TRUNCATE test_part_c;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ TRUNCATE test_part_c;
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||
+2|4|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: Start with a single partition, with RI compatible with the column
+# list, and then attach a partition with incompatible RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ INSERT INTO test_part_d VALUES (1, 2);
+));
+
+# do the same thing on the subscriber (in fact, create both partitions right
+# away, no need to delay that)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ CREATE TABLE test_part_d_2 PARTITION OF test_part_d FOR VALUES IN (2,4);
+ ALTER TABLE test_part_d_2 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_2 REPLICA IDENTITY USING INDEX test_part_d_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub9 FOR TABLE test_part_d (a) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub9
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_d VALUES (3, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_d ORDER BY a, b"),
+ qq(1|
+3|),
+ 'partitions with different replica identities not replicated correctly');
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. So with column lists (a,b) and (a,c) we
+# should replicate (a,b,c).
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+
+ -- initial data
+ INSERT INTO test_mix_1 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_1 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_1 ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES, we should replicate all columns.
+
+# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ test_part, test_part_a, test_part_b, test_part_c, test_part_d;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
+
+ -- initial data
+ INSERT INTO test_mix_2 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_3, pub_mix_4;
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_2 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_2"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES IN SCHEMA, we should replicate all columns.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
+
+ -- initial data
+ INSERT INTO test_mix_3 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_5, pub_mix_6;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_3 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_3"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: Check handling of publish_via_partition_root - if a partition is
+# published through partition root, we should only apply the column list
+# defined for the whole table (not the partitions) - both during the initial
+# sync and when replicating changes. This is what we do for row filters.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+
+ CREATE PUBLICATION pub_root_true FOR TABLE test_root (a) WITH (publish_via_partition_root = true);
+
+ -- initial data
+ INSERT INTO test_root VALUES (1, 2, 3);
+ INSERT INTO test_root VALUES (10, 20, 30);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_root_true;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_root VALUES (2, 3, 4);
+ INSERT INTO test_root VALUES (11, 21, 31);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_root ORDER BY a, b, c"),
+ qq(1||
+2||
+10||
+11||),
+ 'publication via partition root applies column list');
+
+
+# TEST: Multiple publications which publish schema of parent table and
+# partition. The partition is published through two publications, once
+# through a schema (so no column list) containing the parent, and then
+# also directly (with a columns list). The expected outcome is there is
+# no column list.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub1, pub2, pub3, pub4, pub5, pub6, pub7, pub8;
+
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA s1;
+ CREATE PUBLICATION pub2 FOR TABLE t_1(b);
+
+ -- initial data
+ INSERT INTO s1.t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub1, pub2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'two publications, publishing the same relation');
+
+# Now resync the subcription, but with publications in the opposite order.
+# The result should be the same.
+
+$node_subscriber->safe_psql('postgres', qq(
+ TRUNCATE s1.t;
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub1;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (7, 8, 9);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(7|8|9),
+ 'two publications, publishing the same relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub3 FOR TABLE t_1 (a), t_2
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub3;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+# Note: The difference from the preceding test is that in this case both
+# relations have a column list defined.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub4 FOR TABLE t_1 (a), t_2 (b)
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
+
+done_testing();
--
2.34.1
Fix a compiler warning reported by cfbot.
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
0001-Allow-specifying-column-lists-for-logical--20220318b.patchtext/x-patch; charset=UTF-8; name=0001-Allow-specifying-column-lists-for-logical--20220318b.patchDownload
From e7c357ea43868e2ce983c243c98b83b1121a5e6a Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Thu, 17 Mar 2022 19:16:39 +0100
Subject: [PATCH] Allow specifying column lists for logical replication
This allows specifying an optional column list when adding a table to
logical replication. Columns not included on this list are not sent to
the subscriber. The list is specified after the table name, enclosed
in parentheses.
For UPDATE/DELETE publications, the column list needs to cover all
REPLICA IDENTITY columns. For INSERT publications, the column list is
arbitrary and may omit some REPLICA IDENTITY columns. Furthermore, if
the table uses REPLICA IDENTITY FULL, column list is not allowed.
The column list can contain only simple column references. Complex
expressions, function calls etc. are not allowed. This restriction could
be relaxed in the future.
During the initial table synchronization, only columns specified in the
column list are copied to the subscriber. If the subscription has
several publications, containing the same table with different column
lists, columns specified in any of the lists will be copied. This
means all columns are replicated if the table has no column list at
all (which is treated as column list with all columns), of when of the
publications is defined as FOR ALL TABLES (possibly IN SCHEMA for the
schema of the table).
For partitioned tables, publish_via_partition_root determines whether
the column list for the root or leaf relation will be used. If the
parameter is 'false' (the default), the list defined for the leaf
relation is used. Otherwise, the column list for the root partition
will be used.
Psql commands \dRp+ and \d <table-name> now display any column lists.
Author: Tomas Vondra, Rahila Syed
Reviewed-by: Peter Eisentraut, Alvaro Herrera, Vignesh C, Ibrar Ahmed,
Amit Kapila, Hou zj, Peter Smith, Wang wei, Tang, Shi yu
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 15 +-
doc/src/sgml/protocol.sgml | 3 +-
doc/src/sgml/ref/alter_publication.sgml | 18 +-
doc/src/sgml/ref/create_publication.sgml | 17 +-
src/backend/catalog/pg_publication.c | 221 ++++
src/backend/commands/publicationcmds.c | 272 ++++-
src/backend/executor/execReplication.c | 19 +-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 33 +-
src/backend/replication/logical/proto.c | 61 +-
src/backend/replication/logical/tablesync.c | 156 ++-
src/backend/replication/pgoutput/pgoutput.c | 202 +++-
src/backend/utils/cache/relcache.c | 33 +-
src/bin/pg_dump/pg_dump.c | 47 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 60 +
src/bin/psql/describe.c | 40 +-
src/include/catalog/pg_publication.h | 14 +
src/include/catalog/pg_publication_rel.h | 1 +
src/include/commands/publicationcmds.h | 4 +-
src/include/nodes/parsenodes.h | 1 +
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 372 ++++++
src/test/regress/sql/publication.sql | 287 +++++
src/test/subscription/t/030_column_list.pl | 1124 +++++++++++++++++++
26 files changed, 2915 insertions(+), 94 deletions(-)
create mode 100644 src/test/subscription/t/030_column_list.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 4dc5b34d21c..89827c373bd 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -4410,7 +4410,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para>
<para>
This is an array of <structfield>indnatts</structfield> values that
- indicate which table columns this index indexes. For example a value
+ indicate which table columns this index indexes. For example, a value
of <literal>1 3</literal> would mean that the first and the third table
columns make up the index entries. Key columns come before non-key
(included) columns. A zero in this array indicates that the
@@ -6281,6 +6281,19 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to schema
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>prattrs</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+ </para>
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example, a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all columns are published.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 9178c779ba9..fb491e9ebee 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7006,7 +7006,8 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column included in
+ the publication (except generated columns):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 32b75f6c78e..9e9fc19df71 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -30,7 +30,7 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -112,6 +112,14 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ </para>
+
+ <para>
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
If the optional <literal>WHERE</literal> clause is specified, rows for
which the <replaceable class="parameter">expression</replaceable>
evaluates to false or null will not be published. Note that parentheses
@@ -174,7 +182,13 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
<para>
Add some tables to the publication:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
+<programlisting>
+ALTER PUBLICATION mypublication SET TABLE users (user_id, firstname, lastname), TABLE departments;
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 4979b9b646d..fb2d013393b 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -86,6 +86,13 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<literal>TRUNCATE</literal> commands.
</para>
+ <para>
+ When a column list is specified, only the named columns are replicated.
+ If no column list is specified, all columns of the table are replicated
+ through this publication, including any columns added later. If a column
+ list is specified, it must include the replica identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
@@ -327,6 +334,14 @@ CREATE PUBLICATION production_publication FOR TABLE users, departments, ALL TABL
<structname>sales</structname>:
<programlisting>
CREATE PUBLICATION sales_publication FOR ALL TABLES IN SCHEMA marketing, sales;
+</programlisting></para>
+
+ <para>
+ Create a publication that publishes all changes for table <structname>users</structname>,
+ but replicates only columns <structname>user_id</structname> and
+ <structname>firstname</structname>:
+<programlisting>
+CREATE PUBLICATION users_filtered FOR TABLE users (user_id, firstname);
</programlisting></para>
</refsect1>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 789b895db89..54ea8a4cccb 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,6 +45,9 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+static void publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs);
+
/*
* Check if relation can be in given publication and throws appropriate
* error if not.
@@ -345,6 +348,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
Oid relid = RelationGetRelid(targetrel);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ AttrNumber *attarray;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -372,6 +377,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
check_publication_add_relation(targetrel);
+ /*
+ * Translate column names to attnums and make sure the column list contains
+ * only allowed elements (no system or generated columns etc.). Also build
+ * an array of attnums, for storing in the catalog.
+ */
+ publication_translate_columns(pri->relation, pri->columns,
+ &natts, &attarray);
+
/* Form a tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -390,6 +403,12 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
else
nulls[Anum_pg_publication_rel_prqual - 1] = true;
+ /* Add column list, if available */
+ if (pri->columns)
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(buildint2vector(attarray, natts));
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -413,6 +432,13 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
DEPENDENCY_NORMAL, DEPENDENCY_NORMAL,
false);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL);
+ }
+
/* Close the table. */
table_close(rel, RowExclusiveLock);
@@ -432,6 +458,125 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
return myself;
}
+/* qsort comparator for attnums */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list (no system or generated attributes,
+ * no duplicates). Additional checks with replica identity are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ */
+static void
+publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs)
+{
+ AttrNumber *attarray = NULL;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
+
+ /* Bail out when no column list defined. */
+ if (!columns)
+ return;
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference generated column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /* Be tidy, so that the catalog representation is always sorted */
+ qsort(attarray, n, sizeof(AttrNumber), compare_int16);
+
+ *natts = n;
+ *attrs = attarray;
+
+ bms_free(set);
+}
+
+/*
+ * Transform the column list (represented by an array) to a bitmapset.
+ */
+Bitmapset *
+pub_collist_to_bitmapset(Bitmapset *columns, Datum pubcols, MemoryContext mcxt)
+{
+ Bitmapset *result = NULL;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ MemoryContext oldcxt;
+
+ /*
+ * If an existing bitmap was provided, use it. Otherwise just use NULL
+ * and build a new bitmap.
+ */
+ if (columns)
+ result = columns;
+
+ arr = DatumGetArrayTypeP(pubcols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* If a memory context was specified, switch to it. */
+ if (mcxt)
+ oldcxt = MemoryContextSwitchTo(mcxt);
+
+ for (int i = 0; i < nelems; i++)
+ result = bms_add_member(result, elems[i]);
+
+ if (mcxt)
+ MemoryContextSwitchTo(oldcxt);
+
+ return result;
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -539,6 +684,82 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+ Form_pg_publication_rel pubrel;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ /* no column list for this publications/relation */
+ if (isnull)
+ continue;
+
+ pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
+
+ pubs = lappend_oid(pubs, pubrel->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 1aad2e769cb..0c9993a155b 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -296,7 +296,7 @@ contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
* Returns true if any invalid column is found.
*/
bool
-contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
+pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
bool pubviaroot)
{
HeapTuple rftuple;
@@ -368,6 +368,114 @@ contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
return result;
}
+/*
+ * Check if all columns referenced in the REPLICA IDENTITY are covered by
+ * the column list.
+ *
+ * Returns true if any replica identity column is not covered by column list.
+ */
+bool
+pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
+ bool pubviaroot)
+{
+ HeapTuple tuple;
+ Oid relid = RelationGetRelid(relation);
+ Oid publish_as_relid = RelationGetRelid(relation);
+ bool result = false;
+ Datum datum;
+ bool isnull;
+
+ /*
+ * For a partition, if pubviaroot is true, find the topmost ancestor that
+ * is published via this publication as we need to use its column list
+ * for the changes.
+ *
+ * Note that even though the column list used is for an ancestor, the
+ * REPLICA IDENTITY used will be for the actual child table.
+ */
+ if (pubviaroot && relation->rd_rel->relispartition)
+ {
+ publish_as_relid = GetTopMostAncestorInPublication(pubid, ancestors, NULL);
+
+ if (!OidIsValid(publish_as_relid))
+ publish_as_relid = relid;
+ }
+
+ tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tuple))
+ return false;
+
+ datum = SysCacheGetAttr(PUBLICATIONRELMAP, tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ if (!isnull)
+ {
+ int x;
+ Bitmapset *idattrs;
+ Bitmapset *columns = NULL;
+
+ /* With REPLICA IDENTITY FULL, no column list is allowed. */
+ if (relation->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ result = true;
+
+ /* Transform the column list datum to a bitmapset. */
+ columns = pub_collist_to_bitmapset(NULL, datum, NULL);
+
+ /* Remember columns that are part of the REPLICA IDENTITY */
+ idattrs = RelationGetIndexAttrBitmap(relation,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * Attnums in the bitmap returned by RelationGetIndexAttrBitmap are
+ * offset (to handle system columns the usual way), while column list
+ * does not use offset, so we can't do bms_is_subset(). Instead, we have
+ * to loop over the idattrs and check all of them are in the list.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ AttrNumber attnum = (x + FirstLowInvalidHeapAttributeNumber);
+
+ /*
+ * If pubviaroot is true, we are validating the column list of the
+ * parent table, but the bitmap contains the replica identity
+ * information of the child table. The parent/child attnums may not
+ * match, so translate them to the parent - get the attname from
+ * the child, and look it up in the parent.
+ */
+ if (pubviaroot)
+ {
+ /* attribute name in the child table */
+ char *colname = get_attname(relid, attnum, false);
+
+ /*
+ * Determine the attnum for the attribute name in parent (we
+ * are using the column list defined on the parent).
+ */
+ attnum = get_attnum(publish_as_relid, colname);
+ }
+
+ /* replica identity column, not covered by the column list */
+ if (!bms_is_member(attnum, columns))
+ {
+ result = true;
+ break;
+ }
+ }
+
+ bms_free(idattrs);
+ bms_free(columns);
+ }
+
+ ReleaseSysCache(tuple);
+
+ return result;
+}
+
/* check_functions_in_node callback */
static bool
contain_mutable_or_user_functions_checker(Oid func_id, void *context)
@@ -609,6 +717,45 @@ TransformPubWhereClauses(List *tables, const char *queryString,
}
}
+
+/*
+ * Transform the publication column lists expression for all the relations
+ * in the list.
+ *
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow column lists on the leaf relations. So maybe rename it?
+ */
+static void
+TransformPubColumnList(List *tables, const char *queryString,
+ bool pubviaroot)
+{
+ ListCell *lc;
+
+ foreach(lc, tables)
+ {
+ PublicationRelInfo *pri = (PublicationRelInfo *) lfirst(lc);
+
+ if (pri->columns == NIL)
+ continue;
+
+ /*
+ * If the publication doesn't publish changes via the root partitioned
+ * table, the partition's column list will be used. So disallow using
+ * the column list on partitioned table in this case.
+ */
+ if (!pubviaroot &&
+ pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot use publication column list for relation \"%s\"",
+ RelationGetRelationName(pri->relation)),
+ errdetail("column list cannot be used for a partitioned table when %s is false.",
+ "publish_via_partition_root")));
+ }
+}
+
/*
* Create new publication.
*/
@@ -725,6 +872,9 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
TransformPubWhereClauses(rels, pstate->p_sourcetext,
publish_via_partition_root);
+ TransformPubColumnList(rels, pstate->p_sourcetext,
+ publish_via_partition_root);
+
PublicationAddTables(puboid, rels, true, NULL);
CloseTableList(rels);
}
@@ -784,8 +934,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* If the publication doesn't publish changes via the root partitioned
- * table, the partition's row filter will be used. So disallow using WHERE
- * clause on partitioned table in this case.
+ * table, the partition's row filter and column list will be used. So disallow
+ * using WHERE clause and column lists on partitioned table in this case.
*/
if (!pubform->puballtables && publish_via_partition_root_given &&
!publish_via_partition_root)
@@ -793,7 +943,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* Lock the publication so nobody else can do anything with it. This
* prevents concurrent alter to add partitioned table(s) with WHERE
- * clause(s) which we don't allow when not publishing via root.
+ * clause(s) and/or column lists which we don't allow when not
+ * publishing via root.
*/
LockDatabaseObject(PublicationRelationId, pubform->oid, 0,
AccessShareLock);
@@ -805,13 +956,21 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
HeapTuple rftuple;
Oid relid = lfirst_oid(lc);
+ bool has_column_list;
+ bool has_row_filter;
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubform->oid));
+ has_row_filter
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL);
+
+ has_column_list
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prattrs, NULL);
+
if (HeapTupleIsValid(rftuple) &&
- !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL))
+ (has_row_filter || has_column_list))
{
HeapTuple tuple;
@@ -820,7 +979,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
Form_pg_class relform = (Form_pg_class) GETSTRUCT(tuple);
- if (relform->relkind == RELKIND_PARTITIONED_TABLE)
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_row_filter)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot set %s for publication \"%s\"",
@@ -831,6 +991,18 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
NameStr(relform->relname),
"publish_via_partition_root")));
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_column_list)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot set %s for publication \"%s\"",
+ "publish_via_partition_root = false",
+ stmt->pubname),
+ errdetail("The publication contains a column list for a partitioned table \"%s\" "
+ "which is not allowed when %s is false.",
+ NameStr(relform->relname),
+ "publish_via_partition_root")));
+
ReleaseSysCache(tuple);
}
@@ -976,6 +1148,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
PublicationAddTables(pubid, rels, false, stmt);
}
else if (stmt->action == AP_DropObjects)
@@ -992,6 +1166,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
/*
* To recreate the relation list for the publication, look for
* existing relations that do not need to be dropped.
@@ -1003,42 +1179,79 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
PublicationRelInfo *oldrel;
bool found = false;
HeapTuple rftuple;
- bool rfisnull = true;
Node *oldrelwhereclause = NULL;
+ Bitmapset *oldcolumns = NULL;
/* look up the cache for the old relmap */
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(oldrelid),
ObjectIdGetDatum(pubid));
+ /*
+ * See if the existing relation currently has a WHERE clause or a
+ * column list. We need to compare those too.
+ */
if (HeapTupleIsValid(rftuple))
{
+ bool isnull = true;
Datum whereClauseDatum;
+ Datum columnListDatum;
+ /* Load the WHERE clause for this table. */
whereClauseDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
Anum_pg_publication_rel_prqual,
- &rfisnull);
- if (!rfisnull)
+ &isnull);
+ if (!isnull)
oldrelwhereclause = stringToNode(TextDatumGetCString(whereClauseDatum));
+ /* Transform the int2vector column list to a bitmap. */
+ columnListDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ if (!isnull)
+ oldcolumns = pub_collist_to_bitmapset(NULL, columnListDatum, NULL);
+
ReleaseSysCache(rftuple);
}
foreach(newlc, rels)
{
PublicationRelInfo *newpubrel;
+ Oid newrelid;
+ Bitmapset *newcolumns = NULL;
newpubrel = (PublicationRelInfo *) lfirst(newlc);
+ newrelid = RelationGetRelid(newpubrel->relation);
+
+ /*
+ * If the new publication has column list, transform it to
+ * a bitmap too.
+ */
+ if (newpubrel->columns)
+ {
+ ListCell *lc;
+
+ foreach(lc, newpubrel->columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(newrelid, colname);
+
+ newcolumns = bms_add_member(newcolumns, attnum);
+ }
+ }
/*
* Check if any of the new set of relations matches with the
* existing relations in the publication. Additionally, if the
* relation has an associated WHERE clause, check the WHERE
- * expressions also match. Drop the rest.
+ * expressions also match. Same for the column list. Drop the
+ * rest.
*/
if (RelationGetRelid(newpubrel->relation) == oldrelid)
{
- if (equal(oldrelwhereclause, newpubrel->whereClause))
+ if (equal(oldrelwhereclause, newpubrel->whereClause) &&
+ bms_equal(oldcolumns, newcolumns))
{
found = true;
break;
@@ -1057,6 +1270,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
{
oldrel = palloc(sizeof(PublicationRelInfo));
oldrel->whereClause = NULL;
+ oldrel->columns = NIL;
oldrel->relation = table_open(oldrelid,
ShareUpdateExclusiveLock);
delrels = lappend(delrels, oldrel);
@@ -1118,7 +1332,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
}
else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* AP_SetObjects */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -1403,6 +1617,7 @@ OpenTableList(List *tables)
List *rels = NIL;
ListCell *lc;
List *relids_with_rf = NIL;
+ List *relids_with_collist = NIL;
/*
* Open, share-lock, and check all the explicitly-specified relations
@@ -1437,6 +1652,13 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /* Disallow duplicate tables if there are any with column lists. */
+ if (t->columns || list_member_oid(relids_with_collist, myrelid))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
table_close(rel, ShareUpdateExclusiveLock);
continue;
}
@@ -1444,12 +1666,16 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
pub_rel->whereClause = t->whereClause;
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, myrelid);
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, myrelid);
+
/*
* Add children of this rel, if requested, so that they too are added
* to the publication. A partitioned table can't have any inheritance
@@ -1489,6 +1715,18 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /*
+ * We don't allow to specify column list for both parent
+ * and child table at the same time as it is not very
+ * clear which one should be given preference.
+ */
+ if (childrelid != myrelid &&
+ (t->columns || list_member_oid(relids_with_collist, childrelid)))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
continue;
}
@@ -1498,11 +1736,16 @@ OpenTableList(List *tables)
pub_rel->relation = rel;
/* child inherits WHERE clause from parent */
pub_rel->whereClause = t->whereClause;
+ /* child inherits column list from parent */
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, childrelid);
+
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, childrelid);
}
}
}
@@ -1611,6 +1854,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 09f78f22441..3e282ed99ab 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -573,9 +573,6 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
if (cmd != CMD_UPDATE && cmd != CMD_DELETE)
return;
- if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
- return;
-
/*
* It is only safe to execute UPDATE/DELETE when all columns, referenced
* in the row filters from publications which the relation is in, are
@@ -595,17 +592,33 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
errmsg("cannot update table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_UPDATE && !pubdesc.cols_valid_for_update)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot update table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
else if (cmd == CMD_DELETE && !pubdesc.rf_valid_for_delete)
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
errmsg("cannot delete from table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_DELETE && !pubdesc.cols_valid_for_delete)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot delete from table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
/* If relation has replica identity we are always good. */
if (OidIsValid(RelationGetReplicaIndex(rel)))
return;
+ /* REPLICA IDENTITY FULL is also good for UPDATE/DELETE. */
+ if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ return;
+
/*
* This is UPDATE/DELETE and there is no replica identity.
*
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index d4f8455a2bd..a504437873f 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4850,6 +4850,7 @@ _copyPublicationTable(const PublicationTable *from)
COPY_NODE_FIELD(relation);
COPY_NODE_FIELD(whereClause);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f1002afe7a0..4fc16ce04e3 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2322,6 +2322,7 @@ _equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
COMPARE_NODE_FIELD(whereClause);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a03b33b53bd..ff4573390c5 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9751,13 +9751,14 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr OptWhereClause
+ TABLE relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9772,11 +9773,15 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId OptWhereClause
+ | ColId opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- if ($2)
+ /*
+ * If either a row filter or column list is specified, create
+ * a PublicationTable object.
+ */
+ if ($2 || $3)
{
/*
* The OptWhereClause must be stored here but it is
@@ -9786,7 +9791,8 @@ PublicationObjSpec:
*/
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVar(NULL, $1, @1);
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
else
{
@@ -9794,23 +9800,25 @@ PublicationObjSpec:
}
$$->location = @1;
}
- | ColId indirection OptWhereClause
+ | ColId indirection opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr OptWhereClause
+ | extended_relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
| CURRENT_SCHEMA
{
@@ -17488,6 +17496,13 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
errmsg("WHERE clause not allowed for schema"),
parser_errposition(pubobj->location));
+ /* Column list is not allowed on a schema object */
+ if (pubobj->pubtable && pubobj->pubtable->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schema"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index c9b0eeefd7e..f9de1d16dc2 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,16 +29,30 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary);
+ bool binary, Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
static void logicalrep_write_namespace(StringInfo out, Oid nspid);
static const char *logicalrep_read_namespace(StringInfo in);
+/*
+ * Check if a column is covered by a column list.
+ *
+ * Need to be careful about NULL, which is treated as a column list covering
+ * all columns.
+ */
+static bool
+column_in_column_list(int attnum, Bitmapset *columns)
+{
+ return (columns == NULL || bms_is_member(attnum, columns));
+}
+
+
/*
* Write BEGIN to the output stream.
*/
@@ -398,7 +412,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +424,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -443,7 +457,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -464,11 +478,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -537,7 +551,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, NULL);
}
/*
@@ -652,7 +666,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -674,7 +689,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -751,7 +766,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum *values;
@@ -763,8 +778,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ if (!column_in_column_list(att->attnum, columns))
continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -783,6 +804,9 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_column_list(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +928,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -917,8 +941,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* send number of live attributes */
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
continue;
+
+ if (!column_in_column_list(att->attnum, columns))
+ continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -937,6 +967,9 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_column_list(att->attnum, columns))
+ continue;
+
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 1659964571c..caeab853e4c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -112,6 +112,7 @@
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "utils/acl.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -701,12 +702,13 @@ fetch_remote_table_info(char *nspname, char *relname,
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
int natt;
ListCell *lc;
bool first;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
@@ -747,10 +749,110 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+
+ /*
+ * Get column lists for each relation.
+ *
+ * For initial synchronization, column lists can be ignored in following
+ * cases:
+ *
+ * 1) one of the subscribed publications for the table hasn't specified
+ * any column list
+ *
+ * 2) one of the subscribed publications has puballtables set to true
+ *
+ * 3) one of the subscribed publications is declared as ALL TABLES IN
+ * SCHEMA that includes this relation
+ *
+ * We need to do this before fetching info about column names and types,
+ * so that we can skip columns that should not be replicated.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData pub_names;
+ bool first = true;
+
+ initStringInfo(&pub_names);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&pub_names, ", ");
+ appendStringInfoString(&pub_names, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ /*
+ * Fetch info about column lists for the relation (from all the
+ * publications). We unnest the int2vector values, because that
+ * makes it easier to combine lists by simply adding the attnums
+ * to a new bitmap (without having to parse the int2vector data).
+ * This preserves NULL values, so that if one of the publications
+ * has no column list, we'll know that.
+ */
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT DISTINCT unnest"
+ " FROM pg_publication p"
+ " LEFT OUTER JOIN pg_publication_rel pr"
+ " ON (p.oid = pr.prpubid AND pr.prrelid = %u)"
+ " LEFT OUTER JOIN unnest(pr.prattrs) ON TRUE,"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt"
+ " WHERE gpt.relid = %u"
+ " AND p.pubname IN ( %s )",
+ lrel->remoteid,
+ lrel->remoteid,
+ pub_names.data);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch column list info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ /*
+ * Merge the column lists (from different publications) by creating
+ * a single bitmap with all the attnums. If we find a NULL value,
+ * that means one of the publications has no column list for the
+ * table we're syncing.
+ */
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ Datum cfval = slot_getattr(slot, 1, &isnull);
+
+ /* NULL means empty column list, so we're done. */
+ if (isnull)
+ {
+ bms_free(included_cols);
+ included_cols = NULL;
+ break;
+ }
+
+ included_cols = bms_add_member(included_cols,
+ DatumGetInt16(cfval));
+
+ ExecClearTuple(slot);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+
+ walrcv_clear_result(pubres);
+
+ pfree(pub_names.data);
+ }
+
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -778,16 +880,35 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * Store the columns as a list of names. Ignore those that are not
+ * present in the column list, if there is one.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ /* If the column is not in the column list, skip it. */
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ {
+ ExecClearTuple(slot);
+ continue;
+ }
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -821,6 +942,9 @@ fetch_remote_table_info(char *nspname, char *relname,
*
* 3) one of the subscribed publications is declared as ALL TABLES IN
* SCHEMA that includes this relation
+ *
+ * XXX Does this actually handle puballtables and schema publications
+ * correctly?
*/
if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
{
@@ -930,8 +1054,24 @@ copy_table(Relation rel)
/* Regular table with no row filter */
if (lrel.relkind == RELKIND_RELATION && qual == NIL)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+
+ /*
+ * XXX Do we need to list the columns in all cases? Maybe we're replicating
+ * all columns?
+ */
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
+
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
+
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 5fddab3a3d4..f5e7610a172 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -29,6 +29,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -85,7 +86,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -143,9 +145,6 @@ typedef struct RelationSyncEntry
*/
ExprState *exprstate[NUM_ROWFILTER_PUBACTIONS];
EState *estate; /* executor state used for row filter */
- MemoryContext cache_expr_cxt; /* private context for exprstate and
- * estate, if any */
-
TupleTableSlot *new_slot; /* slot for storing new tuple */
TupleTableSlot *old_slot; /* slot for storing old tuple */
@@ -164,6 +163,19 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
AttrMap *attrmap;
+
+ /*
+ * Columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this bitmap are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
+
+ /*
+ * Private context to store additional data for this entry - state for
+ * the row filter expressions, column list, etc.
+ */
+ MemoryContext entry_cxt;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -188,6 +200,7 @@ static EState *create_estate_for_relation(Relation rel);
static void pgoutput_row_filter_init(PGOutputData *data,
List *publications,
RelationSyncEntry *entry);
+
static bool pgoutput_row_filter_exec_expr(ExprState *state,
ExprContext *econtext);
static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
@@ -195,6 +208,11 @@ static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
RelationSyncEntry *entry,
ReorderBufferChangeType *action);
+/* column list routines */
+static void pgoutput_column_list_init(PGOutputData *data,
+ List *publications,
+ RelationSyncEntry *entry);
+
/*
* Specify output plugin callbacks
*/
@@ -603,11 +621,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -620,7 +638,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -643,13 +662,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip this attribute if it's not present in the column list */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -703,6 +726,28 @@ pgoutput_row_filter_exec_expr(ExprState *state, ExprContext *econtext)
return DatumGetBool(ret);
}
+/*
+ * Make sure the per-entry memory context exists.
+ */
+static void
+pgoutput_ensure_entry_cxt(PGOutputData *data, RelationSyncEntry *entry)
+{
+ Relation relation;
+
+ /* The context may already exist, in which case bail out. */
+ if (entry->entry_cxt)
+ return;
+
+ relation = RelationIdGetRelation(entry->publish_as_relid);
+
+ entry->entry_cxt = AllocSetContextCreate(data->cachectx,
+ "entry private context",
+ ALLOCSET_SMALL_SIZES);
+
+ MemoryContextCopyAndSetIdentifier(entry->entry_cxt,
+ RelationGetRelationName(relation));
+}
+
/*
* Initialize the row filter.
*/
@@ -823,21 +868,13 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
{
Relation relation = RelationIdGetRelation(entry->publish_as_relid);
- Assert(entry->cache_expr_cxt == NULL);
-
- /* Create the memory context for row filters */
- entry->cache_expr_cxt = AllocSetContextCreate(data->cachectx,
- "Row filter expressions",
- ALLOCSET_DEFAULT_SIZES);
-
- MemoryContextCopyAndSetIdentifier(entry->cache_expr_cxt,
- RelationGetRelationName(relation));
+ pgoutput_ensure_entry_cxt(data, entry);
/*
* Now all the filters for all pubactions are known. Combine them when
* their pubactions are the same.
*/
- oldctx = MemoryContextSwitchTo(entry->cache_expr_cxt);
+ oldctx = MemoryContextSwitchTo(entry->entry_cxt);
entry->estate = create_estate_for_relation(relation);
for (idx = 0; idx < NUM_ROWFILTER_PUBACTIONS; idx++)
{
@@ -860,6 +897,105 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
}
}
+/*
+ * Initialize the column list.
+ */
+static void
+pgoutput_column_list_init(PGOutputData *data, List *publications,
+ RelationSyncEntry *entry)
+{
+ ListCell *lc;
+
+ /*
+ * Find if there are any column lists for this relation. If there are,
+ * build a bitmap merging all the column lists.
+ *
+ * All the given publication-table mappings must be checked.
+ *
+ * Multiple publications might have multiple column lists for this relation.
+ *
+ * FOR ALL TABLES and FOR ALL TABLES IN SCHEMA implies "don't use column
+ * list" so it takes precedence.
+ */
+ foreach(lc, publications)
+ {
+ Publication *pub = lfirst(lc);
+ HeapTuple cftuple = NULL;
+ Datum cfdatum = 0;
+
+ /*
+ * Assume there's no column list. Only if we find pg_publication_rel
+ * entry with a column list we'll switch it to false.
+ */
+ bool pub_no_list = true;
+
+ /*
+ * If the publication is FOR ALL TABLES then it is treated the same as if
+ * there are no column lists (even if other publications have a list).
+ */
+ if (!pub->alltables)
+ {
+ /*
+ * Check for the presence of a column list in this publication.
+ *
+ * Note: If we find no pg_publication_rel row, it's a publication
+ * defined for a whole schema, so it can't have a column list, just
+ * like a FOR ALL TABLES publication.
+ */
+ cftuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(entry->publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(cftuple))
+ {
+ /*
+ * Lookup the column list attribute.
+ *
+ * Note: We update the pub_no_list value directly, because if
+ * the value is NULL, we have no list (and vice versa).
+ */
+ cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
+ Anum_pg_publication_rel_prattrs,
+ &pub_no_list);
+
+ /*
+ * Build the column list bitmap in the per-entry context.
+ *
+ * We need to merge column lists from all publications, so we
+ * update the same bitmapset. If the column list is null, we
+ * interpret it as replicating all columns.
+ */
+ if (!pub_no_list) /* when not null */
+ {
+ pgoutput_ensure_entry_cxt(data, entry);
+
+ entry->columns = pub_collist_to_bitmapset(entry->columns,
+ cfdatum,
+ entry->entry_cxt);
+ }
+ }
+ }
+
+ /*
+ * Found a publication with no column list, so we're done. But first
+ * discard column list we might have from preceding publications.
+ */
+ if (pub_no_list)
+ {
+ if (cftuple)
+ ReleaseSysCache(cftuple);
+
+ bms_free(entry->columns);
+ entry->columns = NULL;
+
+ break;
+ }
+
+ ReleaseSysCache(cftuple);
+ } /* loop all subscribed publications */
+
+}
+
/*
* Initialize the slot for storing new and old tuples, and build the map that
* will be used to convert the relation's tuples into the ancestor's format.
@@ -1224,7 +1360,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -1278,11 +1414,13 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel,
- new_slot, data->binary);
+ new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel,
- old_slot, new_slot, data->binary);
+ old_slot, new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel,
@@ -1729,8 +1867,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->new_slot = NULL;
entry->old_slot = NULL;
memset(entry->exprstate, 0, sizeof(entry->exprstate));
- entry->cache_expr_cxt = NULL;
+ entry->entry_cxt = NULL;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->attrmap = NULL;
}
@@ -1776,6 +1915,8 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
entry->pubactions.pubinsert = false;
entry->pubactions.pubupdate = false;
entry->pubactions.pubdelete = false;
@@ -1799,17 +1940,18 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/*
* Row filter cache cleanups.
*/
- if (entry->cache_expr_cxt)
- MemoryContextDelete(entry->cache_expr_cxt);
+ if (entry->entry_cxt)
+ MemoryContextDelete(entry->entry_cxt);
- entry->cache_expr_cxt = NULL;
+ entry->entry_cxt = NULL;
entry->estate = NULL;
memset(entry->exprstate, 0, sizeof(entry->exprstate));
/*
* Build publication cache. We can't use one provided by relcache as
- * relcache considers all publications given relation is in, but here
- * we only need to consider ones that the subscriber requested.
+ * relcache considers all publications that the given relation is in,
+ * but here we only need to consider ones that the subscriber
+ * requested.
*/
foreach(lc, data->publications)
{
@@ -1878,6 +2020,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
}
/*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
* Don't publish changes for partitioned tables, because
* publishing those of its partitions suffices, unless partition
* changes won't be published due to pubviaroot being set.
@@ -1938,6 +2083,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/* Initialize the row filter */
pgoutput_row_filter_init(data, rel_publications, entry);
+
+ /* Initialize the column list */
+ pgoutput_column_list_init(data, rel_publications, entry);
}
list_free(pubids);
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index fccffce5729..a2da72f0d48 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -5553,6 +5553,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
return;
}
@@ -5565,6 +5567,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
/* Fetch the publication membership info. */
puboids = GetRelationPublications(relid);
@@ -5616,7 +5620,7 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
*/
if (!pubform->puballtables &&
(pubform->pubupdate || pubform->pubdelete) &&
- contain_invalid_rfcolumn(pubid, relation, ancestors,
+ pub_rf_contains_invalid_column(pubid, relation, ancestors,
pubform->pubviaroot))
{
if (pubform->pubupdate)
@@ -5625,6 +5629,23 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->rf_valid_for_delete = false;
}
+ /*
+ * Check if all columns are part of the REPLICA IDENTITY index or not.
+ *
+ * If the publication is FOR ALL TABLES then it means the table has no
+ * column list and we can skip the validation.
+ */
+ if (!pubform->puballtables &&
+ (pubform->pubupdate || pubform->pubdelete) &&
+ pub_collist_contains_invalid_column(pubid, relation, ancestors,
+ pubform->pubviaroot))
+ {
+ if (pubform->pubupdate)
+ pubdesc->cols_valid_for_update = false;
+ if (pubform->pubdelete)
+ pubdesc->cols_valid_for_delete = false;
+ }
+
ReleaseSysCache(tup);
/*
@@ -5636,6 +5657,16 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
!pubdesc->rf_valid_for_update && !pubdesc->rf_valid_for_delete)
break;
+
+ /*
+ * If we know everything is replicated and the column list is invalid
+ * for update and delete, there is no point to check for other
+ * publications.
+ */
+ if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
+ pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
+ !pubdesc->cols_valid_for_update && !pubdesc->cols_valid_for_delete)
+ break;
}
if (relation->rd_pubdesc)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 725cd2e4ebc..be40acd3e37 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4101,6 +4101,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_prpubid;
int i_prrelid;
int i_prrelqual;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4114,12 +4115,20 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
if (fout->remoteVersion >= 150000)
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual "
- "FROM pg_catalog.pg_publication_rel");
+ "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual, "
+ "(CASE\n"
+ " WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT array_agg(attname)\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) prattrs "
+ "FROM pg_catalog.pg_publication_rel pr");
else
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "NULL AS prrelqual "
+ "NULL AS prrelqual, NULL AS prattrs "
"FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
@@ -4130,6 +4139,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
i_prrelqual = PQfnumber(res, "prrelqual");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4175,6 +4185,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
else
pubrinfo[j].pubrelqual = pg_strdup(PQgetvalue(res, i, i_prrelqual));
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[j].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4249,10 +4281,13 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+
if (pubrinfo->pubrelqual)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 772dc0cf7a2..1d21c2906f1 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -632,6 +632,7 @@ typedef struct _PublicationRelInfo
PublicationInfo *publication;
TableInfo *pubtable;
char *pubrelqual;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index fd1052e5db8..05a7e28bdcc 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2428,6 +2428,28 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'ALTER PUBLICATION pub1 ADD TABLE test_sixth_table (col3, col2)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_sixth_table (col3, col2);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_sixth_table (col2, col3);\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'ALTER PUBLICATION pub1 ADD TABLE test_seventh_table (col3, col2) WHERE (col1 = 1)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_seventh_table (col3, col2) WHERE (col1 = 1);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_seventh_table (col2, col3) WHERE ((col1 = 1));\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'ALTER PUBLICATION pub3 ADD ALL TABLES IN SCHEMA dump_test' => {
create_order => 51,
create_sql =>
@@ -2778,6 +2800,44 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'CREATE TABLE test_sixth_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_sixth_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_sixth_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'CREATE TABLE test_seventh_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_seventh_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_seventh_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'CREATE TABLE test_table_identity' => {
create_order => 3,
create_sql => 'CREATE TABLE dump_test.test_table_identity (
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 991bfc1546b..88bb75ac658 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2892,6 +2892,7 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_namespace pn ON p.oid = pn.pnpubid\n"
" JOIN pg_catalog.pg_class pc ON pc.relnamespace = pn.pnnspid\n"
@@ -2899,6 +2900,12 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , pg_get_expr(pr.prqual, c.oid)\n"
+ " , (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT string_agg(attname, ', ')\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) "
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
" JOIN pg_catalog.pg_class c ON c.oid = pr.prrelid\n"
@@ -2906,6 +2913,7 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2916,12 +2924,14 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
"WHERE pr.prrelid = '%s'\n"
"UNION ALL\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2943,6 +2953,11 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, " \"%s\"",
PQgetvalue(result, i, 0));
+ /* column list (if any) */
+ if (!PQgetisnull(result, i, 2))
+ appendPQExpBuffer(&buf, " (%s)",
+ PQgetvalue(result, i, 2));
+
/* row filter (if any) */
if (!PQgetisnull(result, i, 1))
appendPQExpBuffer(&buf, " WHERE %s",
@@ -5888,7 +5903,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5905,15 +5920,19 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (as_schema)
+ printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
+ else
{
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
+
+ if (!PQgetisnull(res, i, 3))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 3));
+
if (!PQgetisnull(res, i, 2))
appendPQExpBuffer(buf, " WHERE %s", PQgetvalue(res, i, 2));
}
- else
- printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
}
@@ -6042,11 +6061,22 @@ describePublications(const char *pattern)
printfPQExpBuffer(&buf,
"SELECT n.nspname, c.relname");
if (pset.sversion >= 150000)
+ {
appendPQExpBufferStr(&buf,
", pg_get_expr(pr.prqual, c.oid)");
+ appendPQExpBufferStr(&buf,
+ ", (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string("
+ " ARRAY(SELECT attname\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END)");
+ }
else
appendPQExpBufferStr(&buf,
- ", NULL");
+ ", NULL, NULL");
appendPQExpBuffer(&buf,
"\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index fe773cf9b7d..a56c1102463 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -85,6 +85,13 @@ typedef struct PublicationDesc
*/
bool rf_valid_for_update;
bool rf_valid_for_delete;
+
+ /*
+ * true if the columns are part of the replica identity or the publication actions
+ * do not include UPDATE or DELETE.
+ */
+ bool cols_valid_for_update;
+ bool cols_valid_for_delete;
} PublicationDesc;
typedef struct Publication
@@ -100,6 +107,7 @@ typedef struct PublicationRelInfo
{
Relation relation;
Node *whereClause;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -123,8 +131,11 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
+extern void GetActionsInPublication(Oid pubid, PublicationActions *actions);
extern List *GetPublicationSchemas(Oid pubid);
extern List *GetSchemaPublications(Oid schemaid);
extern List *GetSchemaPublicationRelations(Oid schemaid,
@@ -144,6 +155,9 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *pri
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern Bitmapset *pub_collist_to_bitmapset(Bitmapset *columns, Datum pubcols,
+ MemoryContext mcxt);
+
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index 0dd0f425db9..4feb581899e 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -34,6 +34,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
#ifdef CATALOG_VARLEN /* variable-length fields start here */
pg_node_tree prqual; /* qualifications */
+ int2vector prattrs; /* columns to replicate */
#endif
} FormData_pg_publication_rel;
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index 7813cbcb6bb..ae87caf089d 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -31,7 +31,9 @@ extern void RemovePublicationSchemaById(Oid psoid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
extern void AlterPublicationOwner_oid(Oid pubid, Oid newOwnerId);
extern void InvalidatePublicationRels(List *relids);
-extern bool contain_invalid_rfcolumn(Oid pubid, Relation relation,
+extern bool pub_rf_contains_invalid_column(Oid pubid, Relation relation,
+ List *ancestors, bool pubviaroot);
+extern bool pub_collist_contains_invalid_column(Oid pubid, Relation relation,
List *ancestors, bool pubviaroot);
#endif /* PUBLICATIONCMDS_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1617702d9d6..b4479c7049a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3652,6 +3652,7 @@ typedef struct PublicationTable
NodeTag type;
RangeVar *relation; /* relation to be published */
Node *whereClause; /* qualifications */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4d2c881644a..a771ab8ff33 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -209,12 +209,12 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary);
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -231,7 +231,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 4e191c120ac..227b5611915 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -613,6 +613,369 @@ DROP TABLE rf_tbl_abcd_pk;
DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+RESET client_min_messages;
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ERROR: cannot reference generated column "d" in publication column list
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+ERROR: cannot reference system column "ctid" in publication column list
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+ERROR: cannot drop column c of table testpub_tbl5 because other objects depend on it
+DETAIL: publication of table testpub_tbl5 in publication testpub_fortable depends on column c of table testpub_tbl5
+HINT: Use DROP ... CASCADE to drop the dependent objects too.
+-- ok: for insert-only publication, the column list is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: change the replica identity to "b", and column list to (a, c)
+-- then update fails, because (a, c) does not cover replica identity
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+ Publication testpub_table_ins
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | t | f
+Tables:
+ "public.testpub_tbl5" (a)
+
+-- with REPLICA IDENTITY FULL, column lists are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ERROR: cannot update table "testpub_tbl6"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+-- make sure changing the column list is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: update the column list
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, c)
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl8;
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_0"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+ Publication testpub_both_filters
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | t | t | t | f
+Tables:
+ "public.testpub_tbl_both_filters" (a, c) WHERE (c <> 1)
+
+\d+ testpub_tbl_both_filters
+ Table "public.testpub_tbl_both_filters"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+ c | integer | | not null | | plain | |
+Indexes:
+ "testpub_tbl_both_filters_pkey" PRIMARY KEY, btree (a, c) REPLICA IDENTITY
+Publications:
+ "testpub_both_filters" (a, c) WHERE (c <> 1)
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+-- Tests for partitioned table
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+ERROR: cannot use publication column list for relation "rf_tbl_abcd_part_pk"
+DETAIL: column list cannot be used for a partitioned table when publish_via_partition_root is false.
+-- ok - can use column list for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any column list is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+ERROR: cannot set publish_via_partition_root = false for publication "testpub6"
+DETAIL: The publication contains a column list for a partitioned table "rf_tbl_abcd_part_pk" which is not allowed when publish_via_partition_root is false.
+-- Now change the root column list to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have column list for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -1058,6 +1421,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schema
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 5457c56b33f..aeb1b572af8 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -373,6 +373,289 @@ DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+RESET client_min_messages;
+
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+-- ok: for insert-only publication, the column list is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+
+-- error: change the replica identity to "b", and column list to (a, c)
+-- then update fails, because (a, c) does not cover replica identity
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+UPDATE testpub_tbl5 SET a = 1;
+
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+
+-- with REPLICA IDENTITY FULL, column lists are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+
+-- make sure changing the column list is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: update the column list
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl8;
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+\d+ testpub_tbl_both_filters
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Tests for partitioned table
+
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - can use column list for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any column list is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- Now change the root column list to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have column list for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
+
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -614,6 +897,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/030_column_list.pl b/src/test/subscription/t/030_column_list.pl
new file mode 100644
index 00000000000..5ceaec83cdb
--- /dev/null
+++ b/src/test/subscription/t/030_column_list.pl
@@ -0,0 +1,1124 @@
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test partial-column publication of tables
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# create publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+sub wait_for_subscription_sync
+{
+ my ($node) = @_;
+
+ # Also wait for initial table sync to finish
+ my $synced_query = "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+
+ $node->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+}
+
+# setup tables on both nodes
+
+# tab1: simple 1:1 replication
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+# tab2: replication from regular to table with fewer columns
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)
+));
+
+# tab3: simple 1:1 replication with weird column names
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "B" varchar, "c'" int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "c'" int)
+));
+
+# test_part: partitioned tables, with partitioning (including multi-level
+# partitioning, and fewer columns on the subscriber)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+# tab4: table with user-defined enum types
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, d text);
+));
+
+
+# TEST: create publication and subscription for some of the tables with
+# column lists
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub1
+ FOR TABLE tab1 (a, "B"), tab3 ("a'", "c'"), test_part (a, b), tab4 (a, b, d)
+ WITH (publish_via_partition_root = 'true');
+));
+
+# check that we got the right prattrs values for the publication in the
+# pg_publication_rel catalog (order by relname, to get stable ordering)
+my $result = $node_publisher->safe_psql('postgres', qq(
+ SELECT relname, prattrs
+ FROM pg_publication_rel pb JOIN pg_class pc ON(pb.prrelid = pc.oid)
+ ORDER BY relname
+));
+
+is($result, qq(tab1|1 2
+tab3|1 3
+tab4|1 2 4
+test_part|1 2), 'publication relation updated');
+
+# TEST: insert data into the tables, create subscription and see if sync
+# replicates the right columns
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (1, 2, 3);
+ INSERT INTO tab1 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (1, 2, 3);
+ INSERT INTO tab3 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (1, 'red', 3, 'oh my');
+ INSERT INTO tab4 VALUES (2, 'blue', 4, 'hello');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (1, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (2, 'bcd', '2021-07-03 11:12:13');
+ INSERT INTO test_part VALUES (7, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (8, 'bcd', '2021-07-03 11:12:13');
+));
+
+# create subscription for the publication, wait for sync to complete,
+# then check the sync results
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+4|5|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+4|6), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+7|abc
+8|bcd), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: now insert more data into the tables, and wait until we replicate
+# them (not by tablesync, but regular decoding and replication)
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (2, 3, 4);
+ INSERT INTO tab1 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (2, 3, 4);
+ INSERT INTO tab3 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (3, 'red', 5, 'foo');
+ INSERT INTO tab4 VALUES (4, 'blue', 6, 'bar');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (3, 'xxx', '2022-02-01 10:00:00');
+ INSERT INTO test_part VALUES (4, 'yyy', '2022-03-02 15:12:13');
+ INSERT INTO test_part VALUES (9, 'zzz', '2022-04-03 21:00:00');
+ INSERT INTO test_part VALUES (10, 'qqq', '2022-05-04 22:12:13');
+));
+
+# wait for catchup before checking the subscriber
+$node_publisher->wait_for_catchup('sub1');
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+2|3|
+4|5|
+5|6|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+2|4
+4|6
+5|7), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello
+3|red|foo
+4|blue|bar), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+3|xxx
+4|yyy
+7|abc
+8|bcd
+9|zzz
+10|qqq), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: do some updates on some of the tables, both on columns included
+# in the column list and other
+
+# tab1: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET "B" = 2 * "B" where a = 1));
+
+# tab1: update of non-replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET c = 2*c where a = 4));
+
+# tab3: update of non-replicated
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "B" = "B" || ' updated' where "a'" = 4));
+
+# tab3: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "c'" = 2 * "c'" where "a'" = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'blue', c = c * 2, d = d || ' updated' where a = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'red', c = c * 2, d = d || ' updated' where a = 2));
+
+# wait for the replication to catch up, and check the UPDATE results got
+# replicated correctly, with the right column list
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab1 ORDER BY a));
+is($result,
+qq(1|4|
+2|3|
+4|5|
+5|6|), 'only update on column tab1.b is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result,
+qq(1|6
+2|4
+4|6
+5|7), 'only update on column tab3.c is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab4 ORDER BY a));
+
+is($result, qq(1|blue|oh my updated
+2|red|hello updated
+3|red|foo
+4|blue|bar), 'update on column tab4.c is not replicated');
+
+
+# TEST: add table with a column list, insert data, replicate
+
+# insert some data before adding it to the publication
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (1, 'abc', 3);
+));
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2 (a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION");
+
+# wait for the tablesync to complete, add a bit more data and then check
+# the results of the replication
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (2, 'def', 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|def), 'insert on column tab2.c is not replicated');
+
+# do a couple updates, check the correct stuff gets replicated
+$node_publisher->safe_psql('postgres', qq(
+ UPDATE tab2 SET c = 5 where a = 1;
+ UPDATE tab2 SET b = 'xyz' where a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|xyz), 'update on column tab2.c is not replicated');
+
+
+# TEST: add a table to two publications with different column lists, and
+# create a single subscription replicating both publications
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+
+ -- insert a couple initial records
+ INSERT INTO tab5 VALUES (1, 11, 111, 1111);
+ INSERT INTO tab5 VALUES (2, 22, 222, 2222);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub3
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->wait_for_catchup('sub1');
+
+# insert data and make sure all the columns (union of the columns lists)
+# get fully replicated
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (3, 33, 333, 3333);
+ INSERT INTO tab5 VALUES (4, 44, 444, 4444);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111
+2|22|2222
+3|33|3333
+4|44|4444),
+ 'overlapping publications with overlapping column lists');
+
+# and finally, remove the column list for one of the publications, which
+# means replicating all columns (removing the column list), but first add
+# the missing column to the table on subscriber
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub3 SET TABLE tab5;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ ALTER TABLE tab5 ADD COLUMN c INT;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (5, 55, 555, 5555);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111|
+2|22|2222|
+3|33|3333|
+4|44|4444|
+5|55|5555|555),
+ 'overlapping publications with overlapping column lists');
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key (but use a different column in
+# the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub4 FOR TABLE tab6 (a, b);
+
+ -- initial data
+ INSERT INTO tab6 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (2, 33, 444, 5555);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is still covered by the column list, though)
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+# we need to do the same thing on the subscriber
+# XXX What would happen if this happens before the publisher ALTER? Or
+# interleaved, somehow? But that seems unrelated to column lists.
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (3, 55, 666, 8888);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key with a key on multiple columns
+# (all of them covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub5 FOR TABLE tab7 (a, b);
+
+ -- some initial data
+ INSERT INTO tab7 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub5
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (2, 33, 444, 5555);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is not covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ ALTER TABLE tab7 ADD PRIMARY KEY (a, b);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (3, 55, 666, 7777);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+# now switch the primary key again to another columns not covered by the
+# column list, but also generate writes between the drop and creation
+# of the new constraint
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ INSERT INTO tab7 VALUES (4, 77, 888, 9999);
+ -- update/delete is not allowed for tables without RI
+ ALTER TABLE tab7 ADD PRIMARY KEY (b, a);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+ DELETE FROM tab7 WHERE a = 1;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(2|264||
+3|220||
+4|154||),
+ 'replication with the modified primary key');
+
+
+# TEST: partitioned tables (with publish_via_partition_root = false)
+# and replica identity. The (leaf) partitions may have different RI, so
+# we need to check the partition RI (with respect to the column list)
+# while attaching the partition.
+
+# First, let's create a partitioned table with two partitions, each with
+# a different RI, but a column list not covering all those RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+
+ -- initial data, one row in each partition
+ INSERT INTO test_part_a VALUES (1, 3);
+ INSERT INTO test_part_a VALUES (6, 4);
+));
+
+# do the same thing on the subscriber (with the opposite column order)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (b int, a int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+));
+
+# create a publication replicating just the column "a", which is not enough
+# for the second partition
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub6 FOR TABLE test_part_a (b, a) WITH (publish_via_partition_root = true);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_1 (a);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_2 (b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub6
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_a VALUES (2, 5);
+ INSERT INTO test_part_a VALUES (7, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT a, b FROM test_part_a ORDER BY a, b"),
+ qq(1|3
+2|5
+6|4
+7|6),
+ 'partitions with different replica identities not replicated correctly');
+
+# This time start with a column list covering RI for all partitions, but
+# then update the column list to not cover column "b" (needed by the
+# second partition)
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+
+ -- initial data, one row in each partitions
+ INSERT INTO test_part_b VALUES (1, 1);
+ INSERT INTO test_part_b VALUES (6, 2);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub7 FOR TABLE test_part_b (a, b) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub7
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_b VALUES (2, 3);
+ INSERT INTO test_part_b VALUES (7, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_b ORDER BY a, b"),
+ qq(1|1
+2|3
+6|2
+7|4),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: This time start with a column list covering RI for all partitions,
+# but then update RI for one of the partitions to not be covered by the
+# column list anymore.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+
+ -- initial data, one row for each partition
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+));
+
+# create a publication replicating data through partition root, with a column
+# list on the root, and then add the partitions one by one with separate
+# column lists (but those are not applied)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a,c);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SUBSCRIPTION sub1;
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub8;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_c VALUES (3, 7, 8);
+ INSERT INTO test_part_c VALUES (4, 9, 10);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||5
+2|4|
+3||8
+4|9|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# create a publication not replicating data through partition root, without
+# a column list on the root, and then add the partitions one by one with
+# separate column lists
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub8;
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ TRUNCATE test_part_c;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ TRUNCATE test_part_c;
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||
+2|4|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: Start with a single partition, with RI compatible with the column
+# list, and then attach a partition with incompatible RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ INSERT INTO test_part_d VALUES (1, 2);
+));
+
+# do the same thing on the subscriber (in fact, create both partitions right
+# away, no need to delay that)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ CREATE TABLE test_part_d_2 PARTITION OF test_part_d FOR VALUES IN (2,4);
+ ALTER TABLE test_part_d_2 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_2 REPLICA IDENTITY USING INDEX test_part_d_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub9 FOR TABLE test_part_d (a) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub9
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_d VALUES (3, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_d ORDER BY a, b"),
+ qq(1|
+3|),
+ 'partitions with different replica identities not replicated correctly');
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. So with column lists (a,b) and (a,c) we
+# should replicate (a,b,c).
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+
+ -- initial data
+ INSERT INTO test_mix_1 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_1 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_1 ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES, we should replicate all columns.
+
+# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ test_part, test_part_a, test_part_b, test_part_c, test_part_d;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
+
+ -- initial data
+ INSERT INTO test_mix_2 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_3, pub_mix_4;
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_2 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_2"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES IN SCHEMA, we should replicate all columns.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
+
+ -- initial data
+ INSERT INTO test_mix_3 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_5, pub_mix_6;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_3 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_3"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: Check handling of publish_via_partition_root - if a partition is
+# published through partition root, we should only apply the column list
+# defined for the whole table (not the partitions) - both during the initial
+# sync and when replicating changes. This is what we do for row filters.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+
+ CREATE PUBLICATION pub_root_true FOR TABLE test_root (a) WITH (publish_via_partition_root = true);
+
+ -- initial data
+ INSERT INTO test_root VALUES (1, 2, 3);
+ INSERT INTO test_root VALUES (10, 20, 30);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_root_true;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_root VALUES (2, 3, 4);
+ INSERT INTO test_root VALUES (11, 21, 31);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_root ORDER BY a, b, c"),
+ qq(1||
+2||
+10||
+11||),
+ 'publication via partition root applies column list');
+
+
+# TEST: Multiple publications which publish schema of parent table and
+# partition. The partition is published through two publications, once
+# through a schema (so no column list) containing the parent, and then
+# also directly (with a columns list). The expected outcome is there is
+# no column list.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub1, pub2, pub3, pub4, pub5, pub6, pub7, pub8;
+
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA s1;
+ CREATE PUBLICATION pub2 FOR TABLE t_1(b);
+
+ -- initial data
+ INSERT INTO s1.t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub1, pub2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'two publications, publishing the same relation');
+
+# Now resync the subcription, but with publications in the opposite order.
+# The result should be the same.
+
+$node_subscriber->safe_psql('postgres', qq(
+ TRUNCATE s1.t;
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub1;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (7, 8, 9);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(7|8|9),
+ 'two publications, publishing the same relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub3 FOR TABLE t_1 (a), t_2
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub3;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+# Note: The difference from the preceding test is that in this case both
+# relations have a column list defined.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub4 FOR TABLE t_1 (a), t_2 (b)
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
+
+done_testing();
--
2.34.1
On 3/19/22 18:11, Tomas Vondra wrote:
Fix a compiler warning reported by cfbot.
Apologies, I failed to actually commit the fix. So here we go again.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
0001-Allow-specifying-column-lists-for-logical--20220318c.patchtext/x-patch; charset=UTF-8; name=0001-Allow-specifying-column-lists-for-logical--20220318c.patchDownload
From a1bfec22fcb9ca347db1001c0551420721cb10ba Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Thu, 17 Mar 2022 19:16:39 +0100
Subject: [PATCH] Allow specifying column lists for logical replication
This allows specifying an optional column list when adding a table to
logical replication. Columns not included on this list are not sent to
the subscriber. The list is specified after the table name, enclosed
in parentheses.
For UPDATE/DELETE publications, the column list needs to cover all
REPLICA IDENTITY columns. For INSERT publications, the column list is
arbitrary and may omit some REPLICA IDENTITY columns. Furthermore, if
the table uses REPLICA IDENTITY FULL, column list is not allowed.
The column list can contain only simple column references. Complex
expressions, function calls etc. are not allowed. This restriction could
be relaxed in the future.
During the initial table synchronization, only columns specified in the
column list are copied to the subscriber. If the subscription has
several publications, containing the same table with different column
lists, columns specified in any of the lists will be copied. This
means all columns are replicated if the table has no column list at
all (which is treated as column list with all columns), of when of the
publications is defined as FOR ALL TABLES (possibly IN SCHEMA for the
schema of the table).
For partitioned tables, publish_via_partition_root determines whether
the column list for the root or leaf relation will be used. If the
parameter is 'false' (the default), the list defined for the leaf
relation is used. Otherwise, the column list for the root partition
will be used.
Psql commands \dRp+ and \d <table-name> now display any column lists.
Author: Tomas Vondra, Rahila Syed
Reviewed-by: Peter Eisentraut, Alvaro Herrera, Vignesh C, Ibrar Ahmed,
Amit Kapila, Hou zj, Peter Smith, Wang wei, Tang, Shi yu
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 15 +-
doc/src/sgml/protocol.sgml | 3 +-
doc/src/sgml/ref/alter_publication.sgml | 18 +-
doc/src/sgml/ref/create_publication.sgml | 17 +-
src/backend/catalog/pg_publication.c | 221 ++++
src/backend/commands/publicationcmds.c | 272 ++++-
src/backend/executor/execReplication.c | 19 +-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 33 +-
src/backend/replication/logical/proto.c | 61 +-
src/backend/replication/logical/tablesync.c | 156 ++-
src/backend/replication/pgoutput/pgoutput.c | 202 +++-
src/backend/utils/cache/relcache.c | 33 +-
src/bin/pg_dump/pg_dump.c | 47 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 60 +
src/bin/psql/describe.c | 40 +-
src/include/catalog/pg_publication.h | 14 +
src/include/catalog/pg_publication_rel.h | 1 +
src/include/commands/publicationcmds.h | 4 +-
src/include/nodes/parsenodes.h | 1 +
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 372 ++++++
src/test/regress/sql/publication.sql | 287 +++++
src/test/subscription/t/030_column_list.pl | 1124 +++++++++++++++++++
26 files changed, 2915 insertions(+), 94 deletions(-)
create mode 100644 src/test/subscription/t/030_column_list.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 4dc5b34d21c..89827c373bd 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -4410,7 +4410,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para>
<para>
This is an array of <structfield>indnatts</structfield> values that
- indicate which table columns this index indexes. For example a value
+ indicate which table columns this index indexes. For example, a value
of <literal>1 3</literal> would mean that the first and the third table
columns make up the index entries. Key columns come before non-key
(included) columns. A zero in this array indicates that the
@@ -6281,6 +6281,19 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to schema
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>prattrs</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+ </para>
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example, a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all columns are published.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 9178c779ba9..fb491e9ebee 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7006,7 +7006,8 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column included in
+ the publication (except generated columns):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 32b75f6c78e..9e9fc19df71 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -30,7 +30,7 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -112,6 +112,14 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ </para>
+
+ <para>
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
If the optional <literal>WHERE</literal> clause is specified, rows for
which the <replaceable class="parameter">expression</replaceable>
evaluates to false or null will not be published. Note that parentheses
@@ -174,7 +182,13 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
<para>
Add some tables to the publication:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
+<programlisting>
+ALTER PUBLICATION mypublication SET TABLE users (user_id, firstname, lastname), TABLE departments;
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 4979b9b646d..fb2d013393b 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -28,7 +28,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -86,6 +86,13 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<literal>TRUNCATE</literal> commands.
</para>
+ <para>
+ When a column list is specified, only the named columns are replicated.
+ If no column list is specified, all columns of the table are replicated
+ through this publication, including any columns added later. If a column
+ list is specified, it must include the replica identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
@@ -327,6 +334,14 @@ CREATE PUBLICATION production_publication FOR TABLE users, departments, ALL TABL
<structname>sales</structname>:
<programlisting>
CREATE PUBLICATION sales_publication FOR ALL TABLES IN SCHEMA marketing, sales;
+</programlisting></para>
+
+ <para>
+ Create a publication that publishes all changes for table <structname>users</structname>,
+ but replicates only columns <structname>user_id</structname> and
+ <structname>firstname</structname>:
+<programlisting>
+CREATE PUBLICATION users_filtered FOR TABLE users (user_id, firstname);
</programlisting></para>
</refsect1>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 789b895db89..432fc27e591 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,6 +45,9 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+static void publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs);
+
/*
* Check if relation can be in given publication and throws appropriate
* error if not.
@@ -345,6 +348,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
Oid relid = RelationGetRelid(targetrel);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ AttrNumber *attarray = NULL;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -372,6 +377,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
check_publication_add_relation(targetrel);
+ /*
+ * Translate column names to attnums and make sure the column list contains
+ * only allowed elements (no system or generated columns etc.). Also build
+ * an array of attnums, for storing in the catalog.
+ */
+ publication_translate_columns(pri->relation, pri->columns,
+ &natts, &attarray);
+
/* Form a tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -390,6 +403,12 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
else
nulls[Anum_pg_publication_rel_prqual - 1] = true;
+ /* Add column list, if available */
+ if (pri->columns)
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(buildint2vector(attarray, natts));
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -413,6 +432,13 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
DEPENDENCY_NORMAL, DEPENDENCY_NORMAL,
false);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL);
+ }
+
/* Close the table. */
table_close(rel, RowExclusiveLock);
@@ -432,6 +458,125 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
return myself;
}
+/* qsort comparator for attnums */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list (no system or generated attributes,
+ * no duplicates). Additional checks with replica identity are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ */
+static void
+publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs)
+{
+ AttrNumber *attarray = NULL;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
+
+ /* Bail out when no column list defined. */
+ if (!columns)
+ return;
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference generated column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /* Be tidy, so that the catalog representation is always sorted */
+ qsort(attarray, n, sizeof(AttrNumber), compare_int16);
+
+ *natts = n;
+ *attrs = attarray;
+
+ bms_free(set);
+}
+
+/*
+ * Transform the column list (represented by an array) to a bitmapset.
+ */
+Bitmapset *
+pub_collist_to_bitmapset(Bitmapset *columns, Datum pubcols, MemoryContext mcxt)
+{
+ Bitmapset *result = NULL;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ MemoryContext oldcxt;
+
+ /*
+ * If an existing bitmap was provided, use it. Otherwise just use NULL
+ * and build a new bitmap.
+ */
+ if (columns)
+ result = columns;
+
+ arr = DatumGetArrayTypeP(pubcols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* If a memory context was specified, switch to it. */
+ if (mcxt)
+ oldcxt = MemoryContextSwitchTo(mcxt);
+
+ for (int i = 0; i < nelems; i++)
+ result = bms_add_member(result, elems[i]);
+
+ if (mcxt)
+ MemoryContextSwitchTo(oldcxt);
+
+ return result;
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -539,6 +684,82 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+ Form_pg_publication_rel pubrel;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ /* no column list for this publications/relation */
+ if (isnull)
+ continue;
+
+ pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
+
+ pubs = lappend_oid(pubs, pubrel->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 1aad2e769cb..0c9993a155b 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -296,7 +296,7 @@ contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
* Returns true if any invalid column is found.
*/
bool
-contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
+pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
bool pubviaroot)
{
HeapTuple rftuple;
@@ -368,6 +368,114 @@ contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
return result;
}
+/*
+ * Check if all columns referenced in the REPLICA IDENTITY are covered by
+ * the column list.
+ *
+ * Returns true if any replica identity column is not covered by column list.
+ */
+bool
+pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
+ bool pubviaroot)
+{
+ HeapTuple tuple;
+ Oid relid = RelationGetRelid(relation);
+ Oid publish_as_relid = RelationGetRelid(relation);
+ bool result = false;
+ Datum datum;
+ bool isnull;
+
+ /*
+ * For a partition, if pubviaroot is true, find the topmost ancestor that
+ * is published via this publication as we need to use its column list
+ * for the changes.
+ *
+ * Note that even though the column list used is for an ancestor, the
+ * REPLICA IDENTITY used will be for the actual child table.
+ */
+ if (pubviaroot && relation->rd_rel->relispartition)
+ {
+ publish_as_relid = GetTopMostAncestorInPublication(pubid, ancestors, NULL);
+
+ if (!OidIsValid(publish_as_relid))
+ publish_as_relid = relid;
+ }
+
+ tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tuple))
+ return false;
+
+ datum = SysCacheGetAttr(PUBLICATIONRELMAP, tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ if (!isnull)
+ {
+ int x;
+ Bitmapset *idattrs;
+ Bitmapset *columns = NULL;
+
+ /* With REPLICA IDENTITY FULL, no column list is allowed. */
+ if (relation->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ result = true;
+
+ /* Transform the column list datum to a bitmapset. */
+ columns = pub_collist_to_bitmapset(NULL, datum, NULL);
+
+ /* Remember columns that are part of the REPLICA IDENTITY */
+ idattrs = RelationGetIndexAttrBitmap(relation,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * Attnums in the bitmap returned by RelationGetIndexAttrBitmap are
+ * offset (to handle system columns the usual way), while column list
+ * does not use offset, so we can't do bms_is_subset(). Instead, we have
+ * to loop over the idattrs and check all of them are in the list.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ AttrNumber attnum = (x + FirstLowInvalidHeapAttributeNumber);
+
+ /*
+ * If pubviaroot is true, we are validating the column list of the
+ * parent table, but the bitmap contains the replica identity
+ * information of the child table. The parent/child attnums may not
+ * match, so translate them to the parent - get the attname from
+ * the child, and look it up in the parent.
+ */
+ if (pubviaroot)
+ {
+ /* attribute name in the child table */
+ char *colname = get_attname(relid, attnum, false);
+
+ /*
+ * Determine the attnum for the attribute name in parent (we
+ * are using the column list defined on the parent).
+ */
+ attnum = get_attnum(publish_as_relid, colname);
+ }
+
+ /* replica identity column, not covered by the column list */
+ if (!bms_is_member(attnum, columns))
+ {
+ result = true;
+ break;
+ }
+ }
+
+ bms_free(idattrs);
+ bms_free(columns);
+ }
+
+ ReleaseSysCache(tuple);
+
+ return result;
+}
+
/* check_functions_in_node callback */
static bool
contain_mutable_or_user_functions_checker(Oid func_id, void *context)
@@ -609,6 +717,45 @@ TransformPubWhereClauses(List *tables, const char *queryString,
}
}
+
+/*
+ * Transform the publication column lists expression for all the relations
+ * in the list.
+ *
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow column lists on the leaf relations. So maybe rename it?
+ */
+static void
+TransformPubColumnList(List *tables, const char *queryString,
+ bool pubviaroot)
+{
+ ListCell *lc;
+
+ foreach(lc, tables)
+ {
+ PublicationRelInfo *pri = (PublicationRelInfo *) lfirst(lc);
+
+ if (pri->columns == NIL)
+ continue;
+
+ /*
+ * If the publication doesn't publish changes via the root partitioned
+ * table, the partition's column list will be used. So disallow using
+ * the column list on partitioned table in this case.
+ */
+ if (!pubviaroot &&
+ pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot use publication column list for relation \"%s\"",
+ RelationGetRelationName(pri->relation)),
+ errdetail("column list cannot be used for a partitioned table when %s is false.",
+ "publish_via_partition_root")));
+ }
+}
+
/*
* Create new publication.
*/
@@ -725,6 +872,9 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
TransformPubWhereClauses(rels, pstate->p_sourcetext,
publish_via_partition_root);
+ TransformPubColumnList(rels, pstate->p_sourcetext,
+ publish_via_partition_root);
+
PublicationAddTables(puboid, rels, true, NULL);
CloseTableList(rels);
}
@@ -784,8 +934,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* If the publication doesn't publish changes via the root partitioned
- * table, the partition's row filter will be used. So disallow using WHERE
- * clause on partitioned table in this case.
+ * table, the partition's row filter and column list will be used. So disallow
+ * using WHERE clause and column lists on partitioned table in this case.
*/
if (!pubform->puballtables && publish_via_partition_root_given &&
!publish_via_partition_root)
@@ -793,7 +943,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* Lock the publication so nobody else can do anything with it. This
* prevents concurrent alter to add partitioned table(s) with WHERE
- * clause(s) which we don't allow when not publishing via root.
+ * clause(s) and/or column lists which we don't allow when not
+ * publishing via root.
*/
LockDatabaseObject(PublicationRelationId, pubform->oid, 0,
AccessShareLock);
@@ -805,13 +956,21 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
HeapTuple rftuple;
Oid relid = lfirst_oid(lc);
+ bool has_column_list;
+ bool has_row_filter;
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubform->oid));
+ has_row_filter
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL);
+
+ has_column_list
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prattrs, NULL);
+
if (HeapTupleIsValid(rftuple) &&
- !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL))
+ (has_row_filter || has_column_list))
{
HeapTuple tuple;
@@ -820,7 +979,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
Form_pg_class relform = (Form_pg_class) GETSTRUCT(tuple);
- if (relform->relkind == RELKIND_PARTITIONED_TABLE)
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_row_filter)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot set %s for publication \"%s\"",
@@ -831,6 +991,18 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
NameStr(relform->relname),
"publish_via_partition_root")));
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_column_list)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot set %s for publication \"%s\"",
+ "publish_via_partition_root = false",
+ stmt->pubname),
+ errdetail("The publication contains a column list for a partitioned table \"%s\" "
+ "which is not allowed when %s is false.",
+ NameStr(relform->relname),
+ "publish_via_partition_root")));
+
ReleaseSysCache(tuple);
}
@@ -976,6 +1148,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
PublicationAddTables(pubid, rels, false, stmt);
}
else if (stmt->action == AP_DropObjects)
@@ -992,6 +1166,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ TransformPubColumnList(rels, queryString, pubform->pubviaroot);
+
/*
* To recreate the relation list for the publication, look for
* existing relations that do not need to be dropped.
@@ -1003,42 +1179,79 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
PublicationRelInfo *oldrel;
bool found = false;
HeapTuple rftuple;
- bool rfisnull = true;
Node *oldrelwhereclause = NULL;
+ Bitmapset *oldcolumns = NULL;
/* look up the cache for the old relmap */
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(oldrelid),
ObjectIdGetDatum(pubid));
+ /*
+ * See if the existing relation currently has a WHERE clause or a
+ * column list. We need to compare those too.
+ */
if (HeapTupleIsValid(rftuple))
{
+ bool isnull = true;
Datum whereClauseDatum;
+ Datum columnListDatum;
+ /* Load the WHERE clause for this table. */
whereClauseDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
Anum_pg_publication_rel_prqual,
- &rfisnull);
- if (!rfisnull)
+ &isnull);
+ if (!isnull)
oldrelwhereclause = stringToNode(TextDatumGetCString(whereClauseDatum));
+ /* Transform the int2vector column list to a bitmap. */
+ columnListDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ if (!isnull)
+ oldcolumns = pub_collist_to_bitmapset(NULL, columnListDatum, NULL);
+
ReleaseSysCache(rftuple);
}
foreach(newlc, rels)
{
PublicationRelInfo *newpubrel;
+ Oid newrelid;
+ Bitmapset *newcolumns = NULL;
newpubrel = (PublicationRelInfo *) lfirst(newlc);
+ newrelid = RelationGetRelid(newpubrel->relation);
+
+ /*
+ * If the new publication has column list, transform it to
+ * a bitmap too.
+ */
+ if (newpubrel->columns)
+ {
+ ListCell *lc;
+
+ foreach(lc, newpubrel->columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(newrelid, colname);
+
+ newcolumns = bms_add_member(newcolumns, attnum);
+ }
+ }
/*
* Check if any of the new set of relations matches with the
* existing relations in the publication. Additionally, if the
* relation has an associated WHERE clause, check the WHERE
- * expressions also match. Drop the rest.
+ * expressions also match. Same for the column list. Drop the
+ * rest.
*/
if (RelationGetRelid(newpubrel->relation) == oldrelid)
{
- if (equal(oldrelwhereclause, newpubrel->whereClause))
+ if (equal(oldrelwhereclause, newpubrel->whereClause) &&
+ bms_equal(oldcolumns, newcolumns))
{
found = true;
break;
@@ -1057,6 +1270,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
{
oldrel = palloc(sizeof(PublicationRelInfo));
oldrel->whereClause = NULL;
+ oldrel->columns = NIL;
oldrel->relation = table_open(oldrelid,
ShareUpdateExclusiveLock);
delrels = lappend(delrels, oldrel);
@@ -1118,7 +1332,7 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
}
else if (stmt->action == AP_DropObjects)
PublicationDropSchemas(pubform->oid, schemaidlist, false);
- else /* AP_SetObjects */
+ else if (stmt->action == AP_SetObjects)
{
List *oldschemaids = GetPublicationSchemas(pubform->oid);
List *delschemas = NIL;
@@ -1403,6 +1617,7 @@ OpenTableList(List *tables)
List *rels = NIL;
ListCell *lc;
List *relids_with_rf = NIL;
+ List *relids_with_collist = NIL;
/*
* Open, share-lock, and check all the explicitly-specified relations
@@ -1437,6 +1652,13 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /* Disallow duplicate tables if there are any with column lists. */
+ if (t->columns || list_member_oid(relids_with_collist, myrelid))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
table_close(rel, ShareUpdateExclusiveLock);
continue;
}
@@ -1444,12 +1666,16 @@ OpenTableList(List *tables)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
pub_rel->whereClause = t->whereClause;
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, myrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, myrelid);
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, myrelid);
+
/*
* Add children of this rel, if requested, so that they too are added
* to the publication. A partitioned table can't have any inheritance
@@ -1489,6 +1715,18 @@ OpenTableList(List *tables)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /*
+ * We don't allow to specify column list for both parent
+ * and child table at the same time as it is not very
+ * clear which one should be given preference.
+ */
+ if (childrelid != myrelid &&
+ (t->columns || list_member_oid(relids_with_collist, childrelid)))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
continue;
}
@@ -1498,11 +1736,16 @@ OpenTableList(List *tables)
pub_rel->relation = rel;
/* child inherits WHERE clause from parent */
pub_rel->whereClause = t->whereClause;
+ /* child inherits column list from parent */
+ pub_rel->columns = t->columns;
rels = lappend(rels, pub_rel);
relids = lappend_oid(relids, childrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, childrelid);
+
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, childrelid);
}
}
}
@@ -1611,6 +1854,11 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 09f78f22441..3e282ed99ab 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -573,9 +573,6 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
if (cmd != CMD_UPDATE && cmd != CMD_DELETE)
return;
- if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
- return;
-
/*
* It is only safe to execute UPDATE/DELETE when all columns, referenced
* in the row filters from publications which the relation is in, are
@@ -595,17 +592,33 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
errmsg("cannot update table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_UPDATE && !pubdesc.cols_valid_for_update)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot update table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
else if (cmd == CMD_DELETE && !pubdesc.rf_valid_for_delete)
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
errmsg("cannot delete from table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_DELETE && !pubdesc.cols_valid_for_delete)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot delete from table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
/* If relation has replica identity we are always good. */
if (OidIsValid(RelationGetReplicaIndex(rel)))
return;
+ /* REPLICA IDENTITY FULL is also good for UPDATE/DELETE. */
+ if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ return;
+
/*
* This is UPDATE/DELETE and there is no replica identity.
*
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index d4f8455a2bd..a504437873f 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4850,6 +4850,7 @@ _copyPublicationTable(const PublicationTable *from)
COPY_NODE_FIELD(relation);
COPY_NODE_FIELD(whereClause);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f1002afe7a0..4fc16ce04e3 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2322,6 +2322,7 @@ _equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
COMPARE_NODE_FIELD(whereClause);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a03b33b53bd..ff4573390c5 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9751,13 +9751,14 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr OptWhereClause
+ TABLE relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9772,11 +9773,15 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId OptWhereClause
+ | ColId opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- if ($2)
+ /*
+ * If either a row filter or column list is specified, create
+ * a PublicationTable object.
+ */
+ if ($2 || $3)
{
/*
* The OptWhereClause must be stored here but it is
@@ -9786,7 +9791,8 @@ PublicationObjSpec:
*/
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVar(NULL, $1, @1);
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
else
{
@@ -9794,23 +9800,25 @@ PublicationObjSpec:
}
$$->location = @1;
}
- | ColId indirection OptWhereClause
+ | ColId indirection opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr OptWhereClause
+ | extended_relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
| CURRENT_SCHEMA
{
@@ -17488,6 +17496,13 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
errmsg("WHERE clause not allowed for schema"),
parser_errposition(pubobj->location));
+ /* Column list is not allowed on a schema object */
+ if (pubobj->pubtable && pubobj->pubtable->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schema"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index c9b0eeefd7e..f9de1d16dc2 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,16 +29,30 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary);
+ bool binary, Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
static void logicalrep_write_namespace(StringInfo out, Oid nspid);
static const char *logicalrep_read_namespace(StringInfo in);
+/*
+ * Check if a column is covered by a column list.
+ *
+ * Need to be careful about NULL, which is treated as a column list covering
+ * all columns.
+ */
+static bool
+column_in_column_list(int attnum, Bitmapset *columns)
+{
+ return (columns == NULL || bms_is_member(attnum, columns));
+}
+
+
/*
* Write BEGIN to the output stream.
*/
@@ -398,7 +412,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +424,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -443,7 +457,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -464,11 +478,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -537,7 +551,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, NULL);
}
/*
@@ -652,7 +666,8 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -674,7 +689,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -751,7 +766,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum *values;
@@ -763,8 +778,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ if (!column_in_column_list(att->attnum, columns))
continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -783,6 +804,9 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_column_list(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -904,7 +928,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -917,8 +941,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* send number of live attributes */
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
continue;
+
+ if (!column_in_column_list(att->attnum, columns))
+ continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -937,6 +967,9 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_column_list(att->attnum, columns))
+ continue;
+
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 1659964571c..caeab853e4c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -112,6 +112,7 @@
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "utils/acl.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -701,12 +702,13 @@ fetch_remote_table_info(char *nspname, char *relname,
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
int natt;
ListCell *lc;
bool first;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
@@ -747,10 +749,110 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+
+ /*
+ * Get column lists for each relation.
+ *
+ * For initial synchronization, column lists can be ignored in following
+ * cases:
+ *
+ * 1) one of the subscribed publications for the table hasn't specified
+ * any column list
+ *
+ * 2) one of the subscribed publications has puballtables set to true
+ *
+ * 3) one of the subscribed publications is declared as ALL TABLES IN
+ * SCHEMA that includes this relation
+ *
+ * We need to do this before fetching info about column names and types,
+ * so that we can skip columns that should not be replicated.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData pub_names;
+ bool first = true;
+
+ initStringInfo(&pub_names);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&pub_names, ", ");
+ appendStringInfoString(&pub_names, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ /*
+ * Fetch info about column lists for the relation (from all the
+ * publications). We unnest the int2vector values, because that
+ * makes it easier to combine lists by simply adding the attnums
+ * to a new bitmap (without having to parse the int2vector data).
+ * This preserves NULL values, so that if one of the publications
+ * has no column list, we'll know that.
+ */
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT DISTINCT unnest"
+ " FROM pg_publication p"
+ " LEFT OUTER JOIN pg_publication_rel pr"
+ " ON (p.oid = pr.prpubid AND pr.prrelid = %u)"
+ " LEFT OUTER JOIN unnest(pr.prattrs) ON TRUE,"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt"
+ " WHERE gpt.relid = %u"
+ " AND p.pubname IN ( %s )",
+ lrel->remoteid,
+ lrel->remoteid,
+ pub_names.data);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch column list info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ /*
+ * Merge the column lists (from different publications) by creating
+ * a single bitmap with all the attnums. If we find a NULL value,
+ * that means one of the publications has no column list for the
+ * table we're syncing.
+ */
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ Datum cfval = slot_getattr(slot, 1, &isnull);
+
+ /* NULL means empty column list, so we're done. */
+ if (isnull)
+ {
+ bms_free(included_cols);
+ included_cols = NULL;
+ break;
+ }
+
+ included_cols = bms_add_member(included_cols,
+ DatumGetInt16(cfval));
+
+ ExecClearTuple(slot);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+
+ walrcv_clear_result(pubres);
+
+ pfree(pub_names.data);
+ }
+
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -778,16 +880,35 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * Store the columns as a list of names. Ignore those that are not
+ * present in the column list, if there is one.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ /* If the column is not in the column list, skip it. */
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ {
+ ExecClearTuple(slot);
+ continue;
+ }
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -821,6 +942,9 @@ fetch_remote_table_info(char *nspname, char *relname,
*
* 3) one of the subscribed publications is declared as ALL TABLES IN
* SCHEMA that includes this relation
+ *
+ * XXX Does this actually handle puballtables and schema publications
+ * correctly?
*/
if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
{
@@ -930,8 +1054,24 @@ copy_table(Relation rel)
/* Regular table with no row filter */
if (lrel.relkind == RELKIND_RELATION && qual == NIL)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+
+ /*
+ * XXX Do we need to list the columns in all cases? Maybe we're replicating
+ * all columns?
+ */
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
+
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
+
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 5fddab3a3d4..f5e7610a172 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -29,6 +29,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -85,7 +86,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -143,9 +145,6 @@ typedef struct RelationSyncEntry
*/
ExprState *exprstate[NUM_ROWFILTER_PUBACTIONS];
EState *estate; /* executor state used for row filter */
- MemoryContext cache_expr_cxt; /* private context for exprstate and
- * estate, if any */
-
TupleTableSlot *new_slot; /* slot for storing new tuple */
TupleTableSlot *old_slot; /* slot for storing old tuple */
@@ -164,6 +163,19 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
AttrMap *attrmap;
+
+ /*
+ * Columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this bitmap are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
+
+ /*
+ * Private context to store additional data for this entry - state for
+ * the row filter expressions, column list, etc.
+ */
+ MemoryContext entry_cxt;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -188,6 +200,7 @@ static EState *create_estate_for_relation(Relation rel);
static void pgoutput_row_filter_init(PGOutputData *data,
List *publications,
RelationSyncEntry *entry);
+
static bool pgoutput_row_filter_exec_expr(ExprState *state,
ExprContext *econtext);
static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
@@ -195,6 +208,11 @@ static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
RelationSyncEntry *entry,
ReorderBufferChangeType *action);
+/* column list routines */
+static void pgoutput_column_list_init(PGOutputData *data,
+ List *publications,
+ RelationSyncEntry *entry);
+
/*
* Specify output plugin callbacks
*/
@@ -603,11 +621,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -620,7 +638,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -643,13 +662,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip this attribute if it's not present in the column list */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -703,6 +726,28 @@ pgoutput_row_filter_exec_expr(ExprState *state, ExprContext *econtext)
return DatumGetBool(ret);
}
+/*
+ * Make sure the per-entry memory context exists.
+ */
+static void
+pgoutput_ensure_entry_cxt(PGOutputData *data, RelationSyncEntry *entry)
+{
+ Relation relation;
+
+ /* The context may already exist, in which case bail out. */
+ if (entry->entry_cxt)
+ return;
+
+ relation = RelationIdGetRelation(entry->publish_as_relid);
+
+ entry->entry_cxt = AllocSetContextCreate(data->cachectx,
+ "entry private context",
+ ALLOCSET_SMALL_SIZES);
+
+ MemoryContextCopyAndSetIdentifier(entry->entry_cxt,
+ RelationGetRelationName(relation));
+}
+
/*
* Initialize the row filter.
*/
@@ -823,21 +868,13 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
{
Relation relation = RelationIdGetRelation(entry->publish_as_relid);
- Assert(entry->cache_expr_cxt == NULL);
-
- /* Create the memory context for row filters */
- entry->cache_expr_cxt = AllocSetContextCreate(data->cachectx,
- "Row filter expressions",
- ALLOCSET_DEFAULT_SIZES);
-
- MemoryContextCopyAndSetIdentifier(entry->cache_expr_cxt,
- RelationGetRelationName(relation));
+ pgoutput_ensure_entry_cxt(data, entry);
/*
* Now all the filters for all pubactions are known. Combine them when
* their pubactions are the same.
*/
- oldctx = MemoryContextSwitchTo(entry->cache_expr_cxt);
+ oldctx = MemoryContextSwitchTo(entry->entry_cxt);
entry->estate = create_estate_for_relation(relation);
for (idx = 0; idx < NUM_ROWFILTER_PUBACTIONS; idx++)
{
@@ -860,6 +897,105 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
}
}
+/*
+ * Initialize the column list.
+ */
+static void
+pgoutput_column_list_init(PGOutputData *data, List *publications,
+ RelationSyncEntry *entry)
+{
+ ListCell *lc;
+
+ /*
+ * Find if there are any column lists for this relation. If there are,
+ * build a bitmap merging all the column lists.
+ *
+ * All the given publication-table mappings must be checked.
+ *
+ * Multiple publications might have multiple column lists for this relation.
+ *
+ * FOR ALL TABLES and FOR ALL TABLES IN SCHEMA implies "don't use column
+ * list" so it takes precedence.
+ */
+ foreach(lc, publications)
+ {
+ Publication *pub = lfirst(lc);
+ HeapTuple cftuple = NULL;
+ Datum cfdatum = 0;
+
+ /*
+ * Assume there's no column list. Only if we find pg_publication_rel
+ * entry with a column list we'll switch it to false.
+ */
+ bool pub_no_list = true;
+
+ /*
+ * If the publication is FOR ALL TABLES then it is treated the same as if
+ * there are no column lists (even if other publications have a list).
+ */
+ if (!pub->alltables)
+ {
+ /*
+ * Check for the presence of a column list in this publication.
+ *
+ * Note: If we find no pg_publication_rel row, it's a publication
+ * defined for a whole schema, so it can't have a column list, just
+ * like a FOR ALL TABLES publication.
+ */
+ cftuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(entry->publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(cftuple))
+ {
+ /*
+ * Lookup the column list attribute.
+ *
+ * Note: We update the pub_no_list value directly, because if
+ * the value is NULL, we have no list (and vice versa).
+ */
+ cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
+ Anum_pg_publication_rel_prattrs,
+ &pub_no_list);
+
+ /*
+ * Build the column list bitmap in the per-entry context.
+ *
+ * We need to merge column lists from all publications, so we
+ * update the same bitmapset. If the column list is null, we
+ * interpret it as replicating all columns.
+ */
+ if (!pub_no_list) /* when not null */
+ {
+ pgoutput_ensure_entry_cxt(data, entry);
+
+ entry->columns = pub_collist_to_bitmapset(entry->columns,
+ cfdatum,
+ entry->entry_cxt);
+ }
+ }
+ }
+
+ /*
+ * Found a publication with no column list, so we're done. But first
+ * discard column list we might have from preceding publications.
+ */
+ if (pub_no_list)
+ {
+ if (cftuple)
+ ReleaseSysCache(cftuple);
+
+ bms_free(entry->columns);
+ entry->columns = NULL;
+
+ break;
+ }
+
+ ReleaseSysCache(cftuple);
+ } /* loop all subscribed publications */
+
+}
+
/*
* Initialize the slot for storing new and old tuples, and build the map that
* will be used to convert the relation's tuples into the ancestor's format.
@@ -1224,7 +1360,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -1278,11 +1414,13 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel,
- new_slot, data->binary);
+ new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel,
- old_slot, new_slot, data->binary);
+ old_slot, new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel,
@@ -1729,8 +1867,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->new_slot = NULL;
entry->old_slot = NULL;
memset(entry->exprstate, 0, sizeof(entry->exprstate));
- entry->cache_expr_cxt = NULL;
+ entry->entry_cxt = NULL;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->attrmap = NULL;
}
@@ -1776,6 +1915,8 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
entry->pubactions.pubinsert = false;
entry->pubactions.pubupdate = false;
entry->pubactions.pubdelete = false;
@@ -1799,17 +1940,18 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/*
* Row filter cache cleanups.
*/
- if (entry->cache_expr_cxt)
- MemoryContextDelete(entry->cache_expr_cxt);
+ if (entry->entry_cxt)
+ MemoryContextDelete(entry->entry_cxt);
- entry->cache_expr_cxt = NULL;
+ entry->entry_cxt = NULL;
entry->estate = NULL;
memset(entry->exprstate, 0, sizeof(entry->exprstate));
/*
* Build publication cache. We can't use one provided by relcache as
- * relcache considers all publications given relation is in, but here
- * we only need to consider ones that the subscriber requested.
+ * relcache considers all publications that the given relation is in,
+ * but here we only need to consider ones that the subscriber
+ * requested.
*/
foreach(lc, data->publications)
{
@@ -1878,6 +2020,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
}
/*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
* Don't publish changes for partitioned tables, because
* publishing those of its partitions suffices, unless partition
* changes won't be published due to pubviaroot being set.
@@ -1938,6 +2083,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/* Initialize the row filter */
pgoutput_row_filter_init(data, rel_publications, entry);
+
+ /* Initialize the column list */
+ pgoutput_column_list_init(data, rel_publications, entry);
}
list_free(pubids);
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index fccffce5729..a2da72f0d48 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -5553,6 +5553,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
return;
}
@@ -5565,6 +5567,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
/* Fetch the publication membership info. */
puboids = GetRelationPublications(relid);
@@ -5616,7 +5620,7 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
*/
if (!pubform->puballtables &&
(pubform->pubupdate || pubform->pubdelete) &&
- contain_invalid_rfcolumn(pubid, relation, ancestors,
+ pub_rf_contains_invalid_column(pubid, relation, ancestors,
pubform->pubviaroot))
{
if (pubform->pubupdate)
@@ -5625,6 +5629,23 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->rf_valid_for_delete = false;
}
+ /*
+ * Check if all columns are part of the REPLICA IDENTITY index or not.
+ *
+ * If the publication is FOR ALL TABLES then it means the table has no
+ * column list and we can skip the validation.
+ */
+ if (!pubform->puballtables &&
+ (pubform->pubupdate || pubform->pubdelete) &&
+ pub_collist_contains_invalid_column(pubid, relation, ancestors,
+ pubform->pubviaroot))
+ {
+ if (pubform->pubupdate)
+ pubdesc->cols_valid_for_update = false;
+ if (pubform->pubdelete)
+ pubdesc->cols_valid_for_delete = false;
+ }
+
ReleaseSysCache(tup);
/*
@@ -5636,6 +5657,16 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
!pubdesc->rf_valid_for_update && !pubdesc->rf_valid_for_delete)
break;
+
+ /*
+ * If we know everything is replicated and the column list is invalid
+ * for update and delete, there is no point to check for other
+ * publications.
+ */
+ if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
+ pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
+ !pubdesc->cols_valid_for_update && !pubdesc->cols_valid_for_delete)
+ break;
}
if (relation->rd_pubdesc)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 725cd2e4ebc..be40acd3e37 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4101,6 +4101,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_prpubid;
int i_prrelid;
int i_prrelqual;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4114,12 +4115,20 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
if (fout->remoteVersion >= 150000)
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual "
- "FROM pg_catalog.pg_publication_rel");
+ "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual, "
+ "(CASE\n"
+ " WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT array_agg(attname)\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) prattrs "
+ "FROM pg_catalog.pg_publication_rel pr");
else
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "NULL AS prrelqual "
+ "NULL AS prrelqual, NULL AS prattrs "
"FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
@@ -4130,6 +4139,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
i_prrelqual = PQfnumber(res, "prrelqual");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4175,6 +4185,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
else
pubrinfo[j].pubrelqual = pg_strdup(PQgetvalue(res, i, i_prrelqual));
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[j].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4249,10 +4281,13 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY ",
fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, " %s",
- fmtQualifiedDumpable(tbinfo));
+ appendPQExpBufferStr(query, fmtQualifiedDumpable(tbinfo));
+
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+
if (pubrinfo->pubrelqual)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 772dc0cf7a2..1d21c2906f1 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -632,6 +632,7 @@ typedef struct _PublicationRelInfo
PublicationInfo *publication;
TableInfo *pubtable;
char *pubrelqual;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index fd1052e5db8..05a7e28bdcc 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2428,6 +2428,28 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'ALTER PUBLICATION pub1 ADD TABLE test_sixth_table (col3, col2)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_sixth_table (col3, col2);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_sixth_table (col2, col3);\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'ALTER PUBLICATION pub1 ADD TABLE test_seventh_table (col3, col2) WHERE (col1 = 1)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_seventh_table (col3, col2) WHERE (col1 = 1);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_seventh_table (col2, col3) WHERE ((col1 = 1));\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'ALTER PUBLICATION pub3 ADD ALL TABLES IN SCHEMA dump_test' => {
create_order => 51,
create_sql =>
@@ -2778,6 +2800,44 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'CREATE TABLE test_sixth_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_sixth_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_sixth_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'CREATE TABLE test_seventh_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_seventh_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_seventh_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'CREATE TABLE test_table_identity' => {
create_order => 3,
create_sql => 'CREATE TABLE dump_test.test_table_identity (
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 991bfc1546b..88bb75ac658 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2892,6 +2892,7 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_namespace pn ON p.oid = pn.pnpubid\n"
" JOIN pg_catalog.pg_class pc ON pc.relnamespace = pn.pnnspid\n"
@@ -2899,6 +2900,12 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , pg_get_expr(pr.prqual, c.oid)\n"
+ " , (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT string_agg(attname, ', ')\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) "
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
" JOIN pg_catalog.pg_class c ON c.oid = pr.prrelid\n"
@@ -2906,6 +2913,7 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2916,12 +2924,14 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
"WHERE pr.prrelid = '%s'\n"
"UNION ALL\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2943,6 +2953,11 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, " \"%s\"",
PQgetvalue(result, i, 0));
+ /* column list (if any) */
+ if (!PQgetisnull(result, i, 2))
+ appendPQExpBuffer(&buf, " (%s)",
+ PQgetvalue(result, i, 2));
+
/* row filter (if any) */
if (!PQgetisnull(result, i, 1))
appendPQExpBuffer(&buf, " WHERE %s",
@@ -5888,7 +5903,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5905,15 +5920,19 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (as_schema)
+ printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
+ else
{
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
+
+ if (!PQgetisnull(res, i, 3))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 3));
+
if (!PQgetisnull(res, i, 2))
appendPQExpBuffer(buf, " WHERE %s", PQgetvalue(res, i, 2));
}
- else
- printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
}
@@ -6042,11 +6061,22 @@ describePublications(const char *pattern)
printfPQExpBuffer(&buf,
"SELECT n.nspname, c.relname");
if (pset.sversion >= 150000)
+ {
appendPQExpBufferStr(&buf,
", pg_get_expr(pr.prqual, c.oid)");
+ appendPQExpBufferStr(&buf,
+ ", (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string("
+ " ARRAY(SELECT attname\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END)");
+ }
else
appendPQExpBufferStr(&buf,
- ", NULL");
+ ", NULL, NULL");
appendPQExpBuffer(&buf,
"\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index fe773cf9b7d..a56c1102463 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -85,6 +85,13 @@ typedef struct PublicationDesc
*/
bool rf_valid_for_update;
bool rf_valid_for_delete;
+
+ /*
+ * true if the columns are part of the replica identity or the publication actions
+ * do not include UPDATE or DELETE.
+ */
+ bool cols_valid_for_update;
+ bool cols_valid_for_delete;
} PublicationDesc;
typedef struct Publication
@@ -100,6 +107,7 @@ typedef struct PublicationRelInfo
{
Relation relation;
Node *whereClause;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -123,8 +131,11 @@ typedef enum PublicationPartOpt
} PublicationPartOpt;
extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
+extern void GetActionsInPublication(Oid pubid, PublicationActions *actions);
extern List *GetPublicationSchemas(Oid pubid);
extern List *GetSchemaPublications(Oid schemaid);
extern List *GetSchemaPublicationRelations(Oid schemaid,
@@ -144,6 +155,9 @@ extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *pri
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
bool if_not_exists);
+extern Bitmapset *pub_collist_to_bitmapset(Bitmapset *columns, Datum pubcols,
+ MemoryContext mcxt);
+
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index 0dd0f425db9..4feb581899e 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -34,6 +34,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
#ifdef CATALOG_VARLEN /* variable-length fields start here */
pg_node_tree prqual; /* qualifications */
+ int2vector prattrs; /* columns to replicate */
#endif
} FormData_pg_publication_rel;
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index 7813cbcb6bb..ae87caf089d 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -31,7 +31,9 @@ extern void RemovePublicationSchemaById(Oid psoid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
extern void AlterPublicationOwner_oid(Oid pubid, Oid newOwnerId);
extern void InvalidatePublicationRels(List *relids);
-extern bool contain_invalid_rfcolumn(Oid pubid, Relation relation,
+extern bool pub_rf_contains_invalid_column(Oid pubid, Relation relation,
+ List *ancestors, bool pubviaroot);
+extern bool pub_collist_contains_invalid_column(Oid pubid, Relation relation,
List *ancestors, bool pubviaroot);
#endif /* PUBLICATIONCMDS_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1617702d9d6..b4479c7049a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3652,6 +3652,7 @@ typedef struct PublicationTable
NodeTag type;
RangeVar *relation; /* relation to be published */
Node *whereClause; /* qualifications */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4d2c881644a..a771ab8ff33 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -209,12 +209,12 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary);
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -231,7 +231,7 @@ extern List *logicalrep_read_truncate(StringInfo in,
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 4e191c120ac..227b5611915 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -613,6 +613,369 @@ DROP TABLE rf_tbl_abcd_pk;
DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+RESET client_min_messages;
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ERROR: cannot reference generated column "d" in publication column list
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+ERROR: cannot reference system column "ctid" in publication column list
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+ERROR: cannot drop column c of table testpub_tbl5 because other objects depend on it
+DETAIL: publication of table testpub_tbl5 in publication testpub_fortable depends on column c of table testpub_tbl5
+HINT: Use DROP ... CASCADE to drop the dependent objects too.
+-- ok: for insert-only publication, the column list is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: change the replica identity to "b", and column list to (a, c)
+-- then update fails, because (a, c) does not cover replica identity
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+ Publication testpub_table_ins
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | f | f | t | f
+Tables:
+ "public.testpub_tbl5" (a)
+
+-- with REPLICA IDENTITY FULL, column lists are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ERROR: cannot update table "testpub_tbl6"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+-- make sure changing the column list is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: update the column list
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, c)
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl8;
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_0"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+ Publication testpub_both_filters
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f | t | t | t | t | f
+Tables:
+ "public.testpub_tbl_both_filters" (a, c) WHERE (c <> 1)
+
+\d+ testpub_tbl_both_filters
+ Table "public.testpub_tbl_both_filters"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+ c | integer | | not null | | plain | |
+Indexes:
+ "testpub_tbl_both_filters_pkey" PRIMARY KEY, btree (a, c) REPLICA IDENTITY
+Publications:
+ "testpub_both_filters" (a, c) WHERE (c <> 1)
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+-- Tests for partitioned table
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+ERROR: cannot use publication column list for relation "rf_tbl_abcd_part_pk"
+DETAIL: column list cannot be used for a partitioned table when publish_via_partition_root is false.
+-- ok - can use column list for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any column list is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+ERROR: cannot set publish_via_partition_root = false for publication "testpub6"
+DETAIL: The publication contains a column list for a partitioned table "rf_tbl_abcd_part_pk" which is not allowed when publish_via_partition_root is false.
+-- Now change the root column list to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have column list for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -1058,6 +1421,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schema
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 5457c56b33f..aeb1b572af8 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -373,6 +373,289 @@ DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+RESET client_min_messages;
+
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+-- ok: for insert-only publication, the column list is arbitrary
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+
+-- error: change the replica identity to "b", and column list to (a, c)
+-- then update fails, because (a, c) does not cover replica identity
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+UPDATE testpub_tbl5 SET a = 1;
+
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+
+-- with REPLICA IDENTITY FULL, column lists are not allowed
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+
+-- make sure changing the column list is updated in SET TABLE
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: we'll skip this table
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: update the column list
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl8;
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+\d+ testpub_tbl_both_filters
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Tests for partitioned table
+
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - can use column list for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any column list is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- Now change the root column list to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have column list for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
+
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -614,6 +897,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/030_column_list.pl b/src/test/subscription/t/030_column_list.pl
new file mode 100644
index 00000000000..5ceaec83cdb
--- /dev/null
+++ b/src/test/subscription/t/030_column_list.pl
@@ -0,0 +1,1124 @@
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test partial-column publication of tables
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# create publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+sub wait_for_subscription_sync
+{
+ my ($node) = @_;
+
+ # Also wait for initial table sync to finish
+ my $synced_query = "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+
+ $node->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+}
+
+# setup tables on both nodes
+
+# tab1: simple 1:1 replication
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+# tab2: replication from regular to table with fewer columns
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)
+));
+
+# tab3: simple 1:1 replication with weird column names
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "B" varchar, "c'" int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "c'" int)
+));
+
+# test_part: partitioned tables, with partitioning (including multi-level
+# partitioning, and fewer columns on the subscriber)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+# tab4: table with user-defined enum types
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, d text);
+));
+
+
+# TEST: create publication and subscription for some of the tables with
+# column lists
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub1
+ FOR TABLE tab1 (a, "B"), tab3 ("a'", "c'"), test_part (a, b), tab4 (a, b, d)
+ WITH (publish_via_partition_root = 'true');
+));
+
+# check that we got the right prattrs values for the publication in the
+# pg_publication_rel catalog (order by relname, to get stable ordering)
+my $result = $node_publisher->safe_psql('postgres', qq(
+ SELECT relname, prattrs
+ FROM pg_publication_rel pb JOIN pg_class pc ON(pb.prrelid = pc.oid)
+ ORDER BY relname
+));
+
+is($result, qq(tab1|1 2
+tab3|1 3
+tab4|1 2 4
+test_part|1 2), 'publication relation updated');
+
+# TEST: insert data into the tables, create subscription and see if sync
+# replicates the right columns
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (1, 2, 3);
+ INSERT INTO tab1 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (1, 2, 3);
+ INSERT INTO tab3 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (1, 'red', 3, 'oh my');
+ INSERT INTO tab4 VALUES (2, 'blue', 4, 'hello');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (1, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (2, 'bcd', '2021-07-03 11:12:13');
+ INSERT INTO test_part VALUES (7, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (8, 'bcd', '2021-07-03 11:12:13');
+));
+
+# create subscription for the publication, wait for sync to complete,
+# then check the sync results
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+4|5|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+4|6), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+7|abc
+8|bcd), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: now insert more data into the tables, and wait until we replicate
+# them (not by tablesync, but regular decoding and replication)
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (2, 3, 4);
+ INSERT INTO tab1 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (2, 3, 4);
+ INSERT INTO tab3 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (3, 'red', 5, 'foo');
+ INSERT INTO tab4 VALUES (4, 'blue', 6, 'bar');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (3, 'xxx', '2022-02-01 10:00:00');
+ INSERT INTO test_part VALUES (4, 'yyy', '2022-03-02 15:12:13');
+ INSERT INTO test_part VALUES (9, 'zzz', '2022-04-03 21:00:00');
+ INSERT INTO test_part VALUES (10, 'qqq', '2022-05-04 22:12:13');
+));
+
+# wait for catchup before checking the subscriber
+$node_publisher->wait_for_catchup('sub1');
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+2|3|
+4|5|
+5|6|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+2|4
+4|6
+5|7), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello
+3|red|foo
+4|blue|bar), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+3|xxx
+4|yyy
+7|abc
+8|bcd
+9|zzz
+10|qqq), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: do some updates on some of the tables, both on columns included
+# in the column list and other
+
+# tab1: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET "B" = 2 * "B" where a = 1));
+
+# tab1: update of non-replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET c = 2*c where a = 4));
+
+# tab3: update of non-replicated
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "B" = "B" || ' updated' where "a'" = 4));
+
+# tab3: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "c'" = 2 * "c'" where "a'" = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'blue', c = c * 2, d = d || ' updated' where a = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'red', c = c * 2, d = d || ' updated' where a = 2));
+
+# wait for the replication to catch up, and check the UPDATE results got
+# replicated correctly, with the right column list
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab1 ORDER BY a));
+is($result,
+qq(1|4|
+2|3|
+4|5|
+5|6|), 'only update on column tab1.b is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result,
+qq(1|6
+2|4
+4|6
+5|7), 'only update on column tab3.c is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab4 ORDER BY a));
+
+is($result, qq(1|blue|oh my updated
+2|red|hello updated
+3|red|foo
+4|blue|bar), 'update on column tab4.c is not replicated');
+
+
+# TEST: add table with a column list, insert data, replicate
+
+# insert some data before adding it to the publication
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (1, 'abc', 3);
+));
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2 (a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION");
+
+# wait for the tablesync to complete, add a bit more data and then check
+# the results of the replication
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (2, 'def', 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|def), 'insert on column tab2.c is not replicated');
+
+# do a couple updates, check the correct stuff gets replicated
+$node_publisher->safe_psql('postgres', qq(
+ UPDATE tab2 SET c = 5 where a = 1;
+ UPDATE tab2 SET b = 'xyz' where a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|xyz), 'update on column tab2.c is not replicated');
+
+
+# TEST: add a table to two publications with different column lists, and
+# create a single subscription replicating both publications
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+
+ -- insert a couple initial records
+ INSERT INTO tab5 VALUES (1, 11, 111, 1111);
+ INSERT INTO tab5 VALUES (2, 22, 222, 2222);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub3
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->wait_for_catchup('sub1');
+
+# insert data and make sure all the columns (union of the columns lists)
+# get fully replicated
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (3, 33, 333, 3333);
+ INSERT INTO tab5 VALUES (4, 44, 444, 4444);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111
+2|22|2222
+3|33|3333
+4|44|4444),
+ 'overlapping publications with overlapping column lists');
+
+# and finally, remove the column list for one of the publications, which
+# means replicating all columns (removing the column list), but first add
+# the missing column to the table on subscriber
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub3 SET TABLE tab5;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ ALTER TABLE tab5 ADD COLUMN c INT;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (5, 55, 555, 5555);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111|
+2|22|2222|
+3|33|3333|
+4|44|4444|
+5|55|5555|555),
+ 'overlapping publications with overlapping column lists');
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key (but use a different column in
+# the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub4 FOR TABLE tab6 (a, b);
+
+ -- initial data
+ INSERT INTO tab6 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (2, 33, 444, 5555);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is still covered by the column list, though)
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+# we need to do the same thing on the subscriber
+# XXX What would happen if this happens before the publisher ALTER? Or
+# interleaved, somehow? But that seems unrelated to column lists.
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (3, 55, 666, 8888);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key with a key on multiple columns
+# (all of them covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub5 FOR TABLE tab7 (a, b);
+
+ -- some initial data
+ INSERT INTO tab7 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub5
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (2, 33, 444, 5555);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is not covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ ALTER TABLE tab7 ADD PRIMARY KEY (a, b);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (3, 55, 666, 7777);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+# now switch the primary key again to another columns not covered by the
+# column list, but also generate writes between the drop and creation
+# of the new constraint
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ INSERT INTO tab7 VALUES (4, 77, 888, 9999);
+ -- update/delete is not allowed for tables without RI
+ ALTER TABLE tab7 ADD PRIMARY KEY (b, a);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+ DELETE FROM tab7 WHERE a = 1;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(2|264||
+3|220||
+4|154||),
+ 'replication with the modified primary key');
+
+
+# TEST: partitioned tables (with publish_via_partition_root = false)
+# and replica identity. The (leaf) partitions may have different RI, so
+# we need to check the partition RI (with respect to the column list)
+# while attaching the partition.
+
+# First, let's create a partitioned table with two partitions, each with
+# a different RI, but a column list not covering all those RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+
+ -- initial data, one row in each partition
+ INSERT INTO test_part_a VALUES (1, 3);
+ INSERT INTO test_part_a VALUES (6, 4);
+));
+
+# do the same thing on the subscriber (with the opposite column order)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (b int, a int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+));
+
+# create a publication replicating just the column "a", which is not enough
+# for the second partition
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub6 FOR TABLE test_part_a (b, a) WITH (publish_via_partition_root = true);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_1 (a);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_2 (b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub6
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_a VALUES (2, 5);
+ INSERT INTO test_part_a VALUES (7, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT a, b FROM test_part_a ORDER BY a, b"),
+ qq(1|3
+2|5
+6|4
+7|6),
+ 'partitions with different replica identities not replicated correctly');
+
+# This time start with a column list covering RI for all partitions, but
+# then update the column list to not cover column "b" (needed by the
+# second partition)
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+
+ -- initial data, one row in each partitions
+ INSERT INTO test_part_b VALUES (1, 1);
+ INSERT INTO test_part_b VALUES (6, 2);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub7 FOR TABLE test_part_b (a, b) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub7
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_b VALUES (2, 3);
+ INSERT INTO test_part_b VALUES (7, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_b ORDER BY a, b"),
+ qq(1|1
+2|3
+6|2
+7|4),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: This time start with a column list covering RI for all partitions,
+# but then update RI for one of the partitions to not be covered by the
+# column list anymore.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+
+ -- initial data, one row for each partition
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+));
+
+# create a publication replicating data through partition root, with a column
+# list on the root, and then add the partitions one by one with separate
+# column lists (but those are not applied)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a,c);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SUBSCRIPTION sub1;
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub8;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_c VALUES (3, 7, 8);
+ INSERT INTO test_part_c VALUES (4, 9, 10);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||5
+2|4|
+3||8
+4|9|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# create a publication not replicating data through partition root, without
+# a column list on the root, and then add the partitions one by one with
+# separate column lists
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub8;
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ TRUNCATE test_part_c;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ TRUNCATE test_part_c;
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||
+2|4|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: Start with a single partition, with RI compatible with the column
+# list, and then attach a partition with incompatible RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ INSERT INTO test_part_d VALUES (1, 2);
+));
+
+# do the same thing on the subscriber (in fact, create both partitions right
+# away, no need to delay that)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ CREATE TABLE test_part_d_2 PARTITION OF test_part_d FOR VALUES IN (2,4);
+ ALTER TABLE test_part_d_2 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_2 REPLICA IDENTITY USING INDEX test_part_d_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub9 FOR TABLE test_part_d (a) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub9
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_d VALUES (3, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_d ORDER BY a, b"),
+ qq(1|
+3|),
+ 'partitions with different replica identities not replicated correctly');
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. So with column lists (a,b) and (a,c) we
+# should replicate (a,b,c).
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+
+ -- initial data
+ INSERT INTO test_mix_1 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_1 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_1 ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES, we should replicate all columns.
+
+# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ test_part, test_part_a, test_part_b, test_part_c, test_part_d;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
+
+ -- initial data
+ INSERT INTO test_mix_2 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_3, pub_mix_4;
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_2 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_2"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES IN SCHEMA, we should replicate all columns.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
+
+ -- initial data
+ INSERT INTO test_mix_3 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_5, pub_mix_6;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_3 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_3"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: Check handling of publish_via_partition_root - if a partition is
+# published through partition root, we should only apply the column list
+# defined for the whole table (not the partitions) - both during the initial
+# sync and when replicating changes. This is what we do for row filters.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+
+ CREATE PUBLICATION pub_root_true FOR TABLE test_root (a) WITH (publish_via_partition_root = true);
+
+ -- initial data
+ INSERT INTO test_root VALUES (1, 2, 3);
+ INSERT INTO test_root VALUES (10, 20, 30);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_root_true;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_root VALUES (2, 3, 4);
+ INSERT INTO test_root VALUES (11, 21, 31);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_root ORDER BY a, b, c"),
+ qq(1||
+2||
+10||
+11||),
+ 'publication via partition root applies column list');
+
+
+# TEST: Multiple publications which publish schema of parent table and
+# partition. The partition is published through two publications, once
+# through a schema (so no column list) containing the parent, and then
+# also directly (with a columns list). The expected outcome is there is
+# no column list.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub1, pub2, pub3, pub4, pub5, pub6, pub7, pub8;
+
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA s1;
+ CREATE PUBLICATION pub2 FOR TABLE t_1(b);
+
+ -- initial data
+ INSERT INTO s1.t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub1, pub2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'two publications, publishing the same relation');
+
+# Now resync the subcription, but with publications in the opposite order.
+# The result should be the same.
+
+$node_subscriber->safe_psql('postgres', qq(
+ TRUNCATE s1.t;
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub1;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (7, 8, 9);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(7|8|9),
+ 'two publications, publishing the same relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub3 FOR TABLE t_1 (a), t_2
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub3;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+# Note: The difference from the preceding test is that in this case both
+# relations have a column list defined.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub4 FOR TABLE t_1 (a), t_2 (b)
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
+
+done_testing();
--
2.34.1
On Fri, Mar 18, 2022 at 10:42 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 3/18/22 15:43, Tomas Vondra wrote:
Hmmm. So the theory is that in most runs we manage to sync the tables
faster than starting the workers, so we don't hit the limit. But on some
machines the sync worker takes a bit longer, we hit the limit. Seems
possible, yes. Unfortunately we don't seem to log anything when we hit
the limit, so hard to say for sure :-( I suggest we add a WARNING
message to logicalrep_worker_launch or something. Not just because of
this test, it seems useful in general.However, how come we don't retry the sync? Surely we don't just give up
forever, that'd be a pretty annoying behavior. Presumably we just end up
sleeping for a long time before restarting the sync worker, somewhere.I tried lowering the max_sync_workers_per_subscription to 1 and making
the workers to run for a couple seconds (doing some CPU intensive
stuff), but everything still works just fine.
Did the apply worker restarts during that time? If not you can try by
changing some subscription parameters which leads to its restart. This
has to happen before copy_table has finished. In the LOGS, you should
see the message: "logical replication apply worker for subscription
"<subscription_name>" will restart because of a parameter change".
IIUC, the code which doesn't allow to restart the apply worker after
the max_sync_workers_per_subscription is reached is as below:
logicalrep_worker_launch()
{
...
if (nsyncworkers >= max_sync_workers_per_subscription)
{
LWLockRelease(LogicalRepWorkerLock);
return;
}
...
}
This happens before we allocate a worker to apply. So, it can happen
only during the restart of the apply worker because we always first
the apply worker, so in that case, it will never restart.
Looking a bit closer at the logs (from pogona and other), I doubt this
is about hitting the max_sync_workers_per_subscription limit. Notice we
start two sync workers, but neither of them ever completes. So we never
update the sync status or start syncing the remaining tables.
I think they are never completed because they are in a sort of
infinite loop. If you see process_syncing_tables_for_sync(), it will
never mark the status as SUBREL_STATE_SYNCDONE unless apply worker has
set it to SUBREL_STATE_CATCHUP. In LogicalRepSyncTableStart(), we do
wait for a state change to catchup via wait_for_worker_state_change(),
but we bail out in that function if the apply worker has died. After
that tablesync worker won't be able to complete because in our case
apply worker won't be able to restart.
So the question is why those two sync workers never complete - I guess
there's some sort of lock wait (deadlock?) or infinite loop.
It would be a bit tricky to reproduce this even if the above theory is
correct but I'll try it today or tomorrow.
--
With Regards,
Amit Kapila.
On Sun, Mar 20, 2022 at 8:41 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Mar 18, 2022 at 10:42 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:So the question is why those two sync workers never complete - I guess
there's some sort of lock wait (deadlock?) or infinite loop.It would be a bit tricky to reproduce this even if the above theory is
correct but I'll try it today or tomorrow.
I am able to reproduce it with the help of a debugger. Firstly, I have
added the LOG message and some While (true) loops to debug sync and
apply workers. Test setup
Node-1:
create table t1(c1);
create table t2(c1);
insert into t1 values(1);
create publication pub1 for table t1;
create publication pu2;
Node-2:
change max_sync_workers_per_subscription to 1 in potgresql.conf
create table t1(c1);
create table t2(c1);
create subscription sub1 connection 'dbname = postgres' publication pub1;
Till this point, just allow debuggers in both workers just continue.
Node-1:
alter publication pub1 add table t2;
insert into t1 values(2);
Here, we have to debug the apply worker such that when it tries to
apply the insert, stop the debugger in function apply_handle_insert()
after doing begin_replication_step().
Node-2:
alter subscription sub1 set pub1, pub2;
Now, continue the debugger of apply worker, it should first start the
sync worker and then exit because of parameter change. All of these
debugging steps are to just ensure the point that it should first
start the sync worker and then exit. After this point, table sync
worker never finishes and log is filled with messages: "reached
max_sync_workers_per_subscription limit" (a newly added message by me
in the attached debug patch).
Now, it is not completely clear to me how exactly '013_partition.pl'
leads to this situation but there is a possibility based on the LOGs
it shows.
--
With Regards,
Amit Kapila.
Attachments:
debug_sub_workers_1.patchapplication/octet-stream; name=debug_sub_workers_1.patchDownload
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 6f25b2c2ad..efd8852a10 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -351,6 +351,7 @@ retry:
if (nsyncworkers >= max_sync_workers_per_subscription)
{
LWLockRelease(LogicalRepWorkerLock);
+ elog(LOG, "reached max_sync_workers_per_subscription limit");
return;
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 03e069c7cd..e9c2a135d8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2611,6 +2611,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
CHECK_FOR_INTERRUPTS();
MemoryContextSwitchTo(ApplyMessageContext);
+ while (1)
+ {
+ }
len = walrcv_receive(LogRepWorkerWalRcvConn, &buf, &fd);
@@ -3508,6 +3511,13 @@ ApplyWorkerMain(Datum main_arg)
StartTransactionCommand();
oldctx = MemoryContextSwitchTo(ApplyContext);
+ if (am_tablesync_worker())
+ {
+ while (1)
+ {
+ }
+ }
+
MySubscription = GetSubscription(MyLogicalRepWorker->subid, true);
if (!MySubscription)
{
On 3/20/22 07:23, Amit Kapila wrote:
On Sun, Mar 20, 2022 at 8:41 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Mar 18, 2022 at 10:42 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:So the question is why those two sync workers never complete - I guess
there's some sort of lock wait (deadlock?) or infinite loop.It would be a bit tricky to reproduce this even if the above theory is
correct but I'll try it today or tomorrow.I am able to reproduce it with the help of a debugger. Firstly, I have
added the LOG message and some While (true) loops to debug sync and
apply workers. Test setupNode-1:
create table t1(c1);
create table t2(c1);
insert into t1 values(1);
create publication pub1 for table t1;
create publication pu2;Node-2:
change max_sync_workers_per_subscription to 1 in potgresql.conf
create table t1(c1);
create table t2(c1);
create subscription sub1 connection 'dbname = postgres' publication pub1;Till this point, just allow debuggers in both workers just continue.
Node-1:
alter publication pub1 add table t2;
insert into t1 values(2);Here, we have to debug the apply worker such that when it tries to
apply the insert, stop the debugger in function apply_handle_insert()
after doing begin_replication_step().Node-2:
alter subscription sub1 set pub1, pub2;Now, continue the debugger of apply worker, it should first start the
sync worker and then exit because of parameter change. All of these
debugging steps are to just ensure the point that it should first
start the sync worker and then exit. After this point, table sync
worker never finishes and log is filled with messages: "reached
max_sync_workers_per_subscription limit" (a newly added message by me
in the attached debug patch).Now, it is not completely clear to me how exactly '013_partition.pl'
leads to this situation but there is a possibility based on the LOGs
it shows.
Thanks, I'll take a look later. From the description it seems this is an
issue that existed before any of the patches, right? It might be more
likely to hit due to some test changes, but the root cause is older.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Sun, Mar 20, 2022 at 4:53 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 3/20/22 07:23, Amit Kapila wrote:
On Sun, Mar 20, 2022 at 8:41 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Mar 18, 2022 at 10:42 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:So the question is why those two sync workers never complete - I guess
there's some sort of lock wait (deadlock?) or infinite loop.It would be a bit tricky to reproduce this even if the above theory is
correct but I'll try it today or tomorrow.I am able to reproduce it with the help of a debugger. Firstly, I have
added the LOG message and some While (true) loops to debug sync and
apply workers. Test setupNode-1:
create table t1(c1);
create table t2(c1);
insert into t1 values(1);
create publication pub1 for table t1;
create publication pu2;Node-2:
change max_sync_workers_per_subscription to 1 in potgresql.conf
create table t1(c1);
create table t2(c1);
create subscription sub1 connection 'dbname = postgres' publication pub1;Till this point, just allow debuggers in both workers just continue.
Node-1:
alter publication pub1 add table t2;
insert into t1 values(2);Here, we have to debug the apply worker such that when it tries to
apply the insert, stop the debugger in function apply_handle_insert()
after doing begin_replication_step().Node-2:
alter subscription sub1 set pub1, pub2;Now, continue the debugger of apply worker, it should first start the
sync worker and then exit because of parameter change. All of these
debugging steps are to just ensure the point that it should first
start the sync worker and then exit. After this point, table sync
worker never finishes and log is filled with messages: "reached
max_sync_workers_per_subscription limit" (a newly added message by me
in the attached debug patch).Now, it is not completely clear to me how exactly '013_partition.pl'
leads to this situation but there is a possibility based on the LOGs
it shows.Thanks, I'll take a look later. From the description it seems this is an
issue that existed before any of the patches, right? It might be more
likely to hit due to some test changes, but the root cause is older.
Yes, your understanding is correct. If my understanding is correct,
then we need probably just need some changes in the new test to make
it behave as per the current code.
--
With Regards,
Amit Kapila.
On Fri, Mar 18, 2022 at 8:13 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
Ah, thanks for reminding me - it's hard to keep track of all the issues
in threads as long as this one.BTW do you have any opinion on the SET COLUMNS syntax? Peter Smith
proposed to get rid of it in [1] but I'm not sure that's a good idea.
Because if we ditch it, then removing the column list would look like this:ALTER PUBLICATION pub ALTER TABLE tab;
And if we happen to add other per-table options, this would become
pretty ambiguous.Actually, do we even want to allow resetting column lists like this? We
don't allow this for row filters, so if you want to change a row filter
you have to re-add the table, right?
We can use syntax like: "alter publication pub1 set table t1 where (c2
10);" to reset the existing row filter. It seems similar thing works
for column list as well ("alter publication pub1 set table t1 (c2)
where (c2 > 10)"). If I am not missing anything, I don't think we need
additional Alter Table syntax.
So maybe we should just ditch ALTER
TABLE entirely.
Yeah, I agree especially if my above understanding is correct.
--
With Regards,
Amit Kapila.
On Sat, Mar 19, 2022 at 3:56 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 3/18/22 15:43, Tomas Vondra wrote:
As for the issue reported by Shi-San about replica identity full and
column filters, presumably you're referring to this:create table tbl (a int, b int, c int);
create publication pub for table tbl (a, b, c);
alter table tbl replica identity full;postgres=# delete from tbl;
ERROR: cannot delete from table "tbl"
DETAIL: Column list used by the publication does not cover the
replica identity.I believe not allowing column lists with REPLICA IDENTITY FULL is
expected / correct behavior. I mean, for that to work the column list
has to always include all columns anyway, so it's pretty pointless. Of
course, we might check that the column list contains everything, but
considering the list does always have to contain all columns, and it
break as soon as you add any columns, it seems reasonable (cheaper) to
just require no column lists.
Fair point. We can leave this as it is.
I also went through the patch and made the naming more consistent. The
comments used both "column filter" and "column list" randomly, and I
think the agreement is to use "list" so I adopted that wording.However, while looking at how pgoutput, I realized one thing - for row
filters we track them "per operation", depending on which operations are
defined for a given publication. Shouldn't we do the same thing for
column lists, really?I mean, if there are two publications with different column lists, one
for inserts and the other one for updates, isn't it wrong to merge these
two column lists?
The reason we can't combine row filters for inserts with
updates/deletes is that if inserts have some column that is not
present in RI then during update filtering (for old tuple) it will
give an error as the column won't be present in WAL log.
OTOH, the same problem won't be there for the column list/filter patch
because all the RI columns are there in the column list (for
update/delete) and we don't need to apply a column filter for old
tuples in either update or delete.
Basically, the filter rules are slightly different for row filters and
column lists, so we need them (combine of filters) for one but not for
the other. Now, for the sake of consistency with row filters, we can
do it but as such there won't be any problem or maybe we can just add
a comment for the same in code.
--
With Regards,
Amit Kapila.
Hello,
Please add me to the list of authors of this patch. I made a large
number of nontrivial changes to it early on. Thanks. I have modified
the entry in the CF app (which sorts alphabetically, it was not my
intention to put my name first.)
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
On Sat, Mar 19, 2022 at 11:11 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 3/19/22 18:11, Tomas Vondra wrote:
Fix a compiler warning reported by cfbot.
Apologies, I failed to actually commit the fix. So here we go again.
Few comments:
===============
1.
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
{
...
}
...
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
{
...
}
Both these functions are not required now. So, we can remove them.
2.
@@ -464,11 +478,11 @@ logicalrep_write_update(StringInfo out,
TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
As mentioned previously, here, we should pass NULL similar to
logicalrep_write_delete as we don't need to use column list for old
tuples.
3.
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow column lists on the leaf relations. So maybe rename it?
+ */
+static void
+TransformPubColumnList(List *tables, const char *queryString,
+ bool pubviaroot)
The second parameter is not used in this function. As noted in the
comments, I also think it is better to rename this. How about
ValidatePubColumnList?
4.
@@ -821,6 +942,9 @@ fetch_remote_table_info(char *nspname, char *relname,
*
* 3) one of the subscribed publications is declared as ALL TABLES IN
* SCHEMA that includes this relation
+ *
+ * XXX Does this actually handle puballtables and schema publications
+ * correctly?
*/
if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
Why is this comment added in the row filter code? Now, both row filter
and column list are fetched in the same way, so not sure what exactly
this comment is referring to.
5.
+/* qsort comparator for attnums */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}
The exact same code exists in statscmds.c. Do we need a second copy of the same?
6.
static void pgoutput_row_filter_init(PGOutputData *data,
List *publications,
RelationSyncEntry *entry);
+
static bool pgoutput_row_filter_exec_expr(ExprState *state,
Spurious line addition.
7. The tests in 030_column_list.pl take a long time as compared to all
other similar individual tests in the subscription folder. I haven't
checked whether there is any need to reduce some tests but it seems
worth checking.
--
With Regards,
Amit Kapila.
On 2022-Mar-19, Tomas Vondra wrote:
@@ -174,7 +182,13 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete'); <para> Add some tables to the publication: <programlisting> -ALTER PUBLICATION mypublication ADD TABLE users, departments; +ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments; +</programlisting></para> + + <para> + Change the set of columns published for a table: +<programlisting> +ALTER PUBLICATION mypublication SET TABLE users (user_id, firstname, lastname), TABLE departments; </programlisting></para><para>
Hmm, it seems to me that if you've removed the feature to change the set
of columns published for a table, then the second example should be
removed as well.
+/* + * Transform the publication column lists expression for all the relations + * in the list. + * + * XXX The name is a bit misleading, because we don't really transform + * anything here - we merely check the column list is compatible with the + * definition of the publication (with publish_via_partition_root=false) + * we only allow column lists on the leaf relations. So maybe rename it? + */ +static void +TransformPubColumnList(List *tables, const char *queryString, + bool pubviaroot) +{
I agree with renaming this function. Maybe CheckPubRelationColumnList() ?
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"This is a foot just waiting to be shot" (Andrew Dunstan)
On Wed, Mar 23, 2022 at 12:54 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2022-Mar-19, Tomas Vondra wrote:
@@ -174,7 +182,13 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete'); <para> Add some tables to the publication: <programlisting> -ALTER PUBLICATION mypublication ADD TABLE users, departments; +ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments; +</programlisting></para> + + <para> + Change the set of columns published for a table: +<programlisting> +ALTER PUBLICATION mypublication SET TABLE users (user_id, firstname, lastname), TABLE departments; </programlisting></para><para>
Hmm, it seems to me that if you've removed the feature to change the set
of columns published for a table, then the second example should be
removed as well.
As per my understanding, the removed feature is "Alter Publication ...
Alter Table ...". The example here "Alter Publication ... Set Table
.." should still work as mentioned in my email[1]/messages/by-id/CAA4eK1L6YTcx=yJfdudr-y98Wcn4rWX4puHGAa2nxSCRb3fzQw@mail.gmail.com.
[1]: /messages/by-id/CAA4eK1L6YTcx=yJfdudr-y98Wcn4rWX4puHGAa2nxSCRb3fzQw@mail.gmail.com
--
With Regards,
Amit Kapila.
On 2022-Mar-23, Amit Kapila wrote:
On Wed, Mar 23, 2022 at 12:54 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2022-Mar-19, Tomas Vondra wrote:
@@ -174,7 +182,13 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete'); <para> Add some tables to the publication: <programlisting> -ALTER PUBLICATION mypublication ADD TABLE users, departments; +ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments; +</programlisting></para> + + <para> + Change the set of columns published for a table: +<programlisting> +ALTER PUBLICATION mypublication SET TABLE users (user_id, firstname, lastname), TABLE departments; </programlisting></para><para>
Hmm, it seems to me that if you've removed the feature to change the set
of columns published for a table, then the second example should be
removed as well.As per my understanding, the removed feature is "Alter Publication ...
Alter Table ...". The example here "Alter Publication ... Set Table
.." should still work as mentioned in my email[1].
Ah, I see. Yeah, that makes sense. In that case, the leading text
seems a bit confusing. I would suggest "Change the set of tables in the
publication, specifying a different set of columns for one of them:"
I think it would make the example more useful if we table for which the
columns are changing is a different one. Maybe do this:
Add some tables to the publication:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of tables in the publication, keeping the column list
+ in the users table and specifying a different column list for the
+ departments table. Note that previously published tables not mentioned
+ in this command are removed from the publication:
+
+<programlisting>
+ALTER PUBLICATION mypublication SET TABLE users (user_id, firstname), TABLE departments (dept_id, deptname);
</programlisting></para>
so that it is clear that if you want to keep the column list unchanged
in one table, you are forced to specify it again.
(Frankly, this ALTER PUBLICATION SET command seems pretty useless from a
user PoV.)
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Investigación es lo que hago cuando no sé lo que estoy haciendo"
(Wernher von Braun)
On 3/21/22 12:55, Amit Kapila wrote:
On Sat, Mar 19, 2022 at 3:56 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:...
However, while looking at how pgoutput, I realized one thing - for row
filters we track them "per operation", depending on which operations are
defined for a given publication. Shouldn't we do the same thing for
column lists, really?I mean, if there are two publications with different column lists, one
for inserts and the other one for updates, isn't it wrong to merge these
two column lists?The reason we can't combine row filters for inserts with
updates/deletes is that if inserts have some column that is not
present in RI then during update filtering (for old tuple) it will
give an error as the column won't be present in WAL log.OTOH, the same problem won't be there for the column list/filter patch
because all the RI columns are there in the column list (for
update/delete) and we don't need to apply a column filter for old
tuples in either update or delete.Basically, the filter rules are slightly different for row filters and
column lists, so we need them (combine of filters) for one but not for
the other. Now, for the sake of consistency with row filters, we can
do it but as such there won't be any problem or maybe we can just add
a comment for the same in code.
OK, thanks for the explanation. I'll add a comment explaining this to
the function initializing the column filter.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 3/21/22 12:28, Amit Kapila wrote:
On Fri, Mar 18, 2022 at 8:13 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Ah, thanks for reminding me - it's hard to keep track of all the issues
in threads as long as this one.BTW do you have any opinion on the SET COLUMNS syntax? Peter Smith
proposed to get rid of it in [1] but I'm not sure that's a good idea.
Because if we ditch it, then removing the column list would look like this:ALTER PUBLICATION pub ALTER TABLE tab;
And if we happen to add other per-table options, this would become
pretty ambiguous.Actually, do we even want to allow resetting column lists like this? We
don't allow this for row filters, so if you want to change a row filter
you have to re-add the table, right?We can use syntax like: "alter publication pub1 set table t1 where (c2
10);" to reset the existing row filter. It seems similar thing works
for column list as well ("alter publication pub1 set table t1 (c2)
where (c2 > 10)"). If I am not missing anything, I don't think we need
additional Alter Table syntax.So maybe we should just ditch ALTER
TABLE entirely.Yeah, I agree especially if my above understanding is correct.
I think there's a gotcha that
ALTER PUBLICATION pub SET TABLE t ...
also removes all other relations from the publication, and it removes
and re-adds the table anyway. So I'm not sure what's the advantage?
Anyway, I don't see why we would need such ALTER TABLE only for column
filters and not for row filters - either we need to allow this for both
options or none of them.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Thu, Mar 24, 2022 at 4:11 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 3/21/22 12:28, Amit Kapila wrote:
On Fri, Mar 18, 2022 at 8:13 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Ah, thanks for reminding me - it's hard to keep track of all the issues
in threads as long as this one.BTW do you have any opinion on the SET COLUMNS syntax? Peter Smith
proposed to get rid of it in [1] but I'm not sure that's a good idea.
Because if we ditch it, then removing the column list would look like this:ALTER PUBLICATION pub ALTER TABLE tab;
And if we happen to add other per-table options, this would become
pretty ambiguous.Actually, do we even want to allow resetting column lists like this? We
don't allow this for row filters, so if you want to change a row filter
you have to re-add the table, right?We can use syntax like: "alter publication pub1 set table t1 where (c2
10);" to reset the existing row filter. It seems similar thing works
for column list as well ("alter publication pub1 set table t1 (c2)
where (c2 > 10)"). If I am not missing anything, I don't think we need
additional Alter Table syntax.So maybe we should just ditch ALTER
TABLE entirely.Yeah, I agree especially if my above understanding is correct.
I think there's a gotcha that
ALTER PUBLICATION pub SET TABLE t ...
also removes all other relations from the publication, and it removes
and re-adds the table anyway. So I'm not sure what's the advantage?
I think it could be used when the user has fewer tables and she wants
to change the list of published tables or their row/column filters. I
am not sure of the value of this to users but this was a pre-existing
syntax.
Anyway, I don't see why we would need such ALTER TABLE only for column
filters and not for row filters - either we need to allow this for both
options or none of them.
+1. I think for now we can leave this new ALTER TABLE syntax and do it
for both column and row filters together.
--
With Regards,
Amit Kapila.
On 17.03.22 20:11, Tomas Vondra wrote:
But the comment describes the error for the whole block, which looks
like this:-- error: replica identity "a" not included in the column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica
identity.So IMHO the comment is correct.
Ok, that makes sense. I read all the comments in the test file again.
There were a couple that I think could use tweaking; see attached file.
The ones with "???" didn't make sense to me: The first one is before a
command that doesn't seem to change anything, the second one I didn't
understand the meaning. Please take a look.
(The patch is actually based on your 20220318c patch, but I'm adding it
here since we have the discussion here.)
Attachments:
0001-fixup-Allow-specifying-column-lists-for-logical-repl.patchtext/plain; charset=UTF-8; name=0001-fixup-Allow-specifying-column-lists-for-logical-repl.patchDownload
From 2e6352791e5418bb0726a051660d44311046fc28 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Thu, 24 Mar 2022 17:30:32 +0100
Subject: [PATCH] fixup! Allow specifying column lists for logical replication
---
src/test/regress/sql/publication.sql | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index aeb1b572af..d50052ef9d 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -399,14 +399,14 @@ CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
-- ok
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
--- ok: for insert-only publication, the column list is arbitrary
+-- ok: for insert-only publication, any column list is acceptable
ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
/* not all replica identities are good enough */
CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
--- error: replica identity (b,c) is covered by column list (a, c)
+-- error: replica identity (b,c) is not covered by column list (a, c)
UPDATE testpub_tbl5 SET a = 1;
ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
@@ -423,7 +423,7 @@ CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
\dRp+ testpub_table_ins
--- with REPLICA IDENTITY FULL, column lists are not allowed
+-- tests with REPLICA IDENTITY FULL
CREATE TABLE testpub_tbl6 (a int, b text, c text);
ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
@@ -434,11 +434,11 @@ CREATE TABLE testpub_tbl6 (a int, b text, c text);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
UPDATE testpub_tbl6 SET a = 1;
--- make sure changing the column list is updated in SET TABLE
+-- make sure changing the column list is updated in SET TABLE ???
CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
\d+ testpub_tbl7
--- ok: we'll skip this table
+-- ok: we'll skip this table ???
ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
\d+ testpub_tbl7
-- ok: update the column list
--
2.35.1
On 3/24/22 17:33, Peter Eisentraut wrote:
On 17.03.22 20:11, Tomas Vondra wrote:
But the comment describes the error for the whole block, which looks
like this:-- error: replica identity "a" not included in the column list
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
UPDATE testpub_tbl5 SET a = 1;
ERROR: cannot update table "testpub_tbl5"
DETAIL: Column list used by the publication does not cover the replica
identity.So IMHO the comment is correct.
Ok, that makes sense. I read all the comments in the test file again.
There were a couple that I think could use tweaking; see attached file.
The ones with "???" didn't make sense to me: The first one is before a
command that doesn't seem to change anything, the second one I didn't
understand the meaning. Please take a look.
Thanks, the proposed changes seem reasonable. As for the two unclear
tests/comments:
-- make sure changing the column list is updated in SET TABLE (???)
CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
\d+ testpub_tbl7
-- ok: we'll skip this table (???)
ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
\d+ testpub_tbl7
-- ok: update the column list
ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
\d+ testpub_tbl7
The goal of this test is to verify that we handle column lists correctly
in SET TABLE. That is, if the column list matches the currently set one,
we should just skip the table in SET TABLE. If it's different, we need
to update the catalog. That's what the first comment is trying to say.
It's true we can't really check we skip the table in the SetObject code,
but we can at least ensure there's no error and the column list remains
the same.
And we're not replicating any data in regression tests, so it might
happen we discard the new column list, for example. Hence the second
test, which ensures we end up with the modified column list.
Attached is a patch, rebased on top of the sequence decoding stuff I
pushed earlier today, also including the comments rewording, and
renaming the "transform" function.
I'll go over it again and get it pushed soon, unless someone objects.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
0001-Allow-specifying-column-lists-for-logical-re20220325.patchtext/x-patch; charset=UTF-8; name=0001-Allow-specifying-column-lists-for-logical-re20220325.patchDownload
From 9e8931393e57ad1cd5e40c0007980331d1fdbeec Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Thu, 24 Mar 2022 23:23:57 +0100
Subject: [PATCH] Allow specifying column lists for logical replication
This allows specifying an optional column list when adding a table to
logical replication. Columns not included on this list are not sent to
the subscriber. The list is specified after the table name, enclosed
in parentheses.
For UPDATE/DELETE publications, the column list needs to cover all
REPLICA IDENTITY columns. For INSERT publications, the column list is
arbitrary and may omit some REPLICA IDENTITY columns. Furthermore, if
the table uses REPLICA IDENTITY FULL, column list is not allowed.
The column list can contain only simple column references. Complex
expressions, function calls etc. are not allowed. This restriction could
be relaxed in the future.
During the initial table synchronization, only columns specified in the
column list are copied to the subscriber. If the subscription has
several publications, containing the same table with different column
lists, columns specified in any of the lists will be copied. This
means all columns are replicated if the table has no column list at
all (which is treated as column list with all columns), of when of the
publications is defined as FOR ALL TABLES (possibly IN SCHEMA for the
schema of the table).
For partitioned tables, publish_via_partition_root determines whether
the column list for the root or leaf relation will be used. If the
parameter is 'false' (the default), the list defined for the leaf
relation is used. Otherwise, the column list for the root partition
will be used.
Psql commands \dRp+ and \d <table-name> now display any column lists.
Author: Rahila Syed, Alvaro Herrera, Tomas Vondra
Reviewed-by: Peter Eisentraut, Alvaro Herrera, Vignesh C, Ibrar Ahmed,
Amit Kapila, Hou zj, Peter Smith, Wang wei, Tang, Shi yu
Discussion: https://postgr.es/m/CAH2L28vddB_NFdRVpuyRBJEBWjz4BSyTB=_ektNRH8NJ1jf95g@mail.gmail.com
---
doc/src/sgml/catalogs.sgml | 15 +-
doc/src/sgml/protocol.sgml | 3 +-
doc/src/sgml/ref/alter_publication.sgml | 18 +-
doc/src/sgml/ref/create_publication.sgml | 17 +-
src/backend/catalog/pg_publication.c | 221 ++++
src/backend/commands/publicationcmds.c | 271 ++++-
src/backend/executor/execReplication.c | 19 +-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 33 +-
src/backend/replication/logical/proto.c | 61 +-
src/backend/replication/logical/tablesync.c | 156 ++-
src/backend/replication/pgoutput/pgoutput.c | 202 +++-
src/backend/utils/cache/relcache.c | 33 +-
src/bin/pg_dump/pg_dump.c | 41 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 60 +
src/bin/psql/describe.c | 44 +-
src/include/catalog/pg_publication.h | 14 +
src/include/catalog/pg_publication_rel.h | 1 +
src/include/commands/publicationcmds.h | 4 +-
src/include/nodes/parsenodes.h | 1 +
src/include/replication/logicalproto.h | 6 +-
src/test/regress/expected/publication.out | 372 ++++++
src/test/regress/sql/publication.sql | 287 +++++
src/test/subscription/t/030_column_list.pl | 1124 +++++++++++++++++++
26 files changed, 2914 insertions(+), 92 deletions(-)
create mode 100644 src/test/subscription/t/030_column_list.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index b8c954a5547..560e205b95f 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -4410,7 +4410,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para>
<para>
This is an array of <structfield>indnatts</structfield> values that
- indicate which table columns this index indexes. For example a value
+ indicate which table columns this index indexes. For example, a value
of <literal>1 3</literal> would mean that the first and the third table
columns make up the index entries. Key columns come before non-key
(included) columns. A zero in this array indicates that the
@@ -6291,6 +6291,19 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to schema
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>prattrs</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+ </para>
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example, a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all columns are published.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index c61c310e176..635d4cc30a7 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7016,7 +7016,8 @@ Relation
</listitem>
</varlistentry>
</variablelist>
- Next, the following message part appears for each column (except generated columns):
+ Next, the following message part appears for each column included in
+ the publication (except generated columns):
<variablelist>
<varlistentry>
<term>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index a8cc8f3dc25..40366a10fed 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -30,7 +30,7 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
SEQUENCE <replaceable class="parameter">sequence_name</replaceable> [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
ALL SEQUENCES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
@@ -114,6 +114,14 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
specified, the table and all its descendant tables (if any) are
affected. Optionally, <literal>*</literal> can be specified after the table
name to explicitly indicate that descendant tables are included.
+ </para>
+
+ <para>
+ Optionally, a column list can be specified. See <xref
+ linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
If the optional <literal>WHERE</literal> clause is specified, rows for
which the <replaceable class="parameter">expression</replaceable>
evaluates to false or null will not be published. Note that parentheses
@@ -185,7 +193,13 @@ ALTER PUBLICATION noinsert SET (publish = 'update, delete');
<para>
Add some tables to the publication:
<programlisting>
-ALTER PUBLICATION mypublication ADD TABLE users, departments;
+ALTER PUBLICATION mypublication ADD TABLE users (user_id, firstname), departments;
+</programlisting></para>
+
+ <para>
+ Change the set of columns published for a table:
+<programlisting>
+ALTER PUBLICATION mypublication SET TABLE users (user_id, firstname, lastname), TABLE departments;
</programlisting></para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index e5081eb50ea..d2739968d9c 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -33,7 +33,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
- TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
SEQUENCE <replaceable class="parameter">sequence_name</replaceable> [ * ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
ALL SEQUENCES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
@@ -93,6 +93,13 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<literal>TRUNCATE</literal> commands.
</para>
+ <para>
+ When a column list is specified, only the named columns are replicated.
+ If no column list is specified, all columns of the table are replicated
+ through this publication, including any columns added later. If a column
+ list is specified, it must include the replica identity columns.
+ </para>
+
<para>
Only persistent base tables and partitioned tables can be part of a
publication. Temporary tables, unlogged tables, foreign tables,
@@ -348,6 +355,14 @@ CREATE PUBLICATION production_publication FOR TABLE users, departments, ALL TABL
<structname>sales</structname>:
<programlisting>
CREATE PUBLICATION sales_publication FOR ALL TABLES IN SCHEMA marketing, sales;
+</programlisting></para>
+
+ <para>
+ Create a publication that publishes all changes for table <structname>users</structname>,
+ but replicates only columns <structname>user_id</structname> and
+ <structname>firstname</structname>:
+<programlisting>
+CREATE PUBLICATION users_filtered FOR TABLE users (user_id, firstname);
</programlisting></para>
</refsect1>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 5bcfc94e2ba..a346b9ee186 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -45,6 +45,9 @@
#include "utils/rel.h"
#include "utils/syscache.h"
+static void publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs);
+
/*
* Check if relation can be in given publication and throws appropriate
* error if not.
@@ -395,6 +398,8 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
Oid relid = RelationGetRelid(targetrel);
Oid pubreloid;
Publication *pub = GetPublication(pubid);
+ AttrNumber *attarray = NULL;
+ int natts = 0;
ObjectAddress myself,
referenced;
List *relids = NIL;
@@ -422,6 +427,14 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
check_publication_add_relation(targetrel);
+ /*
+ * Translate column names to attnums and make sure the column list contains
+ * only allowed elements (no system or generated columns etc.). Also build
+ * an array of attnums, for storing in the catalog.
+ */
+ publication_translate_columns(pri->relation, pri->columns,
+ &natts, &attarray);
+
/* Form a tuple. */
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
@@ -440,6 +453,12 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
else
nulls[Anum_pg_publication_rel_prqual - 1] = true;
+ /* Add column list, if available */
+ if (pri->columns)
+ values[Anum_pg_publication_rel_prattrs - 1] = PointerGetDatum(buildint2vector(attarray, natts));
+ else
+ nulls[Anum_pg_publication_rel_prattrs - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -463,6 +482,13 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
DEPENDENCY_NORMAL, DEPENDENCY_NORMAL,
false);
+ /* Add dependency on the columns, if any are listed */
+ for (int i = 0; i < natts; i++)
+ {
+ ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL);
+ }
+
/* Close the table. */
table_close(rel, RowExclusiveLock);
@@ -482,6 +508,125 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri,
return myself;
}
+/* qsort comparator for attnums */
+static int
+compare_int16(const void *a, const void *b)
+{
+ int av = *(const int16 *) a;
+ int bv = *(const int16 *) b;
+
+ /* this can't overflow if int is wider than int16 */
+ return (av - bv);
+}
+
+/*
+ * Translate a list of column names to an array of attribute numbers
+ * and a Bitmapset with them; verify that each attribute is appropriate
+ * to have in a publication column list (no system or generated attributes,
+ * no duplicates). Additional checks with replica identity are done later;
+ * see check_publication_columns.
+ *
+ * Note that the attribute numbers are *not* offset by
+ * FirstLowInvalidHeapAttributeNumber; system columns are forbidden so this
+ * is okay.
+ */
+static void
+publication_translate_columns(Relation targetrel, List *columns,
+ int *natts, AttrNumber **attrs)
+{
+ AttrNumber *attarray = NULL;
+ Bitmapset *set = NULL;
+ ListCell *lc;
+ int n = 0;
+ TupleDesc tupdesc = RelationGetDescr(targetrel);
+
+ /* Bail out when no column list defined. */
+ if (!columns)
+ return;
+
+ /*
+ * Translate list of columns to attnums. We prohibit system attributes and
+ * make sure there are no duplicate columns.
+ */
+ attarray = palloc(sizeof(AttrNumber) * list_length(columns));
+ foreach(lc, columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(RelationGetRelid(targetrel), colname);
+
+ if (attnum == InvalidAttrNumber)
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ colname, RelationGetRelationName(targetrel)));
+
+ if (!AttrNumberIsForUserDefinedAttr(attnum))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference system column \"%s\" in publication column list",
+ colname));
+
+ if (TupleDescAttr(tupdesc, attnum - 1)->attgenerated)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot reference generated column \"%s\" in publication column list",
+ colname));
+
+ if (bms_is_member(attnum, set))
+ ereport(ERROR,
+ errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("duplicate column \"%s\" in publication column list",
+ colname));
+
+ set = bms_add_member(set, attnum);
+ attarray[n++] = attnum;
+ }
+
+ /* Be tidy, so that the catalog representation is always sorted */
+ qsort(attarray, n, sizeof(AttrNumber), compare_int16);
+
+ *natts = n;
+ *attrs = attarray;
+
+ bms_free(set);
+}
+
+/*
+ * Transform the column list (represented by an array) to a bitmapset.
+ */
+Bitmapset *
+pub_collist_to_bitmapset(Bitmapset *columns, Datum pubcols, MemoryContext mcxt)
+{
+ Bitmapset *result = NULL;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ MemoryContext oldcxt;
+
+ /*
+ * If an existing bitmap was provided, use it. Otherwise just use NULL
+ * and build a new bitmap.
+ */
+ if (columns)
+ result = columns;
+
+ arr = DatumGetArrayTypeP(pubcols);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ /* If a memory context was specified, switch to it. */
+ if (mcxt)
+ oldcxt = MemoryContextSwitchTo(mcxt);
+
+ for (int i = 0; i < nelems; i++)
+ result = bms_add_member(result, elems[i]);
+
+ if (mcxt)
+ MemoryContextSwitchTo(oldcxt);
+
+ return result;
+}
+
/*
* Insert new publication / schema mapping.
*/
@@ -594,6 +739,82 @@ GetRelationPublications(Oid relid)
return result;
}
+/*
+ * Gets a list of OIDs of all partial-column publications of the given
+ * relation, that is, those that specify a column list.
+ */
+List *
+GetRelationColumnPartialPublications(Oid relid)
+{
+ CatCList *pubrellist;
+ List *pubs = NIL;
+
+ pubrellist = SearchSysCacheList1(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid));
+ for (int i = 0; i < pubrellist->n_members; i++)
+ {
+ HeapTuple tup = &pubrellist->members[i]->tuple;
+ bool isnull;
+ Form_pg_publication_rel pubrel;
+
+ (void) SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ /* no column list for this publications/relation */
+ if (isnull)
+ continue;
+
+ pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
+
+ pubs = lappend_oid(pubs, pubrel->prpubid);
+ }
+
+ ReleaseSysCacheList(pubrellist);
+
+ return pubs;
+}
+
+
+/*
+ * For a relation in a publication that is known to have a non-null column
+ * list, return the list of attribute numbers that are in it.
+ */
+List *
+GetRelationColumnListInPublication(Oid relid, Oid pubid)
+{
+ HeapTuple tup;
+ Datum adatum;
+ bool isnull;
+ ArrayType *arr;
+ int nelems;
+ int16 *elems;
+ List *attnos = NIL;
+
+ tup = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for rel %u of publication %u", relid, pubid);
+
+ adatum = SysCacheGetAttr(PUBLICATIONRELMAP, tup,
+ Anum_pg_publication_rel_prattrs, &isnull);
+ if (isnull)
+ elog(ERROR, "found unexpected null in pg_publication_rel.prattrs");
+
+ arr = DatumGetArrayTypeP(adatum);
+ nelems = ARR_DIMS(arr)[0];
+ elems = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < nelems; i++)
+ attnos = lappend_oid(attnos, elems[i]);
+
+ ReleaseSysCache(tup);
+
+ return attnos;
+}
+
/*
* Gets list of relation oids for a publication.
*
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index f890d3f0baa..9be7f8cf003 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -347,7 +347,7 @@ contain_invalid_rfcolumn_walker(Node *node, rf_context *context)
* Returns true if any invalid column is found.
*/
bool
-contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
+pub_rf_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
bool pubviaroot)
{
HeapTuple rftuple;
@@ -419,6 +419,114 @@ contain_invalid_rfcolumn(Oid pubid, Relation relation, List *ancestors,
return result;
}
+/*
+ * Check if all columns referenced in the REPLICA IDENTITY are covered by
+ * the column list.
+ *
+ * Returns true if any replica identity column is not covered by column list.
+ */
+bool
+pub_collist_contains_invalid_column(Oid pubid, Relation relation, List *ancestors,
+ bool pubviaroot)
+{
+ HeapTuple tuple;
+ Oid relid = RelationGetRelid(relation);
+ Oid publish_as_relid = RelationGetRelid(relation);
+ bool result = false;
+ Datum datum;
+ bool isnull;
+
+ /*
+ * For a partition, if pubviaroot is true, find the topmost ancestor that
+ * is published via this publication as we need to use its column list
+ * for the changes.
+ *
+ * Note that even though the column list used is for an ancestor, the
+ * REPLICA IDENTITY used will be for the actual child table.
+ */
+ if (pubviaroot && relation->rd_rel->relispartition)
+ {
+ publish_as_relid = GetTopMostAncestorInPublication(pubid, ancestors, NULL);
+
+ if (!OidIsValid(publish_as_relid))
+ publish_as_relid = relid;
+ }
+
+ tuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(publish_as_relid),
+ ObjectIdGetDatum(pubid));
+
+ if (!HeapTupleIsValid(tuple))
+ return false;
+
+ datum = SysCacheGetAttr(PUBLICATIONRELMAP, tuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ if (!isnull)
+ {
+ int x;
+ Bitmapset *idattrs;
+ Bitmapset *columns = NULL;
+
+ /* With REPLICA IDENTITY FULL, no column list is allowed. */
+ if (relation->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ result = true;
+
+ /* Transform the column list datum to a bitmapset. */
+ columns = pub_collist_to_bitmapset(NULL, datum, NULL);
+
+ /* Remember columns that are part of the REPLICA IDENTITY */
+ idattrs = RelationGetIndexAttrBitmap(relation,
+ INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ /*
+ * Attnums in the bitmap returned by RelationGetIndexAttrBitmap are
+ * offset (to handle system columns the usual way), while column list
+ * does not use offset, so we can't do bms_is_subset(). Instead, we have
+ * to loop over the idattrs and check all of them are in the list.
+ */
+ x = -1;
+ while ((x = bms_next_member(idattrs, x)) >= 0)
+ {
+ AttrNumber attnum = (x + FirstLowInvalidHeapAttributeNumber);
+
+ /*
+ * If pubviaroot is true, we are validating the column list of the
+ * parent table, but the bitmap contains the replica identity
+ * information of the child table. The parent/child attnums may not
+ * match, so translate them to the parent - get the attname from
+ * the child, and look it up in the parent.
+ */
+ if (pubviaroot)
+ {
+ /* attribute name in the child table */
+ char *colname = get_attname(relid, attnum, false);
+
+ /*
+ * Determine the attnum for the attribute name in parent (we
+ * are using the column list defined on the parent).
+ */
+ attnum = get_attnum(publish_as_relid, colname);
+ }
+
+ /* replica identity column, not covered by the column list */
+ if (!bms_is_member(attnum, columns))
+ {
+ result = true;
+ break;
+ }
+ }
+
+ bms_free(idattrs);
+ bms_free(columns);
+ }
+
+ ReleaseSysCache(tuple);
+
+ return result;
+}
+
/* check_functions_in_node callback */
static bool
contain_mutable_or_user_functions_checker(Oid func_id, void *context)
@@ -660,6 +768,45 @@ TransformPubWhereClauses(List *tables, const char *queryString,
}
}
+
+/*
+ * Transform the publication column lists expression for all the relations
+ * in the list.
+ *
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow column lists on the leaf relations. So maybe rename it?
+ */
+static void
+CheckPubRelationColumnList(List *tables, const char *queryString,
+ bool pubviaroot)
+{
+ ListCell *lc;
+
+ foreach(lc, tables)
+ {
+ PublicationRelInfo *pri = (PublicationRelInfo *) lfirst(lc);
+
+ if (pri->columns == NIL)
+ continue;
+
+ /*
+ * If the publication doesn't publish changes via the root partitioned
+ * table, the partition's column list will be used. So disallow using
+ * the column list on partitioned table in this case.
+ */
+ if (!pubviaroot &&
+ pri->relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot use publication column list for relation \"%s\"",
+ RelationGetRelationName(pri->relation)),
+ errdetail("column list cannot be used for a partitioned table when %s is false.",
+ "publish_via_partition_root")));
+ }
+}
+
/*
* Create new publication.
*/
@@ -812,6 +959,9 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
TransformPubWhereClauses(rels, pstate->p_sourcetext,
publish_via_partition_root);
+ CheckPubRelationColumnList(rels, pstate->p_sourcetext,
+ publish_via_partition_root);
+
PublicationAddRelations(puboid, rels, true, NULL);
CloseRelationList(rels);
}
@@ -899,8 +1049,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* If the publication doesn't publish changes via the root partitioned
- * table, the partition's row filter will be used. So disallow using WHERE
- * clause on partitioned table in this case.
+ * table, the partition's row filter and column list will be used. So disallow
+ * using WHERE clause and column lists on partitioned table in this case.
*/
if (!pubform->puballtables && publish_via_partition_root_given &&
!publish_via_partition_root)
@@ -908,7 +1058,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
/*
* Lock the publication so nobody else can do anything with it. This
* prevents concurrent alter to add partitioned table(s) with WHERE
- * clause(s) which we don't allow when not publishing via root.
+ * clause(s) and/or column lists which we don't allow when not
+ * publishing via root.
*/
LockDatabaseObject(PublicationRelationId, pubform->oid, 0,
AccessShareLock);
@@ -921,13 +1072,21 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
HeapTuple rftuple;
Oid relid = lfirst_oid(lc);
+ bool has_column_list;
+ bool has_row_filter;
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubform->oid));
+ has_row_filter
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL);
+
+ has_column_list
+ = !heap_attisnull(rftuple, Anum_pg_publication_rel_prattrs, NULL);
+
if (HeapTupleIsValid(rftuple) &&
- !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL))
+ (has_row_filter || has_column_list))
{
HeapTuple tuple;
@@ -936,7 +1095,8 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
{
Form_pg_class relform = (Form_pg_class) GETSTRUCT(tuple);
- if (relform->relkind == RELKIND_PARTITIONED_TABLE)
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_row_filter)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot set %s for publication \"%s\"",
@@ -947,6 +1107,18 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
NameStr(relform->relname),
"publish_via_partition_root")));
+ if ((relform->relkind == RELKIND_PARTITIONED_TABLE) &&
+ has_column_list)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot set %s for publication \"%s\"",
+ "publish_via_partition_root = false",
+ stmt->pubname),
+ errdetail("The publication contains a column list for a partitioned table \"%s\" "
+ "which is not allowed when %s is false.",
+ NameStr(relform->relname),
+ "publish_via_partition_root")));
+
ReleaseSysCache(tuple);
}
@@ -1111,6 +1283,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ CheckPubRelationColumnList(rels, queryString, pubform->pubviaroot);
+
PublicationAddRelations(pubid, rels, false, stmt);
}
else if (stmt->action == AP_DropObjects)
@@ -1128,6 +1302,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
TransformPubWhereClauses(rels, queryString, pubform->pubviaroot);
+ CheckPubRelationColumnList(rels, queryString, pubform->pubviaroot);
+
/*
* To recreate the relation list for the publication, look for
* existing relations that do not need to be dropped.
@@ -1139,42 +1315,79 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
PublicationRelInfo *oldrel;
bool found = false;
HeapTuple rftuple;
- bool rfisnull = true;
Node *oldrelwhereclause = NULL;
+ Bitmapset *oldcolumns = NULL;
/* look up the cache for the old relmap */
rftuple = SearchSysCache2(PUBLICATIONRELMAP,
ObjectIdGetDatum(oldrelid),
ObjectIdGetDatum(pubid));
+ /*
+ * See if the existing relation currently has a WHERE clause or a
+ * column list. We need to compare those too.
+ */
if (HeapTupleIsValid(rftuple))
{
+ bool isnull = true;
Datum whereClauseDatum;
+ Datum columnListDatum;
+ /* Load the WHERE clause for this table. */
whereClauseDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
Anum_pg_publication_rel_prqual,
- &rfisnull);
- if (!rfisnull)
+ &isnull);
+ if (!isnull)
oldrelwhereclause = stringToNode(TextDatumGetCString(whereClauseDatum));
+ /* Transform the int2vector column list to a bitmap. */
+ columnListDatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple,
+ Anum_pg_publication_rel_prattrs,
+ &isnull);
+
+ if (!isnull)
+ oldcolumns = pub_collist_to_bitmapset(NULL, columnListDatum, NULL);
+
ReleaseSysCache(rftuple);
}
foreach(newlc, rels)
{
PublicationRelInfo *newpubrel;
+ Oid newrelid;
+ Bitmapset *newcolumns = NULL;
newpubrel = (PublicationRelInfo *) lfirst(newlc);
+ newrelid = RelationGetRelid(newpubrel->relation);
+
+ /*
+ * If the new publication has column list, transform it to
+ * a bitmap too.
+ */
+ if (newpubrel->columns)
+ {
+ ListCell *lc;
+
+ foreach(lc, newpubrel->columns)
+ {
+ char *colname = strVal(lfirst(lc));
+ AttrNumber attnum = get_attnum(newrelid, colname);
+
+ newcolumns = bms_add_member(newcolumns, attnum);
+ }
+ }
/*
* Check if any of the new set of relations matches with the
* existing relations in the publication. Additionally, if the
* relation has an associated WHERE clause, check the WHERE
- * expressions also match. Drop the rest.
+ * expressions also match. Same for the column list. Drop the
+ * rest.
*/
if (RelationGetRelid(newpubrel->relation) == oldrelid)
{
- if (equal(oldrelwhereclause, newpubrel->whereClause))
+ if (equal(oldrelwhereclause, newpubrel->whereClause) &&
+ bms_equal(oldcolumns, newcolumns))
{
found = true;
break;
@@ -1193,6 +1406,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
{
oldrel = palloc(sizeof(PublicationRelInfo));
oldrel->whereClause = NULL;
+ oldrel->columns = NIL;
oldrel->relation = table_open(oldrelid,
ShareUpdateExclusiveLock);
delrels = lappend(delrels, oldrel);
@@ -1408,6 +1622,7 @@ AlterPublicationSequences(AlterPublicationStmt *stmt, HeapTuple tup,
{
oldrel = palloc(sizeof(PublicationRelInfo));
oldrel->whereClause = NULL;
+ oldrel->columns = NULL;
oldrel->relation = table_open(oldrelid,
ShareUpdateExclusiveLock);
delrels = lappend(delrels, oldrel);
@@ -1669,6 +1884,7 @@ OpenRelationList(List *rels, char objectType)
List *result = NIL;
ListCell *lc;
List *relids_with_rf = NIL;
+ List *relids_with_collist = NIL;
/*
* Open, share-lock, and check all the explicitly-specified relations
@@ -1719,6 +1935,13 @@ OpenRelationList(List *rels, char objectType)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /* Disallow duplicate tables if there are any with column lists. */
+ if (t->columns || list_member_oid(relids_with_collist, myrelid))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
table_close(rel, ShareUpdateExclusiveLock);
continue;
}
@@ -1726,12 +1949,16 @@ OpenRelationList(List *rels, char objectType)
pub_rel = palloc(sizeof(PublicationRelInfo));
pub_rel->relation = rel;
pub_rel->whereClause = t->whereClause;
+ pub_rel->columns = t->columns;
result = lappend(result, pub_rel);
relids = lappend_oid(relids, myrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, myrelid);
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, myrelid);
+
/*
* Add children of this rel, if requested, so that they too are added
* to the publication. A partitioned table can't have any inheritance
@@ -1771,6 +1998,18 @@ OpenRelationList(List *rels, char objectType)
errmsg("conflicting or redundant WHERE clauses for table \"%s\"",
RelationGetRelationName(rel))));
+ /*
+ * We don't allow to specify column list for both parent
+ * and child table at the same time as it is not very
+ * clear which one should be given preference.
+ */
+ if (childrelid != myrelid &&
+ (t->columns || list_member_oid(relids_with_collist, childrelid)))
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("conflicting or redundant column lists for table \"%s\"",
+ RelationGetRelationName(rel))));
+
continue;
}
@@ -1780,11 +2019,16 @@ OpenRelationList(List *rels, char objectType)
pub_rel->relation = rel;
/* child inherits WHERE clause from parent */
pub_rel->whereClause = t->whereClause;
+ /* child inherits column list from parent */
+ pub_rel->columns = t->columns;
result = lappend(result, pub_rel);
relids = lappend_oid(relids, childrelid);
if (t->whereClause)
relids_with_rf = lappend_oid(relids_with_rf, childrelid);
+
+ if (t->columns)
+ relids_with_collist = lappend_oid(relids_with_collist, childrelid);
}
}
}
@@ -1893,6 +2137,11 @@ PublicationDropRelations(Oid pubid, List *rels, bool missing_ok)
Relation rel = pubrel->relation;
Oid relid = RelationGetRelid(rel);
+ if (pubrel->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column list must not be specified in ALTER PUBLICATION ... DROP"));
+
prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
ObjectIdGetDatum(relid),
ObjectIdGetDatum(pubid));
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 0df7cf58747..1a4fbdc38c6 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -574,9 +574,6 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
if (cmd != CMD_UPDATE && cmd != CMD_DELETE)
return;
- if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
- return;
-
/*
* It is only safe to execute UPDATE/DELETE when all columns, referenced
* in the row filters from publications which the relation is in, are
@@ -596,17 +593,33 @@ CheckCmdReplicaIdentity(Relation rel, CmdType cmd)
errmsg("cannot update table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_UPDATE && !pubdesc.cols_valid_for_update)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot update table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
else if (cmd == CMD_DELETE && !pubdesc.rf_valid_for_delete)
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
errmsg("cannot delete from table \"%s\"",
RelationGetRelationName(rel)),
errdetail("Column used in the publication WHERE expression is not part of the replica identity.")));
+ else if (cmd == CMD_DELETE && !pubdesc.cols_valid_for_delete)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("cannot delete from table \"%s\"",
+ RelationGetRelationName(rel)),
+ errdetail("Column list used by the publication does not cover the replica identity.")));
/* If relation has replica identity we are always good. */
if (OidIsValid(RelationGetReplicaIndex(rel)))
return;
+ /* REPLICA IDENTITY FULL is also good for UPDATE/DELETE. */
+ if (rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL)
+ return;
+
/*
* This is UPDATE/DELETE and there is no replica identity.
*
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 55f720a88f4..e38ff4000f4 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4850,6 +4850,7 @@ _copyPublicationTable(const PublicationTable *from)
COPY_NODE_FIELD(relation);
COPY_NODE_FIELD(whereClause);
+ COPY_NODE_FIELD(columns);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 82562eb9b87..0f330e3c701 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2322,6 +2322,7 @@ _equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
{
COMPARE_NODE_FIELD(relation);
COMPARE_NODE_FIELD(whereClause);
+ COMPARE_NODE_FIELD(columns);
return true;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index e327bc735fb..a3381780ed9 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -9749,13 +9749,14 @@ CreatePublicationStmt:
* relation_expr here.
*/
PublicationObjSpec:
- TABLE relation_expr OptWhereClause
+ TABLE relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_TABLE;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $2;
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
}
| ALL TABLES IN_P SCHEMA ColId
{
@@ -9790,11 +9791,15 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_SEQUENCES_IN_CUR_SCHEMA;
$$->location = @5;
}
- | ColId OptWhereClause
+ | ColId opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
- if ($2)
+ /*
+ * If either a row filter or column list is specified, create
+ * a PublicationTable object.
+ */
+ if ($2 || $3)
{
/*
* The OptWhereClause must be stored here but it is
@@ -9804,7 +9809,8 @@ PublicationObjSpec:
*/
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVar(NULL, $1, @1);
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
else
{
@@ -9812,23 +9818,25 @@ PublicationObjSpec:
}
$$->location = @1;
}
- | ColId indirection OptWhereClause
+ | ColId indirection opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = makeRangeVarFromQualifiedName($1, $2, @1, yyscanner);
- $$->pubtable->whereClause = $3;
+ $$->pubtable->columns = $3;
+ $$->pubtable->whereClause = $4;
$$->location = @1;
}
/* grammar like tablename * , ONLY tablename, ONLY ( tablename ) */
- | extended_relation_expr OptWhereClause
+ | extended_relation_expr opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
$$->pubobjtype = PUBLICATIONOBJ_CONTINUATION;
$$->pubtable = makeNode(PublicationTable);
$$->pubtable->relation = $1;
- $$->pubtable->whereClause = $2;
+ $$->pubtable->columns = $2;
+ $$->pubtable->whereClause = $3;
}
| CURRENT_SCHEMA
{
@@ -17523,6 +17531,13 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
errmsg("WHERE clause not allowed for schema"),
parser_errposition(pubobj->location));
+ /* Column list is not allowed on a schema object */
+ if (pubobj->pubtable && pubobj->pubtable->columns)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("column specification not allowed for schema"),
+ parser_errposition(pubobj->location));
+
/*
* We can distinguish between the different type of schema
* objects based on whether name and pubtable is set.
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3dbe85d61a2..23ae3d7db83 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -29,16 +29,30 @@
#define TRUNCATE_CASCADE (1<<0)
#define TRUNCATE_RESTART_SEQS (1<<1)
-static void logicalrep_write_attrs(StringInfo out, Relation rel);
+static void logicalrep_write_attrs(StringInfo out, Relation rel,
+ Bitmapset *columns);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
TupleTableSlot *slot,
- bool binary);
+ bool binary, Bitmapset *columns);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
static void logicalrep_write_namespace(StringInfo out, Oid nspid);
static const char *logicalrep_read_namespace(StringInfo in);
+/*
+ * Check if a column is covered by a column list.
+ *
+ * Need to be careful about NULL, which is treated as a column list covering
+ * all columns.
+ */
+static bool
+column_in_column_list(int attnum, Bitmapset *columns)
+{
+ return (columns == NULL || bms_is_member(attnum, columns));
+}
+
+
/*
* Write BEGIN to the output stream.
*/
@@ -398,7 +412,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
*/
void
logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
- TupleTableSlot *newslot, bool binary)
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_INSERT);
@@ -410,7 +424,7 @@ logicalrep_write_insert(StringInfo out, TransactionId xid, Relation rel,
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -443,7 +457,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
void
logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
TupleTableSlot *oldslot, TupleTableSlot *newslot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
pq_sendbyte(out, LOGICAL_REP_MSG_UPDATE);
@@ -464,11 +478,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, columns);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newslot, binary);
+ logicalrep_write_tuple(out, rel, newslot, binary, columns);
}
/*
@@ -537,7 +551,7 @@ logicalrep_write_delete(StringInfo out, TransactionId xid, Relation rel,
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldslot, binary);
+ logicalrep_write_tuple(out, rel, oldslot, binary, NULL);
}
/*
@@ -702,7 +716,8 @@ logicalrep_read_sequence(StringInfo in, LogicalRepSequence *seqdata)
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel,
+ Bitmapset *columns)
{
char *relname;
@@ -724,7 +739,7 @@ logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
pq_sendbyte(out, rel->rd_rel->relreplident);
/* send the attribute info */
- logicalrep_write_attrs(out, rel);
+ logicalrep_write_attrs(out, rel, columns);
}
/*
@@ -801,7 +816,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
*/
static void
logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
- bool binary)
+ bool binary, Bitmapset *columns)
{
TupleDesc desc;
Datum *values;
@@ -813,8 +828,14 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
+ if (!column_in_column_list(att->attnum, columns))
continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -833,6 +854,9 @@ logicalrep_write_tuple(StringInfo out, Relation rel, TupleTableSlot *slot,
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_column_list(att->attnum, columns))
+ continue;
+
if (isnull[i])
{
pq_sendbyte(out, LOGICALREP_COLUMN_NULL);
@@ -954,7 +978,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
* Write relation attribute metadata to the stream.
*/
static void
-logicalrep_write_attrs(StringInfo out, Relation rel)
+logicalrep_write_attrs(StringInfo out, Relation rel, Bitmapset *columns)
{
TupleDesc desc;
int i;
@@ -967,8 +991,14 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
/* send number of live attributes */
for (i = 0; i < desc->natts; i++)
{
- if (TupleDescAttr(desc, i)->attisdropped || TupleDescAttr(desc, i)->attgenerated)
+ Form_pg_attribute att = TupleDescAttr(desc, i);
+
+ if (att->attisdropped || att->attgenerated)
continue;
+
+ if (!column_in_column_list(att->attnum, columns))
+ continue;
+
nliveatts++;
}
pq_sendint16(out, nliveatts);
@@ -987,6 +1017,9 @@ logicalrep_write_attrs(StringInfo out, Relation rel)
if (att->attisdropped || att->attgenerated)
continue;
+ if (!column_in_column_list(att->attnum, columns))
+ continue;
+
/* REPLICA IDENTITY FULL means all columns are sent as part of key. */
if (replidentfull ||
bms_is_member(att->attnum - FirstLowInvalidHeapAttributeNumber,
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index d8b12d94bc3..ee5859dcae0 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -113,6 +113,7 @@
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "utils/acl.h"
+#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -702,12 +703,13 @@ fetch_remote_table_info(char *nspname, char *relname,
StringInfoData cmd;
TupleTableSlot *slot;
Oid tableRow[] = {OIDOID, CHAROID, CHAROID};
- Oid attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+ Oid attrRow[] = {INT2OID, TEXTOID, OIDOID, BOOLOID};
Oid qualRow[] = {TEXTOID};
bool isnull;
int natt;
ListCell *lc;
bool first;
+ Bitmapset *included_cols = NULL;
lrel->nspname = nspname;
lrel->relname = relname;
@@ -748,10 +750,110 @@ fetch_remote_table_info(char *nspname, char *relname,
ExecDropSingleTupleTableSlot(slot);
walrcv_clear_result(res);
- /* Now fetch columns. */
+
+ /*
+ * Get column lists for each relation.
+ *
+ * For initial synchronization, column lists can be ignored in following
+ * cases:
+ *
+ * 1) one of the subscribed publications for the table hasn't specified
+ * any column list
+ *
+ * 2) one of the subscribed publications has puballtables set to true
+ *
+ * 3) one of the subscribed publications is declared as ALL TABLES IN
+ * SCHEMA that includes this relation
+ *
+ * We need to do this before fetching info about column names and types,
+ * so that we can skip columns that should not be replicated.
+ */
+ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+ {
+ WalRcvExecResult *pubres;
+ TupleTableSlot *slot;
+ Oid attrsRow[] = {INT2OID};
+ StringInfoData pub_names;
+ bool first = true;
+
+ initStringInfo(&pub_names);
+ foreach(lc, MySubscription->publications)
+ {
+ if (!first)
+ appendStringInfo(&pub_names, ", ");
+ appendStringInfoString(&pub_names, quote_literal_cstr(strVal(lfirst(lc))));
+ first = false;
+ }
+
+ /*
+ * Fetch info about column lists for the relation (from all the
+ * publications). We unnest the int2vector values, because that
+ * makes it easier to combine lists by simply adding the attnums
+ * to a new bitmap (without having to parse the int2vector data).
+ * This preserves NULL values, so that if one of the publications
+ * has no column list, we'll know that.
+ */
+ resetStringInfo(&cmd);
+ appendStringInfo(&cmd,
+ "SELECT DISTINCT unnest"
+ " FROM pg_publication p"
+ " LEFT OUTER JOIN pg_publication_rel pr"
+ " ON (p.oid = pr.prpubid AND pr.prrelid = %u)"
+ " LEFT OUTER JOIN unnest(pr.prattrs) ON TRUE,"
+ " LATERAL pg_get_publication_tables(p.pubname) gpt"
+ " WHERE gpt.relid = %u"
+ " AND p.pubname IN ( %s )",
+ lrel->remoteid,
+ lrel->remoteid,
+ pub_names.data);
+
+ pubres = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data,
+ lengthof(attrsRow), attrsRow);
+
+ if (pubres->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONNECTION_FAILURE),
+ errmsg("could not fetch column list info for table \"%s.%s\" from publisher: %s",
+ nspname, relname, pubres->err)));
+
+ /*
+ * Merge the column lists (from different publications) by creating
+ * a single bitmap with all the attnums. If we find a NULL value,
+ * that means one of the publications has no column list for the
+ * table we're syncing.
+ */
+ slot = MakeSingleTupleTableSlot(pubres->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(pubres->tuplestore, true, false, slot))
+ {
+ Datum cfval = slot_getattr(slot, 1, &isnull);
+
+ /* NULL means empty column list, so we're done. */
+ if (isnull)
+ {
+ bms_free(included_cols);
+ included_cols = NULL;
+ break;
+ }
+
+ included_cols = bms_add_member(included_cols,
+ DatumGetInt16(cfval));
+
+ ExecClearTuple(slot);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+
+ walrcv_clear_result(pubres);
+
+ pfree(pub_names.data);
+ }
+
+ /*
+ * Now fetch column names and types.
+ */
resetStringInfo(&cmd);
appendStringInfo(&cmd,
- "SELECT a.attname,"
+ "SELECT a.attnum,"
+ " a.attname,"
" a.atttypid,"
" a.attnum = ANY(i.indkey)"
" FROM pg_catalog.pg_attribute a"
@@ -779,16 +881,35 @@ fetch_remote_table_info(char *nspname, char *relname,
lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
lrel->attkeys = NULL;
+ /*
+ * Store the columns as a list of names. Ignore those that are not
+ * present in the column list, if there is one.
+ */
natt = 0;
slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
{
- lrel->attnames[natt] =
- TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ char *rel_colname;
+ AttrNumber attnum;
+
+ attnum = DatumGetInt16(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ /* If the column is not in the column list, skip it. */
+ if (included_cols != NULL && !bms_is_member(attnum, included_cols))
+ {
+ ExecClearTuple(slot);
+ continue;
+ }
+
+ rel_colname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
Assert(!isnull);
- lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+
+ lrel->attnames[natt] = rel_colname;
+ lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 3, &isnull));
Assert(!isnull);
- if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
+
+ if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
lrel->attkeys = bms_add_member(lrel->attkeys, natt);
/* Should never happen. */
@@ -822,6 +943,9 @@ fetch_remote_table_info(char *nspname, char *relname,
*
* 3) one of the subscribed publications is declared as ALL TABLES IN
* SCHEMA that includes this relation
+ *
+ * XXX Does this actually handle puballtables and schema publications
+ * correctly?
*/
if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
{
@@ -931,8 +1055,24 @@ copy_table(Relation rel)
/* Regular table with no row filter */
if (lrel.relkind == RELKIND_RELATION && qual == NIL)
- appendStringInfo(&cmd, "COPY %s TO STDOUT",
+ {
+ appendStringInfo(&cmd, "COPY %s (",
quote_qualified_identifier(lrel.nspname, lrel.relname));
+
+ /*
+ * XXX Do we need to list the columns in all cases? Maybe we're replicating
+ * all columns?
+ */
+ for (int i = 0; i < lrel.natts; i++)
+ {
+ if (i > 0)
+ appendStringInfoString(&cmd, ", ");
+
+ appendStringInfoString(&cmd, quote_identifier(lrel.attnames[i]));
+ }
+
+ appendStringInfo(&cmd, ") TO STDOUT");
+ }
else
{
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 4cdc698cbb3..53bbe247bb4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -30,6 +30,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -90,7 +91,8 @@ static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
static void send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx);
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns);
static void send_repl_origin(LogicalDecodingContext *ctx,
RepOriginId origin_id, XLogRecPtr origin_lsn,
bool send_origin);
@@ -148,9 +150,6 @@ typedef struct RelationSyncEntry
*/
ExprState *exprstate[NUM_ROWFILTER_PUBACTIONS];
EState *estate; /* executor state used for row filter */
- MemoryContext cache_expr_cxt; /* private context for exprstate and
- * estate, if any */
-
TupleTableSlot *new_slot; /* slot for storing new tuple */
TupleTableSlot *old_slot; /* slot for storing old tuple */
@@ -169,6 +168,19 @@ typedef struct RelationSyncEntry
* having identical TupleDesc.
*/
AttrMap *attrmap;
+
+ /*
+ * Columns included in the publication, or NULL if all columns are
+ * included implicitly. Note that the attnums in this bitmap are not
+ * shifted by FirstLowInvalidHeapAttributeNumber.
+ */
+ Bitmapset *columns;
+
+ /*
+ * Private context to store additional data for this entry - state for
+ * the row filter expressions, column list, etc.
+ */
+ MemoryContext entry_cxt;
} RelationSyncEntry;
/* Map used to remember which relation schemas we sent. */
@@ -193,6 +205,7 @@ static EState *create_estate_for_relation(Relation rel);
static void pgoutput_row_filter_init(PGOutputData *data,
List *publications,
RelationSyncEntry *entry);
+
static bool pgoutput_row_filter_exec_expr(ExprState *state,
ExprContext *econtext);
static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
@@ -200,6 +213,11 @@ static bool pgoutput_row_filter(Relation relation, TupleTableSlot *old_slot,
RelationSyncEntry *entry,
ReorderBufferChangeType *action);
+/* column list routines */
+static void pgoutput_column_list_init(PGOutputData *data,
+ List *publications,
+ RelationSyncEntry *entry);
+
/*
* Specify output plugin callbacks
*/
@@ -622,11 +640,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
Relation ancestor = RelationIdGetRelation(relentry->publish_as_relid);
- send_relation_and_attrs(ancestor, xid, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx, relentry->columns);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, xid, ctx);
+ send_relation_and_attrs(relation, xid, ctx, relentry->columns);
if (in_streaming)
set_schema_sent_in_streamed_txn(relentry, topxid);
@@ -639,7 +657,8 @@ maybe_send_schema(LogicalDecodingContext *ctx,
*/
static void
send_relation_and_attrs(Relation relation, TransactionId xid,
- LogicalDecodingContext *ctx)
+ LogicalDecodingContext *ctx,
+ Bitmapset *columns)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -662,13 +681,17 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
if (att->atttypid < FirstGenbkiObjectId)
continue;
+ /* Skip this attribute if it's not present in the column list */
+ if (columns != NULL && !bms_is_member(att->attnum, columns))
+ continue;
+
OutputPluginPrepareWrite(ctx, false);
logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, xid, relation);
+ logicalrep_write_rel(ctx->out, xid, relation, columns);
OutputPluginWrite(ctx, false);
}
@@ -722,6 +745,28 @@ pgoutput_row_filter_exec_expr(ExprState *state, ExprContext *econtext)
return DatumGetBool(ret);
}
+/*
+ * Make sure the per-entry memory context exists.
+ */
+static void
+pgoutput_ensure_entry_cxt(PGOutputData *data, RelationSyncEntry *entry)
+{
+ Relation relation;
+
+ /* The context may already exist, in which case bail out. */
+ if (entry->entry_cxt)
+ return;
+
+ relation = RelationIdGetRelation(entry->publish_as_relid);
+
+ entry->entry_cxt = AllocSetContextCreate(data->cachectx,
+ "entry private context",
+ ALLOCSET_SMALL_SIZES);
+
+ MemoryContextCopyAndSetIdentifier(entry->entry_cxt,
+ RelationGetRelationName(relation));
+}
+
/*
* Initialize the row filter.
*/
@@ -842,21 +887,13 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
{
Relation relation = RelationIdGetRelation(entry->publish_as_relid);
- Assert(entry->cache_expr_cxt == NULL);
-
- /* Create the memory context for row filters */
- entry->cache_expr_cxt = AllocSetContextCreate(data->cachectx,
- "Row filter expressions",
- ALLOCSET_DEFAULT_SIZES);
-
- MemoryContextCopyAndSetIdentifier(entry->cache_expr_cxt,
- RelationGetRelationName(relation));
+ pgoutput_ensure_entry_cxt(data, entry);
/*
* Now all the filters for all pubactions are known. Combine them when
* their pubactions are the same.
*/
- oldctx = MemoryContextSwitchTo(entry->cache_expr_cxt);
+ oldctx = MemoryContextSwitchTo(entry->entry_cxt);
entry->estate = create_estate_for_relation(relation);
for (idx = 0; idx < NUM_ROWFILTER_PUBACTIONS; idx++)
{
@@ -879,6 +916,105 @@ pgoutput_row_filter_init(PGOutputData *data, List *publications,
}
}
+/*
+ * Initialize the column list.
+ */
+static void
+pgoutput_column_list_init(PGOutputData *data, List *publications,
+ RelationSyncEntry *entry)
+{
+ ListCell *lc;
+
+ /*
+ * Find if there are any column lists for this relation. If there are,
+ * build a bitmap merging all the column lists.
+ *
+ * All the given publication-table mappings must be checked.
+ *
+ * Multiple publications might have multiple column lists for this relation.
+ *
+ * FOR ALL TABLES and FOR ALL TABLES IN SCHEMA implies "don't use column
+ * list" so it takes precedence.
+ */
+ foreach(lc, publications)
+ {
+ Publication *pub = lfirst(lc);
+ HeapTuple cftuple = NULL;
+ Datum cfdatum = 0;
+
+ /*
+ * Assume there's no column list. Only if we find pg_publication_rel
+ * entry with a column list we'll switch it to false.
+ */
+ bool pub_no_list = true;
+
+ /*
+ * If the publication is FOR ALL TABLES then it is treated the same as if
+ * there are no column lists (even if other publications have a list).
+ */
+ if (!pub->alltables)
+ {
+ /*
+ * Check for the presence of a column list in this publication.
+ *
+ * Note: If we find no pg_publication_rel row, it's a publication
+ * defined for a whole schema, so it can't have a column list, just
+ * like a FOR ALL TABLES publication.
+ */
+ cftuple = SearchSysCache2(PUBLICATIONRELMAP,
+ ObjectIdGetDatum(entry->publish_as_relid),
+ ObjectIdGetDatum(pub->oid));
+
+ if (HeapTupleIsValid(cftuple))
+ {
+ /*
+ * Lookup the column list attribute.
+ *
+ * Note: We update the pub_no_list value directly, because if
+ * the value is NULL, we have no list (and vice versa).
+ */
+ cfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, cftuple,
+ Anum_pg_publication_rel_prattrs,
+ &pub_no_list);
+
+ /*
+ * Build the column list bitmap in the per-entry context.
+ *
+ * We need to merge column lists from all publications, so we
+ * update the same bitmapset. If the column list is null, we
+ * interpret it as replicating all columns.
+ */
+ if (!pub_no_list) /* when not null */
+ {
+ pgoutput_ensure_entry_cxt(data, entry);
+
+ entry->columns = pub_collist_to_bitmapset(entry->columns,
+ cfdatum,
+ entry->entry_cxt);
+ }
+ }
+ }
+
+ /*
+ * Found a publication with no column list, so we're done. But first
+ * discard column list we might have from preceding publications.
+ */
+ if (pub_no_list)
+ {
+ if (cftuple)
+ ReleaseSysCache(cftuple);
+
+ bms_free(entry->columns);
+ entry->columns = NULL;
+
+ break;
+ }
+
+ ReleaseSysCache(cftuple);
+ } /* loop all subscribed publications */
+
+}
+
/*
* Initialize the slot for storing new and old tuples, and build the map that
* will be used to convert the relation's tuples into the ancestor's format.
@@ -1243,7 +1379,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, xid, targetrel, new_slot,
- data->binary);
+ data->binary, relentry->columns);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -1297,11 +1433,13 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
case REORDER_BUFFER_CHANGE_INSERT:
logicalrep_write_insert(ctx->out, xid, targetrel,
- new_slot, data->binary);
+ new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
logicalrep_write_update(ctx->out, xid, targetrel,
- old_slot, new_slot, data->binary);
+ old_slot, new_slot, data->binary,
+ relentry->columns);
break;
case REORDER_BUFFER_CHANGE_DELETE:
logicalrep_write_delete(ctx->out, xid, targetrel,
@@ -1794,8 +1932,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->new_slot = NULL;
entry->old_slot = NULL;
memset(entry->exprstate, 0, sizeof(entry->exprstate));
- entry->cache_expr_cxt = NULL;
+ entry->entry_cxt = NULL;
entry->publish_as_relid = InvalidOid;
+ entry->columns = NULL;
entry->attrmap = NULL;
}
@@ -1841,6 +1980,8 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->schema_sent = false;
list_free(entry->streamed_txns);
entry->streamed_txns = NIL;
+ bms_free(entry->columns);
+ entry->columns = NULL;
entry->pubactions.pubinsert = false;
entry->pubactions.pubupdate = false;
entry->pubactions.pubdelete = false;
@@ -1865,17 +2006,18 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/*
* Row filter cache cleanups.
*/
- if (entry->cache_expr_cxt)
- MemoryContextDelete(entry->cache_expr_cxt);
+ if (entry->entry_cxt)
+ MemoryContextDelete(entry->entry_cxt);
- entry->cache_expr_cxt = NULL;
+ entry->entry_cxt = NULL;
entry->estate = NULL;
memset(entry->exprstate, 0, sizeof(entry->exprstate));
/*
* Build publication cache. We can't use one provided by relcache as
- * relcache considers all publications given relation is in, but here
- * we only need to consider ones that the subscriber requested.
+ * relcache considers all publications that the given relation is in,
+ * but here we only need to consider ones that the subscriber
+ * requested.
*/
foreach(lc, data->publications)
{
@@ -1946,6 +2088,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
}
/*
+ * If the relation is to be published, determine actions to
+ * publish, and list of columns, if appropriate.
+ *
* Don't publish changes for partitioned tables, because
* publishing those of its partitions suffices, unless partition
* changes won't be published due to pubviaroot being set.
@@ -2007,6 +2152,9 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/* Initialize the row filter */
pgoutput_row_filter_init(data, rel_publications, entry);
+
+ /* Initialize the column list */
+ pgoutput_column_list_init(data, rel_publications, entry);
}
list_free(pubids);
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 4f3fe1118a2..d47fac7bb98 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -5575,6 +5575,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
return;
}
@@ -5587,6 +5589,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
memset(pubdesc, 0, sizeof(PublicationDesc));
pubdesc->rf_valid_for_update = true;
pubdesc->rf_valid_for_delete = true;
+ pubdesc->cols_valid_for_update = true;
+ pubdesc->cols_valid_for_delete = true;
/* Fetch the publication membership info. */
puboids = GetRelationPublications(relid);
@@ -5657,7 +5661,7 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
*/
if (!pubform->puballtables &&
(pubform->pubupdate || pubform->pubdelete) &&
- contain_invalid_rfcolumn(pubid, relation, ancestors,
+ pub_rf_contains_invalid_column(pubid, relation, ancestors,
pubform->pubviaroot))
{
if (pubform->pubupdate)
@@ -5666,6 +5670,23 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->rf_valid_for_delete = false;
}
+ /*
+ * Check if all columns are part of the REPLICA IDENTITY index or not.
+ *
+ * If the publication is FOR ALL TABLES then it means the table has no
+ * column list and we can skip the validation.
+ */
+ if (!pubform->puballtables &&
+ (pubform->pubupdate || pubform->pubdelete) &&
+ pub_collist_contains_invalid_column(pubid, relation, ancestors,
+ pubform->pubviaroot))
+ {
+ if (pubform->pubupdate)
+ pubdesc->cols_valid_for_update = false;
+ if (pubform->pubdelete)
+ pubdesc->cols_valid_for_delete = false;
+ }
+
ReleaseSysCache(tup);
/*
@@ -5677,6 +5698,16 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
!pubdesc->rf_valid_for_update && !pubdesc->rf_valid_for_delete)
break;
+
+ /*
+ * If we know everything is replicated and the column list is invalid
+ * for update and delete, there is no point to check for other
+ * publications.
+ */
+ if (pubdesc->pubactions.pubinsert && pubdesc->pubactions.pubupdate &&
+ pubdesc->pubactions.pubdelete && pubdesc->pubactions.pubtruncate &&
+ !pubdesc->cols_valid_for_update && !pubdesc->cols_valid_for_delete)
+ break;
}
if (relation->rd_pubdesc)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 00629f08362..535b1601655 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4131,6 +4131,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
int i_prpubid;
int i_prrelid;
int i_prrelqual;
+ int i_prattrs;
int i,
j,
ntups;
@@ -4144,12 +4145,20 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
if (fout->remoteVersion >= 150000)
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual "
- "FROM pg_catalog.pg_publication_rel");
+ "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual, "
+ "(CASE\n"
+ " WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT array_agg(attname)\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) prattrs "
+ "FROM pg_catalog.pg_publication_rel pr");
else
appendPQExpBufferStr(query,
"SELECT tableoid, oid, prpubid, prrelid, "
- "NULL AS prrelqual "
+ "NULL AS prrelqual, NULL AS prattrs "
"FROM pg_catalog.pg_publication_rel");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
@@ -4160,6 +4169,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
i_prpubid = PQfnumber(res, "prpubid");
i_prrelid = PQfnumber(res, "prrelid");
i_prrelqual = PQfnumber(res, "prrelqual");
+ i_prattrs = PQfnumber(res, "prattrs");
/* this allocation may be more than we need */
pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4205,6 +4215,28 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
else
pubrinfo[j].pubrelqual = pg_strdup(PQgetvalue(res, i, i_prrelqual));
+ if (!PQgetisnull(res, i, i_prattrs))
+ {
+ char **attnames;
+ int nattnames;
+ PQExpBuffer attribs;
+
+ if (!parsePGArray(PQgetvalue(res, i, i_prattrs),
+ &attnames, &nattnames))
+ fatal("could not parse %s array", "prattrs");
+ attribs = createPQExpBuffer();
+ for (int k = 0; k < nattnames; k++)
+ {
+ if (k > 0)
+ appendPQExpBufferStr(attribs, ", ");
+
+ appendPQExpBufferStr(attribs, fmtId(attnames[k]));
+ }
+ pubrinfo[j].pubrattrs = attribs->data;
+ }
+ else
+ pubrinfo[j].pubrattrs = NULL;
+
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubrinfo[j].dobj), fout);
@@ -4300,6 +4332,9 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
appendPQExpBuffer(query, " %s",
fmtQualifiedDumpable(tbinfo));
+ if (pubrinfo->pubrattrs)
+ appendPQExpBuffer(query, " (%s)", pubrinfo->pubrattrs);
+
if (pubrinfo->pubrelqual)
{
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 893725d121f..688093c55e0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -634,6 +634,7 @@ typedef struct _PublicationRelInfo
PublicationInfo *publication;
TableInfo *pubtable;
char *pubrelqual;
+ char *pubrattrs;
} PublicationRelInfo;
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 0e724b0366d..af5d6fa5a33 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2438,6 +2438,28 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'ALTER PUBLICATION pub1 ADD TABLE test_sixth_table (col3, col2)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_sixth_table (col3, col2);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_sixth_table (col2, col3);\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'ALTER PUBLICATION pub1 ADD TABLE test_seventh_table (col3, col2) WHERE (col1 = 1)' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub1 ADD TABLE dump_test.test_seventh_table (col3, col2) WHERE (col1 = 1);',
+ regexp => qr/^
+ \QALTER PUBLICATION pub1 ADD TABLE ONLY dump_test.test_seventh_table (col2, col3) WHERE ((col1 = 1));\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'ALTER PUBLICATION pub3 ADD ALL TABLES IN SCHEMA dump_test' => {
create_order => 51,
create_sql =>
@@ -2809,6 +2831,44 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'CREATE TABLE test_sixth_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_sixth_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_sixth_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'CREATE TABLE test_seventh_table' => {
+ create_order => 6,
+ create_sql => 'CREATE TABLE dump_test.test_seventh_table (
+ col1 int,
+ col2 text,
+ col3 bytea
+ );',
+ regexp => qr/^
+ \QCREATE TABLE dump_test.test_seventh_table (\E
+ \n\s+\Qcol1 integer,\E
+ \n\s+\Qcol2 text,\E
+ \n\s+\Qcol3 bytea\E
+ \n\);
+ /xm,
+ like =>
+ { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
'CREATE TABLE test_table_identity' => {
create_order => 3,
create_sql => 'CREATE TABLE dump_test.test_table_identity (
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index b8a532a45f7..4dddf087893 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2960,6 +2960,7 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_namespace pn ON p.oid = pn.pnpubid\n"
" JOIN pg_catalog.pg_class pc ON pc.relnamespace = pn.pnnspid\n"
@@ -2967,6 +2968,12 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , pg_get_expr(pr.prqual, c.oid)\n"
+ " , (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " (SELECT string_agg(attname, ', ')\n"
+ " FROM pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = pr.prrelid AND attnum = prattrs[s])\n"
+ " ELSE NULL END) "
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
" JOIN pg_catalog.pg_class c ON c.oid = pr.prrelid\n"
@@ -2974,6 +2981,7 @@ describeOneTableDetails(const char *schemaname,
"UNION\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -2984,12 +2992,14 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf,
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
"WHERE pr.prrelid = '%s'\n"
"UNION ALL\n"
"SELECT pubname\n"
" , NULL\n"
+ " , NULL\n"
"FROM pg_catalog.pg_publication p\n"
"WHERE p.puballtables AND pg_catalog.pg_relation_is_publishable('%s')\n"
"ORDER BY 1;",
@@ -3011,6 +3021,11 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&buf, " \"%s\"",
PQgetvalue(result, i, 0));
+ /* column list (if any) */
+ if (!PQgetisnull(result, i, 2))
+ appendPQExpBuffer(&buf, " (%s)",
+ PQgetvalue(result, i, 2));
+
/* row filter (if any) */
if (!PQgetisnull(result, i, 1))
appendPQExpBuffer(&buf, " WHERE %s",
@@ -5979,7 +5994,7 @@ listPublications(const char *pattern)
*/
static bool
addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
- bool singlecol, printTableContent *cont)
+ bool as_schema, printTableContent *cont)
{
PGresult *res;
int count = 0;
@@ -5996,15 +6011,19 @@ addFooterToPublicationDesc(PQExpBuffer buf, char *footermsg,
for (i = 0; i < count; i++)
{
- if (!singlecol)
+ if (as_schema)
+ printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
+ else
{
printfPQExpBuffer(buf, " \"%s.%s\"", PQgetvalue(res, i, 0),
PQgetvalue(res, i, 1));
+
+ if (!PQgetisnull(res, i, 3))
+ appendPQExpBuffer(buf, " (%s)", PQgetvalue(res, i, 3));
+
if (!PQgetisnull(res, i, 2))
appendPQExpBuffer(buf, " WHERE %s", PQgetvalue(res, i, 2));
}
- else
- printfPQExpBuffer(buf, " \"%s\"", PQgetvalue(res, i, 0));
printTableAddFooter(cont, buf->data);
}
@@ -6155,11 +6174,22 @@ describePublications(const char *pattern)
printfPQExpBuffer(&buf,
"SELECT n.nspname, c.relname");
if (pset.sversion >= 150000)
+ {
appendPQExpBufferStr(&buf,
", pg_get_expr(pr.prqual, c.oid)");
+ appendPQExpBufferStr(&buf,
+ ", (CASE WHEN pr.prattrs IS NOT NULL THEN\n"
+ " pg_catalog.array_to_string("
+ " ARRAY(SELECT attname\n"
+ " FROM\n"
+ " pg_catalog.generate_series(0, pg_catalog.array_upper(pr.prattrs::pg_catalog.int2[], 1)) s,\n"
+ " pg_catalog.pg_attribute\n"
+ " WHERE attrelid = c.oid AND attnum = prattrs[s]), ', ')\n"
+ " ELSE NULL END)");
+ }
else
appendPQExpBufferStr(&buf,
- ", NULL");
+ ", NULL, NULL");
appendPQExpBuffer(&buf,
"\nFROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
@@ -6189,9 +6219,9 @@ describePublications(const char *pattern)
if (!puballsequences)
{
- /* Get the tables for the specified publication */
+ /* Get the sequences for the specified publication */
printfPQExpBuffer(&buf,
- "SELECT n.nspname, c.relname, NULL\n"
+ "SELECT n.nspname, c.relname, NULL, NULL\n"
"FROM pg_catalog.pg_class c,\n"
" pg_catalog.pg_namespace n,\n"
" pg_catalog.pg_publication_rel pr\n"
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 97f26208e1d..847e53fd998 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -95,6 +95,13 @@ typedef struct PublicationDesc
*/
bool rf_valid_for_update;
bool rf_valid_for_delete;
+
+ /*
+ * true if the columns are part of the replica identity or the publication actions
+ * do not include UPDATE or DELETE.
+ */
+ bool cols_valid_for_update;
+ bool cols_valid_for_delete;
} PublicationDesc;
typedef struct Publication
@@ -111,6 +118,7 @@ typedef struct PublicationRelInfo
{
Relation relation;
Node *whereClause;
+ List *columns;
} PublicationRelInfo;
extern Publication *GetPublication(Oid pubid);
@@ -135,8 +143,11 @@ typedef enum PublicationPartOpt
extern List *GetPublicationRelations(Oid pubid, char objectType,
PublicationPartOpt pub_partopt);
+extern List *GetRelationColumnPartialPublications(Oid relid);
+extern List *GetRelationColumnListInPublication(Oid relid, Oid pubid);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
+extern void GetActionsInPublication(Oid pubid, PublicationActions *actions);
extern List *GetPublicationSchemas(Oid pubid, char objectType);
extern List *GetSchemaPublications(Oid schemaid, char objectType);
extern List *GetSchemaPublicationRelations(Oid schemaid, char objectType,
@@ -160,6 +171,9 @@ extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
char objectType,
bool if_not_exists);
+extern Bitmapset *pub_collist_to_bitmapset(Bitmapset *columns, Datum pubcols,
+ MemoryContext mcxt);
+
extern Oid get_publication_oid(const char *pubname, bool missing_ok);
extern char *get_publication_name(Oid pubid, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index 0dd0f425db9..4feb581899e 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -34,6 +34,7 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
#ifdef CATALOG_VARLEN /* variable-length fields start here */
pg_node_tree prqual; /* qualifications */
+ int2vector prattrs; /* columns to replicate */
#endif
} FormData_pg_publication_rel;
diff --git a/src/include/commands/publicationcmds.h b/src/include/commands/publicationcmds.h
index 7813cbcb6bb..ae87caf089d 100644
--- a/src/include/commands/publicationcmds.h
+++ b/src/include/commands/publicationcmds.h
@@ -31,7 +31,9 @@ extern void RemovePublicationSchemaById(Oid psoid);
extern ObjectAddress AlterPublicationOwner(const char *name, Oid newOwnerId);
extern void AlterPublicationOwner_oid(Oid pubid, Oid newOwnerId);
extern void InvalidatePublicationRels(List *relids);
-extern bool contain_invalid_rfcolumn(Oid pubid, Relation relation,
+extern bool pub_rf_contains_invalid_column(Oid pubid, Relation relation,
+ List *ancestors, bool pubviaroot);
+extern bool pub_collist_contains_invalid_column(Oid pubid, Relation relation,
List *ancestors, bool pubviaroot);
#endif /* PUBLICATIONCMDS_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index cb1fcc0ee31..5a458c42e5e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3654,6 +3654,7 @@ typedef struct PublicationTable
NodeTag type;
RangeVar *relation; /* relation to be published */
Node *whereClause; /* qualifications */
+ List *columns; /* List of columns in a publication table */
} PublicationTable;
/*
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index fb86ca022d2..13ee10fdd4e 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -222,12 +222,12 @@ extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *newslot,
- bool binary);
+ bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, TransactionId xid,
Relation rel,
TupleTableSlot *oldslot,
- TupleTableSlot *newslot, bool binary);
+ TupleTableSlot *newslot, bool binary, Bitmapset *columns);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
@@ -250,7 +250,7 @@ extern void logicalrep_write_sequence(StringInfo out, Relation rel,
bool is_called);
extern void logicalrep_read_sequence(StringInfo in, LogicalRepSequence *seqdata);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
- Relation rel);
+ Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
extern void logicalrep_write_typ(StringInfo out, TransactionId xid,
Oid typoid);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 92f6122d409..14f59370f29 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -1119,6 +1119,369 @@ DROP TABLE rf_tbl_abcd_pk;
DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+ERROR: conflicting or redundant column lists for table "testpub_tbl1"
+RESET client_min_messages;
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+ERROR: column "x" of relation "testpub_tbl5" does not exist
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+ERROR: cannot reference generated column "d" in publication column list
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+ERROR: cannot reference system column "ctid" in publication column list
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+ERROR: cannot drop column c of table testpub_tbl5 because other objects depend on it
+DETAIL: publication of table testpub_tbl5 in publication testpub_fortable depends on column c of table testpub_tbl5
+HINT: Use DROP ... CASCADE to drop the dependent objects too.
+-- ok: for insert-only publication, any column list is acceptable
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is not covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: change the replica identity to "b", and column list to (a, c)
+-- then update fails, because (a, c) does not cover replica identity
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+UPDATE testpub_tbl5 SET a = 1;
+ERROR: cannot update table "testpub_tbl5"
+DETAIL: Column list used by the publication does not cover the replica identity.
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+ Publication testpub_table_ins
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | f | f | t | f | f
+Tables:
+ "public.testpub_tbl5" (a)
+
+-- tests with REPLICA IDENTITY FULL
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ERROR: cannot update table "testpub_tbl6"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+-- make sure changing the column list is propagated to the catalog
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: the column list is the same, we should skip this table (or at least not fail)
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, b)
+
+-- ok: the column list changes, make sure the catalog gets updated
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+ Table "public.testpub_tbl7"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+----------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | text | | | | extended | |
+ c | text | | | | extended | |
+Indexes:
+ "testpub_tbl7_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_fortable" (a, c)
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl8;
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+ERROR: cannot update table "testpub_tbl8_0"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+ Publication testpub_both_filters
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
+Tables:
+ "public.testpub_tbl_both_filters" (a, c) WHERE (c <> 1)
+
+\d+ testpub_tbl_both_filters
+ Table "public.testpub_tbl_both_filters"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+ c | integer | | not null | | plain | |
+Indexes:
+ "testpub_tbl_both_filters_pkey" PRIMARY KEY, btree (a, c) REPLICA IDENTITY
+Publications:
+ "testpub_both_filters" (a, c) WHERE (c <> 1)
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk" because it does not have a replica identity and publishes updates
+HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_pk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_nopk"
+DETAIL: Column list used by the publication does not cover the replica identity.
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+-- Tests for partitioned table
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+ERROR: cannot use publication column list for relation "rf_tbl_abcd_part_pk"
+DETAIL: column list cannot be used for a partitioned table when publish_via_partition_root is false.
+-- ok - can use column list for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any column list is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+ERROR: cannot set publish_via_partition_root = false for publication "testpub6"
+DETAIL: The publication contains a column list for a partitioned table "rf_tbl_abcd_part_pk" which is not allowed when publish_via_partition_root is false.
+-- Now change the root column list to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have column list for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+ERROR: cannot update table "rf_tbl_abcd_part_pk_1"
+DETAIL: Column list used by the publication does not cover the replica identity.
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -1564,6 +1927,15 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_tes
Tables from schemas:
"pub_test1"
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ERROR: syntax error at or near "("
+LINE 1: ...TION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ ^
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+ERROR: column specification not allowed for schema
+LINE 1: ... testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b)...
+ ^
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index c195e75c6f0..e62ed835e3c 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -552,6 +552,289 @@ DROP TABLE rf_tbl_abcd_nopk;
DROP TABLE rf_tbl_abcd_part_pk;
-- ======================================================
+-- fail - duplicate tables are not allowed if that table has any column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1 (a), testpub_tbl1 WITH (publish = 'insert');
+CREATE PUBLICATION testpub_dups FOR TABLE testpub_tbl1, testpub_tbl1 (a) WITH (publish = 'insert');
+RESET client_min_messages;
+
+-- test for column lists
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_fortable FOR TABLE testpub_tbl1;
+CREATE PUBLICATION testpub_fortable_insert WITH (publish = 'insert');
+RESET client_min_messages;
+CREATE TABLE testpub_tbl5 (a int PRIMARY KEY, b text, c text,
+ d int generated always as (a + length(b)) stored);
+-- error: column "x" does not exist
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, x);
+-- error: replica identity "a" not included in the column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (b, c);
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+-- error: generated column "d" can't be in list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, d);
+-- error: system attributes "ctid" not allowed in column list
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, ctid);
+-- ok
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+ALTER TABLE testpub_tbl5 DROP COLUMN c; -- no dice
+-- ok: for insert-only publication, any column list is acceptable
+ALTER PUBLICATION testpub_fortable_insert ADD TABLE testpub_tbl5 (b, c);
+
+/* not all replica identities are good enough */
+CREATE UNIQUE INDEX testpub_tbl5_b_key ON testpub_tbl5 (b, c);
+ALTER TABLE testpub_tbl5 ALTER b SET NOT NULL, ALTER c SET NOT NULL;
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+-- error: replica identity (b,c) is not covered by column list (a, c)
+UPDATE testpub_tbl5 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl5;
+
+-- error: change the replica identity to "b", and column list to (a, c)
+-- then update fails, because (a, c) does not cover replica identity
+ALTER TABLE testpub_tbl5 REPLICA IDENTITY USING INDEX testpub_tbl5_b_key;
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl5 (a, c);
+UPDATE testpub_tbl5 SET a = 1;
+
+/* But if upd/del are not published, it works OK */
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
+\dRp+ testpub_table_ins
+
+-- tests with REPLICA IDENTITY FULL
+CREATE TABLE testpub_tbl6 (a int, b text, c text);
+ALTER TABLE testpub_tbl6 REPLICA IDENTITY FULL;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6 (a, b, c);
+UPDATE testpub_tbl6 SET a = 1;
+ALTER PUBLICATION testpub_fortable DROP TABLE testpub_tbl6;
+
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl6; -- ok
+UPDATE testpub_tbl6 SET a = 1;
+
+-- make sure changing the column list is propagated to the catalog
+CREATE TABLE testpub_tbl7 (a int primary key, b text, c text);
+ALTER PUBLICATION testpub_fortable ADD TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: the column list is the same, we should skip this table (or at least not fail)
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, b);
+\d+ testpub_tbl7
+-- ok: the column list changes, make sure the catalog gets updated
+ALTER PUBLICATION testpub_fortable SET TABLE testpub_tbl7 (a, c);
+\d+ testpub_tbl7
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 0);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 PARTITION OF testpub_tbl8 FOR VALUES WITH (modulus 2, remainder 1);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (b);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: column list covers both "a" and "b"
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_col_list FOR TABLE testpub_tbl8 (a, b) WITH (publish_via_partition_root = 'true');
+RESET client_min_messages;
+
+-- ok: the same thing, but try plain ADD TABLE
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: column list does not cover replica identity for the second partition
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- failure: one of the partitions has REPLICA IDENTITY FULL
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, c);
+UPDATE testpub_tbl8 SET a = 1;
+ALTER PUBLICATION testpub_col_list DROP TABLE testpub_tbl8;
+
+-- add table and then try changing replica identity
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+
+-- failure: replica identity full can't be used with a column list
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: replica identity has to be covered by the column list
+ALTER TABLE testpub_tbl8_1 DROP CONSTRAINT testpub_tbl8_1_pkey;
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl8;
+
+-- column list for partitioned tables has to cover replica identities for
+-- all child relations
+CREATE TABLE testpub_tbl8 (a int, b text, c text) PARTITION BY HASH (a);
+ALTER PUBLICATION testpub_col_list ADD TABLE testpub_tbl8 (a, b);
+-- first partition has replica identity "a"
+CREATE TABLE testpub_tbl8_0 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_0 ADD PRIMARY KEY (a);
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY USING INDEX testpub_tbl8_0_pkey;
+-- second partition has replica identity "b"
+CREATE TABLE testpub_tbl8_1 (a int, b text, c text);
+ALTER TABLE testpub_tbl8_1 ADD PRIMARY KEY (c);
+ALTER TABLE testpub_tbl8_1 REPLICA IDENTITY USING INDEX testpub_tbl8_1_pkey;
+
+-- ok: attaching first partition works, because (a) is in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_0 FOR VALUES WITH (modulus 2, remainder 0);
+-- failure: second partition has replica identity (c), which si not in column list
+ALTER TABLE testpub_tbl8 ATTACH PARTITION testpub_tbl8_1 FOR VALUES WITH (modulus 2, remainder 1);
+UPDATE testpub_tbl8 SET a = 1;
+
+-- failure: changing replica identity to FULL for partition fails, because
+-- of the column list on the parent
+ALTER TABLE testpub_tbl8_0 REPLICA IDENTITY FULL;
+UPDATE testpub_tbl8 SET a = 1;
+
+DROP TABLE testpub_tbl5, testpub_tbl6, testpub_tbl7, testpub_tbl8, testpub_tbl8_1;
+DROP PUBLICATION testpub_table_ins, testpub_fortable, testpub_fortable_insert, testpub_col_list;
+-- ======================================================
+
+-- Test combination of column list and row filter
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_both_filters;
+RESET client_min_messages;
+CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
+ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
+ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
+\dRp+ testpub_both_filters
+\d+ testpub_tbl_both_filters
+
+DROP TABLE testpub_tbl_both_filters;
+DROP PUBLICATION testpub_both_filters;
+-- ======================================================
+
+-- More column list tests for validating column references
+CREATE TABLE rf_tbl_abcd_nopk(a int, b int, c int, d int);
+CREATE TABLE rf_tbl_abcd_pk(a int, b int, c int, d int, PRIMARY KEY(a,b));
+CREATE TABLE rf_tbl_abcd_part_pk (a int PRIMARY KEY, b int) PARTITION by RANGE (a);
+CREATE TABLE rf_tbl_abcd_part_pk_1 (b int, a int PRIMARY KEY);
+ALTER TABLE rf_tbl_abcd_part_pk ATTACH PARTITION rf_tbl_abcd_part_pk_1 FOR VALUES FROM (1) TO (10);
+
+-- Case 1. REPLICA IDENTITY DEFAULT (means use primary key or nothing)
+
+-- 1a. REPLICA IDENTITY is DEFAULT and table has a PK.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub6 FOR TABLE rf_tbl_abcd_pk (a, b);
+RESET client_min_messages;
+-- ok - (a,b) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c);
+-- ok - (a,b,c) coverts all PK cols
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - "b" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (b);
+-- fail - "a" is missing from the column list
+UPDATE rf_tbl_abcd_pk SET a = 1;
+
+-- 1b. REPLICA IDENTITY is DEFAULT and table has no PK
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- ok - there's no replica identity, so any column list works
+-- note: it fails anyway, just a bit later because UPDATE requires RI
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 2. REPLICA IDENTITY FULL
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY FULL;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY FULL;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a, b, c, d);
+-- fail - with REPLICA IDENTITY FULL no column list is allowed
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 3. REPLICA IDENTITY NOTHING
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY NOTHING;
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY NOTHING;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a, b, c, d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (d);
+-- ok - REPLICA IDENTITY NOTHING means all column lists are valid
+-- it still fails later because without RI we can't replicate updates
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Case 4. REPLICA IDENTITY INDEX
+ALTER TABLE rf_tbl_abcd_pk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_pk_c ON rf_tbl_abcd_pk(c);
+ALTER TABLE rf_tbl_abcd_pk REPLICA IDENTITY USING INDEX idx_abcd_pk_c;
+ALTER TABLE rf_tbl_abcd_nopk ALTER COLUMN c SET NOT NULL;
+CREATE UNIQUE INDEX idx_abcd_nopk_c ON rf_tbl_abcd_nopk(c);
+ALTER TABLE rf_tbl_abcd_nopk REPLICA IDENTITY USING INDEX idx_abcd_nopk_c;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_pk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_pk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (a);
+-- fail - column list "a" does not cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_nopk (c);
+-- ok - column list "c" does cover the REPLICA IDENTITY INDEX on "c"
+UPDATE rf_tbl_abcd_nopk SET a = 1;
+
+-- Tests for partitioned table
+
+-- set PUBLISH_VIA_PARTITION_ROOT to false and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - cannot use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - can use column list for partition
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true and test column list for partitioned
+-- table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (a);
+-- ok - "a" is a PK col
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any column list is
+-- used for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- Now change the root column list to use a column "b"
+-- (which is not in the replica identity)
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk_1 (b);
+-- ok - we don't have column list for partitioned table.
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+-- set PUBLISH_VIA_PARTITION_ROOT to true
+-- can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=1);
+-- ok - can use column list for partitioned table
+ALTER PUBLICATION testpub6 SET TABLE rf_tbl_abcd_part_pk (b);
+-- fail - "b" is not in REPLICA IDENTITY INDEX
+UPDATE rf_tbl_abcd_part_pk SET a = 1;
+
+DROP PUBLICATION testpub6;
+DROP TABLE rf_tbl_abcd_pk;
+DROP TABLE rf_tbl_abcd_nopk;
+DROP TABLE rf_tbl_abcd_part_pk;
+-- ======================================================
+
-- Test cache invalidation FOR ALL TABLES publication
SET client_min_messages = 'ERROR';
CREATE TABLE testpub_tbl4(a int);
@@ -793,6 +1076,10 @@ ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schem
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
+-- Verify that it fails to add a schema with a column specification
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo (a, b);
+ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA foo, bar (a, b);
+
-- cleanup pub_test1 schema for invalidation tests
ALTER PUBLICATION testpub2_forschema DROP ALL TABLES IN SCHEMA pub_test1;
DROP PUBLICATION testpub3_forschema, testpub4_forschema, testpub5_forschema, testpub6_forschema, testpub_fortable;
diff --git a/src/test/subscription/t/030_column_list.pl b/src/test/subscription/t/030_column_list.pl
new file mode 100644
index 00000000000..5ceaec83cdb
--- /dev/null
+++ b/src/test/subscription/t/030_column_list.pl
@@ -0,0 +1,1124 @@
+# Copyright (c) 2022, PostgreSQL Global Development Group
+
+# Test partial-column publication of tables
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# create publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->append_conf('postgresql.conf',
+ qq(max_logical_replication_workers = 6));
+$node_subscriber->start;
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+
+sub wait_for_subscription_sync
+{
+ my ($node) = @_;
+
+ # Also wait for initial table sync to finish
+ my $synced_query = "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+
+ $node->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+}
+
+# setup tables on both nodes
+
+# tab1: simple 1:1 replication
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab1 (a int PRIMARY KEY, "B" int, c int)
+));
+
+# tab2: replication from regular to table with fewer columns
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar, c int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab2 (a int PRIMARY KEY, b varchar)
+));
+
+# tab3: simple 1:1 replication with weird column names
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "B" varchar, "c'" int)
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab3 ("a'" int PRIMARY KEY, "c'" int)
+));
+
+# test_part: partitioned tables, with partitioning (including multi-level
+# partitioning, and fewer columns on the subscriber)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text, c timestamptz) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part (a int PRIMARY KEY, b text) PARTITION BY LIST (a);
+ CREATE TABLE test_part_1_1 PARTITION OF test_part FOR VALUES IN (1,2,3,4,5,6);
+ CREATE TABLE test_part_2_1 PARTITION OF test_part FOR VALUES IN (7,8,9,10,11,12) PARTITION BY LIST (a);
+ CREATE TABLE test_part_2_2 PARTITION OF test_part_2_1 FOR VALUES IN (7,8,9,10);
+));
+
+# tab4: table with user-defined enum types
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, c int, d text);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TYPE test_typ AS ENUM ('blue', 'red');
+ CREATE TABLE tab4 (a INT PRIMARY KEY, b test_typ, d text);
+));
+
+
+# TEST: create publication and subscription for some of the tables with
+# column lists
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub1
+ FOR TABLE tab1 (a, "B"), tab3 ("a'", "c'"), test_part (a, b), tab4 (a, b, d)
+ WITH (publish_via_partition_root = 'true');
+));
+
+# check that we got the right prattrs values for the publication in the
+# pg_publication_rel catalog (order by relname, to get stable ordering)
+my $result = $node_publisher->safe_psql('postgres', qq(
+ SELECT relname, prattrs
+ FROM pg_publication_rel pb JOIN pg_class pc ON(pb.prrelid = pc.oid)
+ ORDER BY relname
+));
+
+is($result, qq(tab1|1 2
+tab3|1 3
+tab4|1 2 4
+test_part|1 2), 'publication relation updated');
+
+# TEST: insert data into the tables, create subscription and see if sync
+# replicates the right columns
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (1, 2, 3);
+ INSERT INTO tab1 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (1, 2, 3);
+ INSERT INTO tab3 VALUES (4, 5, 6);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (1, 'red', 3, 'oh my');
+ INSERT INTO tab4 VALUES (2, 'blue', 4, 'hello');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (1, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (2, 'bcd', '2021-07-03 11:12:13');
+ INSERT INTO test_part VALUES (7, 'abc', '2021-07-04 12:00:00');
+ INSERT INTO test_part VALUES (8, 'bcd', '2021-07-03 11:12:13');
+));
+
+# create subscription for the publication, wait for sync to complete,
+# then check the sync results
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+4|5|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+4|6), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+7|abc
+8|bcd), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: now insert more data into the tables, and wait until we replicate
+# them (not by tablesync, but regular decoding and replication)
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab1 VALUES (2, 3, 4);
+ INSERT INTO tab1 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab3 VALUES (2, 3, 4);
+ INSERT INTO tab3 VALUES (5, 6, 7);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab4 VALUES (3, 'red', 5, 'foo');
+ INSERT INTO tab4 VALUES (4, 'blue', 6, 'bar');
+));
+
+# replication of partitioned table
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part VALUES (3, 'xxx', '2022-02-01 10:00:00');
+ INSERT INTO test_part VALUES (4, 'yyy', '2022-03-02 15:12:13');
+ INSERT INTO test_part VALUES (9, 'zzz', '2022-04-03 21:00:00');
+ INSERT INTO test_part VALUES (10, 'qqq', '2022-05-04 22:12:13');
+));
+
+# wait for catchup before checking the subscriber
+$node_publisher->wait_for_catchup('sub1');
+
+# tab1: only (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab1 ORDER BY a");
+is($result, qq(1|2|
+2|3|
+4|5|
+5|6|), 'insert on column tab1.c is not replicated');
+
+# tab3: only (a,c) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result, qq(1|3
+2|4
+4|6
+5|7), 'insert on column tab3.b is not replicated');
+
+# tab4: only (a,b,d) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab4 ORDER BY a");
+is($result, qq(1|red|oh my
+2|blue|hello
+3|red|foo
+4|blue|bar), 'insert on column tab4.c is not replicated');
+
+# test_part: (a,b) is replicated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM test_part ORDER BY a");
+is($result, qq(1|abc
+2|bcd
+3|xxx
+4|yyy
+7|abc
+8|bcd
+9|zzz
+10|qqq), 'insert on column test_part.c columns is not replicated');
+
+
+# TEST: do some updates on some of the tables, both on columns included
+# in the column list and other
+
+# tab1: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET "B" = 2 * "B" where a = 1));
+
+# tab1: update of non-replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab1 SET c = 2*c where a = 4));
+
+# tab3: update of non-replicated
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "B" = "B" || ' updated' where "a'" = 4));
+
+# tab3: update of replicated column
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab3 SET "c'" = 2 * "c'" where "a'" = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'blue', c = c * 2, d = d || ' updated' where a = 1));
+
+# tab4
+$node_publisher->safe_psql('postgres',
+ qq(UPDATE tab4 SET b = 'red', c = c * 2, d = d || ' updated' where a = 2));
+
+# wait for the replication to catch up, and check the UPDATE results got
+# replicated correctly, with the right column list
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab1 ORDER BY a));
+is($result,
+qq(1|4|
+2|3|
+4|5|
+5|6|), 'only update on column tab1.b is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab3 ORDER BY "a'"));
+is($result,
+qq(1|6
+2|4
+4|6
+5|7), 'only update on column tab3.c is replicated');
+
+$result = $node_subscriber->safe_psql('postgres',
+ qq(SELECT * FROM tab4 ORDER BY a));
+
+is($result, qq(1|blue|oh my updated
+2|red|hello updated
+3|red|foo
+4|blue|bar), 'update on column tab4.c is not replicated');
+
+
+# TEST: add table with a column list, insert data, replicate
+
+# insert some data before adding it to the publication
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (1, 'abc', 3);
+));
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION pub1 ADD TABLE tab2 (a, b)");
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION");
+
+# wait for the tablesync to complete, add a bit more data and then check
+# the results of the replication
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab2 VALUES (2, 'def', 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|def), 'insert on column tab2.c is not replicated');
+
+# do a couple updates, check the correct stuff gets replicated
+$node_publisher->safe_psql('postgres', qq(
+ UPDATE tab2 SET c = 5 where a = 1;
+ UPDATE tab2 SET b = 'xyz' where a = 2;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT * FROM tab2 ORDER BY a");
+is($result, qq(1|abc
+2|xyz), 'update on column tab2.c is not replicated');
+
+
+# TEST: add a table to two publications with different column lists, and
+# create a single subscription replicating both publications
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub2 FOR TABLE tab5 (a, b);
+ CREATE PUBLICATION pub3 FOR TABLE tab5 (a, d);
+
+ -- insert a couple initial records
+ INSERT INTO tab5 VALUES (1, 11, 111, 1111);
+ INSERT INTO tab5 VALUES (2, 22, 222, 2222);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab5 (a int PRIMARY KEY, b int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub3
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->wait_for_catchup('sub1');
+
+# insert data and make sure all the columns (union of the columns lists)
+# get fully replicated
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (3, 33, 333, 3333);
+ INSERT INTO tab5 VALUES (4, 44, 444, 4444);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111
+2|22|2222
+3|33|3333
+4|44|4444),
+ 'overlapping publications with overlapping column lists');
+
+# and finally, remove the column list for one of the publications, which
+# means replicating all columns (removing the column list), but first add
+# the missing column to the table on subscriber
+$node_publisher->safe_psql('postgres', qq(
+ ALTER PUBLICATION pub3 SET TABLE tab5;
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ ALTER TABLE tab5 ADD COLUMN c INT;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab5 VALUES (5, 55, 555, 5555);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab5 ORDER BY a"),
+ qq(1|11|1111|
+2|22|2222|
+3|33|3333|
+4|44|4444|
+5|55|5555|555),
+ 'overlapping publications with overlapping column lists');
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key (but use a different column in
+# the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub4 FOR TABLE tab6 (a, b);
+
+ -- initial data
+ INSERT INTO tab6 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab6 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (2, 33, 444, 5555);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is still covered by the column list, though)
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+# we need to do the same thing on the subscriber
+# XXX What would happen if this happens before the publisher ALTER? Or
+# interleaved, somehow? But that seems unrelated to column lists.
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER TABLE tab6 DROP CONSTRAINT tab6_pkey;
+ ALTER TABLE tab6 ADD PRIMARY KEY (b);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab6 VALUES (3, 55, 666, 8888);
+ UPDATE tab6 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab6 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+
+# TEST: create a table with a column list, then change the replica
+# identity by replacing a primary key with a key on multiple columns
+# (all of them covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+ CREATE PUBLICATION pub5 FOR TABLE tab7 (a, b);
+
+ -- some initial data
+ INSERT INTO tab7 VALUES (1, 22, 333, 4444);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE tab7 (a int PRIMARY KEY, b int, c int, d int);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub5
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (2, 33, 444, 5555);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|44||
+2|66||), 'replication with the original primary key');
+
+# now redefine the constraint - move the primary key to a different column
+# (which is not covered by the column list)
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ ALTER TABLE tab7 ADD PRIMARY KEY (a, b);
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO tab7 VALUES (3, 55, 666, 7777);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(1|88||
+2|132||
+3|110||),
+ 'replication with the modified primary key');
+
+# now switch the primary key again to another columns not covered by the
+# column list, but also generate writes between the drop and creation
+# of the new constraint
+
+$node_publisher->safe_psql('postgres', qq(
+ ALTER TABLE tab7 DROP CONSTRAINT tab7_pkey;
+ INSERT INTO tab7 VALUES (4, 77, 888, 9999);
+ -- update/delete is not allowed for tables without RI
+ ALTER TABLE tab7 ADD PRIMARY KEY (b, a);
+ UPDATE tab7 SET b = b * 2, c = c * 3, d = d * 4;
+ DELETE FROM tab7 WHERE a = 1;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM tab7 ORDER BY a"),
+ qq(2|264||
+3|220||
+4|154||),
+ 'replication with the modified primary key');
+
+
+# TEST: partitioned tables (with publish_via_partition_root = false)
+# and replica identity. The (leaf) partitions may have different RI, so
+# we need to check the partition RI (with respect to the column list)
+# while attaching the partition.
+
+# First, let's create a partitioned table with two partitions, each with
+# a different RI, but a column list not covering all those RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+
+ -- initial data, one row in each partition
+ INSERT INTO test_part_a VALUES (1, 3);
+ INSERT INTO test_part_a VALUES (6, 4);
+));
+
+# do the same thing on the subscriber (with the opposite column order)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_a (b int, a int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_a_1 PARTITION OF test_part_a FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_a_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_a_1 REPLICA IDENTITY USING INDEX test_part_a_1_pkey;
+
+ CREATE TABLE test_part_a_2 PARTITION OF test_part_a FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_a_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_a_2 REPLICA IDENTITY USING INDEX test_part_a_2_pkey;
+));
+
+# create a publication replicating just the column "a", which is not enough
+# for the second partition
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub6 FOR TABLE test_part_a (b, a) WITH (publish_via_partition_root = true);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_1 (a);
+ ALTER PUBLICATION pub6 ADD TABLE test_part_a_2 (b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub6
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_a VALUES (2, 5);
+ INSERT INTO test_part_a VALUES (7, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT a, b FROM test_part_a ORDER BY a, b"),
+ qq(1|3
+2|5
+6|4
+7|6),
+ 'partitions with different replica identities not replicated correctly');
+
+# This time start with a column list covering RI for all partitions, but
+# then update the column list to not cover column "b" (needed by the
+# second partition)
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+
+ -- initial data, one row in each partitions
+ INSERT INTO test_part_b VALUES (1, 1);
+ INSERT INTO test_part_b VALUES (6, 2);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_b (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_b_1 PARTITION OF test_part_b FOR VALUES IN (1,2,3,4,5);
+ ALTER TABLE test_part_b_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_b_1 REPLICA IDENTITY USING INDEX test_part_b_1_pkey;
+
+ CREATE TABLE test_part_b_2 PARTITION OF test_part_b FOR VALUES IN (6,7,8,9,10);
+ ALTER TABLE test_part_b_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_b_2 REPLICA IDENTITY USING INDEX test_part_b_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub7 FOR TABLE test_part_b (a, b) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub7
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_b VALUES (2, 3);
+ INSERT INTO test_part_b VALUES (7, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_b ORDER BY a, b"),
+ qq(1|1
+2|3
+6|2
+7|4),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: This time start with a column list covering RI for all partitions,
+# but then update RI for one of the partitions to not be covered by the
+# column list anymore.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+
+ -- initial data, one row for each partition
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+# do the same thing on the subscriber
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_c (a int, b int, c int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_c_1 PARTITION OF test_part_c FOR VALUES IN (1,3);
+ ALTER TABLE test_part_c_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_c_1 REPLICA IDENTITY USING INDEX test_part_c_1_pkey;
+
+ CREATE TABLE test_part_c_2 PARTITION OF test_part_c FOR VALUES IN (2,4);
+ ALTER TABLE test_part_c_2 ADD PRIMARY KEY (b);
+ ALTER TABLE test_part_c_2 REPLICA IDENTITY USING INDEX test_part_c_2_pkey;
+));
+
+# create a publication replicating data through partition root, with a column
+# list on the root, and then add the partitions one by one with separate
+# column lists (but those are not applied)
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a,c);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SUBSCRIPTION sub1;
+ CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub8;
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_c VALUES (3, 7, 8);
+ INSERT INTO test_part_c VALUES (4, 9, 10);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||5
+2|4|
+3||8
+4|9|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# create a publication not replicating data through partition root, without
+# a column list on the root, and then add the partitions one by one with
+# separate column lists
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub8;
+ CREATE PUBLICATION pub8 FOR TABLE test_part_c WITH (publish_via_partition_root = false);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_1 (a);
+ ALTER PUBLICATION pub8 ADD TABLE test_part_c_2 (a,b);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+ TRUNCATE test_part_c;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ TRUNCATE test_part_c;
+ INSERT INTO test_part_c VALUES (1, 3, 5);
+ INSERT INTO test_part_c VALUES (2, 4, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_c ORDER BY a, b"),
+ qq(1||
+2|4|),
+ 'partitions with different replica identities not replicated correctly');
+
+
+# TEST: Start with a single partition, with RI compatible with the column
+# list, and then attach a partition with incompatible RI.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ INSERT INTO test_part_d VALUES (1, 2);
+));
+
+# do the same thing on the subscriber (in fact, create both partitions right
+# away, no need to delay that)
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_part_d (a int, b int) PARTITION BY LIST (a);
+
+ CREATE TABLE test_part_d_1 PARTITION OF test_part_d FOR VALUES IN (1,3);
+ ALTER TABLE test_part_d_1 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_1 REPLICA IDENTITY USING INDEX test_part_d_1_pkey;
+
+ CREATE TABLE test_part_d_2 PARTITION OF test_part_d FOR VALUES IN (2,4);
+ ALTER TABLE test_part_d_2 ADD PRIMARY KEY (a);
+ ALTER TABLE test_part_d_2 REPLICA IDENTITY USING INDEX test_part_d_2_pkey;
+));
+
+# create a publication replicating both columns, which is sufficient for
+# both partitions
+$node_publisher->safe_psql('postgres', qq(
+ CREATE PUBLICATION pub9 FOR TABLE test_part_d (a) WITH (publish_via_partition_root = true);
+));
+
+# add the publication to our subscription, wait for sync to complete
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub9
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_part_d VALUES (3, 4);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_part_d ORDER BY a, b"),
+ qq(1|
+3|),
+ 'partitions with different replica identities not replicated correctly');
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. So with column lists (a,b) and (a,c) we
+# should replicate (a,b,c).
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_1 FOR TABLE test_mix_1 (a, b);
+ CREATE PUBLICATION pub_mix_2 FOR TABLE test_mix_1 (a, c);
+
+ -- initial data
+ INSERT INTO test_mix_1 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_1 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_1, pub_mix_2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_1 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_1 ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES, we should replicate all columns.
+
+# drop unnecessary tables, so as not to interfere with the FOR ALL TABLES
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE tab1, tab2, tab3, tab4, tab5, tab6, tab7, test_mix_1,
+ test_part, test_part_a, test_part_b, test_part_c, test_part_d;
+));
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_3 FOR TABLE test_mix_2 (a, b);
+ CREATE PUBLICATION pub_mix_4 FOR ALL TABLES;
+
+ -- initial data
+ INSERT INTO test_mix_2 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_2 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_3, pub_mix_4;
+ ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_2 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_2"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: With a table included in multiple publications, we should use a
+# union of the column lists. If any of the publications is FOR ALL
+# TABLES IN SCHEMA, we should replicate all columns.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ CREATE PUBLICATION pub_mix_5 FOR TABLE test_mix_3 (a, b);
+ CREATE PUBLICATION pub_mix_6 FOR ALL TABLES IN SCHEMA public;
+
+ -- initial data
+ INSERT INTO test_mix_3 VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_mix_3 (a int PRIMARY KEY, b int, c int);
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_mix_5, pub_mix_6;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_mix_3 VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_mix_3"),
+ qq(1|2|3
+4|5|6),
+ 'a mix of publications should use a union of column list');
+
+
+# TEST: Check handling of publish_via_partition_root - if a partition is
+# published through partition root, we should only apply the column list
+# defined for the whole table (not the partitions) - both during the initial
+# sync and when replicating changes. This is what we do for row filters.
+
+$node_publisher->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+
+ CREATE PUBLICATION pub_root_true FOR TABLE test_root (a) WITH (publish_via_partition_root = true);
+
+ -- initial data
+ INSERT INTO test_root VALUES (1, 2, 3);
+ INSERT INTO test_root VALUES (10, 20, 30);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE TABLE test_root (a int PRIMARY KEY, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE test_root_1 PARTITION OF test_root FOR VALUES FROM (1) TO (10);
+ CREATE TABLE test_root_2 PARTITION OF test_root FOR VALUES FROM (10) TO (20);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub_root_true;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO test_root VALUES (2, 3, 4);
+ INSERT INTO test_root VALUES (11, 21, 31);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM test_root ORDER BY a, b, c"),
+ qq(1||
+2||
+10||
+11||),
+ 'publication via partition root applies column list');
+
+
+# TEST: Multiple publications which publish schema of parent table and
+# partition. The partition is published through two publications, once
+# through a schema (so no column list) containing the parent, and then
+# also directly (with a columns list). The expected outcome is there is
+# no column list.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP PUBLICATION pub1, pub2, pub3, pub4, pub5, pub6, pub7, pub8;
+
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub1 FOR ALL TABLES IN SCHEMA s1;
+ CREATE PUBLICATION pub2 FOR TABLE t_1(b);
+
+ -- initial data
+ INSERT INTO s1.t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ CREATE SCHEMA s1;
+ CREATE TABLE s1.t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF s1.t FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub1, pub2;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(1|2|3
+4|5|6),
+ 'two publications, publishing the same relation');
+
+# Now resync the subcription, but with publications in the opposite order.
+# The result should be the same.
+
+$node_subscriber->safe_psql('postgres', qq(
+ TRUNCATE s1.t;
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub2, pub1;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO s1.t VALUES (7, 8, 9);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM s1.t ORDER BY a"),
+ qq(7|8|9),
+ 'two publications, publishing the same relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub3 FOR TABLE t_1 (a), t_2
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP SCHEMA s1 CASCADE;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub3;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+# TEST: One publication, containing both the parent and child relations.
+# The expected outcome is list "a", because that's the column list defined
+# for the top-most ancestor added to the publication.
+# Note: The difference from the preceding test is that in this case both
+# relations have a column list defined.
+
+$node_publisher->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ CREATE PUBLICATION pub4 FOR TABLE t_1 (a), t_2 (b)
+ WITH (PUBLISH_VIA_PARTITION_ROOT);
+
+ -- initial data
+ INSERT INTO t VALUES (1, 2, 3);
+));
+
+$node_subscriber->safe_psql('postgres', qq(
+ DROP TABLE t;
+ CREATE TABLE t (a int, b int, c int) PARTITION BY RANGE (a);
+ CREATE TABLE t_1 PARTITION OF t FOR VALUES FROM (1) TO (10)
+ PARTITION BY RANGE (a);
+ CREATE TABLE t_2 PARTITION OF t_1 FOR VALUES FROM (1) TO (10);
+
+ ALTER SUBSCRIPTION sub1 SET PUBLICATION pub4;
+));
+
+wait_for_subscription_sync($node_subscriber);
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO t VALUES (4, 5, 6);
+));
+
+$node_publisher->wait_for_catchup('sub1');
+
+is($node_subscriber->safe_psql('postgres',"SELECT * FROM t ORDER BY a, b, c"),
+ qq(1||
+4||),
+ 'publication containing both parent and child relation');
+
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
+
+done_testing();
--
2.34.1
On Fri, Mar 25, 2022 at 5:44 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
Attached is a patch, rebased on top of the sequence decoding stuff I
pushed earlier today, also including the comments rewording, and
renaming the "transform" function.I'll go over it again and get it pushed soon, unless someone objects.
You haven't addressed the comments given by me earlier this week. See
/messages/by-id/CAA4eK1LY_JGL7LvdT64ujEiEAVaADuhdej1QNnwxvO_-KPzeEg@mail.gmail.com.
*
+ * XXX The name is a bit misleading, because we don't really transform
+ * anything here - we merely check the column list is compatible with the
+ * definition of the publication (with publish_via_partition_root=false)
+ * we only allow column lists on the leaf relations. So maybe rename it?
+ */
+static void
+CheckPubRelationColumnList(List *tables, const char *queryString,
+ bool pubviaroot)
After changing this function name, the comment above is not required.
--
With Regards,
Amit Kapila.
On 3/25/22 04:10, Amit Kapila wrote:
On Fri, Mar 25, 2022 at 5:44 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Attached is a patch, rebased on top of the sequence decoding stuff I
pushed earlier today, also including the comments rewording, and
renaming the "transform" function.I'll go over it again and get it pushed soon, unless someone objects.
You haven't addressed the comments given by me earlier this week. See
/messages/by-id/CAA4eK1LY_JGL7LvdT64ujEiEAVaADuhdej1QNnwxvO_-KPzeEg@mail.gmail.com.
Thanks for noticing that! Thunderbird did not include that message into
the patch thread for some reason, so I did not notice that!
* + * XXX The name is a bit misleading, because we don't really transform + * anything here - we merely check the column list is compatible with the + * definition of the publication (with publish_via_partition_root=false) + * we only allow column lists on the leaf relations. So maybe rename it? + */ +static void +CheckPubRelationColumnList(List *tables, const char *queryString, + bool pubviaroot)After changing this function name, the comment above is not required.
Thanks, comment updated.
I went over the patch again, polished the commit message a bit, and
pushed. May the buildfarm be merciful!
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 3/21/22 15:12, Amit Kapila wrote:
On Sat, Mar 19, 2022 at 11:11 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 3/19/22 18:11, Tomas Vondra wrote:
Fix a compiler warning reported by cfbot.
Apologies, I failed to actually commit the fix. So here we go again.
Few comments: =============== 1. +/* + * Gets a list of OIDs of all partial-column publications of the given + * relation, that is, those that specify a column list. + */ +List * +GetRelationColumnPartialPublications(Oid relid) { ... }... +/* + * For a relation in a publication that is known to have a non-null column + * list, return the list of attribute numbers that are in it. + */ +List * +GetRelationColumnListInPublication(Oid relid, Oid pubid) { ... }Both these functions are not required now. So, we can remove them.
Good catch, removed.
2. @@ -464,11 +478,11 @@ logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel, pq_sendbyte(out, 'O'); /* old tuple follows */ else pq_sendbyte(out, 'K'); /* old key follows */ - logicalrep_write_tuple(out, rel, oldslot, binary); + logicalrep_write_tuple(out, rel, oldslot, binary, columns); }As mentioned previously, here, we should pass NULL similar to
logicalrep_write_delete as we don't need to use column list for old
tuples.
Fixed.
3. + * XXX The name is a bit misleading, because we don't really transform + * anything here - we merely check the column list is compatible with the + * definition of the publication (with publish_via_partition_root=false) + * we only allow column lists on the leaf relations. So maybe rename it? + */ +static void +TransformPubColumnList(List *tables, const char *queryString, + bool pubviaroot)The second parameter is not used in this function. As noted in the
comments, I also think it is better to rename this. How about
ValidatePubColumnList?4. @@ -821,6 +942,9 @@ fetch_remote_table_info(char *nspname, char *relname, * * 3) one of the subscribed publications is declared as ALL TABLES IN * SCHEMA that includes this relation + * + * XXX Does this actually handle puballtables and schema publications + * correctly? */ if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)Why is this comment added in the row filter code? Now, both row filter
and column list are fetched in the same way, so not sure what exactly
this comment is referring to.
I added that comment as a note to myself while learning about how the
code works, forgot to remove that.
5. +/* qsort comparator for attnums */ +static int +compare_int16(const void *a, const void *b) +{ + int av = *(const int16 *) a; + int bv = *(const int16 *) b; + + /* this can't overflow if int is wider than int16 */ + return (av - bv); +}The exact same code exists in statscmds.c. Do we need a second copy of the same?
Yeah, I thought about moving it to some common header, but I think it's
not really worth it at this point.
6.
static void pgoutput_row_filter_init(PGOutputData *data,
List *publications,
RelationSyncEntry *entry);
+
static bool pgoutput_row_filter_exec_expr(ExprState *state,Spurious line addition.
Fixed.
7. The tests in 030_column_list.pl take a long time as compared to all
other similar individual tests in the subscription folder. I haven't
checked whether there is any need to reduce some tests but it seems
worth checking.
On my machine, 'make check' in src/test/subscription takes ~150 seconds
(with asserts and -O0), and the new script takes ~14 seconds, while most
other tests have 3-6 seconds.
AFAICS that's simply due to the number of tests in the script, and I
don't think there are any unnecessary ones. I was actually adding them
in response to issues reported during development, or to test various
important cases. So I don't think we can remove some of them easily :-(
And it's not like the tests are using massive amounts of data either.
We could split the test, but that obviously won't reduce the duration,
of course.
So I decided to keep the test as is, for now, and maybe we can try
reducing the test after a couple buildfarm runs.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 3/26/22 01:18, Tomas Vondra wrote:
...
I went over the patch again, polished the commit message a bit, and
pushed. May the buildfarm be merciful!
There's a couple failures immediately after the push, which caused me a
minor heart attack. But it seems all of those are strange failures
related to configure (which the patch did not touch at all), on animals
managed by Andres. And a couple animals succeeded since then.
So I guess the animals were reconfigured, or something ...
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Hello,
The 'prattrs' column has been added to the pg_publication_rel catalog,
but the current commit to catalog.sgml seems to have added it to pg_publication_namespace.
The attached patch fixes this.
Regards,
Noriyoshi Shinoda
-----Original Message-----
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
Sent: Saturday, March 26, 2022 9:35 AM
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Peter Eisentraut <peter.eisentraut@enterprisedb.com>; houzj.fnst@fujitsu.com; Alvaro Herrera <alvherre@alvh.no-ip.org>; Justin Pryzby <pryzby@telsasoft.com>; Rahila Syed <rahilasyed90@gmail.com>; Peter Smith <smithpb2250@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org>; shiy.fnst@fujitsu.com
Subject: Re: Column Filtering in Logical Replication
On 3/26/22 01:18, Tomas Vondra wrote:
...
I went over the patch again, polished the commit message a bit, and
pushed. May the buildfarm be merciful!
There's a couple failures immediately after the push, which caused me a minor heart attack. But it seems all of those are strange failures related to configure (which the patch did not touch at all), on animals managed by Andres. And a couple animals succeeded since then.
So I guess the animals were reconfigured, or something ...
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
prattrs_column_fix_v1.diffapplication/octet-stream; name=prattrs_column_fix_v1.diffDownload
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 560e205..94f01e4 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6291,19 +6291,6 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to schema
</para></entry>
</row>
-
- <row>
- <entry role="catalog_table_entry"><para role="column_definition">
- <structfield>prattrs</structfield> <type>int2vector</type>
- (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
- </para>
- <para>
- This is an array of values that indicates which table columns are
- part of the publication. For example, a value of <literal>1 3</literal>
- would mean that the first and the third table columns are published.
- A null value indicates that all columns are published.
- </para></entry>
- </row>
</tbody>
</tgroup>
</table>
@@ -6375,6 +6362,19 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
representation) for the relation's publication qualifying condition. Null
if there is no publication qualifying condition.</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>prattrs</structfield> <type>int2vector</type>
+ (references <link linkend="catalog-pg-attribute"><structname>pg_attribute</structname></link>.<structfield>attnum</structfield>)
+ </para>
+ <para>
+ This is an array of values that indicates which table columns are
+ part of the publication. For example, a value of <literal>1 3</literal>
+ would mean that the first and the third table columns are published.
+ A null value indicates that all columns are published.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
On 3/26/22 05:09, Shinoda, Noriyoshi (PN Japan FSIP) wrote:
Hello,
The 'prattrs' column has been added to the pg_publication_rel catalog,
but the current commit to catalog.sgml seems to have added it to pg_publication_namespace.
The attached patch fixes this.
Thanks, I'll get this pushed.
Sadly, while looking at the catalog docs I realized I forgot to bump the
catversion :-(
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 3/26/22 10:58, Tomas Vondra wrote:
On 3/26/22 05:09, Shinoda, Noriyoshi (PN Japan FSIP) wrote:
Hello,
The 'prattrs' column has been added to the pg_publication_rel catalog,
but the current commit to catalog.sgml seems to have added it to pg_publication_namespace.
The attached patch fixes this.Thanks, I'll get this pushed.
Pushed. Thanks for noticing this!
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Tomas Vondra <tomas.vondra@enterprisedb.com> writes:
I went over the patch again, polished the commit message a bit, and
pushed. May the buildfarm be merciful!
Initial results aren't that great. komodoensis[1]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=komodoensis&dt=2022-03-26%2015%3A54%3A04, petalura[2]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=petalura&dt=2022-03-26%2004%3A20%3A04,
and snapper[3]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=snapper&dt=2022-03-26%2018%3A46%3A28 have all shown variants of
# Failed test 'partitions with different replica identities not replicated correctly'
# at t/031_column_list.pl line 734.
# got: '2|4|
# 4|9|'
# expected: '1||5
# 2|4|
# 3||8
# 4|9|'
# Looks like you failed 1 test of 34.
[18:19:36] t/031_column_list.pl ...............
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/34 subtests
snapper reported different actual output than the other two:
# got: '1||5
# 3||8'
The failure seems intermittent, as both komodoensis and petalura
have also passed cleanly since the commit (snapper's only run once).
This smells like an uninitialized-variable problem, but I've had
no luck finding any problem under valgrind. Not sure how to progress
from here.
regards, tom lane
[1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=komodoensis&dt=2022-03-26%2015%3A54%3A04
[2]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=petalura&dt=2022-03-26%2004%3A20%3A04
[3]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=snapper&dt=2022-03-26%2018%3A46%3A28
On 3/26/22 22:37, Tom Lane wrote:
Tomas Vondra <tomas.vondra@enterprisedb.com> writes:
I went over the patch again, polished the commit message a bit, and
pushed. May the buildfarm be merciful!Initial results aren't that great. komodoensis[1], petalura[2],
and snapper[3] have all shown variants of# Failed test 'partitions with different replica identities not replicated correctly'
# at t/031_column_list.pl line 734.
# got: '2|4|
# 4|9|'
# expected: '1||5
# 2|4|
# 3||8
# 4|9|'
# Looks like you failed 1 test of 34.
[18:19:36] t/031_column_list.pl ...............
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/34 subtestssnapper reported different actual output than the other two:
# got: '1||5
# 3||8'The failure seems intermittent, as both komodoensis and petalura
have also passed cleanly since the commit (snapper's only run once).This smells like an uninitialized-variable problem, but I've had
no luck finding any problem under valgrind. Not sure how to progress
from here.
I think I see the problem - there's a CREATE SUBSCRIPTION but the test
is not waiting for the tablesync to complete, so sometimes it finishes
in time and sometimes not. That'd explain the flaky behavior, and it's
just this one test that misses the sync AFAICS.
FWIW I did run this under valgrind a number of times, and also on
various ARM machines that tend to trip over memory issues.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Tomas Vondra <tomas.vondra@enterprisedb.com> writes:
On 3/26/22 22:37, Tom Lane wrote:
This smells like an uninitialized-variable problem, but I've had
no luck finding any problem under valgrind. Not sure how to progress
from here.
I think I see the problem - there's a CREATE SUBSCRIPTION but the test
is not waiting for the tablesync to complete, so sometimes it finishes
in time and sometimes not. That'd explain the flaky behavior, and it's
just this one test that misses the sync AFAICS.
Ah, that would also fit the symptoms.
regards, tom lane
On 3/26/22 22:55, Tom Lane wrote:
Tomas Vondra <tomas.vondra@enterprisedb.com> writes:
On 3/26/22 22:37, Tom Lane wrote:
This smells like an uninitialized-variable problem, but I've had
no luck finding any problem under valgrind. Not sure how to progress
from here.I think I see the problem - there's a CREATE SUBSCRIPTION but the test
is not waiting for the tablesync to complete, so sometimes it finishes
in time and sometimes not. That'd explain the flaky behavior, and it's
just this one test that misses the sync AFAICS.Ah, that would also fit the symptoms.
I'll go over the test to check if some other test misses that, and
perhaps do a bit of testing, and then push a fix.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 3/26/22 22:58, Tomas Vondra wrote:
On 3/26/22 22:55, Tom Lane wrote:
Tomas Vondra <tomas.vondra@enterprisedb.com> writes:
On 3/26/22 22:37, Tom Lane wrote:
This smells like an uninitialized-variable problem, but I've had
no luck finding any problem under valgrind. Not sure how to progress
from here.I think I see the problem - there's a CREATE SUBSCRIPTION but the test
is not waiting for the tablesync to complete, so sometimes it finishes
in time and sometimes not. That'd explain the flaky behavior, and it's
just this one test that misses the sync AFAICS.Ah, that would also fit the symptoms.
I'll go over the test to check if some other test misses that, and
perhaps do a bit of testing, and then push a fix.
Pushed. I checked the other tests in 031_column_list.pl and I AFAICS all
of them are waiting for the sync correctly.
[rolls eyes] I just noticed I listed the file as .sql in the commit
message. Not great.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Sun, Mar 20, 2022 at 4:53 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 3/20/22 07:23, Amit Kapila wrote:
On Sun, Mar 20, 2022 at 8:41 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Mar 18, 2022 at 10:42 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:So the question is why those two sync workers never complete - I guess
there's some sort of lock wait (deadlock?) or infinite loop.It would be a bit tricky to reproduce this even if the above theory is
correct but I'll try it today or tomorrow.I am able to reproduce it with the help of a debugger. Firstly, I have
added the LOG message and some While (true) loops to debug sync and
apply workers. Test setupNode-1:
create table t1(c1);
create table t2(c1);
insert into t1 values(1);
create publication pub1 for table t1;
create publication pu2;Node-2:
change max_sync_workers_per_subscription to 1 in potgresql.conf
create table t1(c1);
create table t2(c1);
create subscription sub1 connection 'dbname = postgres' publication pub1;Till this point, just allow debuggers in both workers just continue.
Node-1:
alter publication pub1 add table t2;
insert into t1 values(2);Here, we have to debug the apply worker such that when it tries to
apply the insert, stop the debugger in function apply_handle_insert()
after doing begin_replication_step().Node-2:
alter subscription sub1 set pub1, pub2;Now, continue the debugger of apply worker, it should first start the
sync worker and then exit because of parameter change. All of these
debugging steps are to just ensure the point that it should first
start the sync worker and then exit. After this point, table sync
worker never finishes and log is filled with messages: "reached
max_sync_workers_per_subscription limit" (a newly added message by me
in the attached debug patch).Now, it is not completely clear to me how exactly '013_partition.pl'
leads to this situation but there is a possibility based on the LOGs
it shows.Thanks, I'll take a look later.
This is still failing [1]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=florican&dt=2022-03-28%2005%3A16%3A53[2]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-24%2013%3A13%3A08.
[1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=florican&dt=2022-03-28%2005%3A16%3A53
[2]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-24%2013%3A13%3A08
--
With Regards,
Amit Kapila.
On 3/29/22 12:00, Amit Kapila wrote:
On Sun, Mar 20, 2022 at 4:53 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 3/20/22 07:23, Amit Kapila wrote:
On Sun, Mar 20, 2022 at 8:41 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Mar 18, 2022 at 10:42 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:So the question is why those two sync workers never complete - I guess
there's some sort of lock wait (deadlock?) or infinite loop.It would be a bit tricky to reproduce this even if the above theory is
correct but I'll try it today or tomorrow.I am able to reproduce it with the help of a debugger. Firstly, I have
added the LOG message and some While (true) loops to debug sync and
apply workers. Test setupNode-1:
create table t1(c1);
create table t2(c1);
insert into t1 values(1);
create publication pub1 for table t1;
create publication pu2;Node-2:
change max_sync_workers_per_subscription to 1 in potgresql.conf
create table t1(c1);
create table t2(c1);
create subscription sub1 connection 'dbname = postgres' publication pub1;Till this point, just allow debuggers in both workers just continue.
Node-1:
alter publication pub1 add table t2;
insert into t1 values(2);Here, we have to debug the apply worker such that when it tries to
apply the insert, stop the debugger in function apply_handle_insert()
after doing begin_replication_step().Node-2:
alter subscription sub1 set pub1, pub2;Now, continue the debugger of apply worker, it should first start the
sync worker and then exit because of parameter change. All of these
debugging steps are to just ensure the point that it should first
start the sync worker and then exit. After this point, table sync
worker never finishes and log is filled with messages: "reached
max_sync_workers_per_subscription limit" (a newly added message by me
in the attached debug patch).Now, it is not completely clear to me how exactly '013_partition.pl'
leads to this situation but there is a possibility based on the LOGs
it shows.Thanks, I'll take a look later.
This is still failing [1][2].
[1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=florican&dt=2022-03-28%2005%3A16%3A53
[2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-24%2013%3A13%3A08
AFAICS we've concluded this is a pre-existing issue, not something
introduced by a recently committed patch, and I don't think there's any
proposal how to fix that. So I've put that on the back burner until
after the current CF.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Tue, Mar 29, 2022 at 4:33 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 3/29/22 12:00, Amit Kapila wrote:
Thanks, I'll take a look later.
This is still failing [1][2].
[1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=florican&dt=2022-03-28%2005%3A16%3A53
[2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-24%2013%3A13%3A08AFAICS we've concluded this is a pre-existing issue, not something
introduced by a recently committed patch, and I don't think there's any
proposal how to fix that.
I have suggested in email [1]/messages/by-id/CAA4eK1LpBFU49Ohbnk=dv_v9YP+Kqh1+Sf8i++_s-QhD1Gy4Qw@mail.gmail.com that increasing values
max_sync_workers_per_subscription/max_logical_replication_workers
should solve this issue. Now, whether this is a previous issue or
behavior can be debatable but I think it happens for the new test case
added by commit c91f71b9dc.
So I've put that on the back burner until
after the current CF.
Okay, last time you didn't mention that you want to look at it after
CF. I just assumed that you want to take a look after pushing the main
column list patch, so thought of sending a reminder but I am fine if
you want to look at it after CF.
[1]: /messages/by-id/CAA4eK1LpBFU49Ohbnk=dv_v9YP+Kqh1+Sf8i++_s-QhD1Gy4Qw@mail.gmail.com
--
With Regards,
Amit Kapila.
On 3/29/22 13:47, Amit Kapila wrote:
On Tue, Mar 29, 2022 at 4:33 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 3/29/22 12:00, Amit Kapila wrote:
Thanks, I'll take a look later.
This is still failing [1][2].
[1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=florican&dt=2022-03-28%2005%3A16%3A53
[2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-24%2013%3A13%3A08AFAICS we've concluded this is a pre-existing issue, not something
introduced by a recently committed patch, and I don't think there's any
proposal how to fix that.I have suggested in email [1] that increasing values
max_sync_workers_per_subscription/max_logical_replication_workers
should solve this issue. Now, whether this is a previous issue or
behavior can be debatable but I think it happens for the new test case
added by commit c91f71b9dc.
IMHO that'd be just hiding the actual issue, which is the failure to
sync the subscription in some circumstances. We should fix that, not
just make sure the tests don't trigger it.
So I've put that on the back burner until
after the current CF.Okay, last time you didn't mention that you want to look at it after
CF. I just assumed that you want to take a look after pushing the main
column list patch, so thought of sending a reminder but I am fine if
you want to look at it after CF.
OK, sorry for not being clearer in my response.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Tue, Mar 29, 2022 at 6:09 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 3/29/22 13:47, Amit Kapila wrote:
On Tue, Mar 29, 2022 at 4:33 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 3/29/22 12:00, Amit Kapila wrote:
Thanks, I'll take a look later.
This is still failing [1][2].
[1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=florican&dt=2022-03-28%2005%3A16%3A53
[2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-24%2013%3A13%3A08AFAICS we've concluded this is a pre-existing issue, not something
introduced by a recently committed patch, and I don't think there's any
proposal how to fix that.I have suggested in email [1] that increasing values
max_sync_workers_per_subscription/max_logical_replication_workers
should solve this issue. Now, whether this is a previous issue or
behavior can be debatable but I think it happens for the new test case
added by commit c91f71b9dc.IMHO that'd be just hiding the actual issue, which is the failure to
sync the subscription in some circumstances. We should fix that, not
just make sure the tests don't trigger it.
I am in favor of fixing/changing some existing behavior to make it
better and would be ready to help in that investigation as well but
was just not sure if it is a good idea to let some of the buildfarm
member(s) fail for a number of days. Anyway, I leave this judgment to
you.
--
With Regards,
Amit Kapila.
On 3/30/22 04:46, Amit Kapila wrote:
On Tue, Mar 29, 2022 at 6:09 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 3/29/22 13:47, Amit Kapila wrote:
On Tue, Mar 29, 2022 at 4:33 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 3/29/22 12:00, Amit Kapila wrote:
Thanks, I'll take a look later.
This is still failing [1][2].
[1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=florican&dt=2022-03-28%2005%3A16%3A53
[2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2022-03-24%2013%3A13%3A08AFAICS we've concluded this is a pre-existing issue, not something
introduced by a recently committed patch, and I don't think there's any
proposal how to fix that.I have suggested in email [1] that increasing values
max_sync_workers_per_subscription/max_logical_replication_workers
should solve this issue. Now, whether this is a previous issue or
behavior can be debatable but I think it happens for the new test case
added by commit c91f71b9dc.IMHO that'd be just hiding the actual issue, which is the failure to
sync the subscription in some circumstances. We should fix that, not
just make sure the tests don't trigger it.I am in favor of fixing/changing some existing behavior to make it
better and would be ready to help in that investigation as well but
was just not sure if it is a good idea to let some of the buildfarm
member(s) fail for a number of days. Anyway, I leave this judgment to
you.
OK. If it affected more animals, and/or if they were failing more often,
it'd definitely warrant a more active approach. But AFAICS it affects
only a tiny fraction, and even there it fails maybe 1 in 20 runs ...
Plus the symptoms are pretty clear, it's unlikely to cause enigmatic
failures, forcing people to spend time on investigating it.
Of course, that's my assessment and it feels weird as it goes directly
against my instincts to keep all tests working :-/
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Sun, Mar 20, 2022 at 3:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Sun, Mar 20, 2022 at 8:41 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Mar 18, 2022 at 10:42 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:So the question is why those two sync workers never complete - I guess
there's some sort of lock wait (deadlock?) or infinite loop.It would be a bit tricky to reproduce this even if the above theory is
correct but I'll try it today or tomorrow.I am able to reproduce it with the help of a debugger. Firstly, I have
added the LOG message and some While (true) loops to debug sync and
apply workers. Test setupNode-1:
create table t1(c1);
create table t2(c1);
insert into t1 values(1);
create publication pub1 for table t1;
create publication pu2;Node-2:
change max_sync_workers_per_subscription to 1 in potgresql.conf
create table t1(c1);
create table t2(c1);
create subscription sub1 connection 'dbname = postgres' publication pub1;Till this point, just allow debuggers in both workers just continue.
Node-1:
alter publication pub1 add table t2;
insert into t1 values(2);Here, we have to debug the apply worker such that when it tries to
apply the insert, stop the debugger in function apply_handle_insert()
after doing begin_replication_step().Node-2:
alter subscription sub1 set pub1, pub2;Now, continue the debugger of apply worker, it should first start the
sync worker and then exit because of parameter change. All of these
debugging steps are to just ensure the point that it should first
start the sync worker and then exit. After this point, table sync
worker never finishes and log is filled with messages: "reached
max_sync_workers_per_subscription limit" (a newly added message by me
in the attached debug patch).Now, it is not completely clear to me how exactly '013_partition.pl'
leads to this situation but there is a possibility based on the LOGs
I've looked at this issue and had the same analysis. Also, I could
reproduce this issue with the steps shared by Amit.
As I mentioned in another thread[1], the fact that the tablesync
worker doesn't check the return value from
wait_for_worker_state_change() seems a bug to me. So my initial
thought of the solution is that we can have the tablesync worker check
the return value and exit if it's false. That way, the apply worker
can restart and request to launch the tablesync worker again. What do
you think?
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
On Wed, Apr 13, 2022 at 1:41 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
I've looked at this issue and had the same analysis. Also, I could
reproduce this issue with the steps shared by Amit.As I mentioned in another thread[1], the fact that the tablesync
worker doesn't check the return value from
wait_for_worker_state_change() seems a bug to me. So my initial
thought of the solution is that we can have the tablesync worker check
the return value and exit if it's false. That way, the apply worker
can restart and request to launch the tablesync worker again. What do
you think?
I think that will fix this symptom but I am not sure if that would be
the best way to deal with this because we have a mechanism where the
sync worker can continue even if we don't do anything as a result of
wait_for_worker_state_change() provided apply worker restarts.
The other part of the puzzle is the below check in the code:
/*
* If we reached the sync worker limit per subscription, just exit
* silently as we might get here because of an otherwise harmless race
* condition.
*/
if (nsyncworkers >= max_sync_workers_per_subscription)
It is not clear to me why this check is there, if this wouldn't be
there, the user would have got either a WARNING to increase the
max_logical_replication_workers or the apply worker would have been
restarted. Do you have any idea about this?
Yet another option is that we ensure that before launching sync
workers (say in process_syncing_tables_for_apply->FetchTableStates,
when we have to start a new transaction) we again call
maybe_reread_subscription(), which should also fix this symptom. But
again, I am not sure why it should be compulsory to call
maybe_reread_subscription() in such a situation, there are no comments
which suggest it,
Now, the reason why it appeared recently in commit c91f71b9dc is that
I think we have increased the number of initial table syncs in that
test, and probably increasing
max_sync_workers_per_subscription/max_logical_replication_workers
should fix that test.
--
With Regards,
Amit Kapila.
(
On Wed, Apr 13, 2022 at 6:45 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Apr 13, 2022 at 1:41 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
I've looked at this issue and had the same analysis. Also, I could
reproduce this issue with the steps shared by Amit.As I mentioned in another thread[1], the fact that the tablesync
worker doesn't check the return value from
wait_for_worker_state_change() seems a bug to me. So my initial
thought of the solution is that we can have the tablesync worker check
the return value and exit if it's false. That way, the apply worker
can restart and request to launch the tablesync worker again. What do
you think?I think that will fix this symptom but I am not sure if that would be
the best way to deal with this because we have a mechanism where the
sync worker can continue even if we don't do anything as a result of
wait_for_worker_state_change() provided apply worker restarts.
I think we can think this is a separate issue. That is, if tablesync
worker can start streaming changes even without waiting for the apply
worker to set SUBREL_STATE_CATCHUP, do we really need the wait? I'm
not sure it's really safe. If it's safe, the tablesync worker will no
longer need to wait there.
The other part of the puzzle is the below check in the code:
/*
* If we reached the sync worker limit per subscription, just exit
* silently as we might get here because of an otherwise harmless race
* condition.
*/
if (nsyncworkers >= max_sync_workers_per_subscription)It is not clear to me why this check is there, if this wouldn't be
there, the user would have got either a WARNING to increase the
max_logical_replication_workers or the apply worker would have been
restarted. Do you have any idea about this?
Yeah, I'm also puzzled with this check. It seems that this function
doesn't work well when the apply worker is not running and some
tablesync workers are running. I initially thought that the apply
worker calls to this function as many as tables that needs to be
synced, but it checks the max_sync_workers_per_subscription limit
before calling to logicalrep_worker_launch(). So I'm not really sure
we need this check.
Yet another option is that we ensure that before launching sync
workers (say in process_syncing_tables_for_apply->FetchTableStates,
when we have to start a new transaction) we again call
maybe_reread_subscription(), which should also fix this symptom. But
again, I am not sure why it should be compulsory to call
maybe_reread_subscription() in such a situation, there are no comments
which suggest it,
Yes, it will fix this issue.
Now, the reason why it appeared recently in commit c91f71b9dc is that
I think we have increased the number of initial table syncs in that
test, and probably increasing
max_sync_workers_per_subscription/max_logical_replication_workers
should fix that test.
I think so too.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
On Thu, Apr 14, 2022 at 8:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
On Wed, Apr 13, 2022 at 6:45 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Apr 13, 2022 at 1:41 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
I've looked at this issue and had the same analysis. Also, I could
reproduce this issue with the steps shared by Amit.As I mentioned in another thread[1], the fact that the tablesync
worker doesn't check the return value from
wait_for_worker_state_change() seems a bug to me. So my initial
thought of the solution is that we can have the tablesync worker check
the return value and exit if it's false. That way, the apply worker
can restart and request to launch the tablesync worker again. What do
you think?I think that will fix this symptom but I am not sure if that would be
the best way to deal with this because we have a mechanism where the
sync worker can continue even if we don't do anything as a result of
wait_for_worker_state_change() provided apply worker restarts.I think we can think this is a separate issue. That is, if tablesync
worker can start streaming changes even without waiting for the apply
worker to set SUBREL_STATE_CATCHUP, do we really need the wait? I'm
not sure it's really safe. If it's safe, the tablesync worker will no
longer need to wait there.
As per my understanding, it is safe, whatever is streamed by tablesync
worker will be skipped later by apply worker. The wait here avoids
streaming the same data both by the apply worker and table sync worker
which I think is good even if it is not a must.
The other part of the puzzle is the below check in the code:
/*
* If we reached the sync worker limit per subscription, just exit
* silently as we might get here because of an otherwise harmless race
* condition.
*/
if (nsyncworkers >= max_sync_workers_per_subscription)It is not clear to me why this check is there, if this wouldn't be
there, the user would have got either a WARNING to increase the
max_logical_replication_workers or the apply worker would have been
restarted. Do you have any idea about this?Yeah, I'm also puzzled with this check. It seems that this function
doesn't work well when the apply worker is not running and some
tablesync workers are running. I initially thought that the apply
worker calls to this function as many as tables that needs to be
synced, but it checks the max_sync_workers_per_subscription limit
before calling to logicalrep_worker_launch(). So I'm not really sure
we need this check.
I just hope that the original author Petr J. responds to this point. I
have added him to this email. This will help us to find the best
solution for this problem.
Note: I'll be away for the remaining week, so will join the discussion
next week unless we reached the conclusion by that time.
--
With Regards,
Amit Kapila.
On Thu, Apr 14, 2022 at 9:09 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Thu, Apr 14, 2022 at 8:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
The other part of the puzzle is the below check in the code:
/*
* If we reached the sync worker limit per subscription, just exit
* silently as we might get here because of an otherwise harmless race
* condition.
*/
if (nsyncworkers >= max_sync_workers_per_subscription)It is not clear to me why this check is there, if this wouldn't be
there, the user would have got either a WARNING to increase the
max_logical_replication_workers or the apply worker would have been
restarted. Do you have any idea about this?Yeah, I'm also puzzled with this check. It seems that this function
doesn't work well when the apply worker is not running and some
tablesync workers are running. I initially thought that the apply
worker calls to this function as many as tables that needs to be
synced, but it checks the max_sync_workers_per_subscription limit
before calling to logicalrep_worker_launch(). So I'm not really sure
we need this check.I just hope that the original author Petr J. responds to this point. I
have added him to this email. This will help us to find the best
solution for this problem.
I did some more investigation for this code. It is added by commit [1]commit de4389712206d2686e09ad8d6dd112dc4b6c6d42 Author: Peter Eisentraut <peter_e@gmx.net> Date: Wed Apr 26 10:43:04 2017 -0400
and the patch that led to this commit is first time posted on -hackers
in email [2]/messages/by-id/fa387e24-0e26-c02d-ef16-7e46ada200dd@2ndquadrant.com. Now, neither the commit message nor the patch (comments)
gives much idea as to why this part of code is added but I think there
is some hint in the email [2]/messages/by-id/fa387e24-0e26-c02d-ef16-7e46ada200dd@2ndquadrant.com. In particular, read the paragraph in
the email [2]/messages/by-id/fa387e24-0e26-c02d-ef16-7e46ada200dd@2ndquadrant.com that has the lines: ".... and limiting sync workers per
subscription theoretically wasn't either (although I don't think it
could happen in practice).".
It seems that this check has been added to theoretically limit the
sync workers even though that can't happen because apply worker
ensures that before trying to launch the sync worker. Does this theory
make sense to me? If so, I think we can change the check as: "if
(OidIsValid(relid) && nsyncworkers >=
max_sync_workers_per_subscription)" in launcher.c. This will serve the
purpose of the original code and will solve the issue being discussed
here. I think we can even backpatch this. What do you think?
[1]: commit de4389712206d2686e09ad8d6dd112dc4b6c6d42 Author: Peter Eisentraut <peter_e@gmx.net> Date: Wed Apr 26 10:43:04 2017 -0400
commit de4389712206d2686e09ad8d6dd112dc4b6c6d42
Author: Peter Eisentraut <peter_e@gmx.net>
Date: Wed Apr 26 10:43:04 2017 -0400
Fix various concurrency issues in logical replication worker launching
[2]: /messages/by-id/fa387e24-0e26-c02d-ef16-7e46ada200dd@2ndquadrant.com
--
With Regards,
Amit Kapila.
On 4/18/22 13:04, Amit Kapila wrote:
On Thu, Apr 14, 2022 at 9:09 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Thu, Apr 14, 2022 at 8:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
The other part of the puzzle is the below check in the code:
/*
* If we reached the sync worker limit per subscription, just exit
* silently as we might get here because of an otherwise harmless race
* condition.
*/
if (nsyncworkers >= max_sync_workers_per_subscription)It is not clear to me why this check is there, if this wouldn't be
there, the user would have got either a WARNING to increase the
max_logical_replication_workers or the apply worker would have been
restarted. Do you have any idea about this?Yeah, I'm also puzzled with this check. It seems that this function
doesn't work well when the apply worker is not running and some
tablesync workers are running. I initially thought that the apply
worker calls to this function as many as tables that needs to be
synced, but it checks the max_sync_workers_per_subscription limit
before calling to logicalrep_worker_launch(). So I'm not really sure
we need this check.I just hope that the original author Petr J. responds to this point. I
have added him to this email. This will help us to find the best
solution for this problem.I did some more investigation for this code. It is added by commit [1]
and the patch that led to this commit is first time posted on -hackers
in email [2]. Now, neither the commit message nor the patch (comments)
gives much idea as to why this part of code is added but I think there
is some hint in the email [2]. In particular, read the paragraph in
the email [2] that has the lines: ".... and limiting sync workers per
subscription theoretically wasn't either (although I don't think it
could happen in practice).".It seems that this check has been added to theoretically limit the
sync workers even though that can't happen because apply worker
ensures that before trying to launch the sync worker. Does this theory
make sense to me? If so, I think we can change the check as: "if
(OidIsValid(relid) && nsyncworkers >=
max_sync_workers_per_subscription)" in launcher.c. This will serve the
purpose of the original code and will solve the issue being discussed
here. I think we can even backpatch this. What do you think?
Sounds reasonable to me. It's unfortunate there's no explanation of what
exactly is the commit message fixing (and why), but I doubt anyone will
remember the details after 5 years.
+1 to backpatching, I consider this to be a bug
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Mon, Apr 18, 2022 at 8:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Thu, Apr 14, 2022 at 9:09 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Thu, Apr 14, 2022 at 8:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
The other part of the puzzle is the below check in the code:
/*
* If we reached the sync worker limit per subscription, just exit
* silently as we might get here because of an otherwise harmless race
* condition.
*/
if (nsyncworkers >= max_sync_workers_per_subscription)It is not clear to me why this check is there, if this wouldn't be
there, the user would have got either a WARNING to increase the
max_logical_replication_workers or the apply worker would have been
restarted. Do you have any idea about this?Yeah, I'm also puzzled with this check. It seems that this function
doesn't work well when the apply worker is not running and some
tablesync workers are running. I initially thought that the apply
worker calls to this function as many as tables that needs to be
synced, but it checks the max_sync_workers_per_subscription limit
before calling to logicalrep_worker_launch(). So I'm not really sure
we need this check.I just hope that the original author Petr J. responds to this point. I
have added him to this email. This will help us to find the best
solution for this problem.I did some more investigation for this code. It is added by commit [1]
and the patch that led to this commit is first time posted on -hackers
in email [2]. Now, neither the commit message nor the patch (comments)
gives much idea as to why this part of code is added but I think there
is some hint in the email [2]. In particular, read the paragraph in
the email [2] that has the lines: ".... and limiting sync workers per
subscription theoretically wasn't either (although I don't think it
could happen in practice).".It seems that this check has been added to theoretically limit the
sync workers even though that can't happen because apply worker
ensures that before trying to launch the sync worker. Does this theory
make sense to me? If so, I think we can change the check as: "if
(OidIsValid(relid) && nsyncworkers >=
max_sync_workers_per_subscription)" in launcher.c. This will serve the
purpose of the original code and will solve the issue being discussed
here. I think we can even backpatch this. What do you think?
+1. I also think it's a bug so back-patching makes sense to me.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
On Tue, Apr 19, 2022 at 6:58 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
On Mon, Apr 18, 2022 at 8:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Thu, Apr 14, 2022 at 9:09 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Thu, Apr 14, 2022 at 8:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
The other part of the puzzle is the below check in the code:
/*
* If we reached the sync worker limit per subscription, just exit
* silently as we might get here because of an otherwise harmless race
* condition.
*/
if (nsyncworkers >= max_sync_workers_per_subscription)It is not clear to me why this check is there, if this wouldn't be
there, the user would have got either a WARNING to increase the
max_logical_replication_workers or the apply worker would have been
restarted. Do you have any idea about this?Yeah, I'm also puzzled with this check. It seems that this function
doesn't work well when the apply worker is not running and some
tablesync workers are running. I initially thought that the apply
worker calls to this function as many as tables that needs to be
synced, but it checks the max_sync_workers_per_subscription limit
before calling to logicalrep_worker_launch(). So I'm not really sure
we need this check.I just hope that the original author Petr J. responds to this point. I
have added him to this email. This will help us to find the best
solution for this problem.I did some more investigation for this code. It is added by commit [1]
and the patch that led to this commit is first time posted on -hackers
in email [2]. Now, neither the commit message nor the patch (comments)
gives much idea as to why this part of code is added but I think there
is some hint in the email [2]. In particular, read the paragraph in
the email [2] that has the lines: ".... and limiting sync workers per
subscription theoretically wasn't either (although I don't think it
could happen in practice).".It seems that this check has been added to theoretically limit the
sync workers even though that can't happen because apply worker
ensures that before trying to launch the sync worker. Does this theory
make sense to me? If so, I think we can change the check as: "if
(OidIsValid(relid) && nsyncworkers >=
max_sync_workers_per_subscription)" in launcher.c. This will serve the
purpose of the original code and will solve the issue being discussed
here. I think we can even backpatch this. What do you think?+1. I also think it's a bug so back-patching makes sense to me.
Pushed. Thanks Tomas and Sawada-San.
--
With Regards,
Amit Kapila.
Hi,
On 4/19/22 12:53 AM, Amit Kapila wrote:
On Tue, Apr 19, 2022 at 6:58 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
+1. I also think it's a bug so back-patching makes sense to me.
Pushed. Thanks Tomas and Sawada-San.
This is still on the PG15 open items list[1]https://wiki.postgresql.org/wiki/PostgreSQL_15_Open_Items though marked as with a fix.
Did dd4ab6fd resolve the issue, or does this need more work?
Thanks,
Jonathan
[1]: https://wiki.postgresql.org/wiki/PostgreSQL_15_Open_Items
On 5/10/22 15:55, Jonathan S. Katz wrote:
Hi,
On 4/19/22 12:53 AM, Amit Kapila wrote:
On Tue, Apr 19, 2022 at 6:58 AM Masahiko Sawada
<sawada.mshk@gmail.com> wrote:+1. I also think it's a bug so back-patching makes sense to me.
Pushed. Thanks Tomas and Sawada-San.
This is still on the PG15 open items list[1] though marked as with a fix.
Did dd4ab6fd resolve the issue, or does this need more work?
I believe that's fixed, the buildfarm does not seem to show any relevant
failures in subscriptionCheck since dd4ab6fd got committed.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 5/10/22 3:17 PM, Tomas Vondra wrote:
On 5/10/22 15:55, Jonathan S. Katz wrote:
Hi,
On 4/19/22 12:53 AM, Amit Kapila wrote:
On Tue, Apr 19, 2022 at 6:58 AM Masahiko Sawada
<sawada.mshk@gmail.com> wrote:+1. I also think it's a bug so back-patching makes sense to me.
Pushed. Thanks Tomas and Sawada-San.
This is still on the PG15 open items list[1] though marked as with a fix.
Did dd4ab6fd resolve the issue, or does this need more work?
I believe that's fixed, the buildfarm does not seem to show any relevant
failures in subscriptionCheck since dd4ab6fd got committed.
Great. I'm moving it off of open items.
Thanks for confirming!
Jonathan
On Tue, May 10, 2022 at 7:25 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
On 4/19/22 12:53 AM, Amit Kapila wrote:
On Tue, Apr 19, 2022 at 6:58 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
+1. I also think it's a bug so back-patching makes sense to me.
Pushed. Thanks Tomas and Sawada-San.
This is still on the PG15 open items list[1] though marked as with a fix.
Did dd4ab6fd resolve the issue, or does this need more work?
The commit dd4ab6fd resolved this issue. I didn't notice it after that commit.
--
With Regards,
Amit Kapila.
On Sat, Dec 11, 2021 at 12:24 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Dec-10, Peter Eisentraut wrote:
...
There was no documentation, so I wrote a bit (patch 0001). It only touches
the CREATE PUBLICATION and ALTER PUBLICATION pages at the moment. There was
no mention in the Logical Replication chapter that warranted updating.
Perhaps we should revisit that chapter at the end of the release cycle.Thanks. I hadn't looked at the docs yet, so I'll definitely take this.
Was this documentation ever written?
My assumption was that for PG15 there might be a whole new section
added to Chapter 31 [1]https://www.postgresql.org/docs/15/logical-replication.html for describing "Column Lists" (i.e. the Column
List equivalent of the "Row Filters" section)
------
[1]: https://www.postgresql.org/docs/15/logical-replication.html
Kind Regards,
Peter Smith.
Fujitsu Australia
On Mon, Jul 25, 2022 at 1:27 PM Peter Smith <smithpb2250@gmail.com> wrote:
On Sat, Dec 11, 2021 at 12:24 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Dec-10, Peter Eisentraut wrote:
...
There was no documentation, so I wrote a bit (patch 0001). It only touches
the CREATE PUBLICATION and ALTER PUBLICATION pages at the moment. There was
no mention in the Logical Replication chapter that warranted updating.
Perhaps we should revisit that chapter at the end of the release cycle.Thanks. I hadn't looked at the docs yet, so I'll definitely take this.
Was this documentation ever written?
My assumption was that for PG15 there might be a whole new section
added to Chapter 31 [1] for describing "Column Lists" (i.e. the Column
List equivalent of the "Row Filters" section)
+1. I think it makes sense to give more description about this feature
similar to Row Filters. Note that apart from the main feature commit
[1]: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=923def9a533a7d986acfb524139d8b9e5466d0a5
want to cover that as well.
[1]: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=923def9a533a7d986acfb524139d8b9e5466d0a5
[2]: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=fd0b9dcebda7b931a41ce5c8e86d13f2efd0af2e
--
With Regards,
Amit Kapila.
On Tue, Aug 2, 2022 at 6:57 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Jul 25, 2022 at 1:27 PM Peter Smith <smithpb2250@gmail.com> wrote:
On Sat, Dec 11, 2021 at 12:24 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2021-Dec-10, Peter Eisentraut wrote:
...
There was no documentation, so I wrote a bit (patch 0001). It only touches
the CREATE PUBLICATION and ALTER PUBLICATION pages at the moment. There was
no mention in the Logical Replication chapter that warranted updating.
Perhaps we should revisit that chapter at the end of the release cycle.Thanks. I hadn't looked at the docs yet, so I'll definitely take this.
Was this documentation ever written?
My assumption was that for PG15 there might be a whole new section
added to Chapter 31 [1] for describing "Column Lists" (i.e. the Column
List equivalent of the "Row Filters" section)+1. I think it makes sense to give more description about this feature
similar to Row Filters. Note that apart from the main feature commit
[1], we have prohibited certain cases in commit [2]. So, one might
want to cover that as well.[1] - https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=923def9a533a7d986acfb524139d8b9e5466d0a5
[2] - https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=fd0b9dcebda7b931a41ce5c8e86d13f2efd0af2e
OK. Unless somebody else has already started this work then I can do
this. I will post a draft patch in a few days.
------
Kind Regards,
Peter Smith.
Fujitsu Australia
PSA patch version v1* for a new "Column Lists" pgdocs section
This is just a first draft, but I wanted to post it as-is, with the
hope that I can get some feedback while continuing to work on it.
------
Kind Regards,
Peter Smith.
Fujitsu Australia
Attachments:
v1-0001-Column-List-replica-identity-rules.patchapplication/octet-stream; name=v1-0001-Column-List-replica-identity-rules.patchDownload
From 6b6591ee53a33a891449a94ebff65a4545b09590 Mon Sep 17 00:00:00 2001
From: Peter Smith <peter.b.smith@fujitsu.com>
Date: Mon, 8 Aug 2022 17:22:55 +1000
Subject: [PATCH v1] Column List replica identity rules.
It was not strictly correct to say that a column list must always include
replica identity columns.
This patch modifies the CREATE PUBLICATION "Notes" so the column list replica
identity rules are more similar to those documented for row filters.
---
doc/src/sgml/ref/create_publication.sgml | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 5790d76..2d6de8d 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -90,8 +90,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<para>
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated
- through this publication, including any columns added later. If a column
- list is specified, it must include the replica identity columns.
+ through this publication, including any columns added later.
</para>
<para>
@@ -265,6 +264,15 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
</para>
<para>
+ Any column list must include the <literal>REPLICA IDENTITY</literal> columns
+ in order for <command>UPDATE</command> or <command>DELETE</command>
+ operations to be published. There are no such restrictions if the publication
+ publishes only <command>INSERT</command> operations. Furthermore, if the
+ table uses <literal>REPLICA IDENTITY FULL</literal>, specifying a column
+ list is not allowed.
+ </para>
+
+ <para>
For published partitioned tables, the row filter for each
partition is taken from the published partitioned table if the
publication parameter <literal>publish_via_partition_root</literal> is true,
--
1.8.3.1
v1-0002-Column-Lists-new-pgdocs-section.patchapplication/octet-stream; name=v1-0002-Column-Lists-new-pgdocs-section.patchDownload
From 508a24849f7ce177a8fa2d8da8daa9193570ee59 Mon Sep 17 00:00:00 2001
From: Peter Smith <peter.b.smith@fujitsu.com>
Date: Mon, 8 Aug 2022 18:28:06 +1000
Subject: [PATCH v1] Column Lists - new pgdocs section
Add a new logical replication pgdocs section for "Column Lists"
(analogous to the Row Filters page).
Also update xrefs to that new page from CREATE/ALTER PUBLICATION.
---
doc/src/sgml/logical-replication.sgml | 196 +++++++++++++++++++++++++++++++
doc/src/sgml/ref/alter_publication.sgml | 11 +-
doc/src/sgml/ref/create_publication.sgml | 4 +-
3 files changed, 201 insertions(+), 10 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index bdf1e7b..af0310d 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1089,6 +1089,202 @@ test_sub=# SELECT * FROM child ORDER BY a;
</sect1>
+ <sect1 id="logical-replication-col-lists">
+ <title>Column Lists</title>
+
+ <para>
+ By default, all columns of a published table will be replicated to the
+ appropriate subscribers. The subscriber table must have at least all the
+ columns of the published table. However, if a <firstterm>column list</firstterm>
+ is specified then only the columns named in the list will be replicated.
+ This means the subscriber-side table only needs to have those columns named
+ by the column list. A user might choose to use column lists for behavioral,
+ security or performance reasons.
+ </para>
+
+ <sect2 id="logical-replication-col-list-rules">
+ <title>Column List Rules</title>
+
+ <para>
+ A column list is specified per table following the tablename, and enclosed by
+ parenthesis. The column name order of the list has no effect. See
+ <xref linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
+ When a column list is specified, only the named columns are replicated.
+ If no column list is specified, all columns of the table are replicated
+ through this publication, including any columns added later. This means
+ a column list which names all columns is not quite the same as having no
+ column list at all. For example, if additional columns are added to the
+ table, then (after a <literal>REFRESH PUBLICATION</literal>) if there was
+ a column list only those named columns will continue to be replicated.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-restrictions">
+ <title>Column List Restrictions</title>
+
+ <para>
+ Column list can contain only simple column references. Complex
+ expressions, function calls etc. are not allowed.
+ </para>
+
+ <para>
+ If a publication publishes <command>UPDATE</command> or
+ <command>DELETE</command> operations, any column list must include the table's
+ replica identity columns (see <xref linkend="sql-altertable-replica-identity"/>).
+ If a publication publishes only <command>INSERT</command> operations, then
+ the column list is arbitrary and may omit some replica identity columns.
+ </para>
+
+ <para>
+ Furthermore, if the table uses <literal>REPLICA IDENTITY FULL</literal>,
+ specifying a column list is not allowed.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-partitioned">
+ <title>Partitioned Tables</title>
+
+ <para>
+ For partitioned tables, the publication parameter
+ <literal>publish_via_partition_root</literal> determines which column list
+ is used. If <literal>publish_via_partition_root</literal> is
+ <literal>true</literal>, the root partitioned table's column list is used.
+ Otherwise, if <literal>publish_via_partition_root</literal> is
+ <literal>false</literal> (default), each partition's column list is used.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-combining">
+ <title>Combining Multiple Column Lists</title>
+
+ <warning>
+ <para>
+ It is not supported to have a subscription comprising several publications
+ where the same table has been published with different column lists.
+ This means changing the column lists of the tables being subscribed could
+ cause inconsistency of column lists among publications, in which case
+ the <xref linkend="sql-alterpublication"/> will be successful but later the
+ WalSender on the publisher, or the subscriber may throw an error. In this
+ scenario, the user needs to recreate the subscription after adjusting the
+ column list or drop the problematic publication using
+ <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add it
+ back after adjusting the column list.
+ </para>
+ <para>
+ Background: The main purpose of the column list feature is to allow statically
+ different table shapes on publisher and subscriber or hide sensitive
+ column data. In both cases, it doesn't seem to make sense to combine
+ column lists.
+ </para>
+ </warning>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-examples">
+ <title>Examples</title>
+
+ <para>
+ Create a table <literal>t1</literal> to be used in the following example.
+<programlisting>
+test_pub=# CREATE TABLE t1(id int, a text, b text, c text, d text, e text, PRIMARY KEY(id));
+CREATE TABLE
+test_pub=#
+</programlisting></para>
+
+ <para>
+ Create a publication <literal>p1</literal>. A column list is defined for
+ table <literal>t1</literal> to reduce the number of columns that will be
+ replicated.
+<programlisting>
+test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, a, b, c);
+CREATE PUBLICATION
+test_pub=#
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each publication.
+<programlisting>
+test_pub=# \dRp+
+ Publication p1
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+----------+------------+---------+---------+---------+-----------+----------
+ postgres | f | t | t | t | t | f
+Tables:
+ "public.t1" (id, a, b, c)
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each table.
+<programlisting>
+test_pub=# \d t1
+ Table "public.t1"
+ Column | Type | Collation | Nullable | Default
+--------+---------+-----------+----------+---------
+ id | integer | | not null |
+ a | text | | |
+ b | text | | |
+ c | text | | |
+ d | text | | |
+ e | text | | |
+Indexes:
+ "t1_pkey" PRIMARY KEY, btree (id)
+Publications:
+ "p1" (id, a, b, c)
+</programlisting></para>
+
+ <para>
+ On the subscriber node, create a table <literal>t1</literal> which now
+ only needs a subset of the columns that were on the publisher table
+ <literal>t1</literal>, and also create the subscription <literal>s1</literal>
+ that subscribes to the publication <literal>p1</literal>.
+<programlisting>
+test_sub=# CREATE TABLE t1(id int, a text, b text, c text, PRIMARY KEY(id));
+CREATE TABLE
+test_sub=# CREATE SUBSCRIPTION s1
+test_sub-# CONNECTION 'host=localhost dbname=test_pub application_name=s1'
+test_sub-# PUBLICATION p1;
+CREATE SUBSCRIPTION
+</programlisting></para>
+
+ <para>
+ Insert some rows to table <literal>t1</literal>.
+<programlisting>
+test_pub=# INSERT INTO t1 VALUES(1, 'a-1', 'b-1', 'c-1', 'd-1', 'e-1');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(2, 'a-2', 'b-2', 'c-2', 'd-2', 'e-2');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(3, 'a-3', 'b-3', 'c-3', 'd-3', 'e-3');
+INSERT 0 1
+test_pub=# SELECT * FROM t1 ORDER BY id;
+ id | a | b | c | d | e
+----+-----+-----+-----+-----+-----
+ 1 | a-1 | b-1 | c-1 | d-1 | e-1
+ 2 | a-2 | b-2 | c-2 | d-2 | e-2
+ 3 | a-3 | b-3 | c-3 | d-3 | e-3
+(3 rows)
+</programlisting>
+<programlisting>
+test_sub=# SELECT * FROM t1 ORDER BY id;
+ id | a | b | c
+----+-----+-----+-----
+ 1 | a-1 | b-1 | c-1
+ 2 | a-2 | b-2 | c-2
+ 3 | a-3 | b-3 | c-3
+(3 rows)
+</programlisting></para>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="logical-replication-conflicts">
<title>Conflicts</title>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 3e338f4..a8f283d 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -118,15 +118,8 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
Optionally, a column list can be specified. See <xref
linkend="sql-createpublication"/> for details. Note that a subscription
having several publications in which the same table has been published
- with different column lists is not supported. So, changing the column
- lists of the tables being subscribed could cause inconsistency of column
- lists among publications, in which case <command>ALTER PUBLICATION</command>
- will be successful but later the WalSender on the publisher or the
- subscriber may throw an error. In this scenario, the user needs to
- recreate the subscription after adjusting the column list or drop the
- problematic publication using
- <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add
- it back after adjusting the column list.
+ with different column lists is not supported. See <xref linkend="logical-replication-col-list-combining"/>
+ for details of potential problems when altering column lists.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 2d6de8d..5669d51 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -90,7 +90,9 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<para>
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated
- through this publication, including any columns added later.
+ through this publication, including any columns added later. See
+ <xref linkend="logical-replication-col-lists"/> for details about column
+ lists.
</para>
<para>
--
1.8.3.1
On Mon, Aug 8, 2022 at 2:08 PM Peter Smith <smithpb2250@gmail.com> wrote:
PSA patch version v1* for a new "Column Lists" pgdocs section
This is just a first draft, but I wanted to post it as-is, with the
hope that I can get some feedback while continuing to work on it.
Few comments:
1) Row filters mentions that "It has no effect on TRUNCATE commands.",
the same is not present in case of column filters. We should keep the
changes similarly for consistency.
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -90,8 +90,7 @@ CREATE PUBLICATION <replaceable
class="parameter">name</replaceable>
<para>
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated
- through this publication, including any columns added later. If a column
- list is specified, it must include the replica identity columns.
+ through this publication, including any columns added later.
2) The document says that "if the table uses REPLICA IDENTITY FULL,
specifying a column list is not allowed.":
+ publishes only <command>INSERT</command> operations. Furthermore, if the
+ table uses <literal>REPLICA IDENTITY FULL</literal>, specifying a column
+ list is not allowed.
+ </para>
Did you mean specifying a column list during create publication for
REPLICA IDENTITY FULL table like below scenario:
postgres=# create table t2(c1 int, c2 int, c3 int);
CREATE TABLE
postgres=# alter table t2 replica identity full ;
ALTER TABLE
postgres=# create publication pub1 for table t2(c1,c2);
CREATE PUBLICATION
If so, the document says specifying column list is not allowed, but
creating a publication with column list on replica identity full was
successful.
Regards,
Vignesh
Thanks for the view of v1-0001.
On Wed, Aug 17, 2022 at 3:04 AM vignesh C <vignesh21@gmail.com> wrote:
...
1) Row filters mentions that "It has no effect on TRUNCATE commands.", the same is not present in case of column filters. We should keep the changes similarly for consistency. --- a/doc/src/sgml/ref/create_publication.sgml +++ b/doc/src/sgml/ref/create_publication.sgml @@ -90,8 +90,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable> <para> When a column list is specified, only the named columns are replicated. If no column list is specified, all columns of the table are replicated - through this publication, including any columns added later. If a column - list is specified, it must include the replica identity columns. + through this publication, including any columns added later.
Modified as suggested.
2) The document says that "if the table uses REPLICA IDENTITY FULL, specifying a column list is not allowed.": + publishes only <command>INSERT</command> operations. Furthermore, if the + table uses <literal>REPLICA IDENTITY FULL</literal>, specifying a column + list is not allowed. + </para>Did you mean specifying a column list during create publication for
REPLICA IDENTITY FULL table like below scenario:
postgres=# create table t2(c1 int, c2 int, c3 int);
CREATE TABLE
postgres=# alter table t2 replica identity full ;
ALTER TABLE
postgres=# create publication pub1 for table t2(c1,c2);
CREATE PUBLICATIONIf so, the document says specifying column list is not allowed, but
creating a publication with column list on replica identity full was
successful.
That patch v1-0001 was using the same wording from the github commit
message [1]https://github.com/postgres/postgres/commit/923def9a533a7d986acfb524139d8b9e5466d0a5. I agree it was a bit vague.
In fact the replica identity validation is done at DML execution time
so your example will fail as expected when you attempt to do a UPDATE
operation.
e.g.
test_pub=# update t2 set c2=23 where c1=1;
ERROR: cannot update table "t2"
DETAIL: Column list used by the publication does not cover the
replica identity.
I modified the wording for this part of the docs.
~~~
PSA new set of v2* patches.
------
[1]: https://github.com/postgres/postgres/commit/923def9a533a7d986acfb524139d8b9e5466d0a5
Kind Regards,
Peter Smith
Fujitsu Australia
Attachments:
v2-0001-Column-List-replica-identity-rules.patchapplication/octet-stream; name=v2-0001-Column-List-replica-identity-rules.patchDownload
From ffade3916b0ab7b84e11445b706e898a0ca258e6 Mon Sep 17 00:00:00 2001
From: Peter Smith <peter.b.smith@fujitsu.com>
Date: Mon, 22 Aug 2022 16:14:03 +1000
Subject: [PATCH v2] Column List replica identity rules.
It was not strictly correct to say that a column list must always include
replica identity columns.
This patch modifies the CREATE PUBLICATION "Notes" so the column list replica
identity rules are more similar to those documented for row filters.
---
doc/src/sgml/ref/create_publication.sgml | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 5790d76..31f34f1 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -90,8 +90,8 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<para>
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated
- through this publication, including any columns added later. If a column
- list is specified, it must include the replica identity columns.
+ through this publication, including any columns added later. It has no
+ effect on <literal>TRUNCATE</literal> commands.
</para>
<para>
@@ -265,6 +265,16 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
</para>
<para>
+ Any column list must include the <literal>REPLICA IDENTITY</literal> columns
+ in order for <command>UPDATE</command> or <command>DELETE</command>
+ operations to be published. Furthermore, if the table uses
+ <literal>REPLICA IDENTITY FULL</literal>, specifying a column list is not
+ allowed (it will cause publication errors for <command>UPDATE</command> or
+ <command>DELETE</command> operations). There are no column list restrictions
+ if the publication publishes only <command>INSERT</command> operations.
+ </para>
+
+ <para>
For published partitioned tables, the row filter for each
partition is taken from the published partitioned table if the
publication parameter <literal>publish_via_partition_root</literal> is true,
--
1.8.3.1
v2-0002-Column-Lists-new-pgdocs-section.patchapplication/octet-stream; name=v2-0002-Column-Lists-new-pgdocs-section.patchDownload
From 79b0f4c3dbc86ee86b6c07ac0e74339a4b3914e4 Mon Sep 17 00:00:00 2001
From: Peter Smith <peter.b.smith@fujitsu.com>
Date: Mon, 22 Aug 2022 18:05:32 +1000
Subject: [PATCH v2] Column Lists - new pgdocs section
Add a new logical replication pgdocs section for "Column Lists"
(analogous to the Row Filters page).
Also update xrefs to that new page from CREATE/ALTER PUBLICATION.
---
doc/src/sgml/logical-replication.sgml | 198 +++++++++++++++++++++++++++++++
doc/src/sgml/ref/alter_publication.sgml | 12 +-
doc/src/sgml/ref/create_publication.sgml | 6 +-
3 files changed, 205 insertions(+), 11 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index bdf1e7b..1fe0340 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1089,6 +1089,204 @@ test_sub=# SELECT * FROM child ORDER BY a;
</sect1>
+ <sect1 id="logical-replication-col-lists">
+ <title>Column Lists</title>
+
+ <para>
+ By default, all columns of a published table will be replicated to the
+ appropriate subscribers. The subscriber table must have at least all the
+ columns of the published table. However, if a <firstterm>column list</firstterm>
+ is specified then only the columns named in the list will be replicated.
+ This means the subscriber-side table only needs to have those columns named
+ by the column list. A user might choose to use column lists for behavioral,
+ security or performance reasons.
+ </para>
+
+ <sect2 id="logical-replication-col-list-rules">
+ <title>Column List Rules</title>
+
+ <para>
+ A column list is specified per table following the tablename, and enclosed by
+ parenthesis. See <xref linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
+ When a column list is specified, only the named columns are replicated.
+ The list order is not important. If no column list is specified, all columns
+ of the table are replicated through this publication, including any columns
+ added later. This means a column list which names all columns is not quite
+ the same as having no column list at all. For example, if additional columns
+ are added to the table, then (after a <literal>REFRESH PUBLICATION</literal>)
+ if there was a column list only those named columns will continue to be
+ replicated.
+ </para>
+
+ <para>
+ Column lists have no effect for <literal>TRUNCATE</literal> command.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-restrictions">
+ <title>Column List Restrictions</title>
+
+ <para>
+ Column list can contain only simple column references. Complex
+ expressions, function calls etc. are not allowed.
+ </para>
+
+ <para>
+ If a publication publishes <command>UPDATE</command> or
+ <command>DELETE</command> operations, any column list must include the table's
+ replica identity columns (see <xref linkend="sql-altertable-replica-identity"/>).
+ Furthermore, if the table uses <literal>REPLICA IDENTITY FULL</literal>,
+ specifying a column list is not allowed (it will cause publication errors for
+ <command>UPDATE</command> or <command>DELETE</command> operations).
+ If a publication publishes only <command>INSERT</command> operations, then
+ the column list is arbitrary and may omit some replica identity columns.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-partitioned">
+ <title>Partitioned Tables</title>
+
+ <para>
+ For partitioned tables, the publication parameter
+ <literal>publish_via_partition_root</literal> determines which column list
+ is used. If <literal>publish_via_partition_root</literal> is
+ <literal>true</literal>, the root partitioned table's column list is used.
+ Otherwise, if <literal>publish_via_partition_root</literal> is
+ <literal>false</literal> (default), each partition's column list is used.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-combining">
+ <title>Combining Multiple Column Lists</title>
+
+ <warning>
+ <para>
+ It is not supported to have a subscription comprising several publications
+ where the same table has been published with different column lists.
+ This means changing the column lists of the tables being subscribed could
+ cause inconsistency of column lists among publications, in which case
+ the <xref linkend="sql-alterpublication"/> will be successful but later the
+ WalSender on the publisher, or the subscriber may throw an error. In this
+ scenario, the user needs to recreate the subscription after adjusting the
+ column list or drop the problematic publication using
+ <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add it
+ back after adjusting the column list.
+ </para>
+ <para>
+ Background: The main purpose of the column list feature is to allow statically
+ different table shapes on publisher and subscriber or hide sensitive
+ column data. In both cases, it doesn't seem to make sense to combine
+ column lists.
+ </para>
+ </warning>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-examples">
+ <title>Examples</title>
+
+ <para>
+ Create a table <literal>t1</literal> to be used in the following example.
+<programlisting>
+test_pub=# CREATE TABLE t1(id int, a text, b text, c text, d text, e text, PRIMARY KEY(id));
+CREATE TABLE
+test_pub=#
+</programlisting></para>
+
+ <para>
+ Create a publication <literal>p1</literal>. A column list is defined for
+ table <literal>t1</literal> to reduce the number of columns that will be
+ replicated.
+<programlisting>
+test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, a, b, c);
+CREATE PUBLICATION
+test_pub=#
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each publication.
+<programlisting>
+test_pub=# \dRp+
+ Publication p1
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+----------+------------+---------+---------+---------+-----------+----------
+ postgres | f | t | t | t | t | f
+Tables:
+ "public.t1" (id, a, b, c)
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each table.
+<programlisting>
+test_pub=# \d t1
+ Table "public.t1"
+ Column | Type | Collation | Nullable | Default
+--------+---------+-----------+----------+---------
+ id | integer | | not null |
+ a | text | | |
+ b | text | | |
+ c | text | | |
+ d | text | | |
+ e | text | | |
+Indexes:
+ "t1_pkey" PRIMARY KEY, btree (id)
+Publications:
+ "p1" (id, a, b, c)
+</programlisting></para>
+
+ <para>
+ On the subscriber node, create a table <literal>t1</literal> which now
+ only needs a subset of the columns that were on the publisher table
+ <literal>t1</literal>, and also create the subscription <literal>s1</literal>
+ that subscribes to the publication <literal>p1</literal>.
+<programlisting>
+test_sub=# CREATE TABLE t1(id int, a text, b text, c text, PRIMARY KEY(id));
+CREATE TABLE
+test_sub=# CREATE SUBSCRIPTION s1
+test_sub-# CONNECTION 'host=localhost dbname=test_pub application_name=s1'
+test_sub-# PUBLICATION p1;
+CREATE SUBSCRIPTION
+</programlisting></para>
+
+ <para>
+ Insert some rows to table <literal>t1</literal>.
+<programlisting>
+test_pub=# INSERT INTO t1 VALUES(1, 'a-1', 'b-1', 'c-1', 'd-1', 'e-1');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(2, 'a-2', 'b-2', 'c-2', 'd-2', 'e-2');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(3, 'a-3', 'b-3', 'c-3', 'd-3', 'e-3');
+INSERT 0 1
+test_pub=# SELECT * FROM t1 ORDER BY id;
+ id | a | b | c | d | e
+----+-----+-----+-----+-----+-----
+ 1 | a-1 | b-1 | c-1 | d-1 | e-1
+ 2 | a-2 | b-2 | c-2 | d-2 | e-2
+ 3 | a-3 | b-3 | c-3 | d-3 | e-3
+(3 rows)
+</programlisting>
+<programlisting>
+test_sub=# SELECT * FROM t1 ORDER BY id;
+ id | a | b | c
+----+-----+-----+-----
+ 1 | a-1 | b-1 | c-1
+ 2 | a-2 | b-2 | c-2
+ 3 | a-3 | b-3 | c-3
+(3 rows)
+</programlisting></para>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="logical-replication-conflicts">
<title>Conflicts</title>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 3a74973..d8ed89e 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -118,15 +118,9 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
Optionally, a column list can be specified. See <xref
linkend="sql-createpublication"/> for details. Note that a subscription
having several publications in which the same table has been published
- with different column lists is not supported. So, changing the column
- lists of the tables being subscribed could cause inconsistency of column
- lists among publications, in which case <command>ALTER PUBLICATION</command>
- will be successful but later the walsender on the publisher or the
- subscriber may throw an error. In this scenario, the user needs to
- recreate the subscription after adjusting the column list or drop the
- problematic publication using
- <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add
- it back after adjusting the column list.
+ with different column lists is not supported. See
+ <xref linkend="logical-replication-col-list-combining"/> for details of
+ potential problems when altering column lists.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 31f34f1..4a23223 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -91,8 +91,10 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated
through this publication, including any columns added later. It has no
- effect on <literal>TRUNCATE</literal> commands.
- </para>
+ effect on <literal>TRUNCATE</literal> commands. See
+ <xref linkend="logical-replication-col-lists"/> for details about column
+ lists.
+</para>
<para>
Only persistent base tables and partitioned tables can be part of a
--
1.8.3.1
Op 22-08-2022 om 10:27 schreef Peter Smith:
PSA new set of v2* patches.
Hi,
In the second file a small typo, I think:
"enclosed by parenthesis" should be
"enclosed by parentheses"
thanks,
Erik
On Mon, Aug 22, 2022 at 1:58 PM Peter Smith <smithpb2250@gmail.com> wrote:
Thanks for the view of v1-0001.
On Wed, Aug 17, 2022 at 3:04 AM vignesh C <vignesh21@gmail.com> wrote:
...1) Row filters mentions that "It has no effect on TRUNCATE commands.", the same is not present in case of column filters. We should keep the changes similarly for consistency. --- a/doc/src/sgml/ref/create_publication.sgml +++ b/doc/src/sgml/ref/create_publication.sgml @@ -90,8 +90,7 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable> <para> When a column list is specified, only the named columns are replicated. If no column list is specified, all columns of the table are replicated - through this publication, including any columns added later. If a column - list is specified, it must include the replica identity columns. + through this publication, including any columns added later.Modified as suggested.
2) The document says that "if the table uses REPLICA IDENTITY FULL, specifying a column list is not allowed.": + publishes only <command>INSERT</command> operations. Furthermore, if the + table uses <literal>REPLICA IDENTITY FULL</literal>, specifying a column + list is not allowed. + </para>Did you mean specifying a column list during create publication for
REPLICA IDENTITY FULL table like below scenario:
postgres=# create table t2(c1 int, c2 int, c3 int);
CREATE TABLE
postgres=# alter table t2 replica identity full ;
ALTER TABLE
postgres=# create publication pub1 for table t2(c1,c2);
CREATE PUBLICATIONIf so, the document says specifying column list is not allowed, but
creating a publication with column list on replica identity full was
successful.That patch v1-0001 was using the same wording from the github commit
message [1]. I agree it was a bit vague.In fact the replica identity validation is done at DML execution time
so your example will fail as expected when you attempt to do a UPDATE
operation.e.g.
test_pub=# update t2 set c2=23 where c1=1;
ERROR: cannot update table "t2"
DETAIL: Column list used by the publication does not cover the
replica identity.I modified the wording for this part of the docs.
Few comments:
1) I felt no expressions are allowed in case of column filters. Only
column names can be specified. The second part of the sentence
confuses what is allowed and what is not allowed. Won't it be better
to remove the second sentence and mention that only column names can
be specified.
+ <para>
+ Column list can contain only simple column references. Complex
+ expressions, function calls etc. are not allowed.
+ </para>
2) tablename should be table name.
+ <para>
+ A column list is specified per table following the tablename, and
enclosed by
+ parenthesis. See <xref linkend="sql-createpublication"/> for details.
+ </para>
We have used table name in the same page in other instances like:
a) The row filter is defined per table. Use a WHERE clause after the
table name for each published table that requires data to be filtered
out. The WHERE clause must be enclosed by parentheses.
b) The tables are matched between the publisher and the subscriber
using the fully qualified table name.
3) One small whitespace issue:
git am v2-0001-Column-List-replica-identity-rules.patch
Applying: Column List replica identity rules.
.git/rebase-apply/patch:30: trailing whitespace.
if the publication publishes only <command>INSERT</command> operations.
warning: 1 line adds whitespace errors.
Regards,
Vignesh
On Mon, Aug 22, 2022 at 9:25 PM vignesh C <vignesh21@gmail.com> wrote:
...
Few comments: 1) I felt no expressions are allowed in case of column filters. Only column names can be specified. The second part of the sentence confuses what is allowed and what is not allowed. Won't it be better to remove the second sentence and mention that only column names can be specified. + <para> + Column list can contain only simple column references. Complex + expressions, function calls etc. are not allowed. + </para>
This wording was lifted verbatim from the commit message [1]https://github.com/postgres/postgres/commit/923def9a533a7d986acfb524139d8b9e5466d0a5. But I
see your point that it just seems to be overcomplicating a simple
rule. Modified as suggested.
2) tablename should be table name. + <para> + A column list is specified per table following the tablename, and enclosed by + parenthesis. See <xref linkend="sql-createpublication"/> for details. + </para>We have used table name in the same page in other instances like:
a) The row filter is defined per table. Use a WHERE clause after the
table name for each published table that requires data to be filtered
out. The WHERE clause must be enclosed by parentheses.
b) The tables are matched between the publisher and the subscriber
using the fully qualified table name.
Fixed as suggested.
3) One small whitespace issue:
git am v2-0001-Column-List-replica-identity-rules.patch
Applying: Column List replica identity rules.
.git/rebase-apply/patch:30: trailing whitespace.
if the publication publishes only <command>INSERT</command> operations.
warning: 1 line adds whitespace errors.
Fixed.
~~~
PSA the v3* patch set.
------
[1]: https://github.com/postgres/postgres/commit/923def9a533a7d986acfb524139d8b9e5466d0a5
Kind Regards,
Peter Smith.
Fujitsu Australia
Attachments:
v3-0001-Column-List-replica-identity-rules.patchapplication/octet-stream; name=v3-0001-Column-List-replica-identity-rules.patchDownload
From c96e02554d8632ee07cd9fd0f858b217a63eb983 Mon Sep 17 00:00:00 2001
From: Peter Smith <peter.b.smith@fujitsu.com>
Date: Tue, 23 Aug 2022 11:43:51 +1000
Subject: [PATCH v3] Column List replica identity rules.
It was not strictly correct to say that a column list must always include
replica identity columns.
This patch modifies the CREATE PUBLICATION "Notes" so the column list replica
identity rules are more similar to those documented for row filters.
---
doc/src/sgml/ref/create_publication.sgml | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 5790d76..51f8d38 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -90,8 +90,8 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<para>
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated
- through this publication, including any columns added later. If a column
- list is specified, it must include the replica identity columns.
+ through this publication, including any columns added later. It has no
+ effect on <literal>TRUNCATE</literal> commands.
</para>
<para>
@@ -265,6 +265,16 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
</para>
<para>
+ Any column list must include the <literal>REPLICA IDENTITY</literal> columns
+ in order for <command>UPDATE</command> or <command>DELETE</command>
+ operations to be published. Furthermore, if the table uses
+ <literal>REPLICA IDENTITY FULL</literal>, specifying a column list is not
+ allowed (it will cause publication errors for <command>UPDATE</command> or
+ <command>DELETE</command> operations). There are no column list restrictions
+ if the publication publishes only <command>INSERT</command> operations.
+ </para>
+
+ <para>
For published partitioned tables, the row filter for each
partition is taken from the published partitioned table if the
publication parameter <literal>publish_via_partition_root</literal> is true,
--
1.8.3.1
v3-0002-Column-Lists-new-pgdocs-section.patchapplication/octet-stream; name=v3-0002-Column-Lists-new-pgdocs-section.patchDownload
From 250adb86314ff7318ccc82a1306e6f6cc16d3382 Mon Sep 17 00:00:00 2001
From: Peter Smith <peter.b.smith@fujitsu.com>
Date: Tue, 23 Aug 2022 12:10:32 +1000
Subject: [PATCH v3] Column Lists new pgdocs section
Add a new logical replication pgdocs section for "Column Lists"
(analogous to the Row Filters page).
Also update xrefs to that new page from CREATE/ALTER PUBLICATION.
---
doc/src/sgml/logical-replication.sgml | 197 +++++++++++++++++++++++++++++++
doc/src/sgml/ref/alter_publication.sgml | 12 +-
doc/src/sgml/ref/create_publication.sgml | 6 +-
3 files changed, 204 insertions(+), 11 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index bdf1e7b..5d59fd2 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1089,6 +1089,203 @@ test_sub=# SELECT * FROM child ORDER BY a;
</sect1>
+ <sect1 id="logical-replication-col-lists">
+ <title>Column Lists</title>
+
+ <para>
+ By default, all columns of a published table will be replicated to the
+ appropriate subscribers. The subscriber table must have at least all the
+ columns of the published table. However, if a <firstterm>column list</firstterm>
+ is specified then only the columns named in the list will be replicated.
+ This means the subscriber-side table only needs to have those columns named
+ by the column list. A user might choose to use column lists for behavioral,
+ security or performance reasons.
+ </para>
+
+ <sect2 id="logical-replication-col-list-rules">
+ <title>Column List Rules</title>
+
+ <para>
+ A column list is specified per table following the table name, and enclosed by
+ parentheses. See <xref linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
+ When a column list is specified, only the named columns are replicated.
+ The list order is not important. If no column list is specified, all columns
+ of the table are replicated through this publication, including any columns
+ added later. This means a column list which names all columns is not quite
+ the same as having no column list at all. For example, if additional columns
+ are added to the table, then (after a <literal>REFRESH PUBLICATION</literal>)
+ if there was a column list only those named columns will continue to be
+ replicated.
+ </para>
+
+ <para>
+ Column lists have no effect for <literal>TRUNCATE</literal> command.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-restrictions">
+ <title>Column List Restrictions</title>
+
+ <para>
+ A column list can contain only simple column references.
+ </para>
+
+ <para>
+ If a publication publishes <command>UPDATE</command> or
+ <command>DELETE</command> operations, any column list must include the table's
+ replica identity columns (see <xref linkend="sql-altertable-replica-identity"/>).
+ Furthermore, if the table uses <literal>REPLICA IDENTITY FULL</literal>,
+ specifying a column list is not allowed (it will cause publication errors for
+ <command>UPDATE</command> or <command>DELETE</command> operations).
+ If a publication publishes only <command>INSERT</command> operations, then
+ the column list is arbitrary and may omit some replica identity columns.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-partitioned">
+ <title>Partitioned Tables</title>
+
+ <para>
+ For partitioned tables, the publication parameter
+ <literal>publish_via_partition_root</literal> determines which column list
+ is used. If <literal>publish_via_partition_root</literal> is
+ <literal>true</literal>, the root partitioned table's column list is used.
+ Otherwise, if <literal>publish_via_partition_root</literal> is
+ <literal>false</literal> (default), each partition's column list is used.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-combining">
+ <title>Combining Multiple Column Lists</title>
+
+ <warning>
+ <para>
+ It is not supported to have a subscription comprising several publications
+ where the same table has been published with different column lists.
+ This means changing the column lists of the tables being subscribed could
+ cause inconsistency of column lists among publications, in which case
+ the <xref linkend="sql-alterpublication"/> will be successful but later the
+ WalSender on the publisher, or the subscriber may throw an error. In this
+ scenario, the user needs to recreate the subscription after adjusting the
+ column list or drop the problematic publication using
+ <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add it
+ back after adjusting the column list.
+ </para>
+ <para>
+ Background: The main purpose of the column list feature is to allow statically
+ different table shapes on publisher and subscriber or hide sensitive
+ column data. In both cases, it doesn't seem to make sense to combine
+ column lists.
+ </para>
+ </warning>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-examples">
+ <title>Examples</title>
+
+ <para>
+ Create a table <literal>t1</literal> to be used in the following example.
+<programlisting>
+test_pub=# CREATE TABLE t1(id int, a text, b text, c text, d text, e text, PRIMARY KEY(id));
+CREATE TABLE
+test_pub=#
+</programlisting></para>
+
+ <para>
+ Create a publication <literal>p1</literal>. A column list is defined for
+ table <literal>t1</literal> to reduce the number of columns that will be
+ replicated.
+<programlisting>
+test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, a, b, c);
+CREATE PUBLICATION
+test_pub=#
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each publication.
+<programlisting>
+test_pub=# \dRp+
+ Publication p1
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+----------+------------+---------+---------+---------+-----------+----------
+ postgres | f | t | t | t | t | f
+Tables:
+ "public.t1" (id, a, b, c)
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each table.
+<programlisting>
+test_pub=# \d t1
+ Table "public.t1"
+ Column | Type | Collation | Nullable | Default
+--------+---------+-----------+----------+---------
+ id | integer | | not null |
+ a | text | | |
+ b | text | | |
+ c | text | | |
+ d | text | | |
+ e | text | | |
+Indexes:
+ "t1_pkey" PRIMARY KEY, btree (id)
+Publications:
+ "p1" (id, a, b, c)
+</programlisting></para>
+
+ <para>
+ On the subscriber node, create a table <literal>t1</literal> which now
+ only needs a subset of the columns that were on the publisher table
+ <literal>t1</literal>, and also create the subscription <literal>s1</literal>
+ that subscribes to the publication <literal>p1</literal>.
+<programlisting>
+test_sub=# CREATE TABLE t1(id int, a text, b text, c text, PRIMARY KEY(id));
+CREATE TABLE
+test_sub=# CREATE SUBSCRIPTION s1
+test_sub-# CONNECTION 'host=localhost dbname=test_pub application_name=s1'
+test_sub-# PUBLICATION p1;
+CREATE SUBSCRIPTION
+</programlisting></para>
+
+ <para>
+ Insert some rows to table <literal>t1</literal>.
+<programlisting>
+test_pub=# INSERT INTO t1 VALUES(1, 'a-1', 'b-1', 'c-1', 'd-1', 'e-1');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(2, 'a-2', 'b-2', 'c-2', 'd-2', 'e-2');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(3, 'a-3', 'b-3', 'c-3', 'd-3', 'e-3');
+INSERT 0 1
+test_pub=# SELECT * FROM t1 ORDER BY id;
+ id | a | b | c | d | e
+----+-----+-----+-----+-----+-----
+ 1 | a-1 | b-1 | c-1 | d-1 | e-1
+ 2 | a-2 | b-2 | c-2 | d-2 | e-2
+ 3 | a-3 | b-3 | c-3 | d-3 | e-3
+(3 rows)
+</programlisting>
+<programlisting>
+test_sub=# SELECT * FROM t1 ORDER BY id;
+ id | a | b | c
+----+-----+-----+-----
+ 1 | a-1 | b-1 | c-1
+ 2 | a-2 | b-2 | c-2
+ 3 | a-3 | b-3 | c-3
+(3 rows)
+</programlisting></para>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="logical-replication-conflicts">
<title>Conflicts</title>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 3a74973..d8ed89e 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -118,15 +118,9 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
Optionally, a column list can be specified. See <xref
linkend="sql-createpublication"/> for details. Note that a subscription
having several publications in which the same table has been published
- with different column lists is not supported. So, changing the column
- lists of the tables being subscribed could cause inconsistency of column
- lists among publications, in which case <command>ALTER PUBLICATION</command>
- will be successful but later the walsender on the publisher or the
- subscriber may throw an error. In this scenario, the user needs to
- recreate the subscription after adjusting the column list or drop the
- problematic publication using
- <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add
- it back after adjusting the column list.
+ with different column lists is not supported. See
+ <xref linkend="logical-replication-col-list-combining"/> for details of
+ potential problems when altering column lists.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 51f8d38..8492147 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -91,8 +91,10 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated
through this publication, including any columns added later. It has no
- effect on <literal>TRUNCATE</literal> commands.
- </para>
+ effect on <literal>TRUNCATE</literal> commands. See
+ <xref linkend="logical-replication-col-lists"/> for details about column
+ lists.
+</para>
<para>
Only persistent base tables and partitioned tables can be part of a
--
1.8.3.1
On Mon, Aug 22, 2022 at 7:11 PM Erik Rijkers <er@xs4all.nl> wrote:
Op 22-08-2022 om 10:27 schreef Peter Smith:
PSA new set of v2* patches.
Hi,
In the second file a small typo, I think:
"enclosed by parenthesis" should be
"enclosed by parentheses"
Thanks for your feedback.
Fixed in the v3* patches [1]/messages/by-id/CAHut+PtHgQbFs9DDeOoqqQLZmMBD8FQPK2WOXJpR1nyDQy8AGA@mail.gmail.com.
------
[1]: /messages/by-id/CAHut+PtHgQbFs9DDeOoqqQLZmMBD8FQPK2WOXJpR1nyDQy8AGA@mail.gmail.com
Kind Regards,
Peter Smith.
Fujitsu Australia
On Tue, Aug 23, 2022 at 7:52 AM Peter Smith <smithpb2250@gmail.com> wrote:
On Mon, Aug 22, 2022 at 9:25 PM vignesh C <vignesh21@gmail.com> wrote:
...
Few comments: 1) I felt no expressions are allowed in case of column filters. Only column names can be specified. The second part of the sentence confuses what is allowed and what is not allowed. Won't it be better to remove the second sentence and mention that only column names can be specified. + <para> + Column list can contain only simple column references. Complex + expressions, function calls etc. are not allowed. + </para>This wording was lifted verbatim from the commit message [1]. But I
see your point that it just seems to be overcomplicating a simple
rule. Modified as suggested.2) tablename should be table name. + <para> + A column list is specified per table following the tablename, and enclosed by + parenthesis. See <xref linkend="sql-createpublication"/> for details. + </para>We have used table name in the same page in other instances like:
a) The row filter is defined per table. Use a WHERE clause after the
table name for each published table that requires data to be filtered
out. The WHERE clause must be enclosed by parentheses.
b) The tables are matched between the publisher and the subscriber
using the fully qualified table name.Fixed as suggested.
3) One small whitespace issue:
git am v2-0001-Column-List-replica-identity-rules.patch
Applying: Column List replica identity rules.
.git/rebase-apply/patch:30: trailing whitespace.
if the publication publishes only <command>INSERT</command> operations.
warning: 1 line adds whitespace errors.Fixed.
~~~
PSA the v3* patch set.
Thanks for the updated patch.
Few comments:
1) We can shuffle the columns in publisher table and subscriber to
show that the order of the column does not matter
+ <para>
+ Create a publication <literal>p1</literal>. A column list is defined for
+ table <literal>t1</literal> to reduce the number of columns that will be
+ replicated.
+<programlisting>
+test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, a, b, c);
+CREATE PUBLICATION
+test_pub=#
+</programlisting></para>
2) We can try to keep the line content to less than 80 chars
+ <para>
+ A column list is specified per table following the tablename, and
enclosed by
+ parenthesis. See <xref linkend="sql-createpublication"/> for details.
+ </para>
3) tablename should be table name like it is used in other places
+ <para>
+ A column list is specified per table following the tablename, and
enclosed by
+ parenthesis. See <xref linkend="sql-createpublication"/> for details.
+ </para>
4a) In the below, you could include mentioning "Only the column list
data of publication <literal>p1</literal> are replicated."
+ <para>
+ Insert some rows to table <literal>t1</literal>.
+<programlisting>
+test_pub=# INSERT INTO t1 VALUES(1, 'a-1', 'b-1', 'c-1', 'd-1', 'e-1');
+INSERT 0 1
4b) In the above, we could mention that the insert should be done on
the "publisher side" as the previous statements are executed on the
subscriber side.
Regards,
Vignesh
On Thu, Aug 25, 2022 at 7:38 PM vignesh C <vignesh21@gmail.com> wrote:
...
PSA the v3* patch set.
Thanks for the updated patch. Few comments: 1) We can shuffle the columns in publisher table and subscriber to show that the order of the column does not matter + <para> + Create a publication <literal>p1</literal>. A column list is defined for + table <literal>t1</literal> to reduce the number of columns that will be + replicated. +<programlisting> +test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, a, b, c); +CREATE PUBLICATION +test_pub=# +</programlisting></para>
OK. I made the following changes to the example.
- now the subscriber table defines cols in a different order than that
of the publisher table
- now the publisher column list defines col names in a different order
than that of the table
- now the column list avoids using only adjacent column names
2) We can try to keep the line content to less than 80 chars + <para> + A column list is specified per table following the tablename, and enclosed by + parenthesis. See <xref linkend="sql-createpublication"/> for details. + </para>
OK. Modified to use < 80 chars
3) tablename should be table name like it is used in other places + <para> + A column list is specified per table following the tablename, and enclosed by + parenthesis. See <xref linkend="sql-createpublication"/> for details. + </para>
Sorry, I don't see this problem. AFAIK this same issue was already
fixed in the v3* patches. Notice in the cited fragment that
'parenthesis' is misspelt but that was also fixed in v3. Maybe you are
looking at an old patch file (??)
4a) In the below, you could include mentioning "Only the column list data of publication <literal>p1</literal> are replicated." + <para> + Insert some rows to table <literal>t1</literal>. +<programlisting> +test_pub=# INSERT INTO t1 VALUES(1, 'a-1', 'b-1', 'c-1', 'd-1', 'e-1'); +INSERT 0 1
OK. Modified to say this.
4b) In the above, we could mention that the insert should be done on
the "publisher side" as the previous statements are executed on the
subscriber side.
OK. Modified to say this.
~~~
Thanks for the feedback.
PSA patch set v4* where all of the above comments are now addressed.
------
Kind Regards,
Peter Smith.
Fujitsu Australia
Attachments:
v4-0001-Column-List-replica-identity-rules.patchapplication/octet-stream; name=v4-0001-Column-List-replica-identity-rules.patchDownload
From d7d3f099169178d8a1381660f0f764e27e699eeb Mon Sep 17 00:00:00 2001
From: Peter Smith <peter.b.smith@fujitsu.com>
Date: Fri, 26 Aug 2022 09:42:45 +1000
Subject: [PATCH v4] Column List replica identity rules.
It was not strictly correct to say that a column list must always include
replica identity columns.
This patch modifies the CREATE PUBLICATION "Notes" so the column list replica
identity rules are more similar to those documented for row filters.
---
doc/src/sgml/ref/create_publication.sgml | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 5790d76..51f8d38 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -90,8 +90,8 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<para>
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated
- through this publication, including any columns added later. If a column
- list is specified, it must include the replica identity columns.
+ through this publication, including any columns added later. It has no
+ effect on <literal>TRUNCATE</literal> commands.
</para>
<para>
@@ -265,6 +265,16 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
</para>
<para>
+ Any column list must include the <literal>REPLICA IDENTITY</literal> columns
+ in order for <command>UPDATE</command> or <command>DELETE</command>
+ operations to be published. Furthermore, if the table uses
+ <literal>REPLICA IDENTITY FULL</literal>, specifying a column list is not
+ allowed (it will cause publication errors for <command>UPDATE</command> or
+ <command>DELETE</command> operations). There are no column list restrictions
+ if the publication publishes only <command>INSERT</command> operations.
+ </para>
+
+ <para>
For published partitioned tables, the row filter for each
partition is taken from the published partitioned table if the
publication parameter <literal>publish_via_partition_root</literal> is true,
--
1.8.3.1
v4-0002-Column-List-new-pgdocs-section.patchapplication/octet-stream; name=v4-0002-Column-List-new-pgdocs-section.patchDownload
From 98725ca2041646bed5124d4fa96497fe1d9f84b9 Mon Sep 17 00:00:00 2001
From: Peter Smith <peter.b.smith@fujitsu.com>
Date: Fri, 26 Aug 2022 11:52:53 +1000
Subject: [PATCH v4] Column List new pgdocs section
Add a new logical replication pgdocs section for "Column Lists"
(analogous to the Row Filters page).
Also update xrefs to that new page from CREATE/ALTER PUBLICATION.
---
doc/src/sgml/logical-replication.sgml | 205 +++++++++++++++++++++++++++++++
doc/src/sgml/ref/alter_publication.sgml | 12 +-
doc/src/sgml/ref/create_publication.sgml | 6 +-
3 files changed, 212 insertions(+), 11 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index bdf1e7b..b097f3b 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1089,6 +1089,211 @@ test_sub=# SELECT * FROM child ORDER BY a;
</sect1>
+ <sect1 id="logical-replication-col-lists">
+ <title>Column Lists</title>
+
+ <para>
+ By default, all columns of a published table will be replicated to the
+ appropriate subscribers. The subscriber table must have at least all the
+ columns of the published table. However, if a
+ <firstterm>column list</firstterm> is specified then only the columns named
+ in the list will be replicated. This means the subscriber-side table only
+ needs to have those columns named by the column list. A user might choose to
+ use column lists for behavioral, security or performance reasons.
+ </para>
+
+ <sect2 id="logical-replication-col-list-rules">
+ <title>Column List Rules</title>
+
+ <para>
+ A column list is specified per table following the table name, and enclosed
+ by parentheses. See <xref linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
+ When a column list is specified, only the named columns are replicated.
+ The list order is not important. If no column list is specified, all
+ columns of the table are replicated through this publication, including any
+ columns
+ added later. This means a column list which names all columns is not quite
+ the same as having no column list at all. For example, if additional
+ columns are added to the table, then (after a
+ <literal>REFRESH PUBLICATION</literal>) if there was a column list only
+ those named columns will continue to be replicated.
+ </para>
+
+ <para>
+ Column lists have no effect for <literal>TRUNCATE</literal> command.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-restrictions">
+ <title>Column List Restrictions</title>
+
+ <para>
+ A column list can contain only simple column references.
+ </para>
+
+ <para>
+ If a publication publishes <command>UPDATE</command> or
+ <command>DELETE</command> operations, any column list must include the
+ table's replica identity columns (see
+ <xref linkend="sql-altertable-replica-identity"/>).
+ Furthermore, if the table uses <literal>REPLICA IDENTITY FULL</literal>,
+ specifying a column list is not allowed (it will cause publication errors
+ for <command>UPDATE</command> or <command>DELETE</command> operations).
+ If a publication publishes only <command>INSERT</command> operations, then
+ the column list is arbitrary and may omit some replica identity columns.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-partitioned">
+ <title>Partitioned Tables</title>
+
+ <para>
+ For partitioned tables, the publication parameter
+ <literal>publish_via_partition_root</literal> determines which column list
+ is used. If <literal>publish_via_partition_root</literal> is
+ <literal>true</literal>, the root partitioned table's column list is used.
+ Otherwise, if <literal>publish_via_partition_root</literal> is
+ <literal>false</literal> (default), each partition's column list is used.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-combining">
+ <title>Combining Multiple Column Lists</title>
+
+ <warning>
+ <para>
+ It is not supported to have a subscription comprising several publications
+ where the same table has been published with different column lists.
+ This means changing the column lists of the tables being subscribed could
+ cause inconsistency of column lists among publications, in which case
+ the <xref linkend="sql-alterpublication"/> will be successful but later
+ the WalSender on the publisher, or the subscriber may throw an error. In
+ this scenario, the user needs to recreate the subscription after adjusting
+ the column list or drop the problematic publication using
+ <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add it
+ back after adjusting the column list.
+ </para>
+ <para>
+ Background: The main purpose of the column list feature is to allow
+ statically different table shapes on publisher and subscriber, or hide
+ sensitive column data. In both cases, it doesn't seem to make sense to
+ combine column lists.
+ </para>
+ </warning>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-examples">
+ <title>Examples</title>
+
+ <para>
+ Create a table <literal>t1</literal> to be used in the following example.
+<programlisting>
+test_pub=# CREATE TABLE t1(id int, a text, b text, c text, d text, e text, PRIMARY KEY(id));
+CREATE TABLE
+test_pub=#
+</programlisting></para>
+
+ <para>
+ Create a publication <literal>p1</literal>. A column list is defined for
+ table <literal>t1</literal> to reduce the number of columns that will be
+ replicated. Notice that the order of column names in the column list does
+ not matter.
+<programlisting>
+test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, b, a, d);
+CREATE PUBLICATION
+test_pub=#
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each publication.
+<programlisting>
+test_pub=# \dRp+
+ Publication p1
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+----------+------------+---------+---------+---------+-----------+----------
+ postgres | f | t | t | t | t | f
+Tables:
+ "public.t1" (id, a, b, d)
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each table.
+<programlisting>
+test_pub=# \d t1
+ Table "public.t1"
+ Column | Type | Collation | Nullable | Default
+--------+---------+-----------+----------+---------
+ id | integer | | not null |
+ a | text | | |
+ b | text | | |
+ c | text | | |
+ d | text | | |
+ e | text | | |
+Indexes:
+ "t1_pkey" PRIMARY KEY, btree (id)
+Publications:
+ "p1" (id, a, b, d)
+</programlisting></para>
+
+ <para>
+ On the subscriber node, create a table <literal>t1</literal> which now
+ only needs a subset of the columns that were on the publisher table
+ <literal>t1</literal>, and also create the subscription
+ <literal>s1</literal> that subscribes to the publication
+ <literal>p1</literal>.
+<programlisting>
+test_sub=# CREATE TABLE t1(id int, b text, a text, d text, PRIMARY KEY(id));
+CREATE TABLE
+test_sub=# CREATE SUBSCRIPTION s1
+test_sub-# CONNECTION 'host=localhost dbname=test_pub application_name=s1'
+test_sub-# PUBLICATION p1;
+CREATE SUBSCRIPTION
+</programlisting></para>
+
+ <para>
+ On the publisher node, insert some rows to table <literal>t1</literal>.
+<programlisting>
+test_pub=# INSERT INTO t1 VALUES(1, 'a-1', 'b-1', 'c-1', 'd-1', 'e-1');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(2, 'a-2', 'b-2', 'c-2', 'd-2', 'e-2');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(3, 'a-3', 'b-3', 'c-3', 'd-3', 'e-3');
+INSERT 0 1
+test_pub=# SELECT * FROM t1 ORDER BY id;
+ id | a | b | c | d | e
+----+-----+-----+-----+-----+-----
+ 1 | a-1 | b-1 | c-1 | d-1 | e-1
+ 2 | a-2 | b-2 | c-2 | d-2 | e-2
+ 3 | a-3 | b-3 | c-3 | d-3 | e-3
+(3 rows)
+</programlisting></para>
+
+ <para>
+ Only data from the column list of publication <literal>p1</literal> is
+ replicated.
+<programlisting>
+test_sub=# SELECT * FROM t1 ORDER BY id;
+ id | b | a | d
+----+-----+-----+-----
+ 1 | b-1 | a-1 | d-1
+ 2 | b-2 | a-2 | d-2
+ 3 | b-3 | a-3 | d-3
+(3 rows)
+</programlisting></para>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="logical-replication-conflicts">
<title>Conflicts</title>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 3a74973..d8ed89e 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -118,15 +118,9 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
Optionally, a column list can be specified. See <xref
linkend="sql-createpublication"/> for details. Note that a subscription
having several publications in which the same table has been published
- with different column lists is not supported. So, changing the column
- lists of the tables being subscribed could cause inconsistency of column
- lists among publications, in which case <command>ALTER PUBLICATION</command>
- will be successful but later the walsender on the publisher or the
- subscriber may throw an error. In this scenario, the user needs to
- recreate the subscription after adjusting the column list or drop the
- problematic publication using
- <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add
- it back after adjusting the column list.
+ with different column lists is not supported. See
+ <xref linkend="logical-replication-col-list-combining"/> for details of
+ potential problems when altering column lists.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 51f8d38..8492147 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -91,8 +91,10 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated
through this publication, including any columns added later. It has no
- effect on <literal>TRUNCATE</literal> commands.
- </para>
+ effect on <literal>TRUNCATE</literal> commands. See
+ <xref linkend="logical-replication-col-lists"/> for details about column
+ lists.
+</para>
<para>
Only persistent base tables and partitioned tables can be part of a
--
1.8.3.1
On Fri, Aug 26, 2022 at 7:33 AM Peter Smith <smithpb2250@gmail.com> wrote:
Few comments on both the patches:
v4-0001*
=========
1.
Furthermore, if the table uses
+ <literal>REPLICA IDENTITY FULL</literal>, specifying a column list is not
+ allowed (it will cause publication errors for <command>UPDATE</command> or
+ <command>DELETE</command> operations).
This line sounds a bit unclear to me. From this like it appears that
the following operation is not allowed:
postgres=# create table t1(c1 int, c2 int, c3 int);
CREATE TABLE
postgres=# Alter Table t1 replica identity full;
ALTER TABLE
postgres=# create publication pub1 for table t1(c1);
CREATE PUBLICATION
However, what is not allowed is the following:
postgres=# delete from t1;
ERROR: cannot delete from table "t1"
DETAIL: Column list used by the publication does not cover the
replica identity.
I am not sure if we really need this line but if so then please try to
make it more clear. I think the similar text is present in 0002 patch
which should be modified accordingly.
V4-0002*
=========
2.
However, if a
+ <firstterm>column list</firstterm> is specified then only the columns named
+ in the list will be replicated. This means the subscriber-side table only
+ needs to have those columns named by the column list. A user might choose to
+ use column lists for behavioral, security or performance reasons.
+ </para>
+
+ <sect2 id="logical-replication-col-list-rules">
+ <title>Column List Rules</title>
+
+ <para>
+ A column list is specified per table following the table name, and enclosed
+ by parentheses. See <xref linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
+ When a column list is specified, only the named columns are replicated.
+ The list order is not important.
It seems like "When a column list is specified, only the named columns
are replicated." is almost a duplicate of the line in the first para.
So, I think we can remove it. And if we do so then the second line
could be changed to something like: "While specifying column list, the
order of columns is not important."
3. It seems information about initial table synchronization is
missing. We copy only columns specified in the column list. Also, it
would be good to add a Note similar to Row Filter to indicate that
this list won't be used by pre-15 publishers.
--
With Regards,
Amit Kapila.
On Thu, Sep 1, 2022 at 7:53 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Aug 26, 2022 at 7:33 AM Peter Smith <smithpb2250@gmail.com> wrote:
Few comments on both the patches: v4-0001* ========= 1. Furthermore, if the table uses + <literal>REPLICA IDENTITY FULL</literal>, specifying a column list is not + allowed (it will cause publication errors for <command>UPDATE</command> or + <command>DELETE</command> operations).This line sounds a bit unclear to me. From this like it appears that
the following operation is not allowed:postgres=# create table t1(c1 int, c2 int, c3 int);
CREATE TABLE
postgres=# Alter Table t1 replica identity full;
ALTER TABLE
postgres=# create publication pub1 for table t1(c1);
CREATE PUBLICATIONHowever, what is not allowed is the following:
postgres=# delete from t1;
ERROR: cannot delete from table "t1"
DETAIL: Column list used by the publication does not cover the
replica identity.I am not sure if we really need this line but if so then please try to
make it more clear. I think the similar text is present in 0002 patch
which should be modified accordingly.
The "Furthermore…" sentence came from the commit message [1]https://github.com/postgres/postgres/commit/923def9a533a7d986acfb524139d8b9e5466d0a5. But I
agree it seems redundant/ambiguous, so I have removed it (and removed
the same in patch 0002).
V4-0002* ========= 2. However, if a + <firstterm>column list</firstterm> is specified then only the columns named + in the list will be replicated. This means the subscriber-side table only + needs to have those columns named by the column list. A user might choose to + use column lists for behavioral, security or performance reasons. + </para> + + <sect2 id="logical-replication-col-list-rules"> + <title>Column List Rules</title> + + <para> + A column list is specified per table following the table name, and enclosed + by parentheses. See <xref linkend="sql-createpublication"/> for details. + </para> + + <para> + When a column list is specified, only the named columns are replicated. + The list order is not important.It seems like "When a column list is specified, only the named columns
are replicated." is almost a duplicate of the line in the first para.
So, I think we can remove it. And if we do so then the second line
could be changed to something like: "While specifying column list, the
order of columns is not important."
Modified as suggested.
3. It seems information about initial table synchronization is
missing. We copy only columns specified in the column list. Also, it
would be good to add a Note similar to Row Filter to indicate that
this list won't be used by pre-15 publishers.
Done as suggested. Added a new "Initial Data Synchronization" section
with content similar to that of the Row Filters section.
~~~
Thanks for your review comments.
PSA v5* patches where all the above have been addressed.
------
[1]: https://github.com/postgres/postgres/commit/923def9a533a7d986acfb524139d8b9e5466d0a5
Kind Regards,
Peter Smith.
Fujitsu Australia
Attachments:
v5-0002-Column-List-new-pgdocs-section.patchapplication/octet-stream; name=v5-0002-Column-List-new-pgdocs-section.patchDownload
From 8013d5b4dda00ed66162904c0b8b50e817fc78c5 Mon Sep 17 00:00:00 2001
From: Peter Smith <peter.b.smith@fujitsu.com>
Date: Fri, 2 Sep 2022 13:00:30 +1000
Subject: [PATCH v5] Column List new pgdocs section
Add a new logical replication pgdocs section for "Column Lists"
(analogous to the Row Filters page).
Also update xrefs to that new page from CREATE/ALTER PUBLICATION.
---
doc/src/sgml/logical-replication.sgml | 219 +++++++++++++++++++++++++++++++
doc/src/sgml/ref/alter_publication.sgml | 12 +-
doc/src/sgml/ref/create_publication.sgml | 6 +-
3 files changed, 226 insertions(+), 11 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index bdf1e7b..c9a8056 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1089,6 +1089,225 @@ test_sub=# SELECT * FROM child ORDER BY a;
</sect1>
+ <sect1 id="logical-replication-col-lists">
+ <title>Column Lists</title>
+
+ <para>
+ By default, all columns of a published table will be replicated to the
+ appropriate subscribers. The subscriber table must have at least all the
+ columns of the published table. However, if a
+ <firstterm>column list</firstterm> is specified then only the columns named
+ in the list will be replicated. This means the subscriber-side table only
+ needs to have those columns named by the column list. A user might choose to
+ use column lists for behavioral, security or performance reasons.
+ </para>
+
+ <sect2 id="logical-replication-col-list-rules">
+ <title>Column List Rules</title>
+
+ <para>
+ A column list is specified per table following the table name, and enclosed
+ by parentheses. See <xref linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
+ When specifying a column list, the order of columns is not important. If no
+ column list is specified, all columns of the table are replicated through
+ this publication, including any columns added later. This means a column
+ list which names all columns is not quite the same as having no column list
+ at all. For example, if additional columns are added to the table, then
+ (after a <literal>REFRESH PUBLICATION</literal>) if there was a column list
+ only those named columns will continue to be replicated.
+ </para>
+
+ <para>
+ Column lists have no effect for <literal>TRUNCATE</literal> command.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-restrictions">
+ <title>Column List Restrictions</title>
+
+ <para>
+ A column list can contain only simple column references.
+ </para>
+
+ <para>
+ If a publication publishes <command>UPDATE</command> or
+ <command>DELETE</command> operations, any column list must include the
+ table's replica identity columns (see
+ <xref linkend="sql-altertable-replica-identity"/>).
+ If a publication publishes only <command>INSERT</command> operations, then
+ the column list is arbitrary and may omit some replica identity columns.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-partitioned">
+ <title>Partitioned Tables</title>
+
+ <para>
+ For partitioned tables, the publication parameter
+ <literal>publish_via_partition_root</literal> determines which column list
+ is used. If <literal>publish_via_partition_root</literal> is
+ <literal>true</literal>, the root partitioned table's column list is used.
+ Otherwise, if <literal>publish_via_partition_root</literal> is
+ <literal>false</literal> (default), each partition's column list is used.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-initial-data-sync">
+ <title>Initial Data Synchronization</title>
+
+ <para>
+ If the subscription requires copying pre-existing table data and a
+ publication specifies a column list, only data from those columns will be
+ copied.
+ </para>
+
+ <note>
+ <para>
+ If the subscriber is in a release prior to 15, copy pre-existing data
+ doesn't use column lists even if they are defined in the publication.
+ This is because old releases can only copy the entire table data.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-combining">
+ <title>Combining Multiple Column Lists</title>
+
+ <warning>
+ <para>
+ It is not supported to have a subscription comprising several publications
+ where the same table has been published with different column lists.
+ This means changing the column lists of the tables being subscribed could
+ cause inconsistency of column lists among publications, in which case
+ the <xref linkend="sql-alterpublication"/> will be successful but later
+ the WalSender on the publisher, or the subscriber may throw an error. In
+ this scenario, the user needs to recreate the subscription after adjusting
+ the column list or drop the problematic publication using
+ <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add it
+ back after adjusting the column list.
+ </para>
+ <para>
+ Background: The main purpose of the column list feature is to allow
+ statically different table shapes on publisher and subscriber, or hide
+ sensitive column data. In both cases, it doesn't seem to make sense to
+ combine column lists.
+ </para>
+ </warning>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-examples">
+ <title>Examples</title>
+
+ <para>
+ Create a table <literal>t1</literal> to be used in the following example.
+<programlisting>
+test_pub=# CREATE TABLE t1(id int, a text, b text, c text, d text, e text, PRIMARY KEY(id));
+CREATE TABLE
+test_pub=#
+</programlisting></para>
+
+ <para>
+ Create a publication <literal>p1</literal>. A column list is defined for
+ table <literal>t1</literal> to reduce the number of columns that will be
+ replicated. Notice that the order of column names in the column list does
+ not matter.
+<programlisting>
+test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, b, a, d);
+CREATE PUBLICATION
+test_pub=#
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each publication.
+<programlisting>
+test_pub=# \dRp+
+ Publication p1
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+----------+------------+---------+---------+---------+-----------+----------
+ postgres | f | t | t | t | t | f
+Tables:
+ "public.t1" (id, a, b, d)
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each table.
+<programlisting>
+test_pub=# \d t1
+ Table "public.t1"
+ Column | Type | Collation | Nullable | Default
+--------+---------+-----------+----------+---------
+ id | integer | | not null |
+ a | text | | |
+ b | text | | |
+ c | text | | |
+ d | text | | |
+ e | text | | |
+Indexes:
+ "t1_pkey" PRIMARY KEY, btree (id)
+Publications:
+ "p1" (id, a, b, d)
+</programlisting></para>
+
+ <para>
+ On the subscriber node, create a table <literal>t1</literal> which now
+ only needs a subset of the columns that were on the publisher table
+ <literal>t1</literal>, and also create the subscription
+ <literal>s1</literal> that subscribes to the publication
+ <literal>p1</literal>.
+<programlisting>
+test_sub=# CREATE TABLE t1(id int, b text, a text, d text, PRIMARY KEY(id));
+CREATE TABLE
+test_sub=# CREATE SUBSCRIPTION s1
+test_sub-# CONNECTION 'host=localhost dbname=test_pub application_name=s1'
+test_sub-# PUBLICATION p1;
+CREATE SUBSCRIPTION
+</programlisting></para>
+
+ <para>
+ On the publisher node, insert some rows to table <literal>t1</literal>.
+<programlisting>
+test_pub=# INSERT INTO t1 VALUES(1, 'a-1', 'b-1', 'c-1', 'd-1', 'e-1');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(2, 'a-2', 'b-2', 'c-2', 'd-2', 'e-2');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(3, 'a-3', 'b-3', 'c-3', 'd-3', 'e-3');
+INSERT 0 1
+test_pub=# SELECT * FROM t1 ORDER BY id;
+ id | a | b | c | d | e
+----+-----+-----+-----+-----+-----
+ 1 | a-1 | b-1 | c-1 | d-1 | e-1
+ 2 | a-2 | b-2 | c-2 | d-2 | e-2
+ 3 | a-3 | b-3 | c-3 | d-3 | e-3
+(3 rows)
+</programlisting></para>
+
+ <para>
+ Only data from the column list of publication <literal>p1</literal> is
+ replicated.
+<programlisting>
+test_sub=# SELECT * FROM t1 ORDER BY id;
+ id | b | a | d
+----+-----+-----+-----
+ 1 | b-1 | a-1 | d-1
+ 2 | b-2 | a-2 | d-2
+ 3 | b-3 | a-3 | d-3
+(3 rows)
+</programlisting></para>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="logical-replication-conflicts">
<title>Conflicts</title>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 3a74973..d8ed89e 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -118,15 +118,9 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
Optionally, a column list can be specified. See <xref
linkend="sql-createpublication"/> for details. Note that a subscription
having several publications in which the same table has been published
- with different column lists is not supported. So, changing the column
- lists of the tables being subscribed could cause inconsistency of column
- lists among publications, in which case <command>ALTER PUBLICATION</command>
- will be successful but later the walsender on the publisher or the
- subscriber may throw an error. In this scenario, the user needs to
- recreate the subscription after adjusting the column list or drop the
- problematic publication using
- <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add
- it back after adjusting the column list.
+ with different column lists is not supported. See
+ <xref linkend="logical-replication-col-list-combining"/> for details of
+ potential problems when altering column lists.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index b0d59ef..f616418 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -91,8 +91,10 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated
through this publication, including any columns added later. It has no
- effect on <literal>TRUNCATE</literal> commands.
- </para>
+ effect on <literal>TRUNCATE</literal> commands. See
+ <xref linkend="logical-replication-col-lists"/> for details about column
+ lists.
+</para>
<para>
Only persistent base tables and partitioned tables can be part of a
--
1.8.3.1
v5-0001-Column-List-replica-identity-rules.patchapplication/octet-stream; name=v5-0001-Column-List-replica-identity-rules.patchDownload
From da233dc6090b2e2e0ccdfefef7a5d6de43d61b4a Mon Sep 17 00:00:00 2001
From: Peter Smith <peter.b.smith@fujitsu.com>
Date: Fri, 2 Sep 2022 12:09:48 +1000
Subject: [PATCH v5] Column List replica identity rules.
It was not strictly correct to say that a column list must always include
replica identity columns.
This patch modifies the CREATE PUBLICATION "Notes" so the column list replica
identity rules are more similar to those documented for row filters.
---
doc/src/sgml/ref/create_publication.sgml | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 5790d76..b0d59ef 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -90,8 +90,8 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<para>
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated
- through this publication, including any columns added later. If a column
- list is specified, it must include the replica identity columns.
+ through this publication, including any columns added later. It has no
+ effect on <literal>TRUNCATE</literal> commands.
</para>
<para>
@@ -253,6 +253,13 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
</para>
<para>
+ Any column list must include the <literal>REPLICA IDENTITY</literal> columns
+ in order for <command>UPDATE</command> or <command>DELETE</command>
+ operations to be published. There are no column list restrictions if the
+ publication publishes only <command>INSERT</command> operations.
+ </para>
+
+ <para>
A row filter expression (i.e., the <literal>WHERE</literal> clause) must contain only
columns that are covered by the <literal>REPLICA IDENTITY</literal>, in
order for <command>UPDATE</command> and <command>DELETE</command> operations
--
1.8.3.1
On Fri, Sep 2, 2022 at 8:45 AM Peter Smith <smithpb2250@gmail.com> wrote:
On Thu, Sep 1, 2022 at 7:53 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Aug 26, 2022 at 7:33 AM Peter Smith <smithpb2250@gmail.com> wrote:
Few comments on both the patches: v4-0001* ========= 1. Furthermore, if the table uses + <literal>REPLICA IDENTITY FULL</literal>, specifying a column list is not + allowed (it will cause publication errors for <command>UPDATE</command> or + <command>DELETE</command> operations).This line sounds a bit unclear to me. From this like it appears that
the following operation is not allowed:postgres=# create table t1(c1 int, c2 int, c3 int);
CREATE TABLE
postgres=# Alter Table t1 replica identity full;
ALTER TABLE
postgres=# create publication pub1 for table t1(c1);
CREATE PUBLICATIONHowever, what is not allowed is the following:
postgres=# delete from t1;
ERROR: cannot delete from table "t1"
DETAIL: Column list used by the publication does not cover the
replica identity.I am not sure if we really need this line but if so then please try to
make it more clear. I think the similar text is present in 0002 patch
which should be modified accordingly.The "Furthermore…" sentence came from the commit message [1]. But I
agree it seems redundant/ambiguous, so I have removed it (and removed
the same in patch 0002).
Thanks, pushed your first patch.
--
With Regards,
Amit Kapila.
On Fri, Sep 2, 2022 at 11:40 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Sep 2, 2022 at 8:45 AM Peter Smith <smithpb2250@gmail.com> wrote:
On Thu, Sep 1, 2022 at 7:53 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Aug 26, 2022 at 7:33 AM Peter Smith <smithpb2250@gmail.com> wrote:
Few comments on both the patches: v4-0001* ========= 1. Furthermore, if the table uses + <literal>REPLICA IDENTITY FULL</literal>, specifying a column list is not + allowed (it will cause publication errors for <command>UPDATE</command> or + <command>DELETE</command> operations).This line sounds a bit unclear to me. From this like it appears that
the following operation is not allowed:postgres=# create table t1(c1 int, c2 int, c3 int);
CREATE TABLE
postgres=# Alter Table t1 replica identity full;
ALTER TABLE
postgres=# create publication pub1 for table t1(c1);
CREATE PUBLICATIONHowever, what is not allowed is the following:
postgres=# delete from t1;
ERROR: cannot delete from table "t1"
DETAIL: Column list used by the publication does not cover the
replica identity.I am not sure if we really need this line but if so then please try to
make it more clear. I think the similar text is present in 0002 patch
which should be modified accordingly.The "Furthermore…" sentence came from the commit message [1]. But I
agree it seems redundant/ambiguous, so I have removed it (and removed
the same in patch 0002).Thanks, pushed your first patch.
Thanks for the push.
I have rebased the remaining patch (v6-0001 is the same as v5-0002)
------
Kind Regards,
Peter Smith.
Fujitsu Australia
Attachments:
v6-0001-Column-List-new-pgdocs-section.patchapplication/octet-stream; name=v6-0001-Column-List-new-pgdocs-section.patchDownload
From c67a3b64354e529b8c18144b88f4ea973f234d4b Mon Sep 17 00:00:00 2001
From: Peter Smith <peter.b.smith@fujitsu.com>
Date: Mon, 5 Sep 2022 10:22:12 +1000
Subject: [PATCH v6] Column List new pgdocs section
Add a new logical replication pgdocs section for "Column Lists"
(analogous to the Row Filters page).
Also update xrefs to that new page from CREATE/ALTER PUBLICATION.
---
doc/src/sgml/logical-replication.sgml | 219 +++++++++++++++++++++++++++++++
doc/src/sgml/ref/alter_publication.sgml | 12 +-
doc/src/sgml/ref/create_publication.sgml | 6 +-
3 files changed, 226 insertions(+), 11 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index bdf1e7b..c9a8056 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1089,6 +1089,225 @@ test_sub=# SELECT * FROM child ORDER BY a;
</sect1>
+ <sect1 id="logical-replication-col-lists">
+ <title>Column Lists</title>
+
+ <para>
+ By default, all columns of a published table will be replicated to the
+ appropriate subscribers. The subscriber table must have at least all the
+ columns of the published table. However, if a
+ <firstterm>column list</firstterm> is specified then only the columns named
+ in the list will be replicated. This means the subscriber-side table only
+ needs to have those columns named by the column list. A user might choose to
+ use column lists for behavioral, security or performance reasons.
+ </para>
+
+ <sect2 id="logical-replication-col-list-rules">
+ <title>Column List Rules</title>
+
+ <para>
+ A column list is specified per table following the table name, and enclosed
+ by parentheses. See <xref linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
+ When specifying a column list, the order of columns is not important. If no
+ column list is specified, all columns of the table are replicated through
+ this publication, including any columns added later. This means a column
+ list which names all columns is not quite the same as having no column list
+ at all. For example, if additional columns are added to the table, then
+ (after a <literal>REFRESH PUBLICATION</literal>) if there was a column list
+ only those named columns will continue to be replicated.
+ </para>
+
+ <para>
+ Column lists have no effect for <literal>TRUNCATE</literal> command.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-restrictions">
+ <title>Column List Restrictions</title>
+
+ <para>
+ A column list can contain only simple column references.
+ </para>
+
+ <para>
+ If a publication publishes <command>UPDATE</command> or
+ <command>DELETE</command> operations, any column list must include the
+ table's replica identity columns (see
+ <xref linkend="sql-altertable-replica-identity"/>).
+ If a publication publishes only <command>INSERT</command> operations, then
+ the column list is arbitrary and may omit some replica identity columns.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-partitioned">
+ <title>Partitioned Tables</title>
+
+ <para>
+ For partitioned tables, the publication parameter
+ <literal>publish_via_partition_root</literal> determines which column list
+ is used. If <literal>publish_via_partition_root</literal> is
+ <literal>true</literal>, the root partitioned table's column list is used.
+ Otherwise, if <literal>publish_via_partition_root</literal> is
+ <literal>false</literal> (default), each partition's column list is used.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-initial-data-sync">
+ <title>Initial Data Synchronization</title>
+
+ <para>
+ If the subscription requires copying pre-existing table data and a
+ publication specifies a column list, only data from those columns will be
+ copied.
+ </para>
+
+ <note>
+ <para>
+ If the subscriber is in a release prior to 15, copy pre-existing data
+ doesn't use column lists even if they are defined in the publication.
+ This is because old releases can only copy the entire table data.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-combining">
+ <title>Combining Multiple Column Lists</title>
+
+ <warning>
+ <para>
+ It is not supported to have a subscription comprising several publications
+ where the same table has been published with different column lists.
+ This means changing the column lists of the tables being subscribed could
+ cause inconsistency of column lists among publications, in which case
+ the <xref linkend="sql-alterpublication"/> will be successful but later
+ the WalSender on the publisher, or the subscriber may throw an error. In
+ this scenario, the user needs to recreate the subscription after adjusting
+ the column list or drop the problematic publication using
+ <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add it
+ back after adjusting the column list.
+ </para>
+ <para>
+ Background: The main purpose of the column list feature is to allow
+ statically different table shapes on publisher and subscriber, or hide
+ sensitive column data. In both cases, it doesn't seem to make sense to
+ combine column lists.
+ </para>
+ </warning>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-examples">
+ <title>Examples</title>
+
+ <para>
+ Create a table <literal>t1</literal> to be used in the following example.
+<programlisting>
+test_pub=# CREATE TABLE t1(id int, a text, b text, c text, d text, e text, PRIMARY KEY(id));
+CREATE TABLE
+test_pub=#
+</programlisting></para>
+
+ <para>
+ Create a publication <literal>p1</literal>. A column list is defined for
+ table <literal>t1</literal> to reduce the number of columns that will be
+ replicated. Notice that the order of column names in the column list does
+ not matter.
+<programlisting>
+test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, b, a, d);
+CREATE PUBLICATION
+test_pub=#
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each publication.
+<programlisting>
+test_pub=# \dRp+
+ Publication p1
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+----------+------------+---------+---------+---------+-----------+----------
+ postgres | f | t | t | t | t | f
+Tables:
+ "public.t1" (id, a, b, d)
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each table.
+<programlisting>
+test_pub=# \d t1
+ Table "public.t1"
+ Column | Type | Collation | Nullable | Default
+--------+---------+-----------+----------+---------
+ id | integer | | not null |
+ a | text | | |
+ b | text | | |
+ c | text | | |
+ d | text | | |
+ e | text | | |
+Indexes:
+ "t1_pkey" PRIMARY KEY, btree (id)
+Publications:
+ "p1" (id, a, b, d)
+</programlisting></para>
+
+ <para>
+ On the subscriber node, create a table <literal>t1</literal> which now
+ only needs a subset of the columns that were on the publisher table
+ <literal>t1</literal>, and also create the subscription
+ <literal>s1</literal> that subscribes to the publication
+ <literal>p1</literal>.
+<programlisting>
+test_sub=# CREATE TABLE t1(id int, b text, a text, d text, PRIMARY KEY(id));
+CREATE TABLE
+test_sub=# CREATE SUBSCRIPTION s1
+test_sub-# CONNECTION 'host=localhost dbname=test_pub application_name=s1'
+test_sub-# PUBLICATION p1;
+CREATE SUBSCRIPTION
+</programlisting></para>
+
+ <para>
+ On the publisher node, insert some rows to table <literal>t1</literal>.
+<programlisting>
+test_pub=# INSERT INTO t1 VALUES(1, 'a-1', 'b-1', 'c-1', 'd-1', 'e-1');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(2, 'a-2', 'b-2', 'c-2', 'd-2', 'e-2');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(3, 'a-3', 'b-3', 'c-3', 'd-3', 'e-3');
+INSERT 0 1
+test_pub=# SELECT * FROM t1 ORDER BY id;
+ id | a | b | c | d | e
+----+-----+-----+-----+-----+-----
+ 1 | a-1 | b-1 | c-1 | d-1 | e-1
+ 2 | a-2 | b-2 | c-2 | d-2 | e-2
+ 3 | a-3 | b-3 | c-3 | d-3 | e-3
+(3 rows)
+</programlisting></para>
+
+ <para>
+ Only data from the column list of publication <literal>p1</literal> is
+ replicated.
+<programlisting>
+test_sub=# SELECT * FROM t1 ORDER BY id;
+ id | b | a | d
+----+-----+-----+-----
+ 1 | b-1 | a-1 | d-1
+ 2 | b-2 | a-2 | d-2
+ 3 | b-3 | a-3 | d-3
+(3 rows)
+</programlisting></para>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="logical-replication-conflicts">
<title>Conflicts</title>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 3a74973..d8ed89e 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -118,15 +118,9 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
Optionally, a column list can be specified. See <xref
linkend="sql-createpublication"/> for details. Note that a subscription
having several publications in which the same table has been published
- with different column lists is not supported. So, changing the column
- lists of the tables being subscribed could cause inconsistency of column
- lists among publications, in which case <command>ALTER PUBLICATION</command>
- will be successful but later the walsender on the publisher or the
- subscriber may throw an error. In this scenario, the user needs to
- recreate the subscription after adjusting the column list or drop the
- problematic publication using
- <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add
- it back after adjusting the column list.
+ with different column lists is not supported. See
+ <xref linkend="logical-replication-col-list-combining"/> for details of
+ potential problems when altering column lists.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index b0d59ef..f616418 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -91,8 +91,10 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated
through this publication, including any columns added later. It has no
- effect on <literal>TRUNCATE</literal> commands.
- </para>
+ effect on <literal>TRUNCATE</literal> commands. See
+ <xref linkend="logical-replication-col-lists"/> for details about column
+ lists.
+</para>
<para>
Only persistent base tables and partitioned tables can be part of a
--
1.8.3.1
On Mon, Sep 5, 2022 8:28 AM Peter Smith <smithpb2250@gmail.com> wrote:
I have rebased the remaining patch (v6-0001 is the same as v5-0002)
Thanks for updating the patch. Here are some comments.
1.
+ the <xref linkend="sql-alterpublication"/> will be successful but later
+ the WalSender on the publisher, or the subscriber may throw an error. In
+ this scenario, the user needs to recreate the subscription after adjusting
Should "WalSender" be changed to "walsender"? I saw "walsender" is used in other
places in the documentation.
2.
+test_pub=# CREATE TABLE t1(id int, a text, b text, c text, d text, e text, PRIMARY KEY(id));
+CREATE TABLE
+test_pub=#
+test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, b, a, d);
+CREATE PUBLICATION
+test_pub=#
I think the redundant "test_pub=#" can be removed.
Besides, I tested the examples in the patch, there's no problem.
Regards,
Shi yu
On Mon, Sep 5, 2022 at 1:42 PM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
On Mon, Sep 5, 2022 8:28 AM Peter Smith <smithpb2250@gmail.com> wrote:
I have rebased the remaining patch (v6-0001 is the same as v5-0002)
Thanks for updating the patch. Here are some comments.
1. + the <xref linkend="sql-alterpublication"/> will be successful but later + the WalSender on the publisher, or the subscriber may throw an error. In + this scenario, the user needs to recreate the subscription after adjustingShould "WalSender" be changed to "walsender"? I saw "walsender" is used in other
places in the documentation.
Modified.
2. +test_pub=# CREATE TABLE t1(id int, a text, b text, c text, d text, e text, PRIMARY KEY(id)); +CREATE TABLE +test_pub=#+test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, b, a, d); +CREATE PUBLICATION +test_pub=#I think the redundant "test_pub=#" can be removed.
Modified.
Besides, I tested the examples in the patch, there's no problem.
Thanks for the review comments, and testing.
I made both fixes as suggested.
PSA v7.
------
Kind Regards,
Peter Smith.
Fujitsu Australia
Attachments:
v7-0001-Column-List-new-pgdocs-section.patchapplication/octet-stream; name=v7-0001-Column-List-new-pgdocs-section.patchDownload
From ec09c19b8f2b71a79632d8147a77d5d9bbf07b67 Mon Sep 17 00:00:00 2001
From: Peter Smith <peter.b.smith@fujitsu.com>
Date: Mon, 5 Sep 2022 20:09:27 +1000
Subject: [PATCH v7] Column List new pgdocs section
Add a new logical replication pgdocs section for "Column Lists"
(analogous to the Row Filters page).
Also update xrefs to that new page from CREATE/ALTER PUBLICATION.
---
doc/src/sgml/logical-replication.sgml | 217 +++++++++++++++++++++++++++++++
doc/src/sgml/ref/alter_publication.sgml | 12 +-
doc/src/sgml/ref/create_publication.sgml | 6 +-
3 files changed, 224 insertions(+), 11 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index bdf1e7b..9f35fa8 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1089,6 +1089,223 @@ test_sub=# SELECT * FROM child ORDER BY a;
</sect1>
+ <sect1 id="logical-replication-col-lists">
+ <title>Column Lists</title>
+
+ <para>
+ By default, all columns of a published table will be replicated to the
+ appropriate subscribers. The subscriber table must have at least all the
+ columns of the published table. However, if a
+ <firstterm>column list</firstterm> is specified then only the columns named
+ in the list will be replicated. This means the subscriber-side table only
+ needs to have those columns named by the column list. A user might choose to
+ use column lists for behavioral, security or performance reasons.
+ </para>
+
+ <sect2 id="logical-replication-col-list-rules">
+ <title>Column List Rules</title>
+
+ <para>
+ A column list is specified per table following the table name, and enclosed
+ by parentheses. See <xref linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
+ When specifying a column list, the order of columns is not important. If no
+ column list is specified, all columns of the table are replicated through
+ this publication, including any columns added later. This means a column
+ list which names all columns is not quite the same as having no column list
+ at all. For example, if additional columns are added to the table, then
+ (after a <literal>REFRESH PUBLICATION</literal>) if there was a column list
+ only those named columns will continue to be replicated.
+ </para>
+
+ <para>
+ Column lists have no effect for <literal>TRUNCATE</literal> command.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-restrictions">
+ <title>Column List Restrictions</title>
+
+ <para>
+ A column list can contain only simple column references.
+ </para>
+
+ <para>
+ If a publication publishes <command>UPDATE</command> or
+ <command>DELETE</command> operations, any column list must include the
+ table's replica identity columns (see
+ <xref linkend="sql-altertable-replica-identity"/>).
+ If a publication publishes only <command>INSERT</command> operations, then
+ the column list is arbitrary and may omit some replica identity columns.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-partitioned">
+ <title>Partitioned Tables</title>
+
+ <para>
+ For partitioned tables, the publication parameter
+ <literal>publish_via_partition_root</literal> determines which column list
+ is used. If <literal>publish_via_partition_root</literal> is
+ <literal>true</literal>, the root partitioned table's column list is used.
+ Otherwise, if <literal>publish_via_partition_root</literal> is
+ <literal>false</literal> (default), each partition's column list is used.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-initial-data-sync">
+ <title>Initial Data Synchronization</title>
+
+ <para>
+ If the subscription requires copying pre-existing table data and a
+ publication specifies a column list, only data from those columns will be
+ copied.
+ </para>
+
+ <note>
+ <para>
+ If the subscriber is in a release prior to 15, copy pre-existing data
+ doesn't use column lists even if they are defined in the publication.
+ This is because old releases can only copy the entire table data.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-combining">
+ <title>Combining Multiple Column Lists</title>
+
+ <warning>
+ <para>
+ It is not supported to have a subscription comprising several publications
+ where the same table has been published with different column lists.
+ This means changing the column lists of the tables being subscribed could
+ cause inconsistency of column lists among publications, in which case
+ the <xref linkend="sql-alterpublication"/> will be successful but later
+ the walsender on the publisher, or the subscriber may throw an error. In
+ this scenario, the user needs to recreate the subscription after adjusting
+ the column list or drop the problematic publication using
+ <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add it
+ back after adjusting the column list.
+ </para>
+ <para>
+ Background: The main purpose of the column list feature is to allow
+ statically different table shapes on publisher and subscriber, or hide
+ sensitive column data. In both cases, it doesn't seem to make sense to
+ combine column lists.
+ </para>
+ </warning>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-examples">
+ <title>Examples</title>
+
+ <para>
+ Create a table <literal>t1</literal> to be used in the following example.
+<programlisting>
+test_pub=# CREATE TABLE t1(id int, a text, b text, c text, d text, e text, PRIMARY KEY(id));
+CREATE TABLE
+</programlisting></para>
+
+ <para>
+ Create a publication <literal>p1</literal>. A column list is defined for
+ table <literal>t1</literal> to reduce the number of columns that will be
+ replicated. Notice that the order of column names in the column list does
+ not matter.
+<programlisting>
+test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, b, a, d);
+CREATE PUBLICATION
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each publication.
+<programlisting>
+test_pub=# \dRp+
+ Publication p1
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+----------+------------+---------+---------+---------+-----------+----------
+ postgres | f | t | t | t | t | f
+Tables:
+ "public.t1" (id, a, b, d)
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each table.
+<programlisting>
+test_pub=# \d t1
+ Table "public.t1"
+ Column | Type | Collation | Nullable | Default
+--------+---------+-----------+----------+---------
+ id | integer | | not null |
+ a | text | | |
+ b | text | | |
+ c | text | | |
+ d | text | | |
+ e | text | | |
+Indexes:
+ "t1_pkey" PRIMARY KEY, btree (id)
+Publications:
+ "p1" (id, a, b, d)
+</programlisting></para>
+
+ <para>
+ On the subscriber node, create a table <literal>t1</literal> which now
+ only needs a subset of the columns that were on the publisher table
+ <literal>t1</literal>, and also create the subscription
+ <literal>s1</literal> that subscribes to the publication
+ <literal>p1</literal>.
+<programlisting>
+test_sub=# CREATE TABLE t1(id int, b text, a text, d text, PRIMARY KEY(id));
+CREATE TABLE
+test_sub=# CREATE SUBSCRIPTION s1
+test_sub-# CONNECTION 'host=localhost dbname=test_pub application_name=s1'
+test_sub-# PUBLICATION p1;
+CREATE SUBSCRIPTION
+</programlisting></para>
+
+ <para>
+ On the publisher node, insert some rows to table <literal>t1</literal>.
+<programlisting>
+test_pub=# INSERT INTO t1 VALUES(1, 'a-1', 'b-1', 'c-1', 'd-1', 'e-1');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(2, 'a-2', 'b-2', 'c-2', 'd-2', 'e-2');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(3, 'a-3', 'b-3', 'c-3', 'd-3', 'e-3');
+INSERT 0 1
+test_pub=# SELECT * FROM t1 ORDER BY id;
+ id | a | b | c | d | e
+----+-----+-----+-----+-----+-----
+ 1 | a-1 | b-1 | c-1 | d-1 | e-1
+ 2 | a-2 | b-2 | c-2 | d-2 | e-2
+ 3 | a-3 | b-3 | c-3 | d-3 | e-3
+(3 rows)
+</programlisting></para>
+
+ <para>
+ Only data from the column list of publication <literal>p1</literal> is
+ replicated.
+<programlisting>
+test_sub=# SELECT * FROM t1 ORDER BY id;
+ id | b | a | d
+----+-----+-----+-----
+ 1 | b-1 | a-1 | d-1
+ 2 | b-2 | a-2 | d-2
+ 3 | b-3 | a-3 | d-3
+(3 rows)
+</programlisting></para>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="logical-replication-conflicts">
<title>Conflicts</title>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 3a74973..d8ed89e 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -118,15 +118,9 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
Optionally, a column list can be specified. See <xref
linkend="sql-createpublication"/> for details. Note that a subscription
having several publications in which the same table has been published
- with different column lists is not supported. So, changing the column
- lists of the tables being subscribed could cause inconsistency of column
- lists among publications, in which case <command>ALTER PUBLICATION</command>
- will be successful but later the walsender on the publisher or the
- subscriber may throw an error. In this scenario, the user needs to
- recreate the subscription after adjusting the column list or drop the
- problematic publication using
- <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add
- it back after adjusting the column list.
+ with different column lists is not supported. See
+ <xref linkend="logical-replication-col-list-combining"/> for details of
+ potential problems when altering column lists.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index b0d59ef..f616418 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -91,8 +91,10 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated
through this publication, including any columns added later. It has no
- effect on <literal>TRUNCATE</literal> commands.
- </para>
+ effect on <literal>TRUNCATE</literal> commands. See
+ <xref linkend="logical-replication-col-lists"/> for details about column
+ lists.
+</para>
<para>
Only persistent base tables and partitioned tables can be part of a
--
1.8.3.1
On Mon, Sep 5, 2022 at 3:46 PM Peter Smith <smithpb2250@gmail.com> wrote:
PSA v7.
For example, if additional columns are added to the table, then
+ (after a <literal>REFRESH PUBLICATION</literal>) if there was a column list
+ only those named columns will continue to be replicated.
This looks a bit unclear to me w.r.t the refresh publication step. Why
exactly you have used refresh publication in the above para? It is
used to add new tables if any added to the publication, so not clear
to me how it helps in this case. If that is not required then we can
change it to: "For example, if additional columns are added to the
table then only those named columns mentioned in the column list will
continue to be replicated."
--
With Regards,
Amit Kapila.
On Mon, Sep 5, 2022 at 8:46 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Sep 5, 2022 at 3:46 PM Peter Smith <smithpb2250@gmail.com> wrote:
PSA v7.
For example, if additional columns are added to the table, then + (after a <literal>REFRESH PUBLICATION</literal>) if there was a column list + only those named columns will continue to be replicated.This looks a bit unclear to me w.r.t the refresh publication step. Why
exactly you have used refresh publication in the above para? It is
used to add new tables if any added to the publication, so not clear
to me how it helps in this case. If that is not required then we can
change it to: "For example, if additional columns are added to the
table then only those named columns mentioned in the column list will
continue to be replicated."
You are right - that REFRESH PUBLICATION was not necessary for this
example. The patch is modified to use your suggested text.
PSA v8
------
Kind Regards,
Peter Smith.
Fujitsu Australia
Attachments:
v8-0001-Column-List-new-pgdocs-section.patchapplication/octet-stream; name=v8-0001-Column-List-new-pgdocs-section.patchDownload
From cb2601e8bf212a0b29461cbba067e0954b96e301 Mon Sep 17 00:00:00 2001
From: Peter Smith <peter.b.smith@fujitsu.com>
Date: Tue, 6 Sep 2022 09:26:02 +1000
Subject: [PATCH v8] Column List new pgdocs section
Add a new logical replication pgdocs section for "Column Lists"
(analogous to the Row Filters page).
Also update xrefs to that new page from CREATE/ALTER PUBLICATION.
---
doc/src/sgml/logical-replication.sgml | 217 +++++++++++++++++++++++++++++++
doc/src/sgml/ref/alter_publication.sgml | 12 +-
doc/src/sgml/ref/create_publication.sgml | 6 +-
3 files changed, 224 insertions(+), 11 deletions(-)
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index bdf1e7b..0ab191e 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1089,6 +1089,223 @@ test_sub=# SELECT * FROM child ORDER BY a;
</sect1>
+ <sect1 id="logical-replication-col-lists">
+ <title>Column Lists</title>
+
+ <para>
+ By default, all columns of a published table will be replicated to the
+ appropriate subscribers. The subscriber table must have at least all the
+ columns of the published table. However, if a
+ <firstterm>column list</firstterm> is specified then only the columns named
+ in the list will be replicated. This means the subscriber-side table only
+ needs to have those columns named by the column list. A user might choose to
+ use column lists for behavioral, security or performance reasons.
+ </para>
+
+ <sect2 id="logical-replication-col-list-rules">
+ <title>Column List Rules</title>
+
+ <para>
+ A column list is specified per table following the table name, and enclosed
+ by parentheses. See <xref linkend="sql-createpublication"/> for details.
+ </para>
+
+ <para>
+ When specifying a column list, the order of columns is not important. If no
+ column list is specified, all columns of the table are replicated through
+ this publication, including any columns added later. This means a column
+ list which names all columns is not quite the same as having no column list
+ at all. For example, if additional columns are added to the table then only
+ those named columns mentioned in the column list will continue to be
+ replicated.
+ </para>
+
+ <para>
+ Column lists have no effect for <literal>TRUNCATE</literal> command.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-restrictions">
+ <title>Column List Restrictions</title>
+
+ <para>
+ A column list can contain only simple column references.
+ </para>
+
+ <para>
+ If a publication publishes <command>UPDATE</command> or
+ <command>DELETE</command> operations, any column list must include the
+ table's replica identity columns (see
+ <xref linkend="sql-altertable-replica-identity"/>).
+ If a publication publishes only <command>INSERT</command> operations, then
+ the column list is arbitrary and may omit some replica identity columns.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-partitioned">
+ <title>Partitioned Tables</title>
+
+ <para>
+ For partitioned tables, the publication parameter
+ <literal>publish_via_partition_root</literal> determines which column list
+ is used. If <literal>publish_via_partition_root</literal> is
+ <literal>true</literal>, the root partitioned table's column list is used.
+ Otherwise, if <literal>publish_via_partition_root</literal> is
+ <literal>false</literal> (default), each partition's column list is used.
+ </para>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-initial-data-sync">
+ <title>Initial Data Synchronization</title>
+
+ <para>
+ If the subscription requires copying pre-existing table data and a
+ publication specifies a column list, only data from those columns will be
+ copied.
+ </para>
+
+ <note>
+ <para>
+ If the subscriber is in a release prior to 15, copy pre-existing data
+ doesn't use column lists even if they are defined in the publication.
+ This is because old releases can only copy the entire table data.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-combining">
+ <title>Combining Multiple Column Lists</title>
+
+ <warning>
+ <para>
+ It is not supported to have a subscription comprising several publications
+ where the same table has been published with different column lists.
+ This means changing the column lists of the tables being subscribed could
+ cause inconsistency of column lists among publications, in which case
+ the <xref linkend="sql-alterpublication"/> will be successful but later
+ the walsender on the publisher, or the subscriber may throw an error. In
+ this scenario, the user needs to recreate the subscription after adjusting
+ the column list or drop the problematic publication using
+ <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add it
+ back after adjusting the column list.
+ </para>
+ <para>
+ Background: The main purpose of the column list feature is to allow
+ statically different table shapes on publisher and subscriber, or hide
+ sensitive column data. In both cases, it doesn't seem to make sense to
+ combine column lists.
+ </para>
+ </warning>
+
+ </sect2>
+
+ <sect2 id="logical-replication-col-list-examples">
+ <title>Examples</title>
+
+ <para>
+ Create a table <literal>t1</literal> to be used in the following example.
+<programlisting>
+test_pub=# CREATE TABLE t1(id int, a text, b text, c text, d text, e text, PRIMARY KEY(id));
+CREATE TABLE
+</programlisting></para>
+
+ <para>
+ Create a publication <literal>p1</literal>. A column list is defined for
+ table <literal>t1</literal> to reduce the number of columns that will be
+ replicated. Notice that the order of column names in the column list does
+ not matter.
+<programlisting>
+test_pub=# CREATE PUBLICATION p1 FOR TABLE t1 (id, b, a, d);
+CREATE PUBLICATION
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each publication.
+<programlisting>
+test_pub=# \dRp+
+ Publication p1
+ Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
+----------+------------+---------+---------+---------+-----------+----------
+ postgres | f | t | t | t | t | f
+Tables:
+ "public.t1" (id, a, b, d)
+</programlisting></para>
+
+ <para>
+ <literal>psql</literal> can be used to show the column lists (if defined)
+ for each table.
+<programlisting>
+test_pub=# \d t1
+ Table "public.t1"
+ Column | Type | Collation | Nullable | Default
+--------+---------+-----------+----------+---------
+ id | integer | | not null |
+ a | text | | |
+ b | text | | |
+ c | text | | |
+ d | text | | |
+ e | text | | |
+Indexes:
+ "t1_pkey" PRIMARY KEY, btree (id)
+Publications:
+ "p1" (id, a, b, d)
+</programlisting></para>
+
+ <para>
+ On the subscriber node, create a table <literal>t1</literal> which now
+ only needs a subset of the columns that were on the publisher table
+ <literal>t1</literal>, and also create the subscription
+ <literal>s1</literal> that subscribes to the publication
+ <literal>p1</literal>.
+<programlisting>
+test_sub=# CREATE TABLE t1(id int, b text, a text, d text, PRIMARY KEY(id));
+CREATE TABLE
+test_sub=# CREATE SUBSCRIPTION s1
+test_sub-# CONNECTION 'host=localhost dbname=test_pub application_name=s1'
+test_sub-# PUBLICATION p1;
+CREATE SUBSCRIPTION
+</programlisting></para>
+
+ <para>
+ On the publisher node, insert some rows to table <literal>t1</literal>.
+<programlisting>
+test_pub=# INSERT INTO t1 VALUES(1, 'a-1', 'b-1', 'c-1', 'd-1', 'e-1');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(2, 'a-2', 'b-2', 'c-2', 'd-2', 'e-2');
+INSERT 0 1
+test_pub=# INSERT INTO t1 VALUES(3, 'a-3', 'b-3', 'c-3', 'd-3', 'e-3');
+INSERT 0 1
+test_pub=# SELECT * FROM t1 ORDER BY id;
+ id | a | b | c | d | e
+----+-----+-----+-----+-----+-----
+ 1 | a-1 | b-1 | c-1 | d-1 | e-1
+ 2 | a-2 | b-2 | c-2 | d-2 | e-2
+ 3 | a-3 | b-3 | c-3 | d-3 | e-3
+(3 rows)
+</programlisting></para>
+
+ <para>
+ Only data from the column list of publication <literal>p1</literal> is
+ replicated.
+<programlisting>
+test_sub=# SELECT * FROM t1 ORDER BY id;
+ id | b | a | d
+----+-----+-----+-----
+ 1 | b-1 | a-1 | d-1
+ 2 | b-2 | a-2 | d-2
+ 3 | b-3 | a-3 | d-3
+(3 rows)
+</programlisting></para>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="logical-replication-conflicts">
<title>Conflicts</title>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 3a74973..d8ed89e 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -118,15 +118,9 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
Optionally, a column list can be specified. See <xref
linkend="sql-createpublication"/> for details. Note that a subscription
having several publications in which the same table has been published
- with different column lists is not supported. So, changing the column
- lists of the tables being subscribed could cause inconsistency of column
- lists among publications, in which case <command>ALTER PUBLICATION</command>
- will be successful but later the walsender on the publisher or the
- subscriber may throw an error. In this scenario, the user needs to
- recreate the subscription after adjusting the column list or drop the
- problematic publication using
- <literal>ALTER SUBSCRIPTION ... DROP PUBLICATION</literal> and then add
- it back after adjusting the column list.
+ with different column lists is not supported. See
+ <xref linkend="logical-replication-col-list-combining"/> for details of
+ potential problems when altering column lists.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index b0d59ef..f616418 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -91,8 +91,10 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
When a column list is specified, only the named columns are replicated.
If no column list is specified, all columns of the table are replicated
through this publication, including any columns added later. It has no
- effect on <literal>TRUNCATE</literal> commands.
- </para>
+ effect on <literal>TRUNCATE</literal> commands. See
+ <xref linkend="logical-replication-col-lists"/> for details about column
+ lists.
+</para>
<para>
Only persistent base tables and partitioned tables can be part of a
--
1.8.3.1
On Tue, Sep 6, 2022 at 5:08 AM Peter Smith <smithpb2250@gmail.com> wrote:
You are right - that REFRESH PUBLICATION was not necessary for this
example. The patch is modified to use your suggested text.PSA v8
LGTM. I'll push this once the tag appears for v15.
--
With Regards,
Amit Kapila.
On Tue, Sep 6, 2022 at 2:15 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Tue, Sep 6, 2022 at 5:08 AM Peter Smith <smithpb2250@gmail.com> wrote:
You are right - that REFRESH PUBLICATION was not necessary for this
example. The patch is modified to use your suggested text.PSA v8
LGTM. I'll push this once the tag appears for v15.
Pushed!
--
With Regards,
Amit Kapila.
On Wed, Sep 7, 2022 at 8:49 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Tue, Sep 6, 2022 at 2:15 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Tue, Sep 6, 2022 at 5:08 AM Peter Smith <smithpb2250@gmail.com> wrote:
You are right - that REFRESH PUBLICATION was not necessary for this
example. The patch is modified to use your suggested text.PSA v8
LGTM. I'll push this once the tag appears for v15.
Pushed!
Thanks for pushing.
------
Kind Regards,
Peter Smith.
Fujitsu Australia.