Dropped and generated columns might cause wrong data on subs when REPLICA IDENTITY FULL
Hi all,
(cc'ed Amit as he has the context)
While working on [1]/messages/by-id/CACawEhUN=+vjY0+4q416-rAYx6pw-nZMHQYsJZCftf9MjoPN3w@mail.gmail.com, I realized that on HEAD there is a problem with the
$subject. Here is the relevant discussion on the thread [2]/messages/by-id/CACawEhUu6S8E4Oo7+s5iaq=yLRZJb6uOZeEQSGJj-7NVkDzSaw@mail.gmail.com. Quoting my
own notes on that thread below;
I realized that the dropped columns also get into the tuples_equal()
function. And,
the remote sends NULL to for the dropped columns(i.e., remoteslot), but
index_getnext_slot() (or table_scan_getnextslot) indeed fills the dropped
columns on the outslot. So, the dropped columns are not NULL in the outslot
Amit also suggested checking generated columns, which indeed has the same
problem.
Here are the steps to repro the problem with dropped columns:
- pub
CREATE TABLE test (drop_1 jsonb, x int, drop_2 numeric, y text, drop_3
timestamptz);
ALTER TABLE test REPLICA IDENTITY FULL;
INSERT INTO test SELECT NULL, i, i, (i)::text, now() FROM
generate_series(0,1)i;
CREATE PUBLICATION pub FOR ALL TABLES;
-- sub
CREATE TABLE test (drop_1 jsonb, x int, drop_2 numeric, y text, drop_3
timestamptz);
CREATE SUBSCRIPTION sub CONNECTION 'host=localhost port=5432
dbname=postgres' PUBLICATION pub;
-- show that before dropping the columns, the data in the source and
-- target are deleted properly
DELETE FROM test WHERE x = 0;
-- both on the source and target
SELECT count(*) FROM test WHERE x = 0;
┌───────┐
│ count │
├───────┤
│ 0 │
└───────┘
(1 row)
-- drop columns on both the the source
ALTER TABLE test DROP COLUMN drop_1;
ALTER TABLE test DROP COLUMN drop_2;
ALTER TABLE test DROP COLUMN drop_3;
-- drop columns on both the the target
ALTER TABLE test DROP COLUMN drop_1;
ALTER TABLE test DROP COLUMN drop_2;
ALTER TABLE test DROP COLUMN drop_3;
-- on the target
ALTER SUBSCRIPTION sub REFRESH PUBLICATION;
-- after dropping the columns
DELETE FROM test WHERE x = 1;
-- source
SELECT count(*) FROM test WHERE x = 1;
┌───────┐
│ count │
├───────┤
│ 0 │
└───────┘
(1 row)
**-- target, OOPS wrong result!!!!**SELECT count(*) FROM test WHERE x = 1;
┌───────┐
│ count │
├───────┤
│ 1 │
└───────┘
(1 row)
Attaching a patch that could possibly solve the problem.
Thanks,
Onder KALACI
[1]: /messages/by-id/CACawEhUN=+vjY0+4q416-rAYx6pw-nZMHQYsJZCftf9MjoPN3w@mail.gmail.com
/messages/by-id/CACawEhUN=+vjY0+4q416-rAYx6pw-nZMHQYsJZCftf9MjoPN3w@mail.gmail.com
[2]: /messages/by-id/CACawEhUu6S8E4Oo7+s5iaq=yLRZJb6uOZeEQSGJj-7NVkDzSaw@mail.gmail.com
/messages/by-id/CACawEhUu6S8E4Oo7+s5iaq=yLRZJb6uOZeEQSGJj-7NVkDzSaw@mail.gmail.com
Attachments:
v1-0001-Skip-dropped-and-generated-columns-when-REPLICA-I.patchapplication/octet-stream; name=v1-0001-Skip-dropped-and-generated-columns-when-REPLICA-I.patchDownload
From c4aafaedae90fdfee0d482574de11bdb2247c389 Mon Sep 17 00:00:00 2001
From: Onder Kalaci <onderkalaci@gmail.com>
Date: Sat, 11 Mar 2023 12:09:31 +0300
Subject: [PATCH v1] Skip dropped and generated columns when REPLICA IDENTITY
FULL
Dropped columns and generated columns are filled with NULL values on
slot_store_data() but not on table_scan_getnextslot(). With this commit,
we skip such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 11 +++++
src/test/subscription/t/100_bugs.pl | 67 ++++++++++++++++++++++++++
2 files changed, 78 insertions(+)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index c484f5c301..6f90e8af42 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -243,6 +243,17 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
Form_pg_attribute att;
TypeCacheEntry *typentry;
+
+ Form_pg_attribute attr = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
+
+ /*
+ * Dropped columns and generated columns are filled with NULL values on
+ * slot_store_data() but not on table_scan_getnextslot, so ignore
+ * for the comparison.
+ */
+ if (attr->attisdropped || attr->attgenerated)
+ continue;
+
/*
* If one value is NULL and other is not, then they are certainly not
* equal
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 036576acab..e04ecfa6ca 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -373,4 +373,71 @@ is( $result, qq(1
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
+# https://www.postgresql.org/message-id/CACawEhUu6S8E4Oo7%2Bs5iaq%3DyLRZJb6uOZeEQSGJj-7NVkDzSaw%40mail.gmail.com
+
+# The bug was that when when the REPLICA IDENTITY FULL is used with dropped
+# or generated columns, the equality checks for the tuples was trying to
+# compare NULL Datum values (coming from slot_store_data()) with the
+# non-visible actua Datum values (coming from table_scan_getnextslot()).
+my $node_publisher_d_cols = PostgreSQL::Test::Cluster->new('node_publisher_d_cols');
+$node_publisher_d_cols->init(allows_streaming => 'logical');
+$node_publisher_d_cols->start;
+
+my $node_subscriber_d_cols = PostgreSQL::Test::Cluster->new('node_subscriber_d_cols');
+$node_subscriber_d_cols->init(allows_streaming => 'logical');
+$node_subscriber_d_cols->start;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
+
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
+
+ -- some initial data
+ INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a,c) VALUES (1, 1);
+));
+
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+
+));
+
+my $publisher_connstr_d_cols = $node_publisher_d_cols->connstr . ' dbname=postgres';
+$node_subscriber_d_cols->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub_dropped_cols CONNECTION '$publisher_connstr_d_cols' PUBLICATION pub_dropped_cols"
+);
+$node_subscriber_d_cols->wait_for_subscription_sync;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
+));
+$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
+
+is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
+ qq(1), 'replication with RI FULL and dropped columns');
+
+is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1), 'replication with RI FULL and generated columns');
+
+$node_publisher_d_cols->stop('fast');
+$node_subscriber_d_cols->stop('fast');
+
done_testing();
--
2.34.1
On Sun, Mar 12, 2023 4:00 AM Önder Kalacı <onderkalaci@gmail.com> wrote:
Attaching a patch that could possibly solve the problem.
Thanks for your patch. I tried it and it worked well.
Here are some minor comments.
1.
@@ -243,6 +243,17 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
Form_pg_attribute att;
TypeCacheEntry *typentry;
+
+ Form_pg_attribute attr = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
+
I think we can use "att" instead of a new variable. They have the same value.
2.
+# The bug was that when when the REPLICA IDENTITY FULL is used with dropped
There is an extra "when".
Regards,
Shi Yu
Hi Shi Yu,
1.
@@ -243,6 +243,17 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot
*slot2,
Form_pg_attribute att;
TypeCacheEntry *typentry;+ + Form_pg_attribute attr = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum); +I think we can use "att" instead of a new variable. They have the same
value.
ah, of course :)
2.
+# The bug was that when when the REPLICA IDENTITY FULL is used with
droppedThere is an extra "when".
Fixed, thanks
Attaching v2
Attachments:
v2-0001-Skip-dropped-and-generated-columns-when-REPLICA-I.patchapplication/x-patch; name=v2-0001-Skip-dropped-and-generated-columns-when-REPLICA-I.patchDownload
From caae511206ee904db54d0a9300ecef04c86aadb6 Mon Sep 17 00:00:00 2001
From: Onder Kalaci <onderkalaci@gmail.com>
Date: Sat, 11 Mar 2023 12:09:31 +0300
Subject: [PATCH v2] Skip dropped and generated columns when REPLICA IDENTITY
FULL
Dropped columns and generated columns are filled with NULL values on
slot_store_data() but not on table_scan_getnextslot(). With this commit,
we skip such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 10 ++++
src/test/subscription/t/100_bugs.pl | 67 ++++++++++++++++++++++++++
2 files changed, 77 insertions(+)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 4f5083a598..9d447b3c00 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -243,6 +243,16 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
Form_pg_attribute att;
TypeCacheEntry *typentry;
+ att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
+
+ /*
+ * Dropped columns and generated columns are filled with NULL values on
+ * slot_store_data() but not on table_scan_getnextslot, so ignore
+ * for the comparison.
+ */
+ if (att->attisdropped || att->attgenerated)
+ continue;
+
/*
* If one value is NULL and other is not, then they are certainly not
* equal
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 036576acab..5f902d26b6 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -373,4 +373,71 @@ is( $result, qq(1
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
+# https://www.postgresql.org/message-id/CACawEhUu6S8E4Oo7%2Bs5iaq%3DyLRZJb6uOZeEQSGJj-7NVkDzSaw%40mail.gmail.com
+
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped
+# or generated columns, the equality checks for the tuples was trying to
+# compare NULL Datum values (coming from slot_store_data()) with the
+# non-visible actua Datum values (coming from table_scan_getnextslot()).
+my $node_publisher_d_cols = PostgreSQL::Test::Cluster->new('node_publisher_d_cols');
+$node_publisher_d_cols->init(allows_streaming => 'logical');
+$node_publisher_d_cols->start;
+
+my $node_subscriber_d_cols = PostgreSQL::Test::Cluster->new('node_subscriber_d_cols');
+$node_subscriber_d_cols->init(allows_streaming => 'logical');
+$node_subscriber_d_cols->start;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
+
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
+
+ -- some initial data
+ INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a,c) VALUES (1, 1);
+));
+
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+
+));
+
+my $publisher_connstr_d_cols = $node_publisher_d_cols->connstr . ' dbname=postgres';
+$node_subscriber_d_cols->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub_dropped_cols CONNECTION '$publisher_connstr_d_cols' PUBLICATION pub_dropped_cols"
+);
+$node_subscriber_d_cols->wait_for_subscription_sync;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
+));
+$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
+
+is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
+ qq(1), 'replication with RI FULL and dropped columns');
+
+is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1), 'replication with RI FULL and generated columns');
+
+$node_publisher_d_cols->stop('fast');
+$node_subscriber_d_cols->stop('fast');
+
done_testing();
--
2.34.1
On Mon, Mar 13, 2023 at 6:26 PM Önder Kalacı <onderkalaci@gmail.com> wrote:
Attaching v2
Can we change the comment to: "Ignore dropped and generated columns as
the publisher doesn't send those."? After your change, att =
TupleDescAttr(slot1->tts_tupleDescriptor, attrnum); is done twice in
the same function.
In test cases, let's change the comment to: "The bug was that when the
REPLICA IDENTITY FULL is used with dropped or generated columns, we
fail to apply updates and deletes.". Also, I think we don't need to
provide the email link as anyway commit message will have a link to
the discussion.
Did you check this in the back branches?
--
With Regards,
Amit Kapila.
On Thu, Mar 16, 2023 5:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Mar 13, 2023 at 6:26 PM Önder Kalacı <onderkalaci@gmail.com>
wrote:Attaching v2
Can we change the comment to: "Ignore dropped and generated columns as
the publisher doesn't send those."? After your change, att =
TupleDescAttr(slot1->tts_tupleDescriptor, attrnum); is done twice in
the same function.In test cases, let's change the comment to: "The bug was that when the
REPLICA IDENTITY FULL is used with dropped or generated columns, we
fail to apply updates and deletes.". Also, I think we don't need to
provide the email link as anyway commit message will have a link to
the discussion.Did you check this in the back branches?
I tried to reproduce this bug in backbranch.
Generated column is introduced in PG12, and I reproduced generated column problem
in PG12~PG15.
For dropped column problem, I reproduced it in PG10~PG15. (Logical replication
was introduced in PG10)
So I think we should backpatch the fix for generated column to PG12, and
backpatch the fix for dropped column to PG10.
Regards,
Shi Yu
Hi Amit, Shi Yu,
Generated column is introduced in PG12, and I reproduced generated column
problem
in PG12~PG15.
For dropped column problem, I reproduced it in PG10~PG15. (Logical
replication
was introduced in PG10)
So, I'm planning to split the changes into two commits. The first one fixes
for dropped columns, and the second one adds generated columns check/test.
Is that the right approach for such a case?
and backpatch the fix for dropped column to PG10.
Still, even the first commit fails to apply cleanly to PG12 (and below).
What is the procedure here? Should I be creating multiple patches per
version?
Or does the committer prefer to handle the conflicts? Depending on your
reply,
I can work on the followup.
I'm still attaching the dropped column patch for reference.
Thanks,
Onder
shiy.fnst@fujitsu.com <shiy.fnst@fujitsu.com>, 16 Mar 2023 Per, 13:38
tarihinde şunu yazdı:
Show quoted text
On Thu, Mar 16, 2023 5:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Mar 13, 2023 at 6:26 PM Önder Kalacı <onderkalaci@gmail.com>
wrote:Attaching v2
Can we change the comment to: "Ignore dropped and generated columns as
the publisher doesn't send those."? After your change, att =
TupleDescAttr(slot1->tts_tupleDescriptor, attrnum); is done twice in
the same function.In test cases, let's change the comment to: "The bug was that when the
REPLICA IDENTITY FULL is used with dropped or generated columns, we
fail to apply updates and deletes.". Also, I think we don't need to
provide the email link as anyway commit message will have a link to
the discussion.Did you check this in the back branches?
I tried to reproduce this bug in backbranch.
Generated column is introduced in PG12, and I reproduced generated column
problem
in PG12~PG15.For dropped column problem, I reproduced it in PG10~PG15. (Logical
replication
was introduced in PG10)So I think we should backpatch the fix for generated column to PG12, and
backpatch the fix for dropped column to PG10.Regards,
Shi Yu
Attachments:
v1-0001-Ignore-dropped-columns-when-REPLICA-IDENTITY-FULL.patchapplication/x-patch; name=v1-0001-Ignore-dropped-columns-when-REPLICA-IDENTITY-FULL.patchDownload
From d25b2feac3b26646034bf3ace6ab93056c3c29af Mon Sep 17 00:00:00 2001
From: Onder Kalaci <onderkalaci@gmail.com>
Date: Thu, 16 Mar 2023 18:06:54 +0300
Subject: [PATCH v1 1/2] Ignore dropped columns when REPLICA IDENTITY FULL
Dropped columns are filled with NULL values on slot_store_data()
but not on table_scan_getnextslot(). With this commit, we skip
such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 9 ++++-
src/test/subscription/t/100_bugs.pl | 51 ++++++++++++++++++++++++++
2 files changed, 58 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index fa8628e3e1..7c8b0d2667 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -289,6 +289,13 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
Form_pg_attribute att;
TypeCacheEntry *typentry;
+ /*
+ * Ignore dropped columns as the publisher doesn't send those
+ */
+ att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
+ if (att->attisdropped)
+ continue;
+
/*
* If one value is NULL and other is not, then they are certainly not
* equal
@@ -302,8 +309,6 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
if (slot1->tts_isnull[attrnum] || slot2->tts_isnull[attrnum])
continue;
- att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
-
typentry = eq[attrnum];
if (typentry == NULL)
{
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 036576acab..255746094c 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -373,4 +373,55 @@ is( $result, qq(1
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
+# we fail to apply updates and deletes
+my $node_publisher_d_cols = PostgreSQL::Test::Cluster->new('node_publisher_d_cols');
+$node_publisher_d_cols->init(allows_streaming => 'logical');
+$node_publisher_d_cols->start;
+
+my $node_subscriber_d_cols = PostgreSQL::Test::Cluster->new('node_subscriber_d_cols');
+$node_subscriber_d_cols->init(allows_streaming => 'logical');
+$node_subscriber_d_cols->start;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ -- some initial data
+ INSERT INTO dropped_cols VALUES (1, 1, 1);
+));
+
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+));
+
+my $publisher_connstr_d_cols = $node_publisher_d_cols->connstr . ' dbname=postgres';
+$node_subscriber_d_cols->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub_dropped_cols CONNECTION '$publisher_connstr_d_cols' PUBLICATION pub_dropped_cols"
+);
+$node_subscriber_d_cols->wait_for_subscription_sync;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ UPDATE dropped_cols SET a = 100;
+));
+$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
+
+is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
+ qq(1), 'replication with RI FULL and dropped columns');
+
+$node_publisher_d_cols->stop('fast');
+$node_subscriber_d_cols->stop('fast');
+
done_testing();
--
2.34.1
v1-0002-Ignore-generated-columns-when-REPLICA-IDENTITY-FU.patchapplication/x-patch; name=v1-0002-Ignore-generated-columns-when-REPLICA-IDENTITY-FU.patchDownload
From b84f38deac1b02f057bfc05ff868fbf4a621b18b Mon Sep 17 00:00:00 2001
From: Onder Kalaci <onderkalaci@gmail.com>
Date: Thu, 16 Mar 2023 18:18:37 +0300
Subject: [PATCH v1 2/2] Ignore generated columns when REPLICA IDENTITY FULL
Generated columns are filled with NULL values on slot_store_data()
but not on table_scan_getnextslot(). With this commit, we skip
such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 10 +++++++++-
2 files changed, 12 insertions(+), 3 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 7c8b0d2667..482806961a 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -290,10 +290,11 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
TypeCacheEntry *typentry;
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher doesn't
+ * send those
*/
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 255746094c..96126428f3 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -387,14 +387,18 @@ $node_publisher_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a,c) VALUES (1, 1);
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
));
my $publisher_connstr_d_cols = $node_publisher_d_cols->connstr . ' dbname=postgres';
@@ -415,12 +419,16 @@ $node_subscriber_d_cols->safe_psql(
$node_publisher_d_cols->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
qq(1), 'replication with RI FULL and dropped columns');
+is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1), 'replication with RI FULL and generated columns');
+
$node_publisher_d_cols->stop('fast');
$node_subscriber_d_cols->stop('fast');
--
2.34.1
On Thu, Mar 16, 2023 at 9:33 PM Önder Kalacı <onderkalaci@gmail.com> wrote:
Hi Amit, Shi Yu,
Generated column is introduced in PG12, and I reproduced generated column problem
in PG12~PG15.
For dropped column problem, I reproduced it in PG10~PG15. (Logical replication
was introduced in PG10)
So, I'm planning to split the changes into two commits. The first one fixes
for dropped columns, and the second one adds generated columns check/test.Is that the right approach for such a case?
Works for me.
and backpatch the fix for dropped column to PG10.
Still, even the first commit fails to apply cleanly to PG12 (and below).
What is the procedure here? Should I be creating multiple patches per version?
You can first submit the fix for dropped columns with patches till
v10. Once that is committed, then you can send the patches for
generated columns.
Or does the committer prefer to handle the conflicts? Depending on your reply,
I can work on the followup.
Thanks for working on it.
--
With Regards,
Amit Kapila.
Amit Kapila <amit.kapila16@gmail.com> writes:
On Thu, Mar 16, 2023 at 9:33 PM Önder Kalacı <onderkalaci@gmail.com> wrote:
and backpatch the fix for dropped column to PG10.
You can first submit the fix for dropped columns with patches till
v10. Once that is committed, then you can send the patches for
generated columns.
Don't worry about v10 --- that's out of support and shouldn't
get patched for this.
regards, tom lane
On Fri, Mar 17, 2023 at 5:41 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Amit Kapila <amit.kapila16@gmail.com> writes:
On Thu, Mar 16, 2023 at 9:33 PM Önder Kalacı <onderkalaci@gmail.com> wrote:
and backpatch the fix for dropped column to PG10.
You can first submit the fix for dropped columns with patches till
v10. Once that is committed, then you can send the patches for
generated columns.Don't worry about v10 --- that's out of support and shouldn't
get patched for this.
Okay, thanks for reminding me.
--
With Regards,
Amit Kapila.
Hi Amit, all
You can first submit the fix for dropped columns with patches till
v10. Once that is committed, then you can send the patches for
generated columns.
Alright, attaching 2 patches for dropped columns, the names of the files
shows which
versions the patch can be applied to:
v2-0001-Ignore-dropped-columns-HEAD-REL_15-REL_14-REL_13.patch
v2-0001-Ignore-dropped-columns-REL_12-REL_11.patch
And, then on top of that, you can apply the patch for generated columns on
all applicable
versions (HEAD, 15, 14, 13 and 12). It applies cleanly. The name of the
file
is: v2-0001-Ignore-generated-columns.patch
But unfortunately I couldn't test the patch with PG 12 and below. I'm
getting some
unrelated compile errors and Postgrees CI is not available on
these versions . I'll try
to fix that, but I thought it would still be good to share the patches as
you might
already have the environment to run the tests.
Don't worry about v10 --- that's out of support and shouldn't
get patched for this.
Given this information, I skipped the v10 patch.
Thanks,
Onder KALACI
Attachments:
v2-0001-Ignore-dropped-columns-REL_12-REL_11.patchapplication/octet-stream; name=v2-0001-Ignore-dropped-columns-REL_12-REL_11.patchDownload
From dd3682982ba7d1af866f72cbf4bad7b876fa68d9 Mon Sep 17 00:00:00 2001
From: Onder Kalaci <onderkalaci@gmail.com>
Date: Fri, 17 Mar 2023 09:39:15 +0300
Subject: [PATCH v1] Ignore dropped columns when REPLICA IDENTITY FULL
Dropped columns are filled with NULL values on slot_store_data()
but not on table_scan_getnextslot(). With this commit, we skip
such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 7 ++++
src/test/subscription/t/100_bugs.pl | 51 ++++++++++++++++++++++++++
2 files changed, 58 insertions(+)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 6289322127..b3ba1aa6c8 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -242,6 +242,13 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2)
Form_pg_attribute att;
TypeCacheEntry *typentry;
+ /*
+ * Ignore dropped columns as the publisher doesn't send those
+ */
+ att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
+ if (att->attisdropped)
+ continue;
+
/*
* If one value is NULL and other is not, then they are certainly not
* equal
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index cbf46a3d33..3e7071de38 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -69,6 +69,57 @@ pass('index predicates do not cause crash');
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
+# we fail to apply updates and deletes
+my $node_publisher_d_cols = PostgreSQL::Test::Cluster->new('node_publisher_d_cols');
+$node_publisher_d_cols->init(allows_streaming => 'logical');
+$node_publisher_d_cols->start;
+
+my $node_subscriber_d_cols = PostgreSQL::Test::Cluster->new('node_subscriber_d_cols');
+$node_subscriber_d_cols->init(allows_streaming => 'logical');
+$node_subscriber_d_cols->start;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ -- some initial data
+ INSERT INTO dropped_cols VALUES (1, 1, 1);
+));
+
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+));
+
+my $publisher_connstr_d_cols = $node_publisher_d_cols->connstr . ' dbname=postgres';
+$node_subscriber_d_cols->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub_dropped_cols CONNECTION '$publisher_connstr_d_cols' PUBLICATION pub_dropped_cols"
+);
+$node_subscriber_d_cols->wait_for_subscription_sync;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ UPDATE dropped_cols SET a = 100;
+));
+$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
+
+is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
+ qq(1), 'replication with RI FULL and dropped columns');
+
+$node_publisher_d_cols->stop('fast');
+$node_subscriber_d_cols->stop('fast');
+
# Handling of temporary and unlogged tables with FOR ALL TABLES publications
--
2.34.1
v2-0001-Ignore-dropped-columns-HEAD-REL_15-REL_14-REL_13.patchapplication/octet-stream; name=v2-0001-Ignore-dropped-columns-HEAD-REL_15-REL_14-REL_13.patchDownload
From d25b2feac3b26646034bf3ace6ab93056c3c29af Mon Sep 17 00:00:00 2001
From: Onder Kalaci <onderkalaci@gmail.com>
Date: Thu, 16 Mar 2023 18:06:54 +0300
Subject: [PATCH v1 1/2] Ignore dropped columns when REPLICA IDENTITY FULL
Dropped columns are filled with NULL values on slot_store_data()
but not on table_scan_getnextslot(). With this commit, we skip
such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 9 ++++-
src/test/subscription/t/100_bugs.pl | 51 ++++++++++++++++++++++++++
2 files changed, 58 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index fa8628e3e1..7c8b0d2667 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -289,6 +289,13 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
Form_pg_attribute att;
TypeCacheEntry *typentry;
+ /*
+ * Ignore dropped columns as the publisher doesn't send those
+ */
+ att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
+ if (att->attisdropped)
+ continue;
+
/*
* If one value is NULL and other is not, then they are certainly not
* equal
@@ -302,8 +309,6 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
if (slot1->tts_isnull[attrnum] || slot2->tts_isnull[attrnum])
continue;
- att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
-
typentry = eq[attrnum];
if (typentry == NULL)
{
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 036576acab..255746094c 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -373,4 +373,55 @@ is( $result, qq(1
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
+# we fail to apply updates and deletes
+my $node_publisher_d_cols = PostgreSQL::Test::Cluster->new('node_publisher_d_cols');
+$node_publisher_d_cols->init(allows_streaming => 'logical');
+$node_publisher_d_cols->start;
+
+my $node_subscriber_d_cols = PostgreSQL::Test::Cluster->new('node_subscriber_d_cols');
+$node_subscriber_d_cols->init(allows_streaming => 'logical');
+$node_subscriber_d_cols->start;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ -- some initial data
+ INSERT INTO dropped_cols VALUES (1, 1, 1);
+));
+
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+));
+
+my $publisher_connstr_d_cols = $node_publisher_d_cols->connstr . ' dbname=postgres';
+$node_subscriber_d_cols->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub_dropped_cols CONNECTION '$publisher_connstr_d_cols' PUBLICATION pub_dropped_cols"
+);
+$node_subscriber_d_cols->wait_for_subscription_sync;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ UPDATE dropped_cols SET a = 100;
+));
+$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
+
+is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
+ qq(1), 'replication with RI FULL and dropped columns');
+
+$node_publisher_d_cols->stop('fast');
+$node_subscriber_d_cols->stop('fast');
+
done_testing();
--
2.34.1
v2-0001-Ignore-generated-columns.patchapplication/octet-stream; name=v2-0001-Ignore-generated-columns.patchDownload
From b84f38deac1b02f057bfc05ff868fbf4a621b18b Mon Sep 17 00:00:00 2001
From: Onder Kalaci <onderkalaci@gmail.com>
Date: Thu, 16 Mar 2023 18:18:37 +0300
Subject: [PATCH v1 2/2] Ignore generated columns when REPLICA IDENTITY FULL
Generated columns are filled with NULL values on slot_store_data()
but not on table_scan_getnextslot(). With this commit, we skip
such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 10 +++++++++-
2 files changed, 12 insertions(+), 3 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 7c8b0d2667..482806961a 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -290,10 +290,11 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
TypeCacheEntry *typentry;
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher doesn't
+ * send those
*/
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 255746094c..96126428f3 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -387,14 +387,18 @@ $node_publisher_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a,c) VALUES (1, 1);
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
));
my $publisher_connstr_d_cols = $node_publisher_d_cols->connstr . ' dbname=postgres';
@@ -415,12 +419,16 @@ $node_subscriber_d_cols->safe_psql(
$node_publisher_d_cols->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
qq(1), 'replication with RI FULL and dropped columns');
+is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1), 'replication with RI FULL and generated columns');
+
$node_publisher_d_cols->stop('fast');
$node_subscriber_d_cols->stop('fast');
--
2.34.1
On Friday, March 17, 2023 3:38 PM Önder Kalacı <onderkalaci@gmail.com> wrote:
Hi Amit, all
You can first submit the fix for dropped columns with patches till
v10. Once that is committed, then you can send the patches for
generated columns.Alright, attaching 2 patches for dropped columns, the names of the files shows which
versions the patch can be applied to:
v2-0001-Ignore-dropped-columns-HEAD-REL_15-REL_14-REL_13.patch
v2-0001-Ignore-dropped-columns-REL_12-REL_11.patchAnd, then on top of that, you can apply the patch for generated columns on all applicable
versions (HEAD, 15, 14, 13 and 12). It applies cleanly. The name of the file
is: v2-0001-Ignore-generated-columns.patchBut unfortunately I couldn't test the patch with PG 12 and below. I'm getting some
unrelated compile errors and Postgrees CI is not available on these versions . I'll try
to fix that, but I thought it would still be good to share the patches as you might
already have the environment to run the tests.
Thanks for updating the patch.
I couldn't apply v2-0001-Ignore-dropped-columns-HEAD-REL_15-REL_14-REL_13.patch
cleanly in v13 and v14. It looks the patch needs some changes in these versions.
```
Checking patch src/backend/executor/execReplication.c...
Hunk #1 succeeded at 243 (offset -46 lines).
Hunk #2 succeeded at 263 (offset -46 lines).
Checking patch src/test/subscription/t/100_bugs.pl...
error: while searching for:
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
done_testing();
error: patch failed: src/test/subscription/t/100_bugs.pl:373
Applied patch src/backend/executor/execReplication.c cleanly.
Applying patch src/test/subscription/t/100_bugs.pl with 1 reject...
Rejected hunk #1.
```
Besides, I tried v2-0001-Ignore-dropped-columns-REL_12-REL_11.patch in v12. The
test failed and here's some information.
```
Can't locate object method "new" via package "PostgreSQL::Test::Cluster" (perhaps you forgot to load "PostgreSQL::Test::Cluster"?) at t/100_bugs.pl line 74.
# Looks like your test exited with 2 just after 1.
```
+my $node_publisher_d_cols = PostgreSQL::Test::Cluster->new('node_publisher_d_cols');
It seems this usage is not supported in v12 and we should use get_new_node()
like other test cases.
Regards,
Shi Yu
Hi Shi Yu,
Thanks for the review, really appreciate it!
I couldn't apply
v2-0001-Ignore-dropped-columns-HEAD-REL_15-REL_14-REL_13.patch
cleanly in v13 and v14. It looks the patch needs some changes in these
versions.
```
Checking patch src/backend/executor/execReplication.c...
Hunk #1 succeeded at 243 (offset -46 lines).
Hunk #2 succeeded at 263 (offset -46 lines).
Checking patch src/test/subscription/t/100_bugs.pl...
error: while searching for:
$node_publisher->stop('fast');
$node_subscriber->stop('fast');done_testing();
error: patch failed: src/test/subscription/t/100_bugs.pl:373
Applied patch src/backend/executor/execReplication.c cleanly.
Applying patch src/test/subscription/t/100_bugs.pl with 1 reject...
Rejected hunk #1.
```
Hmm, interesting, it behaves differently on Macos and linux. Now attaching
new patches that should apply. Can you please try?
Besides, I tried v2-0001-Ignore-dropped-columns-REL_12-REL_11.patch in v12.
The
test failed and here's some information.```
Can't locate object method "new" via package "PostgreSQL::Test::Cluster"
(perhaps you forgot to load "PostgreSQL::Test::Cluster"?) at t/100_bugs.pl
line 74.
# Looks like your test exited with 2 just after 1.
```+my $node_publisher_d_cols =
PostgreSQL::Test::Cluster->new('node_publisher_d_cols');It seems this usage is not supported in v12 and we should use
get_new_node()
like other test cases.
Thanks for sharing. Fixed
This time I was able to run all the tests with all the patches applied.
Again, the generated column fix also has some minor differences
per version. So, overall we have 6 patches with very minor
differences :)
Thanks,
Onder
Attachments:
v3-0001-Ignore-dropped-columns-REL_11.patchapplication/octet-stream; name=v3-0001-Ignore-dropped-columns-REL_11.patchDownload
From 84ec2ad8c0542d9bafc8ca7c6f4c65380cb9bdc7 Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Fri, 17 Mar 2023 14:20:10 +0000
Subject: [PATCH v3] Ignore dropped columns when REPLICA IDENTITY FULL
Dropped columns are filled with NULL values on slot_store_data()
but not on table_scan_getnextslot(). With this commit, we skip
such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 9 ++++-
src/test/subscription/t/100_bugs.pl | 55 +++++++++++++++++++++++++-
2 files changed, 61 insertions(+), 3 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 108162bb35..910608e41e 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -236,6 +236,13 @@ tuple_equals_slot(TupleDesc desc, HeapTuple tup, TupleTableSlot *slot)
Form_pg_attribute att;
TypeCacheEntry *typentry;
+ /*
+ * Ignore dropped columns as the publisher doesn't send those
+ */
+ att = TupleDescAttr(desc, attrnum);
+ if (att->attisdropped)
+ continue;
+
/*
* If one value is NULL and other is not, then they are certainly not
* equal
@@ -249,8 +256,6 @@ tuple_equals_slot(TupleDesc desc, HeapTuple tup, TupleTableSlot *slot)
if (isnull[attrnum])
continue;
- att = TupleDescAttr(desc, attrnum);
-
typentry = lookup_type_cache(att->atttypid, TYPECACHE_EQ_OPR_FINFO);
if (!OidIsValid(typentry->eq_opr_finfo.fn_oid))
ereport(ERROR,
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 08e5f0bcb0..fa42fc67ea 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -3,7 +3,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 5;
+use Test::More tests => 6;
# Bug #15114
@@ -67,6 +67,59 @@ pass('index predicates do not cause crash');
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
+# we fail to apply updates and deletes
+my $node_publisher_d_cols = get_new_node('node_publisher_d_cols');
+$node_publisher_d_cols->init(allows_streaming => 'logical');
+$node_publisher_d_cols->start;
+
+my $node_subscriber_d_cols = get_new_node('node_subscriber_d_cols');
+$node_subscriber_d_cols->init(allows_streaming => 'logical');
+$node_subscriber_d_cols->start;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ -- some initial data
+ INSERT INTO dropped_cols VALUES (1, 1, 1);
+));
+
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+));
+
+my $publisher_connstr_d_cols = $node_publisher_d_cols->connstr . ' dbname=postgres';
+$node_subscriber_d_cols->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub_dropped_cols CONNECTION '$publisher_connstr_d_cols application_name=sub_dropped_cols' PUBLICATION pub_dropped_cols"
+);
+
+# Wait for initial table sync to finish
+$node_subscriber_d_cols->wait_for_subscription_sync($node_publisher_d_cols, 'sub_dropped_cols');
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ UPDATE dropped_cols SET a = 100;
+));
+$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
+
+is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
+ qq(1), 'replication with RI FULL and dropped columns');
+
+$node_publisher_d_cols->stop('fast');
+$node_subscriber_d_cols->stop('fast');
+
# Handling of temporary and unlogged tables with FOR ALL TABLES publications
--
2.31.1
v3-0001-Ignore-dropped-columns-REL_12.patchapplication/octet-stream; name=v3-0001-Ignore-dropped-columns-REL_12.patchDownload
From 19035da5586048dd5c60b5201275c70fec1941d4 Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Fri, 17 Mar 2023 14:37:30 +0000
Subject: [PATCH v3] Ignore dropped columns when REPLICA IDENTITY FULL
Dropped columns are filled with NULL values on slot_store_data()
but not on table_scan_getnextslot(). With this commit, we skip
such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 9 ++++-
src/test/subscription/t/100_bugs.pl | 55 +++++++++++++++++++++++++-
2 files changed, 61 insertions(+), 3 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 6289322127..6561d44b4f 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -242,6 +242,13 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2)
Form_pg_attribute att;
TypeCacheEntry *typentry;
+ /*
+ * Ignore dropped columns as the publisher doesn't send those
+ */
+ att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
+ if (att->attisdropped)
+ continue;
+
/*
* If one value is NULL and other is not, then they are certainly not
* equal
@@ -255,8 +262,6 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2)
if (slot1->tts_isnull[attrnum] || slot2->tts_isnull[attrnum])
continue;
- att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
-
typentry = lookup_type_cache(att->atttypid, TYPECACHE_EQ_OPR_FINFO);
if (!OidIsValid(typentry->eq_opr_finfo.fn_oid))
ereport(ERROR,
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index cbf46a3d33..0bb6677126 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -3,7 +3,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 5;
+use Test::More tests => 6;
# Bug #15114
@@ -69,6 +69,59 @@ pass('index predicates do not cause crash');
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
+# we fail to apply updates and deletes
+my $node_publisher_d_cols = get_new_node('node_publisher_d_cols');
+$node_publisher_d_cols->init(allows_streaming => 'logical');
+$node_publisher_d_cols->start;
+
+my $node_subscriber_d_cols = get_new_node('node_subscriber_d_cols');
+$node_subscriber_d_cols->init(allows_streaming => 'logical');
+$node_subscriber_d_cols->start;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ -- some initial data
+ INSERT INTO dropped_cols VALUES (1, 1, 1);
+));
+
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+));
+
+my $publisher_connstr_d_cols = $node_publisher_d_cols->connstr . ' dbname=postgres';
+$node_subscriber_d_cols->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub_dropped_cols CONNECTION '$publisher_connstr_d_cols application_name=sub_dropped_cols' PUBLICATION pub_dropped_cols"
+);
+
+# Wait for initial table sync to finish
+$node_subscriber_d_cols->wait_for_subscription_sync($node_publisher_d_cols, 'sub_dropped_cols');
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ UPDATE dropped_cols SET a = 100;
+));
+$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
+
+is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
+ qq(1), 'replication with RI FULL and dropped columns');
+
+$node_publisher_d_cols->stop('fast');
+$node_subscriber_d_cols->stop('fast');
+
# Handling of temporary and unlogged tables with FOR ALL TABLES publications
--
2.31.1
v3-0001-Ignore-dropped-columns-REL_13-REL_14.patchapplication/octet-stream; name=v3-0001-Ignore-dropped-columns-REL_13-REL_14.patchDownload
From 72cc3262d8b2edff2d36e75bad87750709e44d73 Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Fri, 17 Mar 2023 14:54:52 +0000
Subject: [PATCH v3] Ignore dropped columns when REPLICA IDENTITY FULL
Dropped columns are filled with NULL values on slot_store_data()
but not on table_scan_getnextslot(). With this commit, we skip
such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 9 ++++-
src/test/subscription/t/100_bugs.pl | 54 +++++++++++++++++++++++++-
2 files changed, 60 insertions(+), 3 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 16cab1db72..c69daf02b0 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -243,6 +243,13 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
Form_pg_attribute att;
TypeCacheEntry *typentry;
+ /*
+ * Ignore dropped columns as the publisher doesn't send those
+ */
+ att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
+ if (att->attisdropped)
+ continue;
+
/*
* If one value is NULL and other is not, then they are certainly not
* equal
@@ -256,8 +263,6 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
if (slot1->tts_isnull[attrnum] || slot2->tts_isnull[attrnum])
continue;
- att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
-
typentry = eq[attrnum];
if (typentry == NULL)
{
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index a91935e9ee..41a71ca51b 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -3,7 +3,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 7;
+use Test::More tests => 8;
# Bug #15114
@@ -227,3 +227,55 @@ is( $node_subscriber->safe_psql(
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
+
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
+# we fail to apply updates and deletes
+my $node_publisher_d_cols = get_new_node('node_publisher_d_cols');
+$node_publisher_d_cols->init(allows_streaming => 'logical');
+$node_publisher_d_cols->start;
+
+my $node_subscriber_d_cols = get_new_node('node_subscriber_d_cols');
+$node_subscriber_d_cols->init(allows_streaming => 'logical');
+$node_subscriber_d_cols->start;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ -- some initial data
+ INSERT INTO dropped_cols VALUES (1, 1, 1);
+));
+
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+));
+
+my $publisher_connstr_d_cols = $node_publisher_d_cols->connstr . ' dbname=postgres';
+$node_subscriber_d_cols->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub_dropped_cols CONNECTION '$publisher_connstr_d_cols' PUBLICATION pub_dropped_cols"
+);
+$node_subscriber_d_cols->wait_for_subscription_sync;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ UPDATE dropped_cols SET a = 100;
+));
+$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
+
+is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
+ qq(1), 'replication with RI FULL and dropped columns');
+
+$node_publisher_d_cols->stop('fast');
+$node_subscriber_d_cols->stop('fast');
+
--
2.31.1
v3-0001-Ignore-generated-columns-HEAD-REL_15.patchapplication/octet-stream; name=v3-0001-Ignore-generated-columns-HEAD-REL_15.patchDownload
From b84f38deac1b02f057bfc05ff868fbf4a621b18b Mon Sep 17 00:00:00 2001
From: Onder Kalaci <onderkalaci@gmail.com>
Date: Thu, 16 Mar 2023 18:18:37 +0300
Subject: [PATCH v1 2/2] Ignore generated columns when REPLICA IDENTITY FULL
Generated columns are filled with NULL values on slot_store_data()
but not on table_scan_getnextslot(). With this commit, we skip
such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 10 +++++++++-
2 files changed, 12 insertions(+), 3 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 7c8b0d2667..482806961a 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -290,10 +290,11 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
TypeCacheEntry *typentry;
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher doesn't
+ * send those
*/
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 255746094c..96126428f3 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -387,14 +387,18 @@ $node_publisher_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a,c) VALUES (1, 1);
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
));
my $publisher_connstr_d_cols = $node_publisher_d_cols->connstr . ' dbname=postgres';
@@ -415,12 +419,16 @@ $node_subscriber_d_cols->safe_psql(
$node_publisher_d_cols->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
qq(1), 'replication with RI FULL and dropped columns');
+is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1), 'replication with RI FULL and generated columns');
+
$node_publisher_d_cols->stop('fast');
$node_subscriber_d_cols->stop('fast');
--
2.34.1
v3-0001-Ignore-generated-columns-REL_14-REL_13.patchapplication/octet-stream; name=v3-0001-Ignore-generated-columns-REL_14-REL_13.patchDownload
From 30770630d8b96af2b3bc3253c64603dad5f87207 Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Fri, 17 Mar 2023 15:17:13 +0000
Subject: [PATCH v3] Ignore generated columns when REPLICA IDENTITY FULL
Generated columns are filled with NULL values on slot_store_data()
but not on table_scan_getnextslot(). With this commit, we skip
such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 12 ++++++++++--
2 files changed, 13 insertions(+), 4 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index c505cb1df1..672f2fa986 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -244,10 +244,11 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
TypeCacheEntry *typentry;
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher doesn't
+ * send those
*/
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 146a1732f6..0b696bb7a8 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -6,7 +6,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 8;
+use Test::More tests => 9;
# Bug #15114
@@ -313,14 +313,18 @@ $node_publisher_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a,c) VALUES (1, 1);
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
));
my $publisher_connstr_d_cols = $node_publisher_d_cols->connstr . ' dbname=postgres';
@@ -341,12 +345,16 @@ $node_subscriber_d_cols->safe_psql(
$node_publisher_d_cols->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
qq(1), 'replication with RI FULL and dropped columns');
+is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1), 'replication with RI FULL and generated columns');
+
$node_publisher_d_cols->stop('fast');
$node_subscriber_d_cols->stop('fast');
--
2.31.1
v3-0001-Ignore-generated-columns-REL_12.patchapplication/octet-stream; name=v3-0001-Ignore-generated-columns-REL_12.patchDownload
From 76c0b8d82c4b8079e776d51b7c600be6d32bafab Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Fri, 17 Mar 2023 15:24:53 +0000
Subject: [PATCH v3] w
---
src/backend/executor/execReplication.c | 9 ++++-
src/test/subscription/t/100_bugs.pl | 55 +++++++++++++++++++++++++-
2 files changed, 61 insertions(+), 3 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 6289322127..6561d44b4f 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -242,6 +242,13 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2)
Form_pg_attribute att;
TypeCacheEntry *typentry;
+ /*
+ * Ignore dropped columns as the publisher doesn't send those
+ */
+ att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
+ if (att->attisdropped)
+ continue;
+
/*
* If one value is NULL and other is not, then they are certainly not
* equal
@@ -255,8 +262,6 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2)
if (slot1->tts_isnull[attrnum] || slot2->tts_isnull[attrnum])
continue;
- att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
-
typentry = lookup_type_cache(att->atttypid, TYPECACHE_EQ_OPR_FINFO);
if (!OidIsValid(typentry->eq_opr_finfo.fn_oid))
ereport(ERROR,
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index cbf46a3d33..0bb6677126 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -3,7 +3,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 5;
+use Test::More tests => 6;
# Bug #15114
@@ -69,6 +69,59 @@ pass('index predicates do not cause crash');
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
+# we fail to apply updates and deletes
+my $node_publisher_d_cols = get_new_node('node_publisher_d_cols');
+$node_publisher_d_cols->init(allows_streaming => 'logical');
+$node_publisher_d_cols->start;
+
+my $node_subscriber_d_cols = get_new_node('node_subscriber_d_cols');
+$node_subscriber_d_cols->init(allows_streaming => 'logical');
+$node_subscriber_d_cols->start;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ -- some initial data
+ INSERT INTO dropped_cols VALUES (1, 1, 1);
+));
+
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+));
+
+my $publisher_connstr_d_cols = $node_publisher_d_cols->connstr . ' dbname=postgres';
+$node_subscriber_d_cols->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub_dropped_cols CONNECTION '$publisher_connstr_d_cols application_name=sub_dropped_cols' PUBLICATION pub_dropped_cols"
+);
+
+# Wait for initial table sync to finish
+$node_subscriber_d_cols->wait_for_subscription_sync($node_publisher_d_cols, 'sub_dropped_cols');
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ UPDATE dropped_cols SET a = 100;
+));
+$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
+
+is($node_subscriber_d_cols->safe_psql('postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
+ qq(1), 'replication with RI FULL and dropped columns');
+
+$node_publisher_d_cols->stop('fast');
+$node_subscriber_d_cols->stop('fast');
+
# Handling of temporary and unlogged tables with FOR ALL TABLES publications
--
2.31.1
On Fri, Mar 17, 2023 11:29 PM Önder Kalacı <onderkalaci@gmail.com> wrote:
Thanks for sharing. Fixed
This time I was able to run all the tests with all the patches applied.
Again, the generated column fix also has some minor differences
per version. So, overall we have 6 patches with very minor
differences :)
Thanks for updating the patches. It seems you forgot to attach the patches of
dropped columns for HEAD and pg15, I think they are the same as v2.
On HEAD, we can re-use clusters in other test cases, which can save some time.
(see fccaf259f22f4a)
In the patches for pg12 and pg11, I am not sure why not add the test at end of
the file 100_bugs.pl. I think it would be better to be consistent with other
versions.
The attached patches modify these two points. Besides, I made some minor
changes, ran pgindent and pgperltidy. These are patches for dropped columns,
because I think this would be submitted first, and we can discuss the fix for
generated columns later.
Regards,
Shi Yu
Attachments:
v4-pg12-0001-Ignore-dropped-columns-when-REPLICA-IDENTITY-FUL.patchapplication/octet-stream; name=v4-pg12-0001-Ignore-dropped-columns-when-REPLICA-IDENTITY-FUL.patchDownload
From 2630e83ff8e93ee4dea259ebee187fcb4a2a01c3 Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Fri, 17 Mar 2023 14:37:30 +0000
Subject: [PATCH v12] Ignore dropped columns when REPLICA IDENTITY FULL
Dropped columns are filled with NULL values on slot_store_data()
but not on table_scan_getnextslot(). With this commit, we skip
such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 10 ++++-
src/test/subscription/t/100_bugs.pl | 57 +++++++++++++++++++++++++-
2 files changed, 64 insertions(+), 3 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 6289322127..02b3300213 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -242,6 +242,14 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2)
Form_pg_attribute att;
TypeCacheEntry *typentry;
+ att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
+
+ /*
+ * Ignore dropped columns as the publisher doesn't send those
+ */
+ if (att->attisdropped)
+ continue;
+
/*
* If one value is NULL and other is not, then they are certainly not
* equal
@@ -255,8 +263,6 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2)
if (slot1->tts_isnull[attrnum] || slot2->tts_isnull[attrnum])
continue;
- att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
-
typentry = lookup_type_cache(att->atttypid, TYPECACHE_EQ_OPR_FINFO);
if (!OidIsValid(typentry->eq_opr_finfo.fn_oid))
ereport(ERROR,
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index cbf46a3d33..e1a04b1a93 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -3,7 +3,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 5;
+use Test::More tests => 6;
# Bug #15114
@@ -179,3 +179,58 @@ is( $node_subscriber->safe_psql(
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
+
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
+# we fail to apply updates and deletes
+my $node_publisher_d_cols = get_new_node('node_publisher_d_cols');
+$node_publisher_d_cols->init(allows_streaming => 'logical');
+$node_publisher_d_cols->start;
+
+my $node_subscriber_d_cols = get_new_node('node_subscriber_d_cols');
+$node_subscriber_d_cols->init(allows_streaming => 'logical');
+$node_subscriber_d_cols->start;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ -- some initial data
+ INSERT INTO dropped_cols VALUES (1, 1, 1);
+));
+
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+));
+
+my $publisher_connstr_d_cols =
+ $node_publisher_d_cols->connstr . ' dbname=postgres';
+$node_subscriber_d_cols->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub_dropped_cols CONNECTION '$publisher_connstr_d_cols application_name=sub_dropped_cols' PUBLICATION pub_dropped_cols"
+);
+$node_subscriber_d_cols->wait_for_subscription_sync($node_publisher_d_cols,
+ 'sub_dropped_cols');
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ UPDATE dropped_cols SET a = 100;
+));
+$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
+
+is( $node_subscriber_d_cols->safe_psql(
+ 'postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and dropped columns');
+
+$node_publisher_d_cols->stop('fast');
+$node_subscriber_d_cols->stop('fast');
--
2.31.1
v4-pg13-pg14-0001-Ignore-dropped-columns-when-REPLICA-IDENTITY-FUL.patchapplication/octet-stream; name=v4-pg13-pg14-0001-Ignore-dropped-columns-when-REPLICA-IDENTITY-FUL.patchDownload
From 5a64f6a0129350551e44f8f1fdab9fa65faf0af7 Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Fri, 17 Mar 2023 14:54:52 +0000
Subject: [PATCH v14] Ignore dropped columns when REPLICA IDENTITY FULL
Dropped columns are filled with NULL values on slot_store_data()
but not on table_scan_getnextslot(). With this commit, we skip
such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 10 ++++-
src/test/subscription/t/100_bugs.pl | 56 +++++++++++++++++++++++++-
2 files changed, 63 insertions(+), 3 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 8f116b78cc..db7414eaf7 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -243,6 +243,14 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
Form_pg_attribute att;
TypeCacheEntry *typentry;
+ att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
+
+ /*
+ * Ignore dropped columns as the publisher doesn't send those
+ */
+ if (att->attisdropped)
+ continue;
+
/*
* If one value is NULL and other is not, then they are certainly not
* equal
@@ -256,8 +264,6 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
if (slot1->tts_isnull[attrnum] || slot2->tts_isnull[attrnum])
continue;
- att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
-
typentry = eq[attrnum];
if (typentry == NULL)
{
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 91602c4339..b9da72eaf2 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -6,7 +6,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 7;
+use Test::More tests => 8;
# Bug #15114
@@ -298,3 +298,57 @@ is( $node_subscriber->safe_psql(
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
+
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
+# we fail to apply updates and deletes
+my $node_publisher_d_cols = get_new_node('node_publisher_d_cols');
+$node_publisher_d_cols->init(allows_streaming => 'logical');
+$node_publisher_d_cols->start;
+
+my $node_subscriber_d_cols = get_new_node('node_subscriber_d_cols');
+$node_subscriber_d_cols->init(allows_streaming => 'logical');
+$node_subscriber_d_cols->start;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ -- some initial data
+ INSERT INTO dropped_cols VALUES (1, 1, 1);
+));
+
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+));
+
+my $publisher_connstr_d_cols =
+ $node_publisher_d_cols->connstr . ' dbname=postgres';
+$node_subscriber_d_cols->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub_dropped_cols CONNECTION '$publisher_connstr_d_cols' PUBLICATION pub_dropped_cols"
+);
+$node_subscriber_d_cols->wait_for_subscription_sync;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ UPDATE dropped_cols SET a = 100;
+));
+$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
+
+is( $node_subscriber_d_cols->safe_psql(
+ 'postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and dropped columns');
+
+$node_publisher_d_cols->stop('fast');
+$node_subscriber_d_cols->stop('fast');
--
2.31.1
v4-pg15-0001-Ignore-dropped-columns-when-REPLICA-IDENTITY-FUL.patchapplication/octet-stream; name=v4-pg15-0001-Ignore-dropped-columns-when-REPLICA-IDENTITY-FUL.patchDownload
From 7136dc5b58ecf27866a9d4900ad362a53421d8ea Mon Sep 17 00:00:00 2001
From: Onder Kalaci <onderkalaci@gmail.com>
Date: Thu, 16 Mar 2023 18:06:54 +0300
Subject: [PATCH v15] Ignore dropped columns when REPLICA IDENTITY FULL
Dropped columns are filled with NULL values on slot_store_data()
but not on table_scan_getnextslot(). With this commit, we skip
such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 10 ++++-
src/test/subscription/t/100_bugs.pl | 56 ++++++++++++++++++++++++++
2 files changed, 64 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 6014f2e248..f37119670e 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -243,6 +243,14 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
Form_pg_attribute att;
TypeCacheEntry *typentry;
+ att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
+
+ /*
+ * Ignore dropped columns as the publisher doesn't send those
+ */
+ if (att->attisdropped)
+ continue;
+
/*
* If one value is NULL and other is not, then they are certainly not
* equal
@@ -256,8 +264,6 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
if (slot1->tts_isnull[attrnum] || slot2->tts_isnull[attrnum])
continue;
- att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
-
typentry = eq[attrnum];
if (typentry == NULL)
{
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 6247aa7730..e8ad45f426 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -299,4 +299,60 @@ is( $node_subscriber->safe_psql(
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
+# we fail to apply updates and deletes
+my $node_publisher_d_cols =
+ PostgreSQL::Test::Cluster->new('node_publisher_d_cols');
+$node_publisher_d_cols->init(allows_streaming => 'logical');
+$node_publisher_d_cols->start;
+
+my $node_subscriber_d_cols =
+ PostgreSQL::Test::Cluster->new('node_subscriber_d_cols');
+$node_subscriber_d_cols->init(allows_streaming => 'logical');
+$node_subscriber_d_cols->start;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ -- some initial data
+ INSERT INTO dropped_cols VALUES (1, 1, 1);
+));
+
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+));
+
+my $publisher_connstr_d_cols =
+ $node_publisher_d_cols->connstr . ' dbname=postgres';
+$node_subscriber_d_cols->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub_dropped_cols CONNECTION '$publisher_connstr_d_cols' PUBLICATION pub_dropped_cols"
+);
+$node_subscriber_d_cols->wait_for_subscription_sync;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ UPDATE dropped_cols SET a = 100;
+));
+$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
+
+is( $node_subscriber_d_cols->safe_psql(
+ 'postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and dropped columns');
+
+$node_publisher_d_cols->stop('fast');
+$node_subscriber_d_cols->stop('fast');
+
done_testing();
--
2.31.1
v4-HEAD-0001-Ignore-dropped-columns-when-REPLICA-IDENTITY-FULL.patchapplication/octet-stream; name=v4-HEAD-0001-Ignore-dropped-columns-when-REPLICA-IDENTITY-FULL.patchDownload
From 4d8d7ef3a565c53b7b463b609995276fc16f1f82 Mon Sep 17 00:00:00 2001
From: Onder Kalaci <onderkalaci@gmail.com>
Date: Thu, 16 Mar 2023 18:06:54 +0300
Subject: [PATCH v4] Ignore dropped columns when REPLICA IDENTITY FULL
Dropped columns are filled with NULL values on slot_store_data()
but not on table_scan_getnextslot(). With this commit, we skip
such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 10 ++++-
src/test/subscription/t/100_bugs.pl | 55 ++++++++++++++++++++++++++
2 files changed, 63 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index fa8628e3e1..652732c6ed 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -289,6 +289,14 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
Form_pg_attribute att;
TypeCacheEntry *typentry;
+ att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
+
+ /*
+ * Ignore dropped columns as the publisher doesn't send those
+ */
+ if (att->attisdropped)
+ continue;
+
/*
* If one value is NULL and other is not, then they are certainly not
* equal
@@ -302,8 +310,6 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
if (slot1->tts_isnull[attrnum] || slot2->tts_isnull[attrnum])
continue;
- att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
-
typentry = eq[attrnum];
if (typentry == NULL)
{
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 036576acab..549a1b5fe3 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -370,6 +370,61 @@ $result = $node_subscriber->safe_psql('postgres', "SELECT * FROM sch2.t1");
is( $result, qq(1
3), 'check data in subscriber sch2.t1 after schema rename');
+# Again, drop replication state but not tables.
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub_sch");
+$node_publisher->safe_psql('postgres', "DROP PUBLICATION tap_pub_sch");
+
+$node_publisher->stop('fast');
+$node_subscriber->stop('fast');
+
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
+# we fail to apply updates and deletes
+$node_publisher->rotate_logfile();
+$node_publisher->start();
+
+$node_subscriber->rotate_logfile();
+$node_subscriber->start();
+
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ -- some initial data
+ INSERT INTO dropped_cols VALUES (1, 1, 1);
+));
+
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+));
+
+$publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub_dropped_cols CONNECTION '$publisher_connstr' PUBLICATION pub_dropped_cols"
+);
+$node_subscriber->wait_for_subscription_sync;
+
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+$node_subscriber->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ UPDATE dropped_cols SET a = 100;
+));
+$node_publisher->wait_for_catchup('sub_dropped_cols');
+
+is( $node_subscriber->safe_psql(
+ 'postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and dropped columns');
+
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
--
2.31.1
v4-pg11-0001-Ignore-dropped-columns-when-REPLICA-IDENTITY-FUL.patchapplication/octet-stream; name=v4-pg11-0001-Ignore-dropped-columns-when-REPLICA-IDENTITY-FUL.patchDownload
From 5798dbd131485355ffd674dbd6aa86dedfffb032 Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Fri, 17 Mar 2023 14:20:10 +0000
Subject: [PATCH v11] Ignore dropped columns when REPLICA IDENTITY FULL
Dropped columns are filled with NULL values on slot_store_data()
but not on table_scan_getnextslot(). With this commit, we skip
such columns while checking tuple equality.
---
src/backend/executor/execReplication.c | 10 ++++-
src/test/subscription/t/100_bugs.pl | 57 +++++++++++++++++++++++++-
2 files changed, 64 insertions(+), 3 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 108162bb35..f54759425b 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -236,6 +236,14 @@ tuple_equals_slot(TupleDesc desc, HeapTuple tup, TupleTableSlot *slot)
Form_pg_attribute att;
TypeCacheEntry *typentry;
+ att = TupleDescAttr(desc, attrnum);
+
+ /*
+ * Ignore dropped columns as the publisher doesn't send those
+ */
+ if (att->attisdropped)
+ continue;
+
/*
* If one value is NULL and other is not, then they are certainly not
* equal
@@ -249,8 +257,6 @@ tuple_equals_slot(TupleDesc desc, HeapTuple tup, TupleTableSlot *slot)
if (isnull[attrnum])
continue;
- att = TupleDescAttr(desc, attrnum);
-
typentry = lookup_type_cache(att->atttypid, TYPECACHE_EQ_OPR_FINFO);
if (!OidIsValid(typentry->eq_opr_finfo.fn_oid))
ereport(ERROR,
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 08e5f0bcb0..564e099e40 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -3,7 +3,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 5;
+use Test::More tests => 6;
# Bug #15114
@@ -177,3 +177,58 @@ is( $node_subscriber->safe_psql(
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
+
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
+# we fail to apply updates and deletes
+my $node_publisher_d_cols = get_new_node('node_publisher_d_cols');
+$node_publisher_d_cols->init(allows_streaming => 'logical');
+$node_publisher_d_cols->start;
+
+my $node_subscriber_d_cols = get_new_node('node_subscriber_d_cols');
+$node_subscriber_d_cols->init(allows_streaming => 'logical');
+$node_subscriber_d_cols->start;
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ -- some initial data
+ INSERT INTO dropped_cols VALUES (1, 1, 1);
+));
+
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ CREATE TABLE dropped_cols (a int, b_drop int, c int);
+));
+
+my $publisher_connstr_d_cols =
+ $node_publisher_d_cols->connstr . ' dbname=postgres';
+$node_subscriber_d_cols->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub_dropped_cols CONNECTION '$publisher_connstr_d_cols application_name=sub_dropped_cols' PUBLICATION pub_dropped_cols"
+);
+$node_subscriber_d_cols->wait_for_subscription_sync($node_publisher_d_cols,
+ 'sub_dropped_cols');
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+$node_subscriber_d_cols->safe_psql(
+ 'postgres', qq(
+ ALTER TABLE dropped_cols DROP COLUMN b_drop;
+));
+
+$node_publisher_d_cols->safe_psql(
+ 'postgres', qq(
+ UPDATE dropped_cols SET a = 100;
+));
+$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
+
+is( $node_subscriber_d_cols->safe_psql(
+ 'postgres', "SELECT count(*) FROM dropped_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and dropped columns');
+
+$node_publisher_d_cols->stop('fast');
+$node_subscriber_d_cols->stop('fast');
--
2.30.0.windows.2
Hi Shi Yu, all
Thanks for updating the patches. It seems you forgot to attach the patches
of
dropped columns for HEAD and pg15, I think they are the same as v2.
Yes, it seems I forgot. And, yes they were the same as v2.
On HEAD, we can re-use clusters in other test cases, which can save some
time.
(see fccaf259f22f4a)
Thanks for noting.
In the patches for pg12 and pg11, I am not sure why not add the test at
end of
the file 100_bugs.pl. I think it would be better to be consistent with
other
versions.
I applied the same patch that I created for HEAD, and then the "patch"
command
created this version. Given that we are creating new patches per version, I
think
you are right, we should put them at the end.
The attached patches modify these two points. Besides, I made some minor
changes, ran pgindent and pgperltidy.
oh, I didn't know about pgperltidy. And, thanks a lot for making changes and
making it ready for the committer.
These are patches for dropped columns,
because I think this would be submitted first, and we can discuss the fix
for
generated columns later.
Makes sense, even now we have 5 different patches, lets work on generated
columns
when this is fixed.
I applied all patches for all the versions, and re-run the subscription
tests,
all looks good to me.
Thanks,
Onder KALACI
On Mon, Mar 20, 2023 at 2:58 PM Önder Kalacı <onderkalaci@gmail.com> wrote:
I applied all patches for all the versions, and re-run the subscription tests,
all looks good to me.
LGTM. I'll push this tomorrow unless there are more comments.
--
With Regards,
Amit Kapila.
On Mon, Mar 20, 2023 at 6:28 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Mar 20, 2023 at 2:58 PM Önder Kalacı <onderkalaci@gmail.com> wrote:
I applied all patches for all the versions, and re-run the subscription tests,
all looks good to me.LGTM. I'll push this tomorrow unless there are more comments.
Pushed.
--
With Regards,
Amit Kapila.
Hi Amit, Shi Yu
Now attaching the similar patches for generated columns.
Thanks,
Onder KALACI
Amit Kapila <amit.kapila16@gmail.com>, 21 Mar 2023 Sal, 09:07 tarihinde
şunu yazdı:
Show quoted text
On Mon, Mar 20, 2023 at 6:28 PM Amit Kapila <amit.kapila16@gmail.com>
wrote:On Mon, Mar 20, 2023 at 2:58 PM Önder Kalacı <onderkalaci@gmail.com>
wrote:
I applied all patches for all the versions, and re-run the
subscription tests,
all looks good to me.
LGTM. I'll push this tomorrow unless there are more comments.
Pushed.
--
With Regards,
Amit Kapila.
Attachments:
v1-0001-HEAD-Ignore-generated-columns-during-apply-of-update-d.patchapplication/octet-stream; name=v1-0001-HEAD-Ignore-generated-columns-during-apply-of-update-d.patchDownload
From 00ee6b108f816443a04c079fad92b58ef7b3e65f Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Tue, 21 Mar 2023 07:19:42 +0000
Subject: [PATCH v1] Ignore generated columns during apply of update/delete.
We fail to apply updates and deletes when the REPLICA IDENTITY FULL is
used for the table having generated columns. We didn't use to ignore
generated columns while doing tuple comparison among the tuples from
the publisher and subscriber during apply of updates and deletes.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 14 +++++++++++++-
2 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f2bf72b5ff..c73245fee5 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -292,9 +292,10 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher
+ * doesn't send those
*/
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 549a1b5fe3..fa305c3629 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -389,14 +389,18 @@ $node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a,c) VALUES (1, 1);
));
$node_subscriber->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
));
$publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
@@ -408,15 +412,18 @@ $node_subscriber->wait_for_subscription_sync;
$node_publisher->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN b_gen;
));
$node_subscriber->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN b_gen;
));
$node_publisher->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher->wait_for_catchup('sub_dropped_cols');
@@ -425,6 +432,11 @@ is( $node_subscriber->safe_psql(
qq(1),
'replication with RI FULL and dropped columns');
+is( $node_subscriber->safe_psql(
+ 'postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and generated columns');
+
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
--
2.31.1
v1-0001-REL_15-Ignore-generated-columns-during-apply-of-update-d.patchapplication/octet-stream; name=v1-0001-REL_15-Ignore-generated-columns-during-apply-of-update-d.patchDownload
From 47009a3ea990a4a35cd85aaf372820ce7072402b Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Tue, 21 Mar 2023 08:30:33 +0000
Subject: [PATCH v1] Ignore generated columns during apply of update/delete.
We fail to apply updates and deletes when the REPLICA IDENTITY FULL is
used for the table having generated columns. We didn't use to ignore
generated columns while doing tuple comparison among the tuples from
the publisher and subscriber during apply of updates and deletes.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 14 +++++++++++++-
2 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f37119670e..4003101ded 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -246,9 +246,10 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher
+ * doesn't send those
*/
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index e8ad45f426..c25971ac02 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -315,14 +315,18 @@ $node_publisher_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a,c) VALUES (1, 1);
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
));
my $publisher_connstr_d_cols =
@@ -335,15 +339,18 @@ $node_subscriber_d_cols->wait_for_subscription_sync;
$node_publisher_d_cols->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN b_gen;
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN b_gen;
));
$node_publisher_d_cols->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
@@ -352,6 +359,11 @@ is( $node_subscriber_d_cols->safe_psql(
qq(1),
'replication with RI FULL and dropped columns');
+is( $node_subscriber_d_cols->safe_psql(
+ 'postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and generated columns');
+
$node_publisher_d_cols->stop('fast');
$node_subscriber_d_cols->stop('fast');
--
2.31.1
v1-0001-REL_14-REL_13-Ignore-generated-columns-during-apply-of-update-d.patchapplication/octet-stream; name=v1-0001-REL_14-REL_13-Ignore-generated-columns-during-apply-of-update-d.patchDownload
From 59a9d19e4ded1d2b7a97efe8ea042b5b22d901f9 Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Tue, 21 Mar 2023 08:38:15 +0000
Subject: [PATCH v1] Ignore generated columns during apply of update/delete.
We fail to apply updates and deletes when the REPLICA IDENTITY FULL is
used for the table having generated columns. We didn't use to ignore
generated columns while doing tuple comparison among the tuples from
the publisher and subscriber during apply of updates and deletes.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 16 ++++++++++++++--
2 files changed, 17 insertions(+), 4 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index db7414eaf7..84f9655d9e 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -246,9 +246,10 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher
+ * doesn't send those
*/
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index b9da72eaf2..a0f5f2a0eb 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -6,7 +6,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 8;
+use Test::More tests => 9;
# Bug #15114
@@ -313,14 +313,18 @@ $node_publisher_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a,c) VALUES (1, 1);
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
));
my $publisher_connstr_d_cols =
@@ -333,15 +337,18 @@ $node_subscriber_d_cols->wait_for_subscription_sync;
$node_publisher_d_cols->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN b_gen;
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN b_gen;
));
$node_publisher_d_cols->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
@@ -350,5 +357,10 @@ is( $node_subscriber_d_cols->safe_psql(
qq(1),
'replication with RI FULL and dropped columns');
+is( $node_subscriber_d_cols->safe_psql(
+ 'postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and generated columns');
+
$node_publisher_d_cols->stop('fast');
$node_subscriber_d_cols->stop('fast');
--
2.31.1
v1-0001-REL_12-Ignore-generated-columns-during-apply-of-update-d.patchapplication/octet-stream; name=v1-0001-REL_12-Ignore-generated-columns-during-apply-of-update-d.patchDownload
From 0ded82c6e1f8a09d8032069f2663f511b22f1353 Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Tue, 21 Mar 2023 08:49:35 +0000
Subject: [PATCH v1] Ignore generated columns during apply of update/delete.
We fail to apply updates and deletes when the REPLICA IDENTITY FULL is
used for the table having generated columns. We didn't use to ignore
generated columns while doing tuple comparison among the tuples from
the publisher and subscriber during apply of updates and deletes.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 16 ++++++++++++++--
2 files changed, 17 insertions(+), 4 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 02b3300213..16d5e686b9 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -245,9 +245,10 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2)
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher
+ * doesn't send those
*/
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index e1a04b1a93..9e83e8fd21 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -3,7 +3,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 6;
+use Test::More tests => 7;
# Bug #15114
@@ -194,14 +194,18 @@ $node_publisher_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a,c) VALUES (1, 1);
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
));
my $publisher_connstr_d_cols =
@@ -215,15 +219,18 @@ $node_subscriber_d_cols->wait_for_subscription_sync($node_publisher_d_cols,
$node_publisher_d_cols->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN b_gen;
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN b_gen;
));
$node_publisher_d_cols->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
@@ -232,5 +239,10 @@ is( $node_subscriber_d_cols->safe_psql(
qq(1),
'replication with RI FULL and dropped columns');
+is( $node_subscriber_d_cols->safe_psql(
+ 'postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and generated columns');
+
$node_publisher_d_cols->stop('fast');
$node_subscriber_d_cols->stop('fast');
--
2.31.1
On Tue, Mar 21, 2023 4:51 PM Önder Kalacı <onderkalaci@gmail.com> wrote:
Hi Amit, Shi Yu
Now attaching the similar patches for generated columns.
Thanks for your patches. Here are some comments.
1.
$node_publisher->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN b_gen;
));
$node_subscriber->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN b_gen;
));
I think we want to test generated columns, so we don't need to drop columns.
Otherwise the generated column problem can't be detected.
2.
# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
# we fail to apply updates and deletes
Maybe we should mention generated columns in comment of the test.
3.
I ran pgindent and it modified some lines. Maybe we can improve the patch
as the following.
@@ -292,8 +292,8 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
/*
- * Ignore dropped and generated columns as the publisher
- * doesn't send those
+ * Ignore dropped and generated columns as the publisher doesn't send
+ * those
*/
if (att->attisdropped || att->attgenerated)
continue;
Regards,
Shi Yu
Hi Shi Yu,
1. $node_publisher->safe_psql( 'postgres', qq( ALTER TABLE dropped_cols DROP COLUMN b_drop; + ALTER TABLE generated_cols DROP COLUMN b_gen; )); $node_subscriber->safe_psql( 'postgres', qq( ALTER TABLE dropped_cols DROP COLUMN b_drop; + ALTER TABLE generated_cols DROP COLUMN b_gen; ));I think we want to test generated columns, so we don't need to drop
columns.
Otherwise the generated column problem can't be detected.
Ow, what a mistake. Now changed (and ensured that without the patch
the test fails).
2.
# The bug was that when the REPLICA IDENTITY FULL is used with dropped
columns,
# we fail to apply updates and deletesMaybe we should mention generated columns in comment of the test.
makes sense
3.
I ran pgindent and it modified some lines. Maybe we can improve the patch
as the following.@@ -292,8 +292,8 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot
*slot2,
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);/* - * Ignore dropped and generated columns as the publisher - * doesn't send those + * Ignore dropped and generated columns as the publisher doesn't send + * those */ if (att->attisdropped || att->attgenerated) continue;fixed
Attached patches again.
Thanks,
Onder KALACI
Attachments:
v1-0001-REL_12-Ignore-generated-columns-during-apply-of-update-d.patchapplication/octet-stream; name=v1-0001-REL_12-Ignore-generated-columns-during-apply-of-update-d.patchDownload
From 33f4f1ee2ddbf2409b07ed1bcb31dd84587f069e Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Tue, 21 Mar 2023 08:49:35 +0000
Subject: [PATCH v1] Ignore generated columns during apply of update/delete.
We fail to apply updates and deletes when the REPLICA IDENTITY FULL is
used for the table having generated columns. We didn't use to ignore
generated columns while doing tuple comparison among the tuples from
the publisher and subscriber during apply of updates and deletes.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 20 ++++++++++++++++----
2 files changed, 19 insertions(+), 6 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 02b3300213..72ce9fa1a2 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -245,9 +245,10 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2)
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher doesn't send
+ * those
*/
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index e1a04b1a93..f6589be443 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -3,7 +3,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 6;
+use Test::More tests => 7;
# Bug #15114
@@ -180,8 +180,8 @@ is( $node_subscriber->safe_psql(
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
-# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
-# we fail to apply updates and deletes
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped or
+# generated columns, we fail to apply updates and deletes
my $node_publisher_d_cols = get_new_node('node_publisher_d_cols');
$node_publisher_d_cols->init(allows_streaming => 'logical');
$node_publisher_d_cols->start;
@@ -194,14 +194,18 @@ $node_publisher_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c_drop int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a, c_drop) VALUES (1, 1);
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c_drop int);
));
my $publisher_connstr_d_cols =
@@ -215,15 +219,18 @@ $node_subscriber_d_cols->wait_for_subscription_sync($node_publisher_d_cols,
$node_publisher_d_cols->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN c_drop;
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN c_drop;
));
$node_publisher_d_cols->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
@@ -232,5 +239,10 @@ is( $node_subscriber_d_cols->safe_psql(
qq(1),
'replication with RI FULL and dropped columns');
+is( $node_subscriber_d_cols->safe_psql(
+ 'postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and generated columns');
+
$node_publisher_d_cols->stop('fast');
$node_subscriber_d_cols->stop('fast');
--
2.31.1
v1-0001-REL_15-Ignore-generated-columns-during-apply-of-update-d.patchapplication/octet-stream; name=v1-0001-REL_15-Ignore-generated-columns-during-apply-of-update-d.patchDownload
From a610baa315eb096238fea37ba59e591deb73e9c0 Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Tue, 21 Mar 2023 08:30:33 +0000
Subject: [PATCH v1] Ignore generated columns during apply of update/delete.
We fail to apply updates and deletes when the REPLICA IDENTITY FULL is
used for the table having generated columns. We didn't use to ignore
generated columns while doing tuple comparison among the tuples from
the publisher and subscriber during apply of updates and deletes.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 18 +++++++++++++++---
2 files changed, 18 insertions(+), 5 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f37119670e..125f1367d1 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -246,9 +246,10 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher doesn't send
+ * those
*/
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index e8ad45f426..e4ddd9dac4 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -299,8 +299,8 @@ is( $node_subscriber->safe_psql(
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
-# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
-# we fail to apply updates and deletes
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped or
+# generated columns, we fail to apply updates and deletes
my $node_publisher_d_cols =
PostgreSQL::Test::Cluster->new('node_publisher_d_cols');
$node_publisher_d_cols->init(allows_streaming => 'logical');
@@ -315,14 +315,18 @@ $node_publisher_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c_drop int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a, c_drop) VALUES (1, 1);
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c_drop int);
));
my $publisher_connstr_d_cols =
@@ -335,15 +339,18 @@ $node_subscriber_d_cols->wait_for_subscription_sync;
$node_publisher_d_cols->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN c_drop;
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN c_drop;
));
$node_publisher_d_cols->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
@@ -352,6 +359,11 @@ is( $node_subscriber_d_cols->safe_psql(
qq(1),
'replication with RI FULL and dropped columns');
+is( $node_subscriber_d_cols->safe_psql(
+ 'postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and generated columns');
+
$node_publisher_d_cols->stop('fast');
$node_subscriber_d_cols->stop('fast');
--
2.31.1
v1-0001-REL_14-REL_13-Ignore-generated-columns-during-apply-of-update-d.patchapplication/octet-stream; name=v1-0001-REL_14-REL_13-Ignore-generated-columns-during-apply-of-update-d.patchDownload
From ba063f676d2967db6dcb61a29c2944cc987a501f Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Tue, 21 Mar 2023 11:41:05 +0000
Subject: [PATCH v1] Ignore generated columns during apply of update/delete.
We fail to apply updates and deletes when the REPLICA IDENTITY FULL is
used for the table having generated columns. We didn't use to ignore
generated columns while doing tuple comparison among the tuples from
the publisher and subscriber during apply of updates and deletes.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 20 ++++++++++++++++----
2 files changed, 19 insertions(+), 6 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 6c3a93bd3b..43ec111c37 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -246,9 +246,10 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher doesn't send
+ * those
*/
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index a11f6be462..b8e450fcc0 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -3,7 +3,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 8;
+use Test::More tests => 9;
# Bug #15114
@@ -228,8 +228,8 @@ is( $node_subscriber->safe_psql(
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
-# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
-# we fail to apply updates and deletes
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped or
+# generated columns, we fail to apply updates and deletes
my $node_publisher_d_cols = get_new_node('node_publisher_d_cols');
$node_publisher_d_cols->init(allows_streaming => 'logical');
$node_publisher_d_cols->start;
@@ -242,14 +242,18 @@ $node_publisher_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c_drop int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a, c_drop) VALUES (1, 1);
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c_drop int);
));
my $publisher_connstr_d_cols =
@@ -262,15 +266,18 @@ $node_subscriber_d_cols->wait_for_subscription_sync;
$node_publisher_d_cols->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN c_drop;
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN c_drop;
));
$node_publisher_d_cols->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
@@ -279,5 +286,10 @@ is( $node_subscriber_d_cols->safe_psql(
qq(1),
'replication with RI FULL and dropped columns');
+is( $node_subscriber_d_cols->safe_psql(
+ 'postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and generated columns');
+
$node_publisher_d_cols->stop('fast');
$node_subscriber_d_cols->stop('fast');
--
2.31.1
v1-0001-HEAD-Ignore-generated-columns-during-apply-of-update-d.patchapplication/octet-stream; name=v1-0001-HEAD-Ignore-generated-columns-during-apply-of-update-d.patchDownload
From 0bf29056ddab8c3308df4f83900754807bc51acb Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Tue, 21 Mar 2023 07:19:42 +0000
Subject: [PATCH v1] Ignore generated columns during apply of update/delete.
We fail to apply updates and deletes when the REPLICA IDENTITY FULL is
used for the table having generated columns. We didn't use to ignore
generated columns while doing tuple comparison among the tuples from
the publisher and subscriber during apply of updates and deletes.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 18 +++++++++++++++---
2 files changed, 18 insertions(+), 5 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f2bf72b5ff..349bed0f5d 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -292,9 +292,10 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher doesn't send
+ * those
*/
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 549a1b5fe3..03e8bba2e1 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -377,8 +377,8 @@ $node_publisher->safe_psql('postgres', "DROP PUBLICATION tap_pub_sch");
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
-# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
-# we fail to apply updates and deletes
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped or
+# generated columns, we fail to apply updates and deletes
$node_publisher->rotate_logfile();
$node_publisher->start();
@@ -389,14 +389,18 @@ $node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c_drop int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a, c_drop) VALUES (1, 1);
));
$node_subscriber->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c_drop int);
));
$publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
@@ -408,15 +412,18 @@ $node_subscriber->wait_for_subscription_sync;
$node_publisher->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN c_drop;
));
$node_subscriber->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN c_drop;
));
$node_publisher->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher->wait_for_catchup('sub_dropped_cols');
@@ -425,6 +432,11 @@ is( $node_subscriber->safe_psql(
qq(1),
'replication with RI FULL and dropped columns');
+is( $node_subscriber->safe_psql(
+ 'postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and generated columns');
+
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
--
2.31.1
On Tuesday, March 21, 2023 8:03 PM Önder Kalacı <onderkalaci@gmail.com> wrote:
Attached patches again.
Thanks for updating the patch.
@@ -408,15 +412,18 @@ $node_subscriber->wait_for_subscription_sync;
$node_publisher->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN c_drop;
));
$node_subscriber->safe_psql(
'postgres', qq(
ALTER TABLE dropped_cols DROP COLUMN b_drop;
+ ALTER TABLE generated_cols DROP COLUMN c_drop;
));
Is there any reasons why we drop column here? Dropped column case has been
tested on table dropped_cols. The generated column problem can be detected
without dropping columns on my machine.
Regards,
Shi Yu
Hi Shi Yu,
Is there any reasons why we drop column here? Dropped column case has been
tested on table dropped_cols. The generated column problem can be detected
without dropping columns on my machine.
We don't really need to, if you check the first patch, we don't have DROP
for generated case. I mostly
wanted to make the test a little more interesting, but it also seems to be
a little confusing.
Now attaching v2 where we do not drop the columns. I don't have strong
preference on
which patch to proceed with, mostly wanted to attach this version to
progress faster (in case
you/Amit considers this one better).
Thanks,
Onder
Attachments:
v2-0001-REL_12-Ignore-generated-columns-during-apply-of-update-d.patchapplication/octet-stream; name=v2-0001-REL_12-Ignore-generated-columns-during-apply-of-update-d.patchDownload
From f2f2506542273fdfa7de67d6553db2c53b0d9cc6 Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Tue, 21 Mar 2023 08:49:35 +0000
Subject: [PATCH v2] Ignore generated columns during apply of update/delete.
We fail to apply updates and deletes when the REPLICA IDENTITY FULL is
used for the table having generated columns. We didn't use to ignore
generated columns while doing tuple comparison among the tuples from
the publisher and subscriber during apply of updates and deletes.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 18 ++++++++++++++----
2 files changed, 17 insertions(+), 6 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 02b3300213..72ce9fa1a2 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -245,9 +245,10 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2)
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher doesn't send
+ * those
*/
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index e1a04b1a93..f5968ffa97 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -3,7 +3,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 6;
+use Test::More tests => 7;
# Bug #15114
@@ -180,8 +180,8 @@ is( $node_subscriber->safe_psql(
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
-# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
-# we fail to apply updates and deletes
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped or
+# generated columns, we fail to apply updates and deletes
my $node_publisher_d_cols = get_new_node('node_publisher_d_cols');
$node_publisher_d_cols->init(allows_streaming => 'logical');
$node_publisher_d_cols->start;
@@ -194,14 +194,18 @@ $node_publisher_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a, c) VALUES (1, 1);
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
));
my $publisher_connstr_d_cols =
@@ -224,6 +228,7 @@ $node_subscriber_d_cols->safe_psql(
$node_publisher_d_cols->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
@@ -232,5 +237,10 @@ is( $node_subscriber_d_cols->safe_psql(
qq(1),
'replication with RI FULL and dropped columns');
+is( $node_subscriber_d_cols->safe_psql(
+ 'postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and generated columns');
+
$node_publisher_d_cols->stop('fast');
$node_subscriber_d_cols->stop('fast');
--
2.31.1
v2-0001-HEAD-Ignore-generated-columns-during-apply-of-update-d.patchapplication/octet-stream; name=v2-0001-HEAD-Ignore-generated-columns-during-apply-of-update-d.patchDownload
From 865e1f2ac1e5140f2d320d6c55ec2fddaa215a35 Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Tue, 21 Mar 2023 07:19:42 +0000
Subject: [PATCH v2] Ignore generated columns during apply of update/delete.
We fail to apply updates and deletes when the REPLICA IDENTITY FULL is
used for the table having generated columns. We didn't use to ignore
generated columns while doing tuple comparison among the tuples from
the publisher and subscriber during apply of updates and deletes.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 16 +++++++++++++---
2 files changed, 16 insertions(+), 5 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f2bf72b5ff..349bed0f5d 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -292,9 +292,10 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher doesn't send
+ * those
*/
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 549a1b5fe3..b832ddcf63 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -377,8 +377,8 @@ $node_publisher->safe_psql('postgres', "DROP PUBLICATION tap_pub_sch");
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
-# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
-# we fail to apply updates and deletes
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped or
+# generated columns, we fail to apply updates and deletes
$node_publisher->rotate_logfile();
$node_publisher->start();
@@ -389,14 +389,18 @@ $node_publisher->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a, c) VALUES (1, 1);
));
$node_subscriber->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
));
$publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
@@ -417,6 +421,7 @@ $node_subscriber->safe_psql(
$node_publisher->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher->wait_for_catchup('sub_dropped_cols');
@@ -425,6 +430,11 @@ is( $node_subscriber->safe_psql(
qq(1),
'replication with RI FULL and dropped columns');
+is( $node_subscriber->safe_psql(
+ 'postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and generated columns');
+
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
--
2.31.1
v2-0001-REL_15-Ignore-generated-columns-during-apply-of-update-d.patchapplication/octet-stream; name=v2-0001-REL_15-Ignore-generated-columns-during-apply-of-update-d.patchDownload
From 575d0a6ab9f62b693c685ba9e08f8bd95a5561f6 Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Tue, 21 Mar 2023 08:30:33 +0000
Subject: [PATCH v2] Ignore generated columns during apply of update/delete.
We fail to apply updates and deletes when the REPLICA IDENTITY FULL is
used for the table having generated columns. We didn't use to ignore
generated columns while doing tuple comparison among the tuples from
the publisher and subscriber during apply of updates and deletes.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 16 +++++++++++++---
2 files changed, 16 insertions(+), 5 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f37119670e..125f1367d1 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -246,9 +246,10 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher doesn't send
+ * those
*/
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index e8ad45f426..61c9f538e3 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -299,8 +299,8 @@ is( $node_subscriber->safe_psql(
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
-# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
-# we fail to apply updates and deletes
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped or
+# generated columns, we fail to apply updates and deletes
my $node_publisher_d_cols =
PostgreSQL::Test::Cluster->new('node_publisher_d_cols');
$node_publisher_d_cols->init(allows_streaming => 'logical');
@@ -315,14 +315,18 @@ $node_publisher_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a, c) VALUES (1, 1);
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
));
my $publisher_connstr_d_cols =
@@ -344,6 +348,7 @@ $node_subscriber_d_cols->safe_psql(
$node_publisher_d_cols->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
@@ -352,6 +357,11 @@ is( $node_subscriber_d_cols->safe_psql(
qq(1),
'replication with RI FULL and dropped columns');
+is( $node_subscriber_d_cols->safe_psql(
+ 'postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and generated columns');
+
$node_publisher_d_cols->stop('fast');
$node_subscriber_d_cols->stop('fast');
--
2.31.1
v2-0001-REL_14-REL_13-Ignore-generated-columns-during-apply-of-update-d.patchapplication/octet-stream; name=v2-0001-REL_14-REL_13-Ignore-generated-columns-during-apply-of-update-d.patchDownload
From 4d6accb49d7012ad9bc6e739a22a8abab5797488 Mon Sep 17 00:00:00 2001
From: Cloud User
<pguser@citustestclustervm0.qn5vgvlwtv0erjkuazj1o4q4gb.bx.internal.cloudapp.net>
Date: Tue, 21 Mar 2023 11:41:05 +0000
Subject: [PATCH v2] Ignore generated columns during apply of update/delete.
We fail to apply updates and deletes when the REPLICA IDENTITY FULL is
used for the table having generated columns. We didn't use to ignore
generated columns while doing tuple comparison among the tuples from
the publisher and subscriber during apply of updates and deletes.
---
src/backend/executor/execReplication.c | 5 +++--
src/test/subscription/t/100_bugs.pl | 18 ++++++++++++++----
2 files changed, 17 insertions(+), 6 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 6c3a93bd3b..43ec111c37 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -246,9 +246,10 @@ tuples_equal(TupleTableSlot *slot1, TupleTableSlot *slot2,
att = TupleDescAttr(slot1->tts_tupleDescriptor, attrnum);
/*
- * Ignore dropped columns as the publisher doesn't send those
+ * Ignore dropped and generated columns as the publisher doesn't send
+ * those
*/
- if (att->attisdropped)
+ if (att->attisdropped || att->attgenerated)
continue;
/*
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index a11f6be462..7ebb97bbcf 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -3,7 +3,7 @@ use strict;
use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 8;
+use Test::More tests => 9;
# Bug #15114
@@ -228,8 +228,8 @@ is( $node_subscriber->safe_psql(
$node_publisher->stop('fast');
$node_subscriber->stop('fast');
-# The bug was that when the REPLICA IDENTITY FULL is used with dropped columns,
-# we fail to apply updates and deletes
+# The bug was that when the REPLICA IDENTITY FULL is used with dropped or
+# generated columns, we fail to apply updates and deletes
my $node_publisher_d_cols = get_new_node('node_publisher_d_cols');
$node_publisher_d_cols->init(allows_streaming => 'logical');
$node_publisher_d_cols->start;
@@ -242,14 +242,18 @@ $node_publisher_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
ALTER TABLE dropped_cols REPLICA IDENTITY FULL;
- CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols;
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
+ ALTER TABLE generated_cols REPLICA IDENTITY FULL;
+ CREATE PUBLICATION pub_dropped_cols FOR TABLE dropped_cols, generated_cols;
-- some initial data
INSERT INTO dropped_cols VALUES (1, 1, 1);
+ INSERT INTO generated_cols (a, c) VALUES (1, 1);
));
$node_subscriber_d_cols->safe_psql(
'postgres', qq(
CREATE TABLE dropped_cols (a int, b_drop int, c int);
+ CREATE TABLE generated_cols (a int, b_gen int GENERATED ALWAYS AS (5 * a) STORED, c int);
));
my $publisher_connstr_d_cols =
@@ -271,6 +275,7 @@ $node_subscriber_d_cols->safe_psql(
$node_publisher_d_cols->safe_psql(
'postgres', qq(
UPDATE dropped_cols SET a = 100;
+ UPDATE generated_cols SET a = 100;
));
$node_publisher_d_cols->wait_for_catchup('sub_dropped_cols');
@@ -279,5 +284,10 @@ is( $node_subscriber_d_cols->safe_psql(
qq(1),
'replication with RI FULL and dropped columns');
+is( $node_subscriber_d_cols->safe_psql(
+ 'postgres', "SELECT count(*) FROM generated_cols WHERE a = 100"),
+ qq(1),
+ 'replication with RI FULL and generated columns');
+
$node_publisher_d_cols->stop('fast');
$node_subscriber_d_cols->stop('fast');
--
2.31.1
On Wed, Mar 22, 2023 2:53 PM Önder Kalacı <onderkalaci@gmail.com> wrote:
We don't really need to, if you check the first patch, we don't have DROP for generated case. I mostly
wanted to make the test a little more interesting, but it also seems to be a little confusing.Now attaching v2 where we do not drop the columns. I don't have strong preference on
which patch to proceed with, mostly wanted to attach this version to progress faster (in case
you/Amit considers this one better).
Thanks for updating the patches.
The v2 patch LGTM.
Regards,
Shi Yu
On Wed, Mar 22, 2023 at 1:39 PM shiy.fnst@fujitsu.com
<shiy.fnst@fujitsu.com> wrote:
On Wed, Mar 22, 2023 2:53 PM Önder Kalacı <onderkalaci@gmail.com> wrote:
We don't really need to, if you check the first patch, we don't have DROP for generated case. I mostly
wanted to make the test a little more interesting, but it also seems to be a little confusing.Now attaching v2 where we do not drop the columns. I don't have strong preference on
which patch to proceed with, mostly wanted to attach this version to progress faster (in case
you/Amit considers this one better).Thanks for updating the patches.
The v2 patch LGTM.
Pushed.
--
With Regards,
Amit Kapila.