[BUG]Missing REPLICA IDENTITY check when DROP NOT NULL

Started by tanghy.fnst@fujitsu.comabout 4 years ago5 messages
#1tanghy.fnst@fujitsu.com
tanghy.fnst@fujitsu.com
1 attachment(s)

Hi,

I think I found a problem related to replica identity. According to PG doc at [1]https://www.postgresql.org/docs/current/sql-altertable.html, replica identity includes only columns marked NOT NULL.
But in fact users can accidentally break this rule as follows:

create table tbl (a int not null unique);
alter table tbl replica identity using INDEX tbl_a_key;
alter table tbl alter column a drop not null;
insert into tbl values (null);

As a result, some operations on newly added null value will cause unexpected failure as below:

postgres=# delete from tbl;
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.

In the log, I can also see an assertion failure when deleting null value:
TRAP: FailedAssertion("!nulls[i]", File: "heapam.c", Line: 8439, PID: 274656)

To solve the above problem, I think it's better to add a check when executing ALTER COLUMN DROP NOT NULL,
and report an error if this column is part of replica identity.

Attaching a patch that disallow DROP NOT NULL on a column if it's in a REPLICA IDENTITY index. Also added a test in it.
Thanks Hou for helping me write/review this patch.

By the way, replica identity was introduced in PG9.4, so this problem exists in
all supported versions.

[1]: https://www.postgresql.org/docs/current/sql-altertable.html

Regards,
Tang

Attachments:

0001-Disallow-DROP-NOT-NULL-for-column-in-the-REPLICA-IDE.patchapplication/octet-stream; name=0001-Disallow-DROP-NOT-NULL-for-column-in-the-REPLICA-IDE.patchDownload
From 954d1c239c7c0e20e5d4eda937860430d94e827e Mon Sep 17 00:00:00 2001
From: tanghy <tanghy.fnst@fujitsu.com>
Date: Wed, 24 Nov 2021 13:12:05 +0800
Subject: [PATCH] Disallow DROP NOT NULL on a column in the REPLICA IDENTITY
 index

Disallow DROP NOT NULL on a column if it's in a REPLICA IDENTITY index.

Author: Haiying Tang, Hou Zhijie
Reviewed-by: Hou Zhijie
---
 src/backend/commands/tablecmds.c               | 15 ++++++++++-----
 src/test/regress/expected/replica_identity.out |  6 ++++++
 src/test/regress/sql/replica_identity.sql      |  6 ++++++
 3 files changed, 22 insertions(+), 5 deletions(-)

diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 785b282..4c24ff1 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -7164,19 +7164,24 @@ ATExecDropNotNull(Relation rel, const char *colName, LOCKMODE lockmode)
 			elog(ERROR, "cache lookup failed for index %u", indexoid);
 		indexStruct = (Form_pg_index) GETSTRUCT(indexTuple);
 
-		/* If the index is not a primary key, skip the check */
-		if (indexStruct->indisprimary)
+		/*
+		 * If the index is not a primary key or REPLICA IDENTITY index, skip
+		 * the check
+		 */
+		if (indexStruct->indisprimary || indexStruct->indisreplident)
 		{
 			/*
-			 * Loop over each attribute in the primary key and see if it
-			 * matches the to-be-altered attribute
+			 * Loop over each attribute in the primary key or REPLICA IDENTITY
+			 * index and see if it matches the to-be-altered attribute
 			 */
 			for (i = 0; i < indexStruct->indnkeyatts; i++)
 			{
 				if (indexStruct->indkey.values[i] == attnum)
 					ereport(ERROR,
 							(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
-							 errmsg("column \"%s\" is in a primary key",
+							 errmsg(ngettext("column \"%s\" is in a primary key",
+											 "column \"%s\" is in a REPLICA IDENTITY index",
+											 indexStruct->indisprimary),
 									colName)));
 			}
 		}
diff --git a/src/test/regress/expected/replica_identity.out b/src/test/regress/expected/replica_identity.out
index 7900219..8bbf52f 100644
--- a/src/test/regress/expected/replica_identity.out
+++ b/src/test/regress/expected/replica_identity.out
@@ -223,6 +223,12 @@ ALTER TABLE test_replica_identity3 ALTER COLUMN id TYPE bigint;
 Indexes:
     "test_replica_identity3_id_key" UNIQUE, btree (id) REPLICA IDENTITY
 
+---
+-- Test that ALTER TABLE DROP NOT NULL is not allowed for the columns in
+-- replica identity index
+---
+ALTER TABLE test_replica_identity3 ALTER COLUMN id DROP NOT NULL;
+ERROR:  column "id" is in a REPLICA IDENTITY index
 DROP TABLE test_replica_identity;
 DROP TABLE test_replica_identity2;
 DROP TABLE test_replica_identity3;
diff --git a/src/test/regress/sql/replica_identity.sql b/src/test/regress/sql/replica_identity.sql
index a5ac8f5..9684af2 100644
--- a/src/test/regress/sql/replica_identity.sql
+++ b/src/test/regress/sql/replica_identity.sql
@@ -94,6 +94,12 @@ ALTER TABLE test_replica_identity3 REPLICA IDENTITY USING INDEX test_replica_ide
 ALTER TABLE test_replica_identity3 ALTER COLUMN id TYPE bigint;
 \d test_replica_identity3
 
+---
+-- Test that ALTER TABLE DROP NOT NULL is not allowed for the columns in
+-- replica identity index
+---
+ALTER TABLE test_replica_identity3 ALTER COLUMN id DROP NOT NULL;
+
 DROP TABLE test_replica_identity;
 DROP TABLE test_replica_identity2;
 DROP TABLE test_replica_identity3;
-- 
2.7.2.windows.1

#2Michael Paquier
michael@paquier.xyz
In reply to: tanghy.fnst@fujitsu.com (#1)
Re: [BUG]Missing REPLICA IDENTITY check when DROP NOT NULL

On Wed, Nov 24, 2021 at 07:04:51AM +0000, tanghy.fnst@fujitsu.com wrote:

create table tbl (a int not null unique);
alter table tbl replica identity using INDEX tbl_a_key;
alter table tbl alter column a drop not null;
insert into tbl values (null);

Oops. Yes, that's obviously not good.

To solve the above problem, I think it's better to add a check when
executing ALTER COLUMN DROP NOT NULL,
and report an error if this column is part of replica identity.

I'd say that you are right to block the operation. I'll try to play a
bit with this stuff tomorrow.

Attaching a patch that disallow DROP NOT NULL on a column if it's in
a REPLICA IDENTITY index. Also added a test in it.

       if (indexStruct->indkey.values[i] == attnum)
           ereport(ERROR,
                   (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
-                   errmsg("column \"%s\" is in a primary key",
+                   errmsg(ngettext("column \"%s\" is in a primary key",
+                                   "column \"%s\" is in a REPLICA IDENTITY index",
+                                   indexStruct->indisprimary),
                    colName)));
Using ngettext() looks incorrect to me here as it is used to get the
plural form of a string, so you'd better make these completely
separated instead.
--
Michael
#3tanghy.fnst@fujitsu.com
tanghy.fnst@fujitsu.com
In reply to: Michael Paquier (#2)
1 attachment(s)
RE: [BUG]Missing REPLICA IDENTITY check when DROP NOT NULL

On Wed, Nov 24, 2021 7:29 PM, Michael Paquier <michael@paquier.xyz> wrote:

On Wed, Nov 24, 2021 at 07:04:51AM +0000, tanghy.fnst@fujitsu.com wrote:

create table tbl (a int not null unique);
alter table tbl replica identity using INDEX tbl_a_key;
alter table tbl alter column a drop not null;
insert into tbl values (null);

Oops. Yes, that's obviously not good.

To solve the above problem, I think it's better to add a check when
executing ALTER COLUMN DROP NOT NULL,
and report an error if this column is part of replica identity.

I'd say that you are right to block the operation. I'll try to play a
bit with this stuff tomorrow.

Attaching a patch that disallow DROP NOT NULL on a column if it's in
a REPLICA IDENTITY index. Also added a test in it.

if (indexStruct->indkey.values[i] == attnum)
ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
-                   errmsg("column \"%s\" is in a primary key",
+                   errmsg(ngettext("column \"%s\" is in a primary key",
+                                   "column \"%s\" is in a REPLICA IDENTITY index",
+                                   indexStruct->indisprimary),
colName)));
Using ngettext() looks incorrect to me here as it is used to get the
plural form of a string, so you'd better make these completely
separated instead.

Thanks for your comment. I agree with you.
I have fixed it and attached v2 patch.

Regards,
Tang

Attachments:

v2-0001-Disallow-DROP-NOT-NULL-on-a-column-in-the-REPLICA.patchapplication/octet-stream; name=v2-0001-Disallow-DROP-NOT-NULL-on-a-column-in-the-REPLICA.patchDownload
From e641a2a8e4e958241942e289f10ca52c791c91b5 Mon Sep 17 00:00:00 2001
From: tanghy <tanghy.fnst@fujitsu.com>
Date: Wed, 24 Nov 2021 13:12:05 +0800
Subject: [PATCH v2] Disallow DROP NOT NULL on a column in the REPLICA IDENTITY
 index

Disallow DROP NOT NULL on a column if it's in a REPLICA IDENTITY index.

Author: Haiying Tang, Hou Zhijie
Reviewed-by: Hou Zhijie
---
 src/backend/commands/tablecmds.c              | 31 ++++++++++++++-----
 .../regress/expected/replica_identity.out     |  6 ++++
 src/test/regress/sql/replica_identity.sql     |  6 ++++
 3 files changed, 35 insertions(+), 8 deletions(-)

diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 785b282e69..109ec7c180 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -7164,20 +7164,35 @@ ATExecDropNotNull(Relation rel, const char *colName, LOCKMODE lockmode)
 			elog(ERROR, "cache lookup failed for index %u", indexoid);
 		indexStruct = (Form_pg_index) GETSTRUCT(indexTuple);
 
-		/* If the index is not a primary key, skip the check */
-		if (indexStruct->indisprimary)
+		/*
+		 * If the index is not a primary key or REPLICA IDENTITY index, skip
+		 * the check
+		 */
+		if (indexStruct->indisprimary || indexStruct->indisreplident)
 		{
 			/*
-			 * Loop over each attribute in the primary key and see if it
-			 * matches the to-be-altered attribute
+			 * Loop over each attribute in the primary key or REPLICA IDENTITY
+			 * index and see if it matches the to-be-altered attribute
 			 */
 			for (i = 0; i < indexStruct->indnkeyatts; i++)
 			{
 				if (indexStruct->indkey.values[i] == attnum)
-					ereport(ERROR,
-							(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
-							 errmsg("column \"%s\" is in a primary key",
-									colName)));
+				{
+					if (indexStruct->indisprimary)
+					{
+						ereport(ERROR,
+								(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+								 errmsg("column \"%s\" is in a primary key",
+										colName)));
+					}
+					else
+					{
+						ereport(ERROR,
+								(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+								 errmsg("column \"%s\" is in a REPLICA IDENTITY index",
+										colName)));
+					}
+				}
 			}
 		}
 
diff --git a/src/test/regress/expected/replica_identity.out b/src/test/regress/expected/replica_identity.out
index 79002197a7..2a6ecbbf36 100644
--- a/src/test/regress/expected/replica_identity.out
+++ b/src/test/regress/expected/replica_identity.out
@@ -223,6 +223,12 @@ ALTER TABLE test_replica_identity3 ALTER COLUMN id TYPE bigint;
 Indexes:
     "test_replica_identity3_id_key" UNIQUE, btree (id) REPLICA IDENTITY
 
+---
+-- Test that ALTER TABLE DROP NOT NULL is not allowed for the columns in
+-- replica identity index
+---
+ALTER TABLE test_replica_identity3 ALTER COLUMN id DROP NOT NULL;
+ERROR:  column "id" is in a REPLICA IDENTITY index
 DROP TABLE test_replica_identity;
 DROP TABLE test_replica_identity2;
 DROP TABLE test_replica_identity3;
diff --git a/src/test/regress/sql/replica_identity.sql b/src/test/regress/sql/replica_identity.sql
index a5ac8f5567..3ba0e978d5 100644
--- a/src/test/regress/sql/replica_identity.sql
+++ b/src/test/regress/sql/replica_identity.sql
@@ -94,6 +94,12 @@ ALTER TABLE test_replica_identity3 REPLICA IDENTITY USING INDEX test_replica_ide
 ALTER TABLE test_replica_identity3 ALTER COLUMN id TYPE bigint;
 \d test_replica_identity3
 
+---
+-- Test that ALTER TABLE DROP NOT NULL is not allowed for the columns in
+-- replica identity index
+---
+ALTER TABLE test_replica_identity3 ALTER COLUMN id DROP NOT NULL;
+
 DROP TABLE test_replica_identity;
 DROP TABLE test_replica_identity2;
 DROP TABLE test_replica_identity3;
-- 
2.18.4

#4Dilip Kumar
dilipbalaut@gmail.com
In reply to: tanghy.fnst@fujitsu.com (#3)
Re: [BUG]Missing REPLICA IDENTITY check when DROP NOT NULL

On Thu, Nov 25, 2021 at 8:21 AM tanghy.fnst@fujitsu.com
<tanghy.fnst@fujitsu.com> wrote:

On Wed, Nov 24, 2021 7:29 PM, Michael Paquier <michael@paquier.xyz> wrote:

Thanks for your comment. I agree with you.
I have fixed it and attached v2 patch.

Good catch, your patch looks fine to me and solves the reported problem.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

#5Michael Paquier
michael@paquier.xyz
In reply to: Dilip Kumar (#4)
Re: [BUG]Missing REPLICA IDENTITY check when DROP NOT NULL

On Thu, Nov 25, 2021 at 10:44:53AM +0530, Dilip Kumar wrote:

Good catch, your patch looks fine to me and solves the reported problem.

And applied down to 10. A couple of comments in the same area did not
get the call, though.
--
Michael