Can't find not null constraint, but \d+ shows that
Hi Alvaro,
I met an issue related to Catalog not-null commit on HEAD.
postgres=# CREATE TABLE t1(c0 int, c1 int);
CREATE TABLE
postgres=# ALTER TABLE t1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
ALTER TABLE
postgres=# \d+ t1
Table "public.t1"
Column | Type | Collation | Nullable | Default | Storage | Compression
| Stats target | Description
--------+---------+-----------+----------+---------+---------+-------------+--------------+-------------
c0 | integer | | not null | | plain |
| |
c1 | integer | | not null | | plain |
| |
Indexes:
"q" PRIMARY KEY, btree (c0, c1)
Access method: heap
postgres=# ALTER TABLE t1 DROP c1;
ALTER TABLE
postgres=# \d+ t1
Table "public.t1"
Column | Type | Collation | Nullable | Default | Storage | Compression
| Stats target | Description
--------+---------+-----------+----------+---------+---------+-------------+--------------+-------------
c0 | integer | | not null | | plain |
| |
Access method: heap
postgres=# ALTER TABLE t1 ALTER c0 DROP NOT NULL;
ERROR: could not find not-null constraint on column "c0", relation "t1"
postgres=# insert into t1 values (NULL);
ERROR: null value in column "c0" of relation "t1" violates not-null
constraint
DETAIL: Failing row contains (null).
I couldn't reproduce aboved issue on older version(12.12 ...16.1).
to be more precisely, since b0e96f3119 commit.
Without the b0e96f3119, when we drop not null constraint, we just update
the pg_attribute attnotnull to false
in ATExecDropNotNull(). But now we first check pg_constraint if has the
tuple. if attnotnull is ture, but pg_constraint
doesn't has that tuple. Aboved error will report.
It will be confuesed for users. Because \d shows the column c0 has not
null, and we cann't insert NULL value. But it
reports errore when users drop the NOT NULL constraint.
The attached patch is my workaround solution. Look forward your apply.
--
Tender Wang
OpenPie: https://en.openpie.com/
Attachments:
0001-Fix-attnotnull-not-correct-reset-issue.patchapplication/octet-stream; name=0001-Fix-attnotnull-not-correct-reset-issue.patchDownload
From 6239fc48eeaa43b02614ebd96582254d36b5053f Mon Sep 17 00:00:00 2001
From: "tender.wang" <tender.wang@openpie.com>
Date: Tue, 26 Mar 2024 19:33:53 +0800
Subject: [PATCH] Fix attnotnull not correct reset issue.
Since b0e96f3119, ALTER TABLE DROP NOT NULL will first check if
pg_constraint has related tuple. It will report error if the tuple
does not exist, but the pg_attribute attnotnull is true.
In some case, it will happend. So replacing report error, and just
update pg_attribute attnotnull to false.
---
src/backend/commands/tablecmds.c | 16 ++++++++++------
src/test/regress/expected/constraints.out | 5 +++++
src/test/regress/sql/constraints.sql | 6 ++++++
3 files changed, 21 insertions(+), 6 deletions(-)
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 47c900445c..b2a4881373 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -7524,9 +7524,7 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse,
Bitmapset *pkcols;
/*
- * There's no not-null constraint, so throw an error. If the column
- * is in a primary key, we can throw a specific error. Otherwise,
- * this is unexpected.
+ * If the column is in a primary key, we can throw a specific error.
*/
pkcols = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_PRIMARY_KEY);
if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber,
@@ -7535,9 +7533,15 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse,
errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("column \"%s\" is in a primary key", colName));
- /* this shouldn't happen */
- elog(ERROR, "could not find not-null constraint on column \"%s\", relation \"%s\"",
- colName, RelationGetRelationName(rel));
+ /*
+ * If a primary key includes two or more columns, one of the columns
+ * was dropped, so the primary was dropped, too. But Others columns'
+ * attnotnull is still true. So reset it here.
+ */
+ attTup->attnotnull = false;
+ CatalogTupleUpdate(attr_rel, &tuple->t_self, tuple);
+ table_close(attr_rel, RowExclusiveLock);
+ return address;
}
readyRels = NIL;
diff --git a/src/test/regress/expected/constraints.out b/src/test/regress/expected/constraints.out
index b7de50ad6a..be6fe7cfbd 100644
--- a/src/test/regress/expected/constraints.out
+++ b/src/test/regress/expected/constraints.out
@@ -867,6 +867,11 @@ select conname, contype, conkey from pg_constraint where conrelid = 'notnull_tbl
(1 row)
DROP TABLE notnull_tbl1;
+CREATE TABLE notnull_t1(c0 int, c1 int);
+ALTER TABLE notnull_t1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
+ALTER TABLE notnull_t1 DROP c1;
+ALTER TABLE notnull_t1 ALTER c0 DROP NOT NULL;
+DROP TABLE notnull_t1;
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
ERROR: constraint "blah" for relation "notnull_tbl2" already exists
diff --git a/src/test/regress/sql/constraints.sql b/src/test/regress/sql/constraints.sql
index 782699a437..2d4820152c 100644
--- a/src/test/regress/sql/constraints.sql
+++ b/src/test/regress/sql/constraints.sql
@@ -599,6 +599,12 @@ ALTER TABLE notnull_tbl1 ADD CONSTRAINT foobar NOT NULL a;
select conname, contype, conkey from pg_constraint where conrelid = 'notnull_tbl1'::regclass;
DROP TABLE notnull_tbl1;
+CREATE TABLE notnull_t1(c0 int, c1 int);
+ALTER TABLE notnull_t1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
+ALTER TABLE notnull_t1 DROP c1;
+ALTER TABLE notnull_t1 ALTER c0 DROP NOT NULL;
+DROP TABLE notnull_t1;
+
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
--
2.25.1
On 2024-Mar-26, Tender Wang wrote:
postgres=# CREATE TABLE t1(c0 int, c1 int);
postgres=# ALTER TABLE t1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
postgres=# ALTER TABLE t1 DROP c1;postgres=# ALTER TABLE t1 ALTER c0 DROP NOT NULL;
ERROR: could not find not-null constraint on column "c0", relation "t1"
Ooh, hah, what happens here is that we drop the PK constraint
indirectly, so we only go via doDeletion rather than the tablecmds.c
code, so we don't check the attnotnull flags that the PK was protecting.
The attached patch is my workaround solution. Look forward your apply.
Yeah, this is not a very good approach -- I think you're just guessing
that the column is marked NOT NULL because a PK was dropped in the
past -- but really what this catalog state is, is corrupted contents
because the PK drop was mishandled. At least in theory there are other
ways to drop a constraint other than dropping one of its columns (for
example, maybe this could happen if you drop a collation that the PK
depends on). The right approach is to ensure that the PK drop always
does the dance that ATExecDropConstraint does. A good fix probably just
moves some code from dropconstraint_internal to RemoveConstraintById.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
<inflex> really, I see PHP as like a strange amalgamation of C, Perl, Shell
<crab> inflex: you know that "amalgam" means "mixture with mercury",
more or less, right?
<crab> i.e., "deadly poison"
Alvaro Herrera <alvherre@alvh.no-ip.org> 于2024年3月26日周二 23:25写道:
On 2024-Mar-26, Tender Wang wrote:
postgres=# CREATE TABLE t1(c0 int, c1 int);
postgres=# ALTER TABLE t1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
postgres=# ALTER TABLE t1 DROP c1;postgres=# ALTER TABLE t1 ALTER c0 DROP NOT NULL;
ERROR: could not find not-null constraint on column "c0", relation "t1"Ooh, hah, what happens here is that we drop the PK constraint
indirectly, so we only go via doDeletion rather than the tablecmds.c
code, so we don't check the attnotnull flags that the PK was protecting.
Yeah, Indeed, as you said.
The attached patch is my workaround solution. Look forward your apply.
Yeah, this is not a very good approach -- I think you're just guessing
that the column is marked NOT NULL because a PK was dropped in the
past -- but really what this catalog state is, is corrupted contents
because the PK drop was mishandled. At least in theory there are other
ways to drop a constraint other than dropping one of its columns (for
example, maybe this could happen if you drop a collation that the PK
depends on). The right approach is to ensure that the PK drop always
does the dance that ATExecDropConstraint does. A good fix probably just
moves some code from dropconstraint_internal to RemoveConstraintById.
Agreed. It is look better. But it will not work if simply move some codes
from dropconstraint_internal
to RemoveConstraintById. I have tried this fix before 0001 patch, but
failed.
For example:
create table skip_wal_skip_rewrite_index (c varchar(10) primary key);
alter table skip_wal_skip_rewrite_index alter c type varchar(20);
ERROR: primary key column "c" is not marked NOT NULL
index_check_primary_key() in index.c has below comments;
"We check for a pre-existing primary key, and that all columns of the index
are simple column references (not expressions), and that all those columns
are marked NOT NULL. If not, fail."
So in aboved example, RemoveConstraintById() can't reset attnotnull. We can
pass some information to
RemoveConstraintById() like a bool var to indicate that attnotnull should
be reset or not.
--
Tender Wang
OpenPie: https://en.openpie.com/
Alvaro Herrera <alvherre@alvh.no-ip.org> 于2024年3月26日周二 23:25写道:
On 2024-Mar-26, Tender Wang wrote:
postgres=# CREATE TABLE t1(c0 int, c1 int);
postgres=# ALTER TABLE t1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
postgres=# ALTER TABLE t1 DROP c1;postgres=# ALTER TABLE t1 ALTER c0 DROP NOT NULL;
ERROR: could not find not-null constraint on column "c0", relation "t1"Ooh, hah, what happens here is that we drop the PK constraint
indirectly, so we only go via doDeletion rather than the tablecmds.c
code, so we don't check the attnotnull flags that the PK was protecting.The attached patch is my workaround solution. Look forward your apply.
Yeah, this is not a very good approach -- I think you're just guessing
that the column is marked NOT NULL because a PK was dropped in the
past -- but really what this catalog state is, is corrupted contents
because the PK drop was mishandled. At least in theory there are other
ways to drop a constraint other than dropping one of its columns (for
example, maybe this could happen if you drop a collation that the PK
depends on). The right approach is to ensure that the PK drop always
does the dance that ATExecDropConstraint does. A good fix probably just
moves some code from dropconstraint_internal to RemoveConstraintById.
I think again, below solutin maybe looks more better:
i. move some code from dropconstraint_internal to RemoveConstraintById,
not change the RemoveConstraintById interface. Ensure that the PK drop
always
does the dance that ATExecDropConstraint does.
ii. After i phase, the attnotnull of some column of primary key may be
reset to false as I provided example in last email.
We can set attnotnull to true again in some place.
--
Tender Wang
OpenPie: https://en.openpie.com/
Alvaro Herrera <alvherre@alvh.no-ip.org> 于2024年3月26日周二 23:25写道:
On 2024-Mar-26, Tender Wang wrote:
postgres=# CREATE TABLE t1(c0 int, c1 int);
postgres=# ALTER TABLE t1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
postgres=# ALTER TABLE t1 DROP c1;postgres=# ALTER TABLE t1 ALTER c0 DROP NOT NULL;
ERROR: could not find not-null constraint on column "c0", relation "t1"Ooh, hah, what happens here is that we drop the PK constraint
indirectly, so we only go via doDeletion rather than the tablecmds.c
code, so we don't check the attnotnull flags that the PK was protecting.The attached patch is my workaround solution. Look forward your apply.
Yeah, this is not a very good approach -- I think you're just guessing
that the column is marked NOT NULL because a PK was dropped in the
past -- but really what this catalog state is, is corrupted contents
because the PK drop was mishandled. At least in theory there are other
ways to drop a constraint other than dropping one of its columns (for
example, maybe this could happen if you drop a collation that the PK
depends on). The right approach is to ensure that the PK drop always
does the dance that ATExecDropConstraint does. A good fix probably just
moves some code from dropconstraint_internal to RemoveConstraintById.
I found some types ddl would check the attnotnull of column is true, for
example: AT_ReAddIndex, AT_ReplicaIdentity.
So we should add AT_SetAttNotNull sub-command to the wqueue. I add a
new AT_PASS_OLD_COL_ATTRS to make sure
AT_SetAttNotNull will have done when do AT_ReAddIndex or
AT_ReplicaIdentity.
--
Tender Wang
OpenPie: https://en.openpie.com/
Attachments:
v2-0001-Fix-pg_attribute-attnotnull-not-reset-when-droppe.patchapplication/octet-stream; name=v2-0001-Fix-pg_attribute-attnotnull-not-reset-when-droppe.patchDownload
From 9c84e69b94f9a0094d224e5af3b40f51a8fac01d Mon Sep 17 00:00:00 2001
From: "tender.wang" <tender.wang@openpie.com>
Date: Wed, 27 Mar 2024 22:00:17 +0800
Subject: [PATCH v2] Fix pg_attribute attnotnull not reset when dropped PK
indirectly.
Since b0e96f3119, we will reset pg_attribute attnotnull when drop
pk constraint and not difine NOT NULL. But in some cases, PK will
be dropped indirectly, which will not call tablecmds.c codes.
The result is \d will show not null, but we can't drop NOT NULL.
Moving some code from dropconstraint_internal to RemoveConstraintById
can fix this issue.
We should add SetAttrNotNull sub-command for some Alter Table. Because
those DDLs will check attnotnull is true during DDL.
---
src/backend/catalog/pg_constraint.c | 123 ++++++++++++++++++++++
src/backend/commands/tablecmds.c | 11 +-
src/test/regress/expected/constraints.out | 6 ++
src/test/regress/sql/constraints.sql | 7 ++
4 files changed, 142 insertions(+), 5 deletions(-)
diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c
index 604280d322..8c270909bc 100644
--- a/src/backend/catalog/pg_constraint.c
+++ b/src/backend/catalog/pg_constraint.c
@@ -19,6 +19,7 @@
#include "access/htup_details.h"
#include "access/sysattr.h"
#include "access/table.h"
+#include "access/xact.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/heap.h"
@@ -916,6 +917,8 @@ RemoveConstraintById(Oid conId)
Relation conDesc;
HeapTuple tup;
Form_pg_constraint con;
+ List *unconstrained_cols = NIL;
+ bool dropping_pk = false;
conDesc = table_open(ConstraintRelationId, RowExclusiveLock);
@@ -967,6 +970,32 @@ RemoveConstraintById(Oid conId)
table_close(pgrel, RowExclusiveLock);
}
+ else if (con->contype == CONSTRAINT_PRIMARY)
+ {
+ Datum adatum;
+ ArrayType *arr;
+ int numkeys;
+ bool isNull;
+ int16 *attnums;
+
+ dropping_pk = true;
+
+ adatum = heap_getattr(tup, Anum_pg_constraint_conkey,
+ RelationGetDescr(conDesc), &isNull);
+ if (isNull)
+ elog(ERROR, "null conkey for constraint %u", con->oid);
+ arr = DatumGetArrayTypeP(adatum); /* ensure not toasted */
+ numkeys = ARR_DIMS(arr)[0];
+ if (ARR_NDIM(arr) != 1 ||
+ numkeys < 0 ||
+ ARR_HASNULL(arr) ||
+ ARR_ELEMTYPE(arr) != INT2OID)
+ elog(ERROR, "conkey is not a 1-D smallint array");
+ attnums = (int16 *)ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < numkeys; i++)
+ unconstrained_cols = lappend_int(unconstrained_cols, attnums[i]);
+ }
/* Keep lock on constraint's rel until end of xact */
table_close(rel, NoLock);
@@ -986,6 +1015,100 @@ RemoveConstraintById(Oid conId)
/* Fry the constraint itself */
CatalogTupleDelete(conDesc, &tup->t_self);
+ /*
+ * If this was a NOT NULL or the primary key, the constrained columns must
+ * have had pg_attribute.attnotnull set. See if we need to reset it, and
+ * do so.
+ */
+ if (unconstrained_cols)
+ {
+ Relation attrel;
+ Relation rel;
+ Bitmapset *pkcols;
+ Bitmapset *ircols;
+ ListCell *lc;
+
+ /* Make the above deletion visible */
+ CommandCounterIncrement();
+
+ attrel = table_open(AttributeRelationId, RowExclusiveLock);
+ rel = table_open(con->conrelid, RowExclusiveLock);
+
+ /*
+ * We want to test columns for their presence in the primary key, but
+ * only if we're not dropping it.
+ */
+ pkcols = dropping_pk ? NULL :
+ RelationGetIndexAttrBitmap(rel,
+ INDEX_ATTR_BITMAP_PRIMARY_KEY);
+ ircols = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ foreach(lc, unconstrained_cols)
+ {
+ AttrNumber attnum = lfirst_int(lc);
+ HeapTuple atttup;
+ HeapTuple contup;
+ Form_pg_attribute attForm;
+
+ /*
+ * Obtain pg_attribute tuple and verify conditions on it. We use
+ * a copy we can scribble on.
+ */
+ atttup = SearchSysCacheCopyAttNum(RelationGetRelid(rel), attnum);
+ if (!HeapTupleIsValid(atttup))
+ elog(ERROR, "cache lookup failed for attribute %d of relation %u",
+ attnum, RelationGetRelid(rel));
+ attForm = (Form_pg_attribute) GETSTRUCT(atttup);
+
+ /*
+ * Since the above deletion has been made visible, we can now
+ * search for any remaining constraints on this column (or these
+ * columns, in the case we're dropping a multicol primary key.)
+ * Then, verify whether any further NOT NULL or primary key
+ * exists, and reset attnotnull if none.
+ *
+ * However, if this is a generated identity column, abort the
+ * whole thing with a specific error message, because the
+ * constraint is required in that case.
+ */
+ contup = findNotNullConstraintAttnum(RelationGetRelid(rel), attnum);
+ if (contup ||
+ bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber,
+ pkcols))
+ continue;
+
+ /*
+ * It's not valid to drop the not-null constraint for a GENERATED
+ * AS IDENTITY column.
+ */
+ if (attForm->attidentity)
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("column \"%s\" of relation \"%s\" is an identity column",
+ get_attname(RelationGetRelid(rel), attnum,
+ false),
+ RelationGetRelationName(rel)));
+
+ /*
+ * It's not valid to drop the not-null constraint for a column in
+ * the replica identity index, either. (FULL is not affected.)
+ */
+ if (bms_is_member(lfirst_int(lc) - FirstLowInvalidHeapAttributeNumber, ircols))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("column \"%s\" is in index used as replica identity",
+ get_attname(RelationGetRelid(rel), lfirst_int(lc), false)));
+
+ /* Reset attnotnull */
+ if (attForm->attnotnull)
+ {
+ attForm->attnotnull = false;
+ CatalogTupleUpdate(attrel, &atttup->t_self, atttup);
+ }
+ }
+ table_close(rel, RowExclusiveLock);
+ table_close(attrel, RowExclusiveLock);
+ }
/* Clean up */
ReleaseSysCache(tup);
table_close(conDesc, RowExclusiveLock);
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 259b4237a2..3f40f81621 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -147,6 +147,7 @@ typedef enum AlterTablePass
AT_PASS_ALTER_TYPE, /* ALTER COLUMN TYPE */
AT_PASS_ADD_COL, /* ADD COLUMN */
AT_PASS_SET_EXPRESSION, /* ALTER SET EXPRESSION */
+ AT_PASS_OLD_COL_ATTRS, /* re-set column NOT NULL */
AT_PASS_OLD_INDEX, /* re-add existing indexes */
AT_PASS_OLD_CONSTR, /* re-add existing constraints */
/* We could support a RENAME COLUMN pass here, but not currently used */
@@ -14629,12 +14630,12 @@ ATPostAlterTypeParse(Oid oldId, Oid oldRelId, Oid refRelId, char *cmd,
else if (cmd->subtype == AT_SetAttNotNull)
{
/*
- * The parser will create AT_AttSetNotNull subcommands for
- * columns of PRIMARY KEY indexes/constraints, but we need
- * not do anything with them here, because the columns'
- * NOT NULL marks will already have been propagated into
- * the new table definition.
+ * PK drop now will reset pg_attribute attnotnull to false.
+ * We should set attnotnull to true again.
*/
+ cmd->subtype = AT_SetAttNotNull;
+ tab->subcmds[AT_PASS_OLD_COL_ATTRS] =
+ lappend(tab->subcmds[AT_PASS_OLD_COL_ATTRS], cmd);
}
else
elog(ERROR, "unexpected statement subtype: %d",
diff --git a/src/test/regress/expected/constraints.out b/src/test/regress/expected/constraints.out
index d9d8408e86..00fab56bf8 100644
--- a/src/test/regress/expected/constraints.out
+++ b/src/test/regress/expected/constraints.out
@@ -867,6 +867,12 @@ select conname, contype, conkey from pg_constraint where conrelid = 'notnull_tbl
(1 row)
DROP TABLE notnull_tbl1;
+-- NOT NULL mask should be reset when drop PK
+CREATE TABLE notnull_t1(c0 int, c1 int);
+ALTER TABLE notnull_t1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
+ALTER TABLE notnull_t1 DROP c1;
+ALTER TABLE notnull_t1 ALTER c0 DROP NOT NULL;
+DROP TABLE notnull_t1;
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
ERROR: constraint "blah" for relation "notnull_tbl2" already exists
diff --git a/src/test/regress/sql/constraints.sql b/src/test/regress/sql/constraints.sql
index 87d685ae39..a146b67416 100644
--- a/src/test/regress/sql/constraints.sql
+++ b/src/test/regress/sql/constraints.sql
@@ -599,6 +599,13 @@ ALTER TABLE notnull_tbl1 ADD CONSTRAINT foobar NOT NULL a;
select conname, contype, conkey from pg_constraint where conrelid = 'notnull_tbl1'::regclass;
DROP TABLE notnull_tbl1;
+-- NOT NULL mask should be reset when drop PK
+CREATE TABLE notnull_t1(c0 int, c1 int);
+ALTER TABLE notnull_t1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
+ALTER TABLE notnull_t1 DROP c1;
+ALTER TABLE notnull_t1 ALTER c0 DROP NOT NULL;
+DROP TABLE notnull_t1;
+
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
--
2.25.1
On Wed, Mar 27, 2024 at 10:26 PM Tender Wang <tndrwang@gmail.com> wrote:
Alvaro Herrera <alvherre@alvh.no-ip.org> 于2024年3月26日周二 23:25写道:
On 2024-Mar-26, Tender Wang wrote:
postgres=# CREATE TABLE t1(c0 int, c1 int);
postgres=# ALTER TABLE t1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
postgres=# ALTER TABLE t1 DROP c1;postgres=# ALTER TABLE t1 ALTER c0 DROP NOT NULL;
ERROR: could not find not-null constraint on column "c0", relation "t1"Ooh, hah, what happens here is that we drop the PK constraint
indirectly, so we only go via doDeletion rather than the tablecmds.c
code, so we don't check the attnotnull flags that the PK was protecting.The attached patch is my workaround solution. Look forward your apply.
after applying v2-0001-Fix-pg_attribute-attnotnull-not-reset-when-droppe.patch
something is off, now i cannot drop a table.
demo:
CREATE TABLE t2(c0 int, c1 int);
ALTER TABLE t2 ADD CONSTRAINT t2_pk PRIMARY KEY(c0, c1);
ALTER TABLE t2 ALTER COLUMN c0 ADD GENERATED ALWAYS AS IDENTITY;
DROP TABLE t2 cascade;
Similarly, maybe there will be some issue with replica identity.
+ /*
+ * If this was a NOT NULL or the primary key, the constrained columns must
+ * have had pg_attribute.attnotnull set. See if we need to reset it, and
+ * do so.
+ */
+ if (unconstrained_cols)
it should be if (unconstrained_cols != NIL)?,
given unconstrained_cols is a List, also "unconstrained_cols" naming
seems not intuitive.
maybe pk_attnums or pk_cols or pk_columns.
+ attrel = table_open(AttributeRelationId, RowExclusiveLock);
+ rel = table_open(con->conrelid, RowExclusiveLock);
I am not sure why we using RowExclusiveLock for con->conrelid?
given we use AccessExclusiveLock at:
/*
* If the constraint is for a relation, open and exclusive-lock the
* relation it's for.
*/
rel = table_open(con->conrelid, AccessExclusiveLock);
+ /*
+ * Since the above deletion has been made visible, we can now
+ * search for any remaining constraints on this column (or these
+ * columns, in the case we're dropping a multicol primary key.)
+ * Then, verify whether any further NOT NULL or primary key
+ * exists, and reset attnotnull if none.
+ *
+ * However, if this is a generated identity column, abort the
+ * whole thing with a specific error message, because the
+ * constraint is required in that case.
+ */
+ contup = findNotNullConstraintAttnum(RelationGetRelid(rel), attnum);
+ if (contup ||
+ bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber,
+ pkcols))
+ continue;
I didn't get this part.
if you drop delete a primary key,
the "NOT NULL" constraint within pg_constraint should definitely be removed?
therefore contup should be pretty sure is NULL?
/*
- * The parser will create AT_AttSetNotNull subcommands for
- * columns of PRIMARY KEY indexes/constraints, but we need
- * not do anything with them here, because the columns'
- * NOT NULL marks will already have been propagated into
- * the new table definition.
+ * PK drop now will reset pg_attribute attnotnull to false.
+ * We should set attnotnull to true again.
*/
PK drop now will reset pg_attribute attnotnull to false,
which is what we should be expecting.
the comment didn't explain why should set attnotnull to true again?
jian he <jian.universality@gmail.com> 于2024年3月28日周四 13:18写道:
On Wed, Mar 27, 2024 at 10:26 PM Tender Wang <tndrwang@gmail.com> wrote:
Alvaro Herrera <alvherre@alvh.no-ip.org> 于2024年3月26日周二 23:25写道:
On 2024-Mar-26, Tender Wang wrote:
postgres=# CREATE TABLE t1(c0 int, c1 int);
postgres=# ALTER TABLE t1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
postgres=# ALTER TABLE t1 DROP c1;postgres=# ALTER TABLE t1 ALTER c0 DROP NOT NULL;
ERROR: could not find not-null constraint on column "c0", relation"t1"
Ooh, hah, what happens here is that we drop the PK constraint
indirectly, so we only go via doDeletion rather than the tablecmds.c
code, so we don't check the attnotnull flags that the PK was protecting.The attached patch is my workaround solution. Look forward your
apply.
after applying
v2-0001-Fix-pg_attribute-attnotnull-not-reset-when-droppe.patchsomething is off, now i cannot drop a table.
demo:
CREATE TABLE t2(c0 int, c1 int);
ALTER TABLE t2 ADD CONSTRAINT t2_pk PRIMARY KEY(c0, c1);
ALTER TABLE t2 ALTER COLUMN c0 ADD GENERATED ALWAYS AS IDENTITY;
DROP TABLE t2 cascade;
Similarly, maybe there will be some issue with replica identity.
Thanks for review this patch. Yeah, I can reproduce it. The error reported
in RemoveConstraintById(), where I moved
some codes from dropconstraint_internal(). But some check seems to no need
in RemoveConstraintById(). For example:
/*
* It's not valid to drop the not-null constraint for a GENERATED
* AS IDENTITY column.
*/
if (attForm->attidentity)
ereport(ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("column \"%s\" of relation \"%s\" is an identity column",
get_attname(RelationGetRelid(rel), attnum,
false),
RelationGetRelationName(rel)));
/*
* It's not valid to drop the not-null constraint for a column in
* the replica identity index, either. (FULL is not affected.)
*/
if (bms_is_member(lfirst_int(lc) - FirstLowInvalidHeapAttributeNumber,
ircols))
ereport(ERROR,
errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("column \"%s\" is in index used as replica identity",
get_attname(RelationGetRelid(rel), lfirst_int(lc), false)));
Above two check can remove from RemoveConstraintById()? I need more test.
+ /* + * If this was a NOT NULL or the primary key, the constrained columns must + * have had pg_attribute.attnotnull set. See if we need to reset it, and + * do so. + */ + if (unconstrained_cols) it should be if (unconstrained_cols != NIL)?, given unconstrained_cols is a List, also "unconstrained_cols" naming seems not intuitive. maybe pk_attnums or pk_cols or pk_columns.
As I said above, the codes were copied from dropconstraint_internal(). NOT
NULL columns were not alwayls PK.
So I thinks "unconstrained_cols" is OK.
+ attrel = table_open(AttributeRelationId, RowExclusiveLock); + rel = table_open(con->conrelid, RowExclusiveLock); I am not sure why we using RowExclusiveLock for con->conrelid? given we use AccessExclusiveLock at: /* * If the constraint is for a relation, open and exclusive-lock the * relation it's for. */ rel = table_open(con->conrelid, AccessExclusiveLock);
Yeah, you are right.
+ /* + * Since the above deletion has been made visible, we can now + * search for any remaining constraints on this column (or these + * columns, in the case we're dropping a multicol primary key.) + * Then, verify whether any further NOT NULL or primary key + * exists, and reset attnotnull if none. + * + * However, if this is a generated identity column, abort the + * whole thing with a specific error message, because the + * constraint is required in that case. + */ + contup = findNotNullConstraintAttnum(RelationGetRelid(rel), attnum); + if (contup || + bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, + pkcols)) + continue;I didn't get this part.
if you drop delete a primary key,
the "NOT NULL" constraint within pg_constraint should definitely be
removed?
therefore contup should be pretty sure is NULL?
No, If the original definaiton of column includes NOT NULL, we can't reset
attnotnull to false when
We we drop PK.
/* - * The parser will create AT_AttSetNotNull subcommands for - * columns of PRIMARY KEY indexes/constraints, but we need - * not do anything with them here, because the columns' - * NOT NULL marks will already have been propagated into - * the new table definition. + * PK drop now will reset pg_attribute attnotnull to false. + * We should set attnotnull to true again. */ PK drop now will reset pg_attribute attnotnull to false, which is what we should be expecting. the comment didn't explain why should set attnotnull to true again?
The V2 patch still needs more cases to test, Probably not right solution.
Anyway, I will send a v3 version patch after I do more test.
--
Tender Wang
OpenPie: https://en.openpie.com/
Alvaro Herrera <alvherre@alvh.no-ip.org> 于2024年3月26日周二 23:25写道:
On 2024-Mar-26, Tender Wang wrote:
postgres=# CREATE TABLE t1(c0 int, c1 int);
postgres=# ALTER TABLE t1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
postgres=# ALTER TABLE t1 DROP c1;postgres=# ALTER TABLE t1 ALTER c0 DROP NOT NULL;
ERROR: could not find not-null constraint on column "c0", relation "t1"Ooh, hah, what happens here is that we drop the PK constraint
indirectly, so we only go via doDeletion rather than the tablecmds.c
code, so we don't check the attnotnull flags that the PK was protecting.The attached patch is my workaround solution. Look forward your apply.
Yeah, this is not a very good approach -- I think you're just guessing
that the column is marked NOT NULL because a PK was dropped in the
past -- but really what this catalog state is, is corrupted contents
because the PK drop was mishandled. At least in theory there are other
ways to drop a constraint other than dropping one of its columns (for
example, maybe this could happen if you drop a collation that the PK
depends on). The right approach is to ensure that the PK drop always
does the dance that ATExecDropConstraint does. A good fix probably just
moves some code from dropconstraint_internal to RemoveConstraintById.
RemoveConstraintById() should think recurse(e.g. partition table)? I'm not
sure now.
If we should think process recurse in RemoveConstraintById(), the
function will look complicated than before.
--
Tender Wang
OpenPie: https://en.openpie.com/
On 2024-Mar-28, Tender Wang wrote:
RemoveConstraintById() should think recurse(e.g. partition table)? I'm not
sure now.
If we should think process recurse in RemoveConstraintById(), the
function will look complicated than before.
No -- this function handles just a single constraint, as identified by
OID. The recursion is handled by upper layers, which can be either
dependency.c or tablecmds.c. I think the problem you found is caused by
the fact that I worked with the tablecmds.c recursion and neglected the
one in dependency.c.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"No nos atrevemos a muchas cosas porque son difíciles,
pero son difíciles porque no nos atrevemos a hacerlas" (Séneca)
Alvaro Herrera <alvherre@alvh.no-ip.org> 于2024年3月28日周四 17:18写道:
On 2024-Mar-28, Tender Wang wrote:
RemoveConstraintById() should think recurse(e.g. partition table)? I'm
not
sure now.
If we should think process recurse in RemoveConstraintById(), the
function will look complicated than before.No -- this function handles just a single constraint, as identified by
OID. The recursion is handled by upper layers, which can be either
dependency.c or tablecmds.c. I think the problem you found is caused by
the fact that I worked with the tablecmds.c recursion and neglected the
one in dependency.c.
Indeed.
create table skip_wal_skip_rewrite_index (c varchar(10) primary key);
alter table skip_wal_skip_rewrite_index alter c type varchar(20);
Above SQL need attnotnull to be true when re-add index, but
RemoveConstraintById() is hard to recognize this scenario as I know.
We should re-set attnotnull to be true before re-add index. I add a new
AT_PASS in attached patch.
Any thoughts?
--
Tender Wang
OpenPie: https://en.openpie.com/
Attachments:
v4-0001-Fix-pg_attribute-attnotnull-not-reset-when-droppe.patchapplication/octet-stream; name=v4-0001-Fix-pg_attribute-attnotnull-not-reset-when-droppe.patchDownload
From afb98ba63161f9e7b88aaff4b9d21d7ca64c8244 Mon Sep 17 00:00:00 2001
From: "tender.wang" <tender.wang@openpie.com>
Date: Wed, 27 Mar 2024 22:00:17 +0800
Subject: [PATCH v4] Fix pg_attribute attnotnull not reset when dropped PK
indirectly.
Since b0e96f3119, we will reset pg_attribute attnotnull when drop
pk constraint and not difine NOT NULL. But in some cases, PK will
be dropped indirectly, which will not call tablecmds.c codes.
The result is \d will show not null, but we can't drop NOT NULL.
Moving some code from dropconstraint_internal to RemoveConstraintById
can fix this issue.
We should add SetAttrNotNull sub-command for some Alter Table. Because
those DDLs will check attnotnull is true during DDL.
---
src/backend/catalog/pg_constraint.c | 105 +++++++++++++++++++++-
src/backend/commands/tablecmds.c | 13 +--
src/test/regress/expected/constraints.out | 6 ++
src/test/regress/sql/constraints.sql | 7 ++
4 files changed, 125 insertions(+), 6 deletions(-)
diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c
index 604280d322..f16f21985d 100644
--- a/src/backend/catalog/pg_constraint.c
+++ b/src/backend/catalog/pg_constraint.c
@@ -19,6 +19,7 @@
#include "access/htup_details.h"
#include "access/sysattr.h"
#include "access/table.h"
+#include "access/xact.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/heap.h"
@@ -906,7 +907,6 @@ RelationGetNotNullConstraints(Oid relid, bool cooked)
return notnulls;
}
-
/*
* Delete a single constraint record.
*/
@@ -916,6 +916,7 @@ RemoveConstraintById(Oid conId)
Relation conDesc;
HeapTuple tup;
Form_pg_constraint con;
+ List *unconstrained_cols = NIL;
conDesc = table_open(ConstraintRelationId, RowExclusiveLock);
@@ -967,6 +968,30 @@ RemoveConstraintById(Oid conId)
table_close(pgrel, RowExclusiveLock);
}
+ else if (con->contype == CONSTRAINT_PRIMARY)
+ {
+ Datum adatum;
+ ArrayType *arr;
+ int numkeys;
+ bool isNull;
+ int16 *attnums;
+
+ adatum = heap_getattr(tup, Anum_pg_constraint_conkey,
+ RelationGetDescr(conDesc), &isNull);
+ if (isNull)
+ elog(ERROR, "null conkey for constraint %u", con->oid);
+ arr = DatumGetArrayTypeP(adatum); /* ensure not toasted */
+ numkeys = ARR_DIMS(arr)[0];
+ if (ARR_NDIM(arr) != 1 ||
+ numkeys < 0 ||
+ ARR_HASNULL(arr) ||
+ ARR_ELEMTYPE(arr) != INT2OID)
+ elog(ERROR, "conkey is not a 1-D smallint array");
+ attnums = (int16 *)ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < numkeys; i++)
+ unconstrained_cols = lappend_int(unconstrained_cols, attnums[i]);
+ }
/* Keep lock on constraint's rel until end of xact */
table_close(rel, NoLock);
@@ -986,6 +1011,84 @@ RemoveConstraintById(Oid conId)
/* Fry the constraint itself */
CatalogTupleDelete(conDesc, &tup->t_self);
+ /*
+ * If this was a NOT NULL or the primary key, the constrained columns must
+ * have had pg_attribute.attnotnull set. See if we need to reset it, and
+ * do so.
+ */
+ if (unconstrained_cols != NIL)
+ {
+ Relation attrel;
+ Relation rel;
+ Bitmapset *ircols;
+ ListCell *lc;
+
+ /* Make the above deletion visible */
+ CommandCounterIncrement();
+
+ attrel = table_open(AttributeRelationId, RowExclusiveLock);
+ rel = table_open(con->conrelid, AccessExclusiveLock);
+
+ ircols = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_IDENTITY_KEY);
+
+ foreach(lc, unconstrained_cols)
+ {
+ AttrNumber attnum = lfirst_int(lc);
+ HeapTuple atttup;
+ HeapTuple contup;
+ Form_pg_attribute attForm;
+
+ /*
+ * Obtain pg_attribute tuple and verify conditions on it. We use
+ * a copy we can scribble on.
+ */
+ atttup = SearchSysCacheCopyAttNum(RelationGetRelid(rel), attnum);
+ if (!HeapTupleIsValid(atttup))
+ elog(ERROR, "cache lookup failed for attribute %d of relation %u",
+ attnum, RelationGetRelid(rel));
+ attForm = (Form_pg_attribute) GETSTRUCT(atttup);
+
+ /* GENERATED AS IDENTITY column need not null */
+ if (attForm->attidentity)
+ {
+ heap_freetuple(atttup);
+ continue;
+ }
+
+ /* A column in the replica identity index need not null */
+ if (bms_is_member(lfirst_int(lc) - FirstLowInvalidHeapAttributeNumber, ircols))
+ {
+ heap_freetuple(atttup);
+ continue;
+ }
+
+ /*
+ * Since the above deletion has been made visible, we can now
+ * search for any remaining constraints on this column (or these
+ * columns, in the case we're dropping a multicol primary key.)
+ * Then, verify whether any further NOT NULL exists, and reset
+ * attnotnull if none.
+ */
+ contup = findNotNullConstraintAttnum(RelationGetRelid(rel), attnum);
+ if (HeapTupleIsValid(contup))
+ {
+ heap_freetuple(contup);
+ heap_freetuple(atttup);
+ continue;
+ }
+
+ /* Reset attnotnull */
+ if (attForm->attnotnull)
+ {
+ attForm->attnotnull = false;
+ CatalogTupleUpdate(attrel, &atttup->t_self, atttup);
+ }
+
+ heap_freetuple(atttup);
+ }
+ table_close(rel, AccessExclusiveLock);
+ table_close(attrel, RowExclusiveLock);
+ }
/* Clean up */
ReleaseSysCache(tup);
table_close(conDesc, RowExclusiveLock);
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 259b4237a2..57fafb839a 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -147,6 +147,7 @@ typedef enum AlterTablePass
AT_PASS_ALTER_TYPE, /* ALTER COLUMN TYPE */
AT_PASS_ADD_COL, /* ADD COLUMN */
AT_PASS_SET_EXPRESSION, /* ALTER SET EXPRESSION */
+ AT_PASS_OLD_COL_ATTRS, /* re-set column NOT NULL */
AT_PASS_OLD_INDEX, /* re-add existing indexes */
AT_PASS_OLD_CONSTR, /* re-add existing constraints */
/* We could support a RENAME COLUMN pass here, but not currently used */
@@ -14629,12 +14630,14 @@ ATPostAlterTypeParse(Oid oldId, Oid oldRelId, Oid refRelId, char *cmd,
else if (cmd->subtype == AT_SetAttNotNull)
{
/*
- * The parser will create AT_AttSetNotNull subcommands for
- * columns of PRIMARY KEY indexes/constraints, but we need
- * not do anything with them here, because the columns'
- * NOT NULL marks will already have been propagated into
- * the new table definition.
+ * When doing alter primary key column type, Removing old PK
+ * constraint as sub-command will reset attnotnull to false in
+ * RemoveConstraintById(). We must set attnotnull back to true
+ * before doing re-add the index sub-command.
*/
+ cmd->subtype = AT_SetAttNotNull;
+ tab->subcmds[AT_PASS_OLD_COL_ATTRS] =
+ lappend(tab->subcmds[AT_PASS_OLD_COL_ATTRS], cmd);
}
else
elog(ERROR, "unexpected statement subtype: %d",
diff --git a/src/test/regress/expected/constraints.out b/src/test/regress/expected/constraints.out
index d9d8408e86..00fab56bf8 100644
--- a/src/test/regress/expected/constraints.out
+++ b/src/test/regress/expected/constraints.out
@@ -867,6 +867,12 @@ select conname, contype, conkey from pg_constraint where conrelid = 'notnull_tbl
(1 row)
DROP TABLE notnull_tbl1;
+-- NOT NULL mask should be reset when drop PK
+CREATE TABLE notnull_t1(c0 int, c1 int);
+ALTER TABLE notnull_t1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
+ALTER TABLE notnull_t1 DROP c1;
+ALTER TABLE notnull_t1 ALTER c0 DROP NOT NULL;
+DROP TABLE notnull_t1;
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
ERROR: constraint "blah" for relation "notnull_tbl2" already exists
diff --git a/src/test/regress/sql/constraints.sql b/src/test/regress/sql/constraints.sql
index 87d685ae39..a146b67416 100644
--- a/src/test/regress/sql/constraints.sql
+++ b/src/test/regress/sql/constraints.sql
@@ -599,6 +599,13 @@ ALTER TABLE notnull_tbl1 ADD CONSTRAINT foobar NOT NULL a;
select conname, contype, conkey from pg_constraint where conrelid = 'notnull_tbl1'::regclass;
DROP TABLE notnull_tbl1;
+-- NOT NULL mask should be reset when drop PK
+CREATE TABLE notnull_t1(c0 int, c1 int);
+ALTER TABLE notnull_t1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
+ALTER TABLE notnull_t1 DROP c1;
+ALTER TABLE notnull_t1 ALTER c0 DROP NOT NULL;
+DROP TABLE notnull_t1;
+
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
--
2.25.1
hi.
about v4, i think, i understand the changes you made.
RemoveConstraintById(Oid conId)
will drop a single constraint record.
if the constraint is primary key, then primary key associated
attnotnull should set to false.
but sometimes it shouldn't.
for example:
drop table if exists t2;
CREATE TABLE t2(c0 int, c1 int);
ALTER TABLE t2 ADD CONSTRAINT t2_pk PRIMARY KEY(c0, c1);
ALTER TABLE t2 ALTER COLUMN c0 ADD GENERATED ALWAYS AS IDENTITY;
ALTER TABLE t2 DROP c1;
+ * If this was a NOT NULL or the primary key, the constrained columns must
+ * have had pg_attribute.attnotnull set. See if we need to reset it, and
+ * do so.
+ */
+ if (unconstrained_cols != NIL)
unconstrained_cols is not NIL, which means we have dropped a primary key.
I am confused by "If this was a NOT NULL or the primary key".
+
+ /*
+ * Since the above deletion has been made visible, we can now
+ * search for any remaining constraints on this column (or these
+ * columns, in the case we're dropping a multicol primary key.)
+ * Then, verify whether any further NOT NULL exists, and reset
+ * attnotnull if none.
+ */
+ contup = findNotNullConstraintAttnum(RelationGetRelid(rel), attnum);
+ if (HeapTupleIsValid(contup))
+ {
+ heap_freetuple(contup);
+ heap_freetuple(atttup);
+ continue;
+ }
I am a little bit confused by the above comment.
I think the comments should say,
if contup is valid, that means, we already have one "not null"
constraint associate with the attnum
in that condition, we must not set attnotnull, otherwise the
consistency between attnotnull and "not null"
table constraint will be broken.
other than that, the change in RemoveConstraintById looks sane.
jian he <jian.universality@gmail.com> 于2024年3月29日周五 14:56写道:
hi.
about v4, i think, i understand the changes you made.
RemoveConstraintById(Oid conId)
will drop a single constraint record.
if the constraint is primary key, then primary key associated
attnotnull should set to false.
but sometimes it shouldn't.
Yeah, indeed.
for example:
drop table if exists t2;
CREATE TABLE t2(c0 int, c1 int);
ALTER TABLE t2 ADD CONSTRAINT t2_pk PRIMARY KEY(c0, c1);
ALTER TABLE t2 ALTER COLUMN c0 ADD GENERATED ALWAYS AS IDENTITY;
ALTER TABLE t2 DROP c1;+ * If this was a NOT NULL or the primary key, the constrained columns must + * have had pg_attribute.attnotnull set. See if we need to reset it, and + * do so. + */ + if (unconstrained_cols != NIL)unconstrained_cols is not NIL, which means we have dropped a primary key.
I am confused by "If this was a NOT NULL or the primary key".
NOT NULL means the definition of column having not-null constranit. For
example:
create table t1(a int not null);
the pg_attribute.attnotnull is set to true.
primary key case like below:
create table t2(a int primary key);
the pg_attribute.attnotnull is set to true.
I think aboved case can explain what's meaning about comments in
dropconstraint_internal.
But here, in RemoveConstraintById() , we only care about primary key case,
so NOT NULL is better
to removed from comments.
+ + /* + * Since the above deletion has been made visible, we can now + * search for any remaining constraints on this column (or these + * columns, in the case we're dropping a multicol primary key.) + * Then, verify whether any further NOT NULL exists, and reset + * attnotnull if none. + */ + contup = findNotNullConstraintAttnum(RelationGetRelid(rel), attnum); + if (HeapTupleIsValid(contup)) + { + heap_freetuple(contup); + heap_freetuple(atttup); + continue; + }I am a little bit confused by the above comment.
I think the comments should say,
if contup is valid, that means, we already have one "not null"
constraint associate with the attnum
in that condition, we must not set attnotnull, otherwise the
consistency between attnotnull and "not null"
table constraint will be broken.other than that, the change in RemoveConstraintById looks sane.
Above comments want to say that after pk constranit dropped, if there are
tuples in
pg_constraint, that means the definition of column has not-null constraint.
So we can't
set pg_attribute.attnotnull to false.
For example:
create table t1(a int not null);
alter table t1 add constraint t1_pk primary key(a);
alter table t1 drop constraint t1_pk;
--
Tender Wang
OpenPie: https://en.openpie.com/
It has been several days since the last email. Do you have any
suggestions, please?
What I'm concerned about is that adding a new AT_PASS is good fix? Is it
a big try?
More concerned is that it can cover all ALTER TABLE cases?
Any thoughts.
--
Tender Wang
OpenPie: https://en.openpie.com/
On 2024-Mar-29, Tender Wang wrote:
I think aboved case can explain what's meaning about comments in
dropconstraint_internal.
But here, in RemoveConstraintById() , we only care about primary key case,
so NOT NULL is better to removed from comments.
Actually, I think it's better if all the resets of attnotnull occur in
RemoveConstraintById, for both types of constraints; we would keep that
block in dropconstraint_internal only to raise errors in the cases where
the constraint is protecting a replica identity or a generated column.
Something like the attached, perhaps, may need more polish.
I'm not really sure about the business of adding a new pass value
-- it's clear that things don't work if we don't do *something* -- I'm
just not sure if this is the something that we want to do.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Tiene valor aquel que admite que es un cobarde" (Fernandel)
Attachments:
0001-Correctly-reset-attnotnull-when-constraints-dropped-.patchtext/x-diff; charset=utf-8Download
From 397d424cd64020bcb4aff219deefca7fc7b6f88d Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 9 Apr 2024 19:18:57 +0200
Subject: [PATCH] Correctly reset attnotnull when constraints dropped
indirectly
---
src/backend/catalog/pg_constraint.c | 105 +++++++++++++++++++++-
src/backend/commands/tablecmds.c | 67 ++++++--------
src/test/regress/expected/constraints.out | 11 +++
src/test/regress/sql/constraints.sql | 7 ++
4 files changed, 150 insertions(+), 40 deletions(-)
diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c
index 604280d322..17c7a2d0f3 100644
--- a/src/backend/catalog/pg_constraint.c
+++ b/src/backend/catalog/pg_constraint.c
@@ -19,6 +19,7 @@
#include "access/htup_details.h"
#include "access/sysattr.h"
#include "access/table.h"
+#include "access/xact.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/heap.h"
@@ -916,6 +917,8 @@ RemoveConstraintById(Oid conId)
Relation conDesc;
HeapTuple tup;
Form_pg_constraint con;
+ bool dropping_pk = false;
+ List *unconstrained_cols = NIL;
conDesc = table_open(ConstraintRelationId, RowExclusiveLock);
@@ -940,7 +943,9 @@ RemoveConstraintById(Oid conId)
/*
* We need to update the relchecks count if it is a check constraint
* being dropped. This update will force backends to rebuild relcache
- * entries when we commit.
+ * entries when we commit. For not-null and primary key constraints,
+ * obtain the list of columns affected, so that we can reset their
+ * attnotnull flags below.
*/
if (con->contype == CONSTRAINT_CHECK)
{
@@ -967,6 +972,36 @@ RemoveConstraintById(Oid conId)
table_close(pgrel, RowExclusiveLock);
}
+ else if (con->contype == CONSTRAINT_NOTNULL)
+ {
+ unconstrained_cols = list_make1_int(extractNotNullColumn(tup));
+ }
+ else if (con->contype == CONSTRAINT_PRIMARY)
+ {
+ Datum adatum;
+ ArrayType *arr;
+ int numkeys;
+ bool isNull;
+ int16 *attnums;
+
+ dropping_pk = true;
+
+ adatum = heap_getattr(tup, Anum_pg_constraint_conkey,
+ RelationGetDescr(conDesc), &isNull);
+ if (isNull)
+ elog(ERROR, "null conkey for constraint %u", con->oid);
+ arr = DatumGetArrayTypeP(adatum); /* ensure not toasted */
+ numkeys = ARR_DIMS(arr)[0];
+ if (ARR_NDIM(arr) != 1 ||
+ numkeys < 0 ||
+ ARR_HASNULL(arr) ||
+ ARR_ELEMTYPE(arr) != INT2OID)
+ elog(ERROR, "conkey is not a 1-D smallint array");
+ attnums = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < numkeys; i++)
+ unconstrained_cols = lappend_int(unconstrained_cols, attnums[i]);
+ }
/* Keep lock on constraint's rel until end of xact */
table_close(rel, NoLock);
@@ -986,6 +1021,74 @@ RemoveConstraintById(Oid conId)
/* Fry the constraint itself */
CatalogTupleDelete(conDesc, &tup->t_self);
+ /*
+ * If this was a NOT NULL or the primary key, the constrained columns must
+ * have had pg_attribute.attnotnull set. See if we need to reset it, and
+ * do so.
+ */
+ if (unconstrained_cols)
+ {
+ Relation tablerel;
+ Relation attrel;
+ Bitmapset *pkcols;
+ ListCell *lc;
+
+ /* Make the above deletion visible */
+ CommandCounterIncrement();
+
+ tablerel = table_open(con->conrelid, NoLock); /* already have lock */
+ attrel = table_open(AttributeRelationId, RowExclusiveLock);
+
+ /*
+ * We want to test columns for their presence in the primary key, but
+ * only if we're not dropping it.
+ */
+ pkcols = dropping_pk ? NULL :
+ RelationGetIndexAttrBitmap(tablerel,
+ INDEX_ATTR_BITMAP_PRIMARY_KEY);
+
+ foreach(lc, unconstrained_cols)
+ {
+ AttrNumber attnum = lfirst_int(lc);
+ HeapTuple atttup;
+ HeapTuple contup;
+ Form_pg_attribute attForm;
+
+ /*
+ * Obtain pg_attribute tuple and verify conditions on it. We use
+ * a copy we can scribble on.
+ */
+ atttup = SearchSysCacheCopyAttNum(con->conrelid, attnum);
+ if (!HeapTupleIsValid(atttup))
+ elog(ERROR, "cache lookup failed for attribute %d of relation %u",
+ attnum, con->conrelid);
+ attForm = (Form_pg_attribute) GETSTRUCT(atttup);
+
+ /*
+ * Since the above deletion has been made visible, we can now
+ * search for any remaining constraints on this column (or these
+ * columns, in the case we're dropping a multicol primary key.)
+ * Then, verify whether any further NOT NULL or primary key
+ * exists, and reset attnotnull if none.
+ */
+ contup = findNotNullConstraintAttnum(con->conrelid, attnum);
+ if (contup ||
+ bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber,
+ pkcols))
+ continue;
+
+ /* Reset attnotnull */
+ if (attForm->attnotnull)
+ {
+ attForm->attnotnull = false;
+ CatalogTupleUpdate(attrel, &atttup->t_self, atttup);
+ }
+ }
+
+ table_close(attrel, RowExclusiveLock);
+ table_close(tablerel, NoLock);
+ }
+
/* Clean up */
ReleaseSysCache(tup);
table_close(conDesc, RowExclusiveLock);
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 865c6331c1..25d3613fe4 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -149,6 +149,7 @@ typedef enum AlterTablePass
AT_PASS_ALTER_TYPE, /* ALTER COLUMN TYPE */
AT_PASS_ADD_COL, /* ADD COLUMN */
AT_PASS_SET_EXPRESSION, /* ALTER SET EXPRESSION */
+ AT_PASS_OLD_COL_ATTRS, /* re-install attnotnull */
AT_PASS_OLD_INDEX, /* re-add existing indexes */
AT_PASS_OLD_CONSTR, /* re-add existing constraints */
/* We could support a RENAME COLUMN pass here, but not currently used */
@@ -12933,10 +12934,10 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
List *children;
ListCell *child;
bool is_no_inherit_constraint = false;
- bool dropping_pk = false;
char *constrName;
List *unconstrained_cols = NIL;
- char *colname;
+ char *colname = NULL;
+ bool dropping_pk = false;
if (list_member_oid(*readyRels, RelationGetRelid(rel)))
return InvalidObjectAddress;
@@ -12968,10 +12969,12 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
*/
if (con->contype == CONSTRAINT_NOTNULL)
{
- AttrNumber colnum = extractNotNullColumn(constraintTup);
+ AttrNumber colnum;
- if (colnum != InvalidAttrNumber)
- unconstrained_cols = list_make1_int(colnum);
+ colnum = extractNotNullColumn(constraintTup);
+ unconstrained_cols = list_make1_int(colnum);
+ colname = NameStr(TupleDescAttr(RelationGetDescr(rel),
+ colnum - 1)->attname);
}
else if (con->contype == CONSTRAINT_PRIMARY)
{
@@ -13027,9 +13030,8 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
performDeletion(&conobj, behavior, 0);
/*
- * If this was a NOT NULL or the primary key, the constrained columns must
- * have had pg_attribute.attnotnull set. See if we need to reset it, and
- * do so.
+ * If this was a NOT NULL or the primary key, verify that we still have
+ * constraints to support GENERATED AS IDENTITY or the replica identity.
*/
if (unconstrained_cols)
{
@@ -13038,7 +13040,7 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
Bitmapset *ircols;
ListCell *lc;
- /* Make the above deletion visible */
+ /* Make implicit attnotnull changes visible */
CommandCounterIncrement();
attrel = table_open(AttributeRelationId, RowExclusiveLock);
@@ -13058,27 +13060,26 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
HeapTuple atttup;
HeapTuple contup;
Form_pg_attribute attForm;
+ char attidentity;
/*
- * Obtain pg_attribute tuple and verify conditions on it. We use
- * a copy we can scribble on.
+ * Obtain pg_attribute tuple and verify conditions on it.
*/
- atttup = SearchSysCacheCopyAttNum(RelationGetRelid(rel), attnum);
+ atttup = SearchSysCacheAttNum(RelationGetRelid(rel), attnum);
if (!HeapTupleIsValid(atttup))
elog(ERROR, "cache lookup failed for attribute %d of relation %u",
attnum, RelationGetRelid(rel));
attForm = (Form_pg_attribute) GETSTRUCT(atttup);
+ attidentity = attForm->attidentity;
+ ReleaseSysCache(atttup);
/*
* Since the above deletion has been made visible, we can now
* search for any remaining constraints on this column (or these
* columns, in the case we're dropping a multicol primary key.)
* Then, verify whether any further NOT NULL or primary key
- * exists, and reset attnotnull if none.
- *
- * However, if this is a generated identity column, abort the
- * whole thing with a specific error message, because the
- * constraint is required in that case.
+ * exists: if none and this is a generated identity column or the
+ * replica identity, abort the whole thing.
*/
contup = findNotNullConstraintAttnum(RelationGetRelid(rel), attnum);
if (contup ||
@@ -13090,7 +13091,7 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
* It's not valid to drop the not-null constraint for a GENERATED
* AS IDENTITY column.
*/
- if (attForm->attidentity)
+ if (attidentity != '\0')
ereport(ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("column \"%s\" of relation \"%s\" is an identity column",
@@ -13107,13 +13108,6 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("column \"%s\" is in index used as replica identity",
get_attname(RelationGetRelid(rel), lfirst_int(lc), false)));
-
- /* Reset attnotnull */
- if (attForm->attnotnull)
- {
- attForm->attnotnull = false;
- CatalogTupleUpdate(attrel, &atttup->t_self, atttup);
- }
}
table_close(attrel, RowExclusiveLock);
}
@@ -13152,13 +13146,6 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
errmsg("cannot remove constraint from only the partitioned table when partitions exist"),
errhint("Do not specify the ONLY keyword.")));
- /* For not-null constraints we recurse by column name */
- if (con->contype == CONSTRAINT_NOTNULL)
- colname = NameStr(TupleDescAttr(RelationGetDescr(rel),
- linitial_int(unconstrained_cols) - 1)->attname);
- else
- colname = NULL; /* keep compiler quiet */
-
foreach(child, children)
{
Oid childrelid = lfirst_oid(child);
@@ -13174,8 +13161,8 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
CheckTableNotInUse(childrel, "ALTER TABLE");
/*
- * We search for not-null constraint by column number, and other
- * constraints by name.
+ * We search for not-null constraints by column name, and others by
+ * constraint name.
*/
if (con->contype == CONSTRAINT_NOTNULL)
{
@@ -14656,12 +14643,14 @@ ATPostAlterTypeParse(Oid oldId, Oid oldRelId, Oid refRelId, char *cmd,
else if (cmd->subtype == AT_SetAttNotNull)
{
/*
- * The parser will create AT_AttSetNotNull subcommands for
- * columns of PRIMARY KEY indexes/constraints, but we need
- * not do anything with them here, because the columns'
- * NOT NULL marks will already have been propagated into
- * the new table definition.
+ * When doing alter primary key column type, Removing old
+ * PK constraint as sub-command will reset attnotnull to
+ * false in RemoveConstraintById(). We must set attnotnull
+ * back to true before doing re-add the index sub-command.
*/
+ cmd->subtype = AT_SetAttNotNull;
+ tab->subcmds[AT_PASS_OLD_COL_ATTRS] =
+ lappend(tab->subcmds[AT_PASS_OLD_COL_ATTRS], cmd);
}
else
elog(ERROR, "unexpected statement subtype: %d",
diff --git a/src/test/regress/expected/constraints.out b/src/test/regress/expected/constraints.out
index d9d8408e86..ce8ba73a87 100644
--- a/src/test/regress/expected/constraints.out
+++ b/src/test/regress/expected/constraints.out
@@ -866,6 +866,17 @@ select conname, contype, conkey from pg_constraint where conrelid = 'notnull_tbl
foobar | n | {1}
(1 row)
+DROP TABLE notnull_tbl1;
+-- make sure attnotnull is reset correctly when a PK is dropped indirectly
+CREATE TABLE notnull_tbl1 (c0 int, c1 int);
+ALTER TABLE notnull_tbl1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
+ALTER TABLE notnull_tbl1 DROP c1;
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c0 | integer | | | | plain | |
+
DROP TABLE notnull_tbl1;
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
diff --git a/src/test/regress/sql/constraints.sql b/src/test/regress/sql/constraints.sql
index 87d685ae39..b25107da46 100644
--- a/src/test/regress/sql/constraints.sql
+++ b/src/test/regress/sql/constraints.sql
@@ -599,6 +599,13 @@ ALTER TABLE notnull_tbl1 ADD CONSTRAINT foobar NOT NULL a;
select conname, contype, conkey from pg_constraint where conrelid = 'notnull_tbl1'::regclass;
DROP TABLE notnull_tbl1;
+-- make sure attnotnull is reset correctly when a PK is dropped indirectly
+CREATE TABLE notnull_tbl1 (c0 int, c1 int);
+ALTER TABLE notnull_tbl1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
+ALTER TABLE notnull_tbl1 DROP c1;
+\d+ notnull_tbl1
+DROP TABLE notnull_tbl1;
+
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
--
2.39.2
On Wed, Apr 10, 2024 at 1:29 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2024-Mar-29, Tender Wang wrote:
I think aboved case can explain what's meaning about comments in
dropconstraint_internal.
But here, in RemoveConstraintById() , we only care about primary key case,
so NOT NULL is better to removed from comments.Actually, I think it's better if all the resets of attnotnull occur in
RemoveConstraintById, for both types of constraints; we would keep that
block in dropconstraint_internal only to raise errors in the cases where
the constraint is protecting a replica identity or a generated column.
Something like the attached, perhaps, may need more polish.
DROP TABLE if exists notnull_tbl2;
CREATE TABLE notnull_tbl2 (c0 int generated by default as IDENTITY, c1 int);
ALTER TABLE notnull_tbl2 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
ALTER TABLE notnull_tbl2 DROP CONSTRAINT notnull_tbl2_c0_not_null;
ALTER TABLE notnull_tbl2 DROP c1;
\d notnull_tbl2
last "\d notnull_tbl2" command, master output is:
Table "public.notnull_tbl2"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+----------------------------------
c0 | integer | | not null | generated by default as identity
last "\d notnull_tbl2" command, applying
0001-Correctly-reset-attnotnull-when-constraints-dropped-.patch
output:
Table "public.notnull_tbl2"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+----------------------------------
c0 | integer | | | generated by default as identity
there may also have similar issues with the replicate identity.
jian he <jian.universality@gmail.com> 于2024年4月10日周三 14:10写道:
On Wed, Apr 10, 2024 at 1:29 AM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:On 2024-Mar-29, Tender Wang wrote:
I think aboved case can explain what's meaning about comments in
dropconstraint_internal.
But here, in RemoveConstraintById() , we only care about primary keycase,
so NOT NULL is better to removed from comments.
Actually, I think it's better if all the resets of attnotnull occur in
RemoveConstraintById, for both types of constraints; we would keep that
block in dropconstraint_internal only to raise errors in the cases where
the constraint is protecting a replica identity or a generated column.
Something like the attached, perhaps, may need more polish.DROP TABLE if exists notnull_tbl2;
CREATE TABLE notnull_tbl2 (c0 int generated by default as IDENTITY, c1
int);
ALTER TABLE notnull_tbl2 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
ALTER TABLE notnull_tbl2 DROP CONSTRAINT notnull_tbl2_c0_not_null;
ALTER TABLE notnull_tbl2 DROP c1;
\d notnull_tbl2last "\d notnull_tbl2" command, master output is:
Table "public.notnull_tbl2"
Column | Type | Collation | Nullable | Default--------+---------+-----------+----------+----------------------------------
c0 | integer | | not null | generated by default as identitylast "\d notnull_tbl2" command, applying
0001-Correctly-reset-attnotnull-when-constraints-dropped-.patch
output:
Table "public.notnull_tbl2"
Column | Type | Collation | Nullable | Default--------+---------+-----------+----------+----------------------------------
c0 | integer | | | generated by default as identity
Hmm,
ALTER TABLE notnull_tbl2 DROP c1; will not call dropconstraint_internal().
When dropping PK constraint indirectly, c0's attnotnull was set to false in
RemoveConstraintById().
--
Tender Wang
OpenPie: https://en.openpie.com/
another related bug, in master.
drop table if exists notnull_tbl1;
CREATE TABLE notnull_tbl1 (c0 int not null, c1 int);
ALTER TABLE notnull_tbl1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
\d+ notnull_tbl1
ALTER TABLE notnull_tbl1 ALTER c0 DROP NOT NULL;
ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL;
"ALTER TABLE notnull_tbl1 ALTER c0 DROP NOT NULL;"
should fail?
I didn't investigate deep enough.
jian he <jian.universality@gmail.com> 于2024年4月10日周三 17:34写道:
another related bug, in master.
drop table if exists notnull_tbl1;
CREATE TABLE notnull_tbl1 (c0 int not null, c1 int);
ALTER TABLE notnull_tbl1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
\d+ notnull_tbl1
ALTER TABLE notnull_tbl1 ALTER c0 DROP NOT NULL;
ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL;"ALTER TABLE notnull_tbl1 ALTER c0 DROP NOT NULL;"
should fail?
Yeah, it should fail as before, because c0 is primary key.
In master, although c0's pg_attribute.attnotnull is still true, but its
not-null constraint has been deleted
in dropconstraint_internal().
If we drop column c1 after dropping c0 not null, the primary key will be
dropped indirectly.
And now you can see c0 is still not-null if you do \d+ notnull_tbl1. But it
will report error "not found not-null"
if you alter c0 drop not null.
postgres=# ALTER TABLE notnull_tbl1 ALTER c0 DROP NOT NULL;
ALTER TABLE
postgres=# \d+ notnull_tbl1
Table "public.notnull_tbl1"
Column | Type | Collation | Nullable | Default | Storage | Compression
| Stats target | Description
--------+---------+-----------+----------+---------+---------+-------------+--------------+-------------
c0 | integer | | not null | | plain |
| |
c1 | integer | | not null | | plain |
| |
Indexes:
"q" PRIMARY KEY, btree (c0, c1)
Access method: heap
postgres=# alter table notnull_tbl1 drop c1;
ALTER TABLE
postgres=# \d notnull_tbl1
Table "public.notnull_tbl1"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------
c0 | integer | | not null |
postgres=# alter table notnull_tbl1 alter c0 drop not null;
ERROR: could not find not-null constraint on column "c0", relation
"notnull_tbl1"
--
Tender Wang
OpenPie: https://en.openpie.com/
On 2024-Apr-10, Tender Wang wrote:
Yeah, it should fail as before, because c0 is primary key.
In master, although c0's pg_attribute.attnotnull is still true, but its
not-null constraint has been deleted
in dropconstraint_internal().
Yeah, the problem here is that we need to do the checks whenever the
constraints are dropped, either directly or indirectly ... but we cannot
do them in RemoveConstraintById, because if you elog(ERROR) there, it
won't let you use DROP TABLE (which will also arrive at
RemoveConstraintById):
55490 17devel 2220048=# drop table notnull_tbl2;
ERROR: column "c0" of relation "notnull_tbl2" is an identity column
... which is of course kinda ridiculous, so this is not a viable
alternative. The problem is that RemoveConstraintById doesn't have
sufficient context about the whole operation. Worse: it cannot feed
its operations back into the alter table state.
I had a thought yesterday about making the resets of attnotnull and the
tests for replica identity and PKs to a separate ALTER TABLE pass,
independent of RemoveConstraintById (which would continue to be
responsible only for dropping the catalog row, as currently).
This may make the whole thing simpler. I'm on it.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"Hay que recordar que la existencia en el cosmos, y particularmente la
elaboración de civilizaciones dentro de él no son, por desgracia,
nada idílicas" (Ijon Tichy)
On 2024-Apr-10, jian he wrote:
another related bug, in master.
drop table if exists notnull_tbl1;
CREATE TABLE notnull_tbl1 (c0 int not null, c1 int);
ALTER TABLE notnull_tbl1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
\d+ notnull_tbl1
ALTER TABLE notnull_tbl1 ALTER c0 DROP NOT NULL;
ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL;"ALTER TABLE notnull_tbl1 ALTER c0 DROP NOT NULL;"
should fail?
No, this should not fail, and it is working correctly in master. You
can drop the not-null constraint, but the column will still be
non-nullable, because the primary key still exists. If you drop the
primary key later, then the column becomes nullable. This is by design.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"El miedo atento y previsor es la madre de la seguridad" (E. Burke)
It turns out that trying to close all holes that lead to columns marked
not-null without a pg_constraint row is not possible within the ALTER
TABLE framework, because it can happen outside it also. Consider this
CREATE DOMAIN dom1 AS integer;
CREATE TABLE notnull_tbl (a dom1, b int, PRIMARY KEY (a, b));
DROP DOMAIN dom1 CASCADE;
In this case you'll end up with b having attnotnull=true and no
constraint; and no amount of messing with tablecmds.c will fix it.
So I propose to instead allow those constraints, and treat them as
second-class citizens. We allow dropping them with ALTER TABLE DROP NOT
NULL, and we allow to create a backing full-fledged constraint with SET
NOT NULL or ADD CONSTRAINT. So here's a partial crude initial patch to
do that.
One thing missing here is pg_dump support. If you just dump this table,
it'll end up with no constraint at all. That's obviously bad, so I
propose we have pg_dump add a regular NOT NULL constraint for those, to
avoid perpetuating the weird situation further.
Another thing I wonder if whether I should use the existing
set_attnotnull() instead of adding drop_orphaned_notnull(). Or we could
just inline the code in ATExecDropNotNull, since it's small and
self-contained.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"Postgres is bloatware by design: it was built to house
PhD theses." (Joey Hellerstein, SIGMOD annual conference 2002)
Attachments:
0001-Handle-ALTER-.-DROP-NOT-NULL-when-no-pg_constraint-r.patchtext/x-diff; charset=utf-8Download
From 3a00358847f1792c01e608909dc8a4a0edb8739c Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Wed, 10 Apr 2024 13:02:35 +0200
Subject: [PATCH] Handle ALTER .. DROP NOT NULL when no pg_constraint row
exists
---
src/backend/commands/tablecmds.c | 67 +++++++++++++++++++----
src/test/regress/expected/constraints.out | 55 +++++++++++++++++++
src/test/regress/sql/constraints.sql | 29 ++++++++++
3 files changed, 139 insertions(+), 12 deletions(-)
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 865c6331c1..f692001e55 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -448,6 +448,7 @@ static ObjectAddress ATExecDropNotNull(Relation rel, const char *colName, bool r
LOCKMODE lockmode);
static bool set_attnotnull(List **wqueue, Relation rel,
AttrNumber attnum, bool recurse, LOCKMODE lockmode);
+static void drop_orphaned_notnull(Oid relationOid, AttrNumber attnum);
static ObjectAddress ATExecSetNotNull(List **wqueue, Relation rel,
char *constrname, char *colName,
bool recurse, bool recursing,
@@ -7678,17 +7679,23 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse,
}
/*
- * Find the constraint that makes this column NOT NULL.
+ * Find the constraint that makes this column NOT NULL, and drop it if we
+ * see one. dropconstraint_internal() will do necessary consistency
+ * checking. If there isn't one, there are two possibilities: either the
+ * column is marked attnotnull because it's part of the primary key, and
+ * then we just throw an appropriate error; or it's a leftover marking that
+ * we can remove. However, before doing the latter, to avoid breaking
+ * consistency any further, prevent this if the column is part of the
+ * replica identity.
*/
conTup = findNotNullConstraint(RelationGetRelid(rel), colName);
if (conTup == NULL)
{
Bitmapset *pkcols;
+ Bitmapset *ircols;
/*
- * There's no not-null constraint, so throw an error. If the column
- * is in a primary key, we can throw a specific error. Otherwise,
- * this is unexpected.
+ * If the column is in a primary key, throw a specific error message.
*/
pkcols = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_PRIMARY_KEY);
if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber,
@@ -7697,16 +7704,25 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse,
errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("column \"%s\" is in a primary key", colName));
- /* this shouldn't happen */
- elog(ERROR, "could not find not-null constraint on column \"%s\", relation \"%s\"",
- colName, RelationGetRelationName(rel));
+ /* Also throw an error if the column is in the replica identity */
+ ircols = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_IDENTITY_KEY);
+ if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, ircols))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("column \"%s\" is in index used as replica identity",
+ get_attname(RelationGetRelid(rel), attnum, false)));
+
+ /* Otherwise, just remove the attnotnull marking and do nothing else. */
+ drop_orphaned_notnull(RelationGetRelid(rel), attnum);
}
+ else
+ {
+ readyRels = NIL;
+ dropconstraint_internal(rel, conTup, DROP_RESTRICT, recurse, false,
+ false, &readyRels, lockmode);
- readyRels = NIL;
- dropconstraint_internal(rel, conTup, DROP_RESTRICT, recurse, false,
- false, &readyRels, lockmode);
-
- heap_freetuple(conTup);
+ heap_freetuple(conTup);
+ }
InvokeObjectPostAlterHook(RelationRelationId,
RelationGetRelid(rel), attnum);
@@ -7796,6 +7812,33 @@ set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, bool recurse,
return retval;
}
+/*
+ * In rare cases, a column can end up with an "orphaned" attnotnull marking
+ * with no corresponding pg_constraint row. If the user then does ALTER TABLE
+ * DROP NOT NULL, this takes care of resetting that.
+ */
+static void
+drop_orphaned_notnull(Oid relationOid, AttrNumber attnum)
+{
+ Relation attr_rel;
+ HeapTuple tuple;
+ Form_pg_attribute attForm;
+
+ attr_rel = table_open(AttributeRelationId, RowExclusiveLock);
+
+ tuple = SearchSysCacheCopyAttNum(relationOid, attnum);
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for attribute %d of relation %u",
+ attnum, relationOid);
+ attForm = (Form_pg_attribute) GETSTRUCT(tuple);
+
+ attForm->attnotnull = false;
+ CatalogTupleUpdate(attr_rel, &tuple->t_self, tuple);
+
+ table_close(attr_rel, RowExclusiveLock);
+}
+
+
/*
* ALTER TABLE ALTER COLUMN SET NOT NULL
*
diff --git a/src/test/regress/expected/constraints.out b/src/test/regress/expected/constraints.out
index d9d8408e86..f0c166dc4f 100644
--- a/src/test/regress/expected/constraints.out
+++ b/src/test/regress/expected/constraints.out
@@ -866,6 +866,61 @@ select conname, contype, conkey from pg_constraint where conrelid = 'notnull_tbl
foobar | n | {1}
(1 row)
+DROP TABLE notnull_tbl1;
+-- make sure attnotnull is reset correctly when a PK is dropped indirectly
+CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1));
+ALTER TABLE notnull_tbl1 DROP c1;
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c0 | integer | | not null | | plain | |
+
+ALTER TABLE notnull_tbl1 ALTER c0 DROP NOT NULL;
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c0 | integer | | | | plain | |
+
+DROP TABLE notnull_tbl1;
+-- again
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int, PRIMARY KEY (c0, c1));
+DROP DOMAIN notnull_dom1 CASCADE;
+NOTICE: drop cascades to column c0 of table notnull_tbl1
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c1 | integer | | not null | | plain | |
+
+ALTER TABLE notnull_tbl1 ALTER c1 SET NOT NULL;
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c1 | integer | | not null | | plain | |
+Not-null constraints:
+ "notnull_tbl1_c1_not_null" NOT NULL "c1"
+
+DROP TABLE notnull_tbl1;
+-- with a REPLICA IDENTITY column
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int UNIQUE, PRIMARY KEY (c0, c1));
+ALTER TABLE notnull_tbl1 REPLICA IDENTITY USING INDEX notnull_tbl1_c1_key;
+DROP DOMAIN notnull_dom1 CASCADE;
+NOTICE: drop cascades to column c0 of table notnull_tbl1
+ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL;
+ERROR: column "c1" is in index used as replica identity
+DROP TABLE notnull_tbl1;
+-- with an identity generated column
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int generated by default as identity, PRIMARY KEY (c0, c1));
+DROP DOMAIN notnull_dom1 CASCADE;
+NOTICE: drop cascades to column c0 of table notnull_tbl1
+ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL;
+ERROR: column "c1" of relation "notnull_tbl1" is an identity column
DROP TABLE notnull_tbl1;
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
diff --git a/src/test/regress/sql/constraints.sql b/src/test/regress/sql/constraints.sql
index 87d685ae39..e946c29f18 100644
--- a/src/test/regress/sql/constraints.sql
+++ b/src/test/regress/sql/constraints.sql
@@ -599,6 +599,35 @@ ALTER TABLE notnull_tbl1 ADD CONSTRAINT foobar NOT NULL a;
select conname, contype, conkey from pg_constraint where conrelid = 'notnull_tbl1'::regclass;
DROP TABLE notnull_tbl1;
+-- make sure attnotnull is reset correctly when a PK is dropped indirectly
+CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1));
+ALTER TABLE notnull_tbl1 DROP c1;
+\d+ notnull_tbl1
+ALTER TABLE notnull_tbl1 ALTER c0 DROP NOT NULL;
+\d+ notnull_tbl1
+DROP TABLE notnull_tbl1;
+-- again
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int, PRIMARY KEY (c0, c1));
+DROP DOMAIN notnull_dom1 CASCADE;
+\d+ notnull_tbl1
+ALTER TABLE notnull_tbl1 ALTER c1 SET NOT NULL;
+\d+ notnull_tbl1
+DROP TABLE notnull_tbl1;
+-- with a REPLICA IDENTITY column
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int UNIQUE, PRIMARY KEY (c0, c1));
+ALTER TABLE notnull_tbl1 REPLICA IDENTITY USING INDEX notnull_tbl1_c1_key;
+DROP DOMAIN notnull_dom1 CASCADE;
+ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL;
+DROP TABLE notnull_tbl1;
+-- with an identity generated column
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int generated by default as identity, PRIMARY KEY (c0, c1));
+DROP DOMAIN notnull_dom1 CASCADE;
+ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL;
+DROP TABLE notnull_tbl1;
+
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
--
2.39.2
On Wed, Apr 10, 2024 at 7:01 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2024-Apr-10, jian he wrote:
another related bug, in master.
drop table if exists notnull_tbl1;
CREATE TABLE notnull_tbl1 (c0 int not null, c1 int);
ALTER TABLE notnull_tbl1 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
\d+ notnull_tbl1
ALTER TABLE notnull_tbl1 ALTER c0 DROP NOT NULL;
ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL;"ALTER TABLE notnull_tbl1 ALTER c0 DROP NOT NULL;"
should fail?No, this should not fail, and it is working correctly in master. You
can drop the not-null constraint, but the column will still be
non-nullable, because the primary key still exists. If you drop the
primary key later, then the column becomes nullable. This is by design.
now I got it. the second time, it will fail.
it should be the expected behavior.
per commit:
https://git.postgresql.org/cgit/postgresql.git/commit/?id=14dd0f27d7cd56ffae9ecdbe324965073d01a9ff
In the function dropconstraint_internal, I changed "foreach" to
"foreach_int" in some places,
and other minor cosmetic changes within the function
dropconstraint_internal only.
Since I saw your changes in dropconstraint_internal, I posted here.
I will review your latest patch later.
Attachments:
v1-0001-minor-coesmetuic-refactor-in-dropconstraint_inter.patchapplication/x-patch; name=v1-0001-minor-coesmetuic-refactor-in-dropconstraint_inter.patchDownload
From dd41bf8d1f2dea5725909150609f8e565e11efcf Mon Sep 17 00:00:00 2001
From: jian he <jian.universality@gmail.com>
Date: Wed, 10 Apr 2024 20:49:04 +0800
Subject: [PATCH v1 1/1] minor coesmetuic refactor in dropconstraint_internal
mainly change `foreach` to `foreach_int` while iterating through list unconstrained_cols.
---
src/backend/commands/tablecmds.c | 14 ++++++--------
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 8a02c5b0..8dc623e8 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -12968,12 +12968,11 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
* have had pg_attribute.attnotnull set. See if we need to reset it, and
* do so.
*/
- if (unconstrained_cols)
+ if (unconstrained_cols != NIL)
{
Relation attrel;
Bitmapset *pkcols;
Bitmapset *ircols;
- ListCell *lc;
/* Make the above deletion visible */
CommandCounterIncrement();
@@ -12989,9 +12988,8 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
INDEX_ATTR_BITMAP_PRIMARY_KEY);
ircols = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_IDENTITY_KEY);
- foreach(lc, unconstrained_cols)
+ foreach_int(attnum, unconstrained_cols)
{
- AttrNumber attnum = lfirst_int(lc);
HeapTuple atttup;
HeapTuple contup;
Form_pg_attribute attForm;
@@ -13039,11 +13037,11 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
* It's not valid to drop the not-null constraint for a column in
* the replica identity index, either. (FULL is not affected.)
*/
- if (bms_is_member(lfirst_int(lc) - FirstLowInvalidHeapAttributeNumber, ircols))
+ if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, ircols))
ereport(ERROR,
errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("column \"%s\" is in index used as replica identity",
- get_attname(RelationGetRelid(rel), lfirst_int(lc), false)));
+ get_attname(RelationGetRelid(rel), attnum, false)));
/* Reset attnotnull */
if (attForm->attnotnull)
@@ -13233,10 +13231,10 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
* Find out the list of column names to process. Fortunately, we
* already have the list of column numbers.
*/
- foreach(lc, unconstrained_cols)
+ foreach_int(attnum, unconstrained_cols)
{
colnames = lappend(colnames, get_attname(RelationGetRelid(rel),
- lfirst_int(lc), false));
+ attnum, false));
}
foreach(child, children)
--
2.34.1
On 2024-Apr-10, Alvaro Herrera wrote:
One thing missing here is pg_dump support. If you just dump this table,
it'll end up with no constraint at all. That's obviously bad, so I
propose we have pg_dump add a regular NOT NULL constraint for those, to
avoid perpetuating the weird situation further.
Here's another crude patchset, this time including the pg_dump aspect.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"On the other flipper, one wrong move and we're Fatal Exceptions"
(T.U.X.: Term Unit X - http://www.thelinuxreview.com/TUX/)
Attachments:
v2-0001-Handle-ALTER-.-DROP-NOT-NULL-when-no-pg_constrain.patchtext/x-diff; charset=utf-8Download
From 3a00358847f1792c01e608909dc8a4a0edb8739c Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Wed, 10 Apr 2024 13:02:35 +0200
Subject: [PATCH v2 1/2] Handle ALTER .. DROP NOT NULL when no pg_constraint
row exists
---
src/backend/commands/tablecmds.c | 67 +++++++++++++++++++----
src/test/regress/expected/constraints.out | 55 +++++++++++++++++++
src/test/regress/sql/constraints.sql | 29 ++++++++++
3 files changed, 139 insertions(+), 12 deletions(-)
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 865c6331c1..f692001e55 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -448,6 +448,7 @@ static ObjectAddress ATExecDropNotNull(Relation rel, const char *colName, bool r
LOCKMODE lockmode);
static bool set_attnotnull(List **wqueue, Relation rel,
AttrNumber attnum, bool recurse, LOCKMODE lockmode);
+static void drop_orphaned_notnull(Oid relationOid, AttrNumber attnum);
static ObjectAddress ATExecSetNotNull(List **wqueue, Relation rel,
char *constrname, char *colName,
bool recurse, bool recursing,
@@ -7678,17 +7679,23 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse,
}
/*
- * Find the constraint that makes this column NOT NULL.
+ * Find the constraint that makes this column NOT NULL, and drop it if we
+ * see one. dropconstraint_internal() will do necessary consistency
+ * checking. If there isn't one, there are two possibilities: either the
+ * column is marked attnotnull because it's part of the primary key, and
+ * then we just throw an appropriate error; or it's a leftover marking that
+ * we can remove. However, before doing the latter, to avoid breaking
+ * consistency any further, prevent this if the column is part of the
+ * replica identity.
*/
conTup = findNotNullConstraint(RelationGetRelid(rel), colName);
if (conTup == NULL)
{
Bitmapset *pkcols;
+ Bitmapset *ircols;
/*
- * There's no not-null constraint, so throw an error. If the column
- * is in a primary key, we can throw a specific error. Otherwise,
- * this is unexpected.
+ * If the column is in a primary key, throw a specific error message.
*/
pkcols = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_PRIMARY_KEY);
if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber,
@@ -7697,16 +7704,25 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse,
errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("column \"%s\" is in a primary key", colName));
- /* this shouldn't happen */
- elog(ERROR, "could not find not-null constraint on column \"%s\", relation \"%s\"",
- colName, RelationGetRelationName(rel));
+ /* Also throw an error if the column is in the replica identity */
+ ircols = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_IDENTITY_KEY);
+ if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, ircols))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("column \"%s\" is in index used as replica identity",
+ get_attname(RelationGetRelid(rel), attnum, false)));
+
+ /* Otherwise, just remove the attnotnull marking and do nothing else. */
+ drop_orphaned_notnull(RelationGetRelid(rel), attnum);
}
+ else
+ {
+ readyRels = NIL;
+ dropconstraint_internal(rel, conTup, DROP_RESTRICT, recurse, false,
+ false, &readyRels, lockmode);
- readyRels = NIL;
- dropconstraint_internal(rel, conTup, DROP_RESTRICT, recurse, false,
- false, &readyRels, lockmode);
-
- heap_freetuple(conTup);
+ heap_freetuple(conTup);
+ }
InvokeObjectPostAlterHook(RelationRelationId,
RelationGetRelid(rel), attnum);
@@ -7796,6 +7812,33 @@ set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, bool recurse,
return retval;
}
+/*
+ * In rare cases, a column can end up with an "orphaned" attnotnull marking
+ * with no corresponding pg_constraint row. If the user then does ALTER TABLE
+ * DROP NOT NULL, this takes care of resetting that.
+ */
+static void
+drop_orphaned_notnull(Oid relationOid, AttrNumber attnum)
+{
+ Relation attr_rel;
+ HeapTuple tuple;
+ Form_pg_attribute attForm;
+
+ attr_rel = table_open(AttributeRelationId, RowExclusiveLock);
+
+ tuple = SearchSysCacheCopyAttNum(relationOid, attnum);
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for attribute %d of relation %u",
+ attnum, relationOid);
+ attForm = (Form_pg_attribute) GETSTRUCT(tuple);
+
+ attForm->attnotnull = false;
+ CatalogTupleUpdate(attr_rel, &tuple->t_self, tuple);
+
+ table_close(attr_rel, RowExclusiveLock);
+}
+
+
/*
* ALTER TABLE ALTER COLUMN SET NOT NULL
*
diff --git a/src/test/regress/expected/constraints.out b/src/test/regress/expected/constraints.out
index d9d8408e86..f0c166dc4f 100644
--- a/src/test/regress/expected/constraints.out
+++ b/src/test/regress/expected/constraints.out
@@ -866,6 +866,61 @@ select conname, contype, conkey from pg_constraint where conrelid = 'notnull_tbl
foobar | n | {1}
(1 row)
+DROP TABLE notnull_tbl1;
+-- make sure attnotnull is reset correctly when a PK is dropped indirectly
+CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1));
+ALTER TABLE notnull_tbl1 DROP c1;
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c0 | integer | | not null | | plain | |
+
+ALTER TABLE notnull_tbl1 ALTER c0 DROP NOT NULL;
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c0 | integer | | | | plain | |
+
+DROP TABLE notnull_tbl1;
+-- again
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int, PRIMARY KEY (c0, c1));
+DROP DOMAIN notnull_dom1 CASCADE;
+NOTICE: drop cascades to column c0 of table notnull_tbl1
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c1 | integer | | not null | | plain | |
+
+ALTER TABLE notnull_tbl1 ALTER c1 SET NOT NULL;
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c1 | integer | | not null | | plain | |
+Not-null constraints:
+ "notnull_tbl1_c1_not_null" NOT NULL "c1"
+
+DROP TABLE notnull_tbl1;
+-- with a REPLICA IDENTITY column
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int UNIQUE, PRIMARY KEY (c0, c1));
+ALTER TABLE notnull_tbl1 REPLICA IDENTITY USING INDEX notnull_tbl1_c1_key;
+DROP DOMAIN notnull_dom1 CASCADE;
+NOTICE: drop cascades to column c0 of table notnull_tbl1
+ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL;
+ERROR: column "c1" is in index used as replica identity
+DROP TABLE notnull_tbl1;
+-- with an identity generated column
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int generated by default as identity, PRIMARY KEY (c0, c1));
+DROP DOMAIN notnull_dom1 CASCADE;
+NOTICE: drop cascades to column c0 of table notnull_tbl1
+ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL;
+ERROR: column "c1" of relation "notnull_tbl1" is an identity column
DROP TABLE notnull_tbl1;
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
diff --git a/src/test/regress/sql/constraints.sql b/src/test/regress/sql/constraints.sql
index 87d685ae39..e946c29f18 100644
--- a/src/test/regress/sql/constraints.sql
+++ b/src/test/regress/sql/constraints.sql
@@ -599,6 +599,35 @@ ALTER TABLE notnull_tbl1 ADD CONSTRAINT foobar NOT NULL a;
select conname, contype, conkey from pg_constraint where conrelid = 'notnull_tbl1'::regclass;
DROP TABLE notnull_tbl1;
+-- make sure attnotnull is reset correctly when a PK is dropped indirectly
+CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1));
+ALTER TABLE notnull_tbl1 DROP c1;
+\d+ notnull_tbl1
+ALTER TABLE notnull_tbl1 ALTER c0 DROP NOT NULL;
+\d+ notnull_tbl1
+DROP TABLE notnull_tbl1;
+-- again
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int, PRIMARY KEY (c0, c1));
+DROP DOMAIN notnull_dom1 CASCADE;
+\d+ notnull_tbl1
+ALTER TABLE notnull_tbl1 ALTER c1 SET NOT NULL;
+\d+ notnull_tbl1
+DROP TABLE notnull_tbl1;
+-- with a REPLICA IDENTITY column
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int UNIQUE, PRIMARY KEY (c0, c1));
+ALTER TABLE notnull_tbl1 REPLICA IDENTITY USING INDEX notnull_tbl1_c1_key;
+DROP DOMAIN notnull_dom1 CASCADE;
+ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL;
+DROP TABLE notnull_tbl1;
+-- with an identity generated column
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int generated by default as identity, PRIMARY KEY (c0, c1));
+DROP DOMAIN notnull_dom1 CASCADE;
+ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL;
+DROP TABLE notnull_tbl1;
+
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
--
2.39.2
v2-0002-support-this-in-pg_dump.patchtext/x-diff; charset=utf-8Download
From fe2c032c958b5855434232c9b699ef75a887e249 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Wed, 10 Apr 2024 19:10:52 +0200
Subject: [PATCH v2 2/2] support this in pg_dump
---
src/bin/pg_dump/pg_dump.c | 30 ++++++++++++++++++++++--------
1 file changed, 22 insertions(+), 8 deletions(-)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index c52e961b30..7dcbfaf8dc 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -8788,12 +8788,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
"), E',\n ') AS attfdwoptions,\n");
/*
- * Find out any NOT NULL markings for each column. In 17 and up we have
- * to read pg_constraint, and keep track whether it's NO INHERIT; in older
- * versions we rely on pg_attribute.attnotnull.
+ * Find out any NOT NULL markings for each column. In 17 and up we
+ * read pg_constraint to obtain the constraint name. notnull_noinherit
+ * is set according to the NO INHERIT property. For versions prior to 17,
+ * we store an empty string as the name when a constraint is marked as
+ * attnotnull (this cues dumpTableSchema to print the NOT NULL clause
+ * without a name); also, such cases are never NO INHERIT.
*
- * We also track whether the constraint was defined directly in this table
- * or via an ancestor, for binary upgrade.
+ * We track in notnull_inh whether the constraint was defined directly in
+ * this table or via an ancestor, for binary upgrade.
*
* Lastly, we need to know if the PK for the table involves each column;
* for columns that are there we need a NOT NULL marking even if there's
@@ -8801,13 +8804,24 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
* NULLs after the data is loaded when the PK is created, later in the
* dump; for this case we add throwaway constraints that are dropped once
* the PK is created.
+ *
+ * Another complication arises from columns that have attnotnull set, but
+ * for which no corresponding not-null nor PK constraint exists. This can
+ * happen if, for example, a primary key is dropped indirectly -- say,
+ * because one of its columns is dropped. This is an irregular condition,
+ * so we don't work hard to preserve it, and instead act as though an
+ * unnamed not-null constraint exists.
*/
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(q,
- "co.conname AS notnull_name,\n"
- "co.connoinherit AS notnull_noinherit,\n"
+ "CASE WHEN co.conname IS NOT NULL THEN co.conname "
+ " WHEN a.attnotnull AND copk.conname IS NULL THEN '' ELSE NULL END AS notnull_name,\n"
+ "CASE WHEN co.conname IS NOT NULL THEN co.connoinherit "
+ " WHEN a.attnotnull THEN false ELSE NULL END AS notnull_noinherit,\n"
"copk.conname IS NOT NULL as notnull_is_pk,\n"
- "coalesce(NOT co.conislocal, true) AS notnull_inh,\n");
+ "CASE WHEN co.conname IS NOT NULL THEN "
+ " coalesce(NOT co.conislocal, true) "
+ "ELSE false END as notnull_inh,\n");
else
appendPQExpBufferStr(q,
"CASE WHEN a.attnotnull THEN '' ELSE NULL END AS notnull_name,\n"
--
2.39.2
Alvaro Herrera <alvherre@alvh.no-ip.org> 于2024年4月10日周三 21:58写道:
It turns out that trying to close all holes that lead to columns marked
not-null without a pg_constraint row is not possible within the ALTER
TABLE framework, because it can happen outside it also. Consider thisCREATE DOMAIN dom1 AS integer;
CREATE TABLE notnull_tbl (a dom1, b int, PRIMARY KEY (a, b));
DROP DOMAIN dom1 CASCADE;In this case you'll end up with b having attnotnull=true and no
constraint; and no amount of messing with tablecmds.c will fix it.
I try above case on my v4 patch[1]/messages/by-id/CAHewXNn_So7LUCxxxyDNfdvCQp1TnD3gTVECBZX2bT_nbPgraQ@mail.gmail.com -- Tender Wang OpenPie: https://en.openpie.com/, and it seems no result as what you said.
But, anyway, I now don't like updating other catalog in
RemoveConstraintById().
Because it will not be friendly for others who call RemoveConstraintById()
want only
to remove pg_constraint tuple, but actually it do more works stealthily.
So I propose to instead allow those constraints, and treat them as
second-class citizens. We allow dropping them with ALTER TABLE DROP NOT
NULL, and we allow to create a backing full-fledged constraint with SET
NOT NULL or ADD CONSTRAINT. So here's a partial crude initial patch to
do that.
Hmm, the patch looks like the patch in my first email in this thread. But
my v1 patch seem
a poc at most.
One thing missing here is pg_dump support. If you just dump this table,
it'll end up with no constraint at all. That's obviously bad, so I
propose we have pg_dump add a regular NOT NULL constraint for those, to
avoid perpetuating the weird situation further.Another thing I wonder if whether I should use the existing
set_attnotnull() instead of adding drop_orphaned_notnull(). Or we could
just inline the code in ATExecDropNotNull, since it's small and
self-contained.
I like just inline the code in ATExecDropNotNull, as you said, it's small
and self-contained.
in ATExecDropNotNull(), we had open the pg_attributed table and hold
RowExclusiveLock,
the tuple we also get.
What we do is set attnotnull = false, and call CatalogTupleUpdate.
--
Álvaro Herrera 48°01'N 7°57'E —
https://www.EnterpriseDB.com/
"Postgres is bloatware by design: it was built to house
PhD theses." (Joey Hellerstein, SIGMOD annual conference 2002)
[1]: /messages/by-id/CAHewXNn_So7LUCxxxyDNfdvCQp1TnD3gTVECBZX2bT_nbPgraQ@mail.gmail.com -- Tender Wang OpenPie: https://en.openpie.com/
/messages/by-id/CAHewXNn_So7LUCxxxyDNfdvCQp1TnD3gTVECBZX2bT_nbPgraQ@mail.gmail.com
--
Tender Wang
OpenPie: https://en.openpie.com/
On Wed, Apr 10, 2024 at 2:10 PM jian he <jian.universality@gmail.com> wrote:
DROP TABLE if exists notnull_tbl2;
CREATE TABLE notnull_tbl2 (c0 int generated by default as IDENTITY, c1 int);
ALTER TABLE notnull_tbl2 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
ALTER TABLE notnull_tbl2 DROP CONSTRAINT notnull_tbl2_c0_not_null;
ALTER TABLE notnull_tbl2 DROP c1;
\d notnull_tbl2
ALTER TABLE notnull_tbl2 DROP CONSTRAINT notnull_tbl2_c0_not_null;
per above sequence execution order, this should error out?
otherwise which "not null" (attribute|constraint) to anchor "generated
by default as identity" not null property?
"DROP c1" will drop the not null property for "c0" and "c1".
if "DROP CONSTRAINT notnull_tbl2_c0_not_nul" not error out, then
" ALTER TABLE notnull_tbl2 DROP c1;"
should either error out
or transform "c0" from "c0 int generated by default as identity"
to
"c0 int"
On Thu, Apr 11, 2024 at 1:23 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2024-Apr-10, Alvaro Herrera wrote:
One thing missing here is pg_dump support. If you just dump this table,
it'll end up with no constraint at all. That's obviously bad, so I
propose we have pg_dump add a regular NOT NULL constraint for those, to
avoid perpetuating the weird situation further.Here's another crude patchset, this time including the pg_dump aspect.
+DROP TABLE notnull_tbl1;
+-- make sure attnotnull is reset correctly when a PK is dropped indirectly
+CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1));
+ALTER TABLE notnull_tbl1 DROP c1;
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats
target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c0 | integer | | not null | | plain | |
+
this is not what we expected?
"not null" for "c0" now should be false?
am I missing something?
jian he <jian.universality@gmail.com> 于2024年4月11日周四 14:40写道:
On Wed, Apr 10, 2024 at 2:10 PM jian he <jian.universality@gmail.com>
wrote:DROP TABLE if exists notnull_tbl2;
CREATE TABLE notnull_tbl2 (c0 int generated by default as IDENTITY, c1int);
ALTER TABLE notnull_tbl2 ADD CONSTRAINT Q PRIMARY KEY(c0, c1);
ALTER TABLE notnull_tbl2 DROP CONSTRAINT notnull_tbl2_c0_not_null;
ALTER TABLE notnull_tbl2 DROP c1;
\d notnull_tbl2ALTER TABLE notnull_tbl2 DROP CONSTRAINT notnull_tbl2_c0_not_null;
per above sequence execution order, this should error out?
otherwise which "not null" (attribute|constraint) to anchor "generated
by default as identity" not null property?
"DROP c1" will drop the not null property for "c0" and "c1".
if "DROP CONSTRAINT notnull_tbl2_c0_not_nul" not error out, then
" ALTER TABLE notnull_tbl2 DROP c1;"
should either error out
or transform "c0" from "c0 int generated by default as identity"
to
"c0 int"I try above case on MASTER and MASTER with Alvaro V2 patch, and all work
correctly.
\d+ notnull_tbl2 will see not-null of "c0".
On Thu, Apr 11, 2024 at 1:23 AM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:On 2024-Apr-10, Alvaro Herrera wrote:
One thing missing here is pg_dump support. If you just dump this
table,
it'll end up with no constraint at all. That's obviously bad, so I
propose we have pg_dump add a regular NOT NULL constraint for those, to
avoid perpetuating the weird situation further.Here's another crude patchset, this time including the pg_dump aspect.
+DROP TABLE notnull_tbl1; +-- make sure attnotnull is reset correctly when a PK is dropped indirectly +CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1)); +ALTER TABLE notnull_tbl1 DROP c1; +\d+ notnull_tbl1 + Table "public.notnull_tbl1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description+--------+---------+-----------+----------+---------+---------+--------------+------------- + c0 | integer | | not null | | plain | | +this is not what we expected?
"not null" for "c0" now should be false?
am I missing something?
Yeah, now this is expected behavior.
Users can drop manually not-null of "c0" if they want, and no error
reporte.
--
Tender Wang
OpenPie: https://en.openpie.com/
On Thu, Apr 11, 2024 at 3:19 PM Tender Wang <tndrwang@gmail.com> wrote:
+DROP TABLE notnull_tbl1; +-- make sure attnotnull is reset correctly when a PK is dropped indirectly +CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1)); +ALTER TABLE notnull_tbl1 DROP c1; +\d+ notnull_tbl1 + Table "public.notnull_tbl1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+---------+-----------+----------+---------+---------+--------------+------------- + c0 | integer | | not null | | plain | | +this is not what we expected?
"not null" for "c0" now should be false?
am I missing something?Yeah, now this is expected behavior.
Users can drop manually not-null of "c0" if they want, and no error reporte.
sorry for the noise.
these two past patches confused me:
0001-Correctly-reset-attnotnull-when-constraints-dropped-.patch
v4-0001-Fix-pg_attribute-attnotnull-not-reset-when-droppe.patch
I thought dropping a column of primary key (ALTER TABLE notnull_tbl2 DROP c1)
will make the others key columns to not have "not null" property.
now I figured out that
dropping a column of primary key columns will not change other key
columns' "not null" property.
dropping the primary key associated constraint will make all key
columns "not null" property disappear.
v2-0001-Handle-ALTER-.-DROP-NOT-NULL-when-no-pg_constrain.patch
behavior looks fine to me now.
inline drop_orphaned_notnull in ATExecDropNotNull looks fine to me.
On 2024-Apr-11, jian he wrote:
now I figured out that
dropping a column of primary key columns will not change other key
columns' "not null" property.
dropping the primary key associated constraint will make all key
columns "not null" property disappear.
Well, I think you were right that we should try to handle the situation
of unmarking attnotnull as much as possible, to decrease the chances
that the problematic situation occurs. That means, we can use the
earlier code to handle DROP COLUMN when it causes a PK to be dropped --
even though we still need to handle the situation of an attnotnull flag
set with no pg_constraint row. I mean, we still have to handle DROP
DOMAIN correctly (and there might be other cases that I haven't thought
about) ... but surely this is a much less common situation than the one
you reported. So I'll merge everything and post an updated patch.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
Bob [Floyd] used to say that he was planning to get a Ph.D. by the "green
stamp method," namely by saving envelopes addressed to him as 'Dr. Floyd'.
After collecting 500 such letters, he mused, a university somewhere in
Arizona would probably grant him a degree. (Don Knuth)
On 2024-Apr-11, Alvaro Herrera wrote:
Well, I think you were right that we should try to handle the situation
of unmarking attnotnull as much as possible, to decrease the chances
that the problematic situation occurs. That means, we can use the
earlier code to handle DROP COLUMN when it causes a PK to be dropped --
even though we still need to handle the situation of an attnotnull flag
set with no pg_constraint row. I mean, we still have to handle DROP
DOMAIN correctly (and there might be other cases that I haven't thought
about) ... but surely this is a much less common situation than the one
you reported. So I'll merge everything and post an updated patch.
Here's roughly what I'm thinking. If we drop a constraint, we can still
reset attnotnull in RemoveConstraintById(), but only after checking that
it's not a generated column or a replica identity. If they are, we
don't error out -- just skip the attnotnull update.
Now, about the code to allow ALTER TABLE DROP NOT NULL in case there's
no pg_constraint row, I think at this point it's mostly dead code,
because it can only happen when you have a replica identity or generated
column ... and the DROP NOT NULL should still prevent you from dropping
the flag anyway. But the case can still arise, if you change the
replica identity or ALTER TABLE ALTER COLUMN DROP DEFAULT, respectively.
I'm still not ready with this -- still not convinced about the new AT
pass. Also, I want to add a test for the pg_dump behavior, and there's
an XXX comment.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"The eagle never lost so much time, as
when he submitted to learn of the crow." (William Blake)
Attachments:
0001-Better-handle-indirect-constraint-drops.patchtext/x-diff; charset=utf-8Download
From ce8fff0ccb568a601ac9f765da7b7891066f522b Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Thu, 11 Apr 2024 12:29:34 +0200
Subject: [PATCH] Better handle indirect constraint drops
It is possible for certain cases to remove constraints without
maintaining the attnotnull in its correct state; for example if you drop
a column that's part of the primary key, and the remaining columns don't
have not-null constraints, then we should reset the attnotnull flags for
those other columns, but we don't. We can handle those cases better by
doing the attnotnull reset in RemoveConstraintById() instead of in
dropconstraint_internal().
However, there are some cases where we must not do so. For example if
those other columns are in replica identity indexes or are generated
identity columns, we must keep the flag, even though it results in the
catalog inconsistency that no constraint supports the flag.
Because of these exceptions, we also include a pg_dump hack to include a
not-null constraint when the attnotnull flag is set even if no
pg_constraint row exists. This part is undesirable but necessary,
because failing to handle the case can result in unrestorable dumps.
---
src/backend/catalog/pg_constraint.c | 116 +++++++++++++++++++++-
src/backend/commands/tablecmds.c | 108 ++++++++++----------
src/bin/pg_dump/pg_dump.c | 30 ++++--
src/test/regress/expected/constraints.out | 73 ++++++++++++++
src/test/regress/sql/constraints.sql | 34 +++++++
5 files changed, 301 insertions(+), 60 deletions(-)
diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c
index 604280d322..c48acf00b0 100644
--- a/src/backend/catalog/pg_constraint.c
+++ b/src/backend/catalog/pg_constraint.c
@@ -19,6 +19,7 @@
#include "access/htup_details.h"
#include "access/sysattr.h"
#include "access/table.h"
+#include "access/xact.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/heap.h"
@@ -916,6 +917,8 @@ RemoveConstraintById(Oid conId)
Relation conDesc;
HeapTuple tup;
Form_pg_constraint con;
+ bool dropping_pk = false;
+ List *unconstrained_cols = NIL;
conDesc = table_open(ConstraintRelationId, RowExclusiveLock);
@@ -940,7 +943,9 @@ RemoveConstraintById(Oid conId)
/*
* We need to update the relchecks count if it is a check constraint
* being dropped. This update will force backends to rebuild relcache
- * entries when we commit.
+ * entries when we commit. For not-null and primary key constraints,
+ * obtain the list of columns affected, so that we can reset their
+ * attnotnull flags below.
*/
if (con->contype == CONSTRAINT_CHECK)
{
@@ -967,6 +972,36 @@ RemoveConstraintById(Oid conId)
table_close(pgrel, RowExclusiveLock);
}
+ else if (con->contype == CONSTRAINT_NOTNULL)
+ {
+ unconstrained_cols = list_make1_int(extractNotNullColumn(tup));
+ }
+ else if (con->contype == CONSTRAINT_PRIMARY)
+ {
+ Datum adatum;
+ ArrayType *arr;
+ int numkeys;
+ bool isNull;
+ int16 *attnums;
+
+ dropping_pk = true;
+
+ adatum = heap_getattr(tup, Anum_pg_constraint_conkey,
+ RelationGetDescr(conDesc), &isNull);
+ if (isNull)
+ elog(ERROR, "null conkey for constraint %u", con->oid);
+ arr = DatumGetArrayTypeP(adatum); /* ensure not toasted */
+ numkeys = ARR_DIMS(arr)[0];
+ if (ARR_NDIM(arr) != 1 ||
+ numkeys < 0 ||
+ ARR_HASNULL(arr) ||
+ ARR_ELEMTYPE(arr) != INT2OID)
+ elog(ERROR, "conkey is not a 1-D smallint array");
+ attnums = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < numkeys; i++)
+ unconstrained_cols = lappend_int(unconstrained_cols, attnums[i]);
+ }
/* Keep lock on constraint's rel until end of xact */
table_close(rel, NoLock);
@@ -986,6 +1021,85 @@ RemoveConstraintById(Oid conId)
/* Fry the constraint itself */
CatalogTupleDelete(conDesc, &tup->t_self);
+ /*
+ * If this was a NOT NULL or the primary key, the constrained columns must
+ * have had pg_attribute.attnotnull set. See if we need to reset it, and
+ * do so.
+ */
+ if (unconstrained_cols)
+ {
+ Relation tablerel;
+ Relation attrel;
+ Bitmapset *pkcols;
+ ListCell *lc;
+
+ /* Make the above deletion visible */
+ CommandCounterIncrement();
+
+ tablerel = table_open(con->conrelid, NoLock); /* already have lock */
+ attrel = table_open(AttributeRelationId, RowExclusiveLock);
+
+ /*
+ * We want to test columns for their presence in the primary key, but
+ * only if we're not dropping it.
+ */
+ pkcols = dropping_pk ? NULL :
+ RelationGetIndexAttrBitmap(tablerel,
+ INDEX_ATTR_BITMAP_PRIMARY_KEY);
+
+ foreach(lc, unconstrained_cols)
+ {
+ AttrNumber attnum = lfirst_int(lc);
+ HeapTuple atttup;
+ HeapTuple contup;
+ Bitmapset *ircols;
+ Form_pg_attribute attForm;
+
+ /*
+ * Obtain pg_attribute tuple and verify conditions on it. We use
+ * a copy we can scribble on.
+ */
+ atttup = SearchSysCacheCopyAttNum(con->conrelid, attnum);
+ if (!HeapTupleIsValid(atttup))
+ elog(ERROR, "cache lookup failed for attribute %d of relation %u",
+ attnum, con->conrelid);
+ attForm = (Form_pg_attribute) GETSTRUCT(atttup);
+
+ /*
+ * Since the above deletion has been made visible, we can now
+ * search for any remaining constraints on this column (or these
+ * columns, in the case we're dropping a multicol primary key.)
+ * Then, verify whether any further NOT NULL or primary key
+ * exists, and reset attnotnull if none.
+ */
+ contup = findNotNullConstraintAttnum(con->conrelid, attnum);
+ if (contup ||
+ bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber,
+ pkcols))
+ continue;
+
+ /*
+ * ... but don't do it if the column is in the replica identity or
+ * it's a generated column
+ */
+ if (attForm->attidentity != '\0')
+ continue;
+ ircols = RelationGetIndexAttrBitmap(tablerel, INDEX_ATTR_BITMAP_IDENTITY_KEY); /* XXX bms_free */
+ if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, ircols))
+ continue;
+
+ /* Reset attnotnull */
+ if (attForm->attnotnull)
+ {
+ attForm->attnotnull = false;
+ CatalogTupleUpdate(attrel, &atttup->t_self, atttup);
+ }
+ }
+
+ table_close(attrel, RowExclusiveLock);
+ table_close(tablerel, NoLock);
+ }
+
/* Clean up */
ReleaseSysCache(tup);
table_close(conDesc, RowExclusiveLock);
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 8a98a0af48..aa9c11760f 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -149,6 +149,7 @@ typedef enum AlterTablePass
AT_PASS_ALTER_TYPE, /* ALTER COLUMN TYPE */
AT_PASS_ADD_COL, /* ADD COLUMN */
AT_PASS_SET_EXPRESSION, /* ALTER SET EXPRESSION */
+ AT_PASS_OLD_COL_ATTRS, /* re-install attnotnull */
AT_PASS_OLD_INDEX, /* re-add existing indexes */
AT_PASS_OLD_CONSTR, /* re-add existing constraints */
/* We could support a RENAME COLUMN pass here, but not currently used */
@@ -7678,17 +7679,23 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse,
}
/*
- * Find the constraint that makes this column NOT NULL.
+ * Find the constraint that makes this column NOT NULL, and drop it if we
+ * see one. dropconstraint_internal() will do necessary consistency
+ * checking. If there isn't one, there are two possibilities: either the
+ * column is marked attnotnull because it's part of the primary key, and
+ * then we just throw an appropriate error; or it's a leftover marking that
+ * we can remove. However, before doing the latter, to avoid breaking
+ * consistency any further, prevent this if the column is part of the
+ * replica identity.
*/
conTup = findNotNullConstraint(RelationGetRelid(rel), colName);
if (conTup == NULL)
{
Bitmapset *pkcols;
+ Bitmapset *ircols;
/*
- * There's no not-null constraint, so throw an error. If the column
- * is in a primary key, we can throw a specific error. Otherwise,
- * this is unexpected.
+ * If the column is in a primary key, throw a specific error message.
*/
pkcols = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_PRIMARY_KEY);
if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber,
@@ -7697,16 +7704,27 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse,
errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("column \"%s\" is in a primary key", colName));
- /* this shouldn't happen */
- elog(ERROR, "could not find not-null constraint on column \"%s\", relation \"%s\"",
- colName, RelationGetRelationName(rel));
+ /* Also throw an error if the column is in the replica identity */
+ ircols = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_IDENTITY_KEY);
+ if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, ircols))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("column \"%s\" is in index used as replica identity",
+ get_attname(RelationGetRelid(rel), attnum, false)));
+
+ /* Otherwise, just remove the attnotnull marking and do nothing else. */
+ attTup->attnotnull = false;
+ CatalogTupleUpdate(attr_rel, &tuple->t_self, tuple);
}
+ else
+ {
+ /* The normal case: we have a pg_constraint row, remove it */
+ readyRels = NIL;
+ dropconstraint_internal(rel, conTup, DROP_RESTRICT, recurse, false,
+ false, &readyRels, lockmode);
- readyRels = NIL;
- dropconstraint_internal(rel, conTup, DROP_RESTRICT, recurse, false,
- false, &readyRels, lockmode);
-
- heap_freetuple(conTup);
+ heap_freetuple(conTup);
+ }
InvokeObjectPostAlterHook(RelationRelationId,
RelationGetRelid(rel), attnum);
@@ -12933,10 +12951,10 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
List *children;
ListCell *child;
bool is_no_inherit_constraint = false;
- bool dropping_pk = false;
char *constrName;
List *unconstrained_cols = NIL;
- char *colname;
+ char *colname = NULL;
+ bool dropping_pk = false;
if (list_member_oid(*readyRels, RelationGetRelid(rel)))
return InvalidObjectAddress;
@@ -12968,10 +12986,12 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
*/
if (con->contype == CONSTRAINT_NOTNULL)
{
- AttrNumber colnum = extractNotNullColumn(constraintTup);
+ AttrNumber colnum;
- if (colnum != InvalidAttrNumber)
- unconstrained_cols = list_make1_int(colnum);
+ colnum = extractNotNullColumn(constraintTup);
+ unconstrained_cols = list_make1_int(colnum);
+ colname = NameStr(TupleDescAttr(RelationGetDescr(rel),
+ colnum - 1)->attname);
}
else if (con->contype == CONSTRAINT_PRIMARY)
{
@@ -13027,9 +13047,8 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
performDeletion(&conobj, behavior, 0);
/*
- * If this was a NOT NULL or the primary key, the constrained columns must
- * have had pg_attribute.attnotnull set. See if we need to reset it, and
- * do so.
+ * If this was a NOT NULL or the primary key, verify that we still have
+ * constraints to support GENERATED AS IDENTITY or the replica identity.
*/
if (unconstrained_cols)
{
@@ -13038,7 +13057,7 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
Bitmapset *ircols;
ListCell *lc;
- /* Make the above deletion visible */
+ /* Make implicit attnotnull changes visible */
CommandCounterIncrement();
attrel = table_open(AttributeRelationId, RowExclusiveLock);
@@ -13058,27 +13077,26 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
HeapTuple atttup;
HeapTuple contup;
Form_pg_attribute attForm;
+ char attidentity;
/*
- * Obtain pg_attribute tuple and verify conditions on it. We use
- * a copy we can scribble on.
+ * Obtain pg_attribute tuple and verify conditions on it.
*/
- atttup = SearchSysCacheCopyAttNum(RelationGetRelid(rel), attnum);
+ atttup = SearchSysCacheAttNum(RelationGetRelid(rel), attnum);
if (!HeapTupleIsValid(atttup))
elog(ERROR, "cache lookup failed for attribute %d of relation %u",
attnum, RelationGetRelid(rel));
attForm = (Form_pg_attribute) GETSTRUCT(atttup);
+ attidentity = attForm->attidentity;
+ ReleaseSysCache(atttup);
/*
* Since the above deletion has been made visible, we can now
* search for any remaining constraints on this column (or these
* columns, in the case we're dropping a multicol primary key.)
* Then, verify whether any further NOT NULL or primary key
- * exists, and reset attnotnull if none.
- *
- * However, if this is a generated identity column, abort the
- * whole thing with a specific error message, because the
- * constraint is required in that case.
+ * exists: if none and this is a generated identity column or the
+ * replica identity, abort the whole thing.
*/
contup = findNotNullConstraintAttnum(RelationGetRelid(rel), attnum);
if (contup ||
@@ -13090,7 +13108,7 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
* It's not valid to drop the not-null constraint for a GENERATED
* AS IDENTITY column.
*/
- if (attForm->attidentity)
+ if (attidentity != '\0')
ereport(ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("column \"%s\" of relation \"%s\" is an identity column",
@@ -13107,13 +13125,6 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("column \"%s\" is in index used as replica identity",
get_attname(RelationGetRelid(rel), lfirst_int(lc), false)));
-
- /* Reset attnotnull */
- if (attForm->attnotnull)
- {
- attForm->attnotnull = false;
- CatalogTupleUpdate(attrel, &atttup->t_self, atttup);
- }
}
table_close(attrel, RowExclusiveLock);
}
@@ -13152,13 +13163,6 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
errmsg("cannot remove constraint from only the partitioned table when partitions exist"),
errhint("Do not specify the ONLY keyword.")));
- /* For not-null constraints we recurse by column name */
- if (con->contype == CONSTRAINT_NOTNULL)
- colname = NameStr(TupleDescAttr(RelationGetDescr(rel),
- linitial_int(unconstrained_cols) - 1)->attname);
- else
- colname = NULL; /* keep compiler quiet */
-
foreach(child, children)
{
Oid childrelid = lfirst_oid(child);
@@ -13174,8 +13178,8 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
CheckTableNotInUse(childrel, "ALTER TABLE");
/*
- * We search for not-null constraint by column number, and other
- * constraints by name.
+ * We search for not-null constraints by column name, and others by
+ * constraint name.
*/
if (con->contype == CONSTRAINT_NOTNULL)
{
@@ -14656,12 +14660,14 @@ ATPostAlterTypeParse(Oid oldId, Oid oldRelId, Oid refRelId, char *cmd,
else if (cmd->subtype == AT_SetAttNotNull)
{
/*
- * The parser will create AT_AttSetNotNull subcommands for
- * columns of PRIMARY KEY indexes/constraints, but we need
- * not do anything with them here, because the columns'
- * NOT NULL marks will already have been propagated into
- * the new table definition.
+ * When doing alter primary key column type, Removing old
+ * PK constraint as sub-command will reset attnotnull to
+ * false in RemoveConstraintById(). We must set attnotnull
+ * back to true before doing re-add the index sub-command.
*/
+ cmd->subtype = AT_SetAttNotNull;
+ tab->subcmds[AT_PASS_OLD_COL_ATTRS] =
+ lappend(tab->subcmds[AT_PASS_OLD_COL_ATTRS], cmd);
}
else
elog(ERROR, "unexpected statement subtype: %d",
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index c52e961b30..7dcbfaf8dc 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -8788,12 +8788,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
"), E',\n ') AS attfdwoptions,\n");
/*
- * Find out any NOT NULL markings for each column. In 17 and up we have
- * to read pg_constraint, and keep track whether it's NO INHERIT; in older
- * versions we rely on pg_attribute.attnotnull.
+ * Find out any NOT NULL markings for each column. In 17 and up we
+ * read pg_constraint to obtain the constraint name. notnull_noinherit
+ * is set according to the NO INHERIT property. For versions prior to 17,
+ * we store an empty string as the name when a constraint is marked as
+ * attnotnull (this cues dumpTableSchema to print the NOT NULL clause
+ * without a name); also, such cases are never NO INHERIT.
*
- * We also track whether the constraint was defined directly in this table
- * or via an ancestor, for binary upgrade.
+ * We track in notnull_inh whether the constraint was defined directly in
+ * this table or via an ancestor, for binary upgrade.
*
* Lastly, we need to know if the PK for the table involves each column;
* for columns that are there we need a NOT NULL marking even if there's
@@ -8801,13 +8804,24 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
* NULLs after the data is loaded when the PK is created, later in the
* dump; for this case we add throwaway constraints that are dropped once
* the PK is created.
+ *
+ * Another complication arises from columns that have attnotnull set, but
+ * for which no corresponding not-null nor PK constraint exists. This can
+ * happen if, for example, a primary key is dropped indirectly -- say,
+ * because one of its columns is dropped. This is an irregular condition,
+ * so we don't work hard to preserve it, and instead act as though an
+ * unnamed not-null constraint exists.
*/
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(q,
- "co.conname AS notnull_name,\n"
- "co.connoinherit AS notnull_noinherit,\n"
+ "CASE WHEN co.conname IS NOT NULL THEN co.conname "
+ " WHEN a.attnotnull AND copk.conname IS NULL THEN '' ELSE NULL END AS notnull_name,\n"
+ "CASE WHEN co.conname IS NOT NULL THEN co.connoinherit "
+ " WHEN a.attnotnull THEN false ELSE NULL END AS notnull_noinherit,\n"
"copk.conname IS NOT NULL as notnull_is_pk,\n"
- "coalesce(NOT co.conislocal, true) AS notnull_inh,\n");
+ "CASE WHEN co.conname IS NOT NULL THEN "
+ " coalesce(NOT co.conislocal, true) "
+ "ELSE false END as notnull_inh,\n");
else
appendPQExpBufferStr(q,
"CASE WHEN a.attnotnull THEN '' ELSE NULL END AS notnull_name,\n"
diff --git a/src/test/regress/expected/constraints.out b/src/test/regress/expected/constraints.out
index d9d8408e86..34927f7afc 100644
--- a/src/test/regress/expected/constraints.out
+++ b/src/test/regress/expected/constraints.out
@@ -867,6 +867,79 @@ select conname, contype, conkey from pg_constraint where conrelid = 'notnull_tbl
(1 row)
DROP TABLE notnull_tbl1;
+-- make sure attnotnull is reset correctly when a PK is dropped indirectly,
+-- or kept if there's a reason for that
+CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1));
+ALTER TABLE notnull_tbl1 DROP c1;
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c0 | integer | | | | plain | |
+
+DROP TABLE notnull_tbl1;
+-- same, via dropping a domain
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int, PRIMARY KEY (c0, c1));
+DROP DOMAIN notnull_dom1 CASCADE;
+NOTICE: drop cascades to column c0 of table notnull_tbl1
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c1 | integer | | | | plain | |
+
+DROP TABLE notnull_tbl1;
+-- with a REPLICA IDENTITY column. Here the not-nulls must be kept
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int UNIQUE, c2 int generated by default as identity, PRIMARY KEY (c0, c1, c2));
+ALTER TABLE notnull_tbl1 DROP CONSTRAINT notnull_tbl1_c2_not_null;
+ALTER TABLE notnull_tbl1 REPLICA IDENTITY USING INDEX notnull_tbl1_c1_key;
+DROP DOMAIN notnull_dom1 CASCADE;
+NOTICE: drop cascades to column c0 of table notnull_tbl1
+ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL; -- can't be dropped
+ERROR: column "c1" is in index used as replica identity
+ALTER TABLE notnull_tbl1 ALTER c1 SET NOT NULL; -- can be set right
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+----------------------------------+---------+--------------+-------------
+ c1 | integer | | not null | | plain | |
+ c2 | integer | | not null | generated by default as identity | plain | |
+Indexes:
+ "notnull_tbl1_c1_key" UNIQUE CONSTRAINT, btree (c1) REPLICA IDENTITY
+Not-null constraints:
+ "notnull_tbl1_c1_not_null" NOT NULL "c1"
+
+-- Leave this table around for pg_upgrade testing
+CREATE DOMAIN notnull_dom2 AS INTEGER;
+CREATE TABLE notnull_tbl2 (c0 notnull_dom2, c1 int UNIQUE, c2 int generated by default as identity, PRIMARY KEY (c0, c1, c2));
+ALTER TABLE notnull_tbl2 DROP CONSTRAINT notnull_tbl2_c2_not_null;
+ALTER TABLE notnull_tbl2 REPLICA IDENTITY USING INDEX notnull_tbl2_c1_key;
+DROP DOMAIN notnull_dom2 CASCADE;
+NOTICE: drop cascades to column c0 of table notnull_tbl2
+\d+ notnull_tbl2
+ Table "public.notnull_tbl2"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+----------------------------------+---------+--------------+-------------
+ c1 | integer | | not null | | plain | |
+ c2 | integer | | not null | generated by default as identity | plain | |
+Indexes:
+ "notnull_tbl2_c1_key" UNIQUE CONSTRAINT, btree (c1) REPLICA IDENTITY
+
+ALTER TABLE notnull_tbl2 REPLICA IDENTITY FULL, ALTER c2 DROP IDENTITY;
+ALTER TABLE notnull_tbl2 ALTER c1 DROP NOT NULL, ALTER c2 DROP NOT NULL;
+\d+ notnull_tbl2
+ Table "public.notnull_tbl2"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c1 | integer | | | | plain | |
+ c2 | integer | | | | plain | |
+Indexes:
+ "notnull_tbl2_c1_key" UNIQUE CONSTRAINT, btree (c1)
+Replica Identity: FULL
+
+DROP TABLE notnull_tbl2;
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
ERROR: constraint "blah" for relation "notnull_tbl2" already exists
diff --git a/src/test/regress/sql/constraints.sql b/src/test/regress/sql/constraints.sql
index 87d685ae39..ecc7fb8dcf 100644
--- a/src/test/regress/sql/constraints.sql
+++ b/src/test/regress/sql/constraints.sql
@@ -599,6 +599,40 @@ ALTER TABLE notnull_tbl1 ADD CONSTRAINT foobar NOT NULL a;
select conname, contype, conkey from pg_constraint where conrelid = 'notnull_tbl1'::regclass;
DROP TABLE notnull_tbl1;
+-- make sure attnotnull is reset correctly when a PK is dropped indirectly,
+-- or kept if there's a reason for that
+CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1));
+ALTER TABLE notnull_tbl1 DROP c1;
+\d+ notnull_tbl1
+DROP TABLE notnull_tbl1;
+-- same, via dropping a domain
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int, PRIMARY KEY (c0, c1));
+DROP DOMAIN notnull_dom1 CASCADE;
+\d+ notnull_tbl1
+DROP TABLE notnull_tbl1;
+-- with a REPLICA IDENTITY column. Here the not-nulls must be kept
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int UNIQUE, c2 int generated by default as identity, PRIMARY KEY (c0, c1, c2));
+ALTER TABLE notnull_tbl1 DROP CONSTRAINT notnull_tbl1_c2_not_null;
+ALTER TABLE notnull_tbl1 REPLICA IDENTITY USING INDEX notnull_tbl1_c1_key;
+DROP DOMAIN notnull_dom1 CASCADE;
+ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL; -- can't be dropped
+ALTER TABLE notnull_tbl1 ALTER c1 SET NOT NULL; -- can be set right
+\d+ notnull_tbl1
+-- Leave this table around for pg_upgrade testing
+
+CREATE DOMAIN notnull_dom2 AS INTEGER;
+CREATE TABLE notnull_tbl2 (c0 notnull_dom2, c1 int UNIQUE, c2 int generated by default as identity, PRIMARY KEY (c0, c1, c2));
+ALTER TABLE notnull_tbl2 DROP CONSTRAINT notnull_tbl2_c2_not_null;
+ALTER TABLE notnull_tbl2 REPLICA IDENTITY USING INDEX notnull_tbl2_c1_key;
+DROP DOMAIN notnull_dom2 CASCADE;
+\d+ notnull_tbl2
+ALTER TABLE notnull_tbl2 REPLICA IDENTITY FULL, ALTER c2 DROP IDENTITY;
+ALTER TABLE notnull_tbl2 ALTER c1 DROP NOT NULL, ALTER c2 DROP NOT NULL;
+\d+ notnull_tbl2
+DROP TABLE notnull_tbl2;
+
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
--
2.39.2
On Thu, Apr 11, 2024 at 10:48 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
I'm still not ready with this -- still not convinced about the new AT
pass. Also, I want to add a test for the pg_dump behavior, and there's
an XXX comment.
Now I am more confused...
+-- make sure attnotnull is reset correctly when a PK is dropped indirectly,
+-- or kept if there's a reason for that
+CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1));
+ALTER TABLE notnull_tbl1 DROP c1;
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats
target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c0 | integer | | | | plain | |
+
+DROP TABLE notnull_tbl1;
same query, mysql make let "c0" be not null
mysql https://dbfiddle.uk/_ltoU7PO
for postgre
https://dbfiddle.uk/ZHJXEqL1
from 9.3 to 16 (click the link (https://dbfiddle.uk/ZHJXEqL1), then
click "9.3" choose which version you like)
all will make the remaining column "co" be not null.
latest
0001-Better-handle-indirect-constraint-drops.patch make c0 attnotnull be false.
previous patches:
v2-0001-Handle-ALTER-.-DROP-NOT-NULL-when-no-pg_constrain.patch make
c0 attnotnull be true.
0001-Correctly-reset-attnotnull-when-constraints-dropped-.patch make
c0 attnotnull be false.
jian he <jian.universality@gmail.com> 于2024年4月12日周五 10:12写道:
On Thu, Apr 11, 2024 at 10:48 PM Alvaro Herrera <alvherre@alvh.no-ip.org>
wrote:I'm still not ready with this -- still not convinced about the new AT
pass. Also, I want to add a test for the pg_dump behavior, and there's
an XXX comment.Now I am more confused...
+-- make sure attnotnull is reset correctly when a PK is dropped indirectly, +-- or kept if there's a reason for that +CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1)); +ALTER TABLE notnull_tbl1 DROP c1; +\d+ notnull_tbl1 + Table "public.notnull_tbl1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description+--------+---------+-----------+----------+---------+---------+--------------+------------- + c0 | integer | | | | plain | | + +DROP TABLE notnull_tbl1;same query, mysql make let "c0" be not null
mysql https://dbfiddle.uk/_ltoU7POfor postgre
https://dbfiddle.uk/ZHJXEqL1
from 9.3 to 16 (click the link (https://dbfiddle.uk/ZHJXEqL1), then
click "9.3" choose which version you like)
all will make the remaining column "co" be not null.latest
0001-Better-handle-indirect-constraint-drops.patch make c0 attnotnull be
false.previous patches:
v2-0001-Handle-ALTER-.-DROP-NOT-NULL-when-no-pg_constrain.patch make
c0 attnotnull be true.
0001-Correctly-reset-attnotnull-when-constraints-dropped-.patch make
c0 attnotnull be false.
I'm not sure that SQL standard specifies what database must do for this
case.
If the standard does not specify, then it depends on each database vendor's
decision.
Some people like not-null retained, other people may like not-null removed.
I think it will be ok if people can drop not-null or add not-null back
again after dropping pk.
In Master, not-null will reset when we drop PK directly. I hope dropping pk
indirectly
is consistent with dropping PK directly.
--
Tender Wang
OpenPie: https://en.openpie.com/
Alvaro Herrera <alvherre@alvh.no-ip.org> 于2024年4月11日周四 22:48写道:
On 2024-Apr-11, Alvaro Herrera wrote:
Well, I think you were right that we should try to handle the situation
of unmarking attnotnull as much as possible, to decrease the chances
that the problematic situation occurs. That means, we can use the
earlier code to handle DROP COLUMN when it causes a PK to be dropped --
even though we still need to handle the situation of an attnotnull flag
set with no pg_constraint row. I mean, we still have to handle DROP
DOMAIN correctly (and there might be other cases that I haven't thought
about) ... but surely this is a much less common situation than the one
you reported. So I'll merge everything and post an updated patch.Here's roughly what I'm thinking. If we drop a constraint, we can still
reset attnotnull in RemoveConstraintById(), but only after checking that
it's not a generated column or a replica identity. If they are, we
don't error out -- just skip the attnotnull update.Now, about the code to allow ALTER TABLE DROP NOT NULL in case there's
no pg_constraint row, I think at this point it's mostly dead code,
because it can only happen when you have a replica identity or generated
column ... and the DROP NOT NULL should still prevent you from dropping
the flag anyway. But the case can still arise, if you change the
replica identity or ALTER TABLE ALTER COLUMN DROP DEFAULT, respectively.I'm still not ready with this -- still not convinced about the new AT
pass.
Yeah, at first, I was also hesitant. Two reasons make me convinced.
in ATPostAlterTypeParse()
-----
else if (cmd->subtype == AT_SetAttNotNull)
{
/*
* The parser will create
AT_AttSetNotNull subcommands for
* columns of PRIMARY KEY
indexes/constraints, but we need
* not do anything with them here,
because the columns'
* NOT NULL marks will already have
been propagated into
* the new table definition.
*/
}
-------
The new table difinition continues to use old column not-null, so here does
nothing.
If we reset NOT NULL marks in RemoveConstrainById() when dropping PK
indirectly,
we need to do something here or somewhere else.
Except AT_SetAttNotNull type, other types add a AT pass to tab->subcmds.
Because
not-null should be added before re-adding index, there is no right AT pass
in current AlterTablePass.
So a new AT pass ahead AT_PASS_OLD_INDEX is needed.
Another reason is that it can use ALTER TABLE frame to set not-null.
This way looks simpler and better than hardcode to re-install not-null in
some funciton.
--
Tender Wang
OpenPie: https://en.openpie.com/
On 2024-Apr-12, jian he wrote:
Now I am more confused...
+CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1)); +ALTER TABLE notnull_tbl1 DROP c1;
same query, mysql make let "c0" be not null
Yes, that was Postgres' old model. But the way we think of it now, is
that a column is marked attnotnull when a pg_constraint entry exists to
support that flag, which can be a not-null constraint, or a primary key
constraint. In the old Postgres model, you're right that we would
continue to have c0 as not-null, just like mysql. In the new model,
that flag no longer has no reason to be there, because the backing
primary key constraint has been removed, which is why we reset it.
So what I was saying in the cases with replica identity and generated
columns, is that there's an attnotnull flag we cannot remove, because of
either of those things, but we don't have any backing constraint for it,
which is an inconsistency with the view of the world that I described
above. I would like to manufacture one not-null constraint at that
point, or just abort the drop of the PK ... but I don't see how to do
either of those things.
If you want the c0 column to be still not-null after dropping the
primary key, you need to SET NOT NULL:
CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1));
ALTER TABLE notnull_tbl1 ALTER c0 SET NOT NULL;
ALTER TABLE notnull_tbl1 DROP c1;
\d+ notnull_tbl1
Table "public.notnull_tbl1"
Column │ Type │ Collation │ Nullable │ Default │ Storage │ Compression │ Stats target │ Description
────────┼─────────┼───────────┼──────────┼─────────┼─────────┼─────────────┼──────────────┼─────────────
c0 │ integer │ │ not null │ │ plain │ │ │
Not-null constraints:
"notnull_tbl1_c0_not_null" NOT NULL "c0"
Access method: heap
One thing that's not quite ideal, is that the "Nullable" column doesn't
make it obvious that the flag is going to be removed if you drop the PK;
you have to infer that that's going to happen by noticing that there's
no explicit not-null constraint listed for that column -- maybe too
subtle, especially if you have a lot of columns (luckily, PKs normally
don't have too many columns). This is why I suggested to change the
contents of that column if the flag is sustained by the PK. Something
like this, perhaps:
=# CREATE TABLE notnull_tbl1 (c0 int not null, c1 int, PRIMARY KEY (c0, c1));
=# \d+ notnull_tbl1
Table "public.notnull_tbl1"
Column │ Type │ Collation │ Nullable │ Default │ Storage │ Compression │ Stats target │ Description
────────┼─────────┼───────────┼─────────────┼─────────┼─────────┼─────────────┼──────────────┼─────────────
c0 │ integer │ │ not null │ │ plain │ │ │
c1 │ integer │ │ primary key │ │ plain │ │ │
Indexes:
"notnull_tbl1_pkey" PRIMARY KEY, btree (c0, c1)
Not-null constraints:
"notnull_tbl1_c0_not_null" NOT NULL "c0"
Access method: heap
which should make it obvious.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Right now the sectors on the hard disk run clockwise, but I heard a rumor that
you can squeeze 0.2% more throughput by running them counterclockwise.
It's worth the effort. Recommended." (Gerry Pourwelle)
On Fri, Apr 12, 2024 at 3:52 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2024-Apr-12, jian he wrote:
Now I am more confused...
+CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1)); +ALTER TABLE notnull_tbl1 DROP c1;same query, mysql make let "c0" be not null
Yes, that was Postgres' old model. But the way we think of it now, is
that a column is marked attnotnull when a pg_constraint entry exists to
support that flag, which can be a not-null constraint, or a primary key
constraint. In the old Postgres model, you're right that we would
continue to have c0 as not-null, just like mysql. In the new model,
that flag no longer has no reason to be there, because the backing
primary key constraint has been removed, which is why we reset it.So what I was saying in the cases with replica identity and generated
columns, is that there's an attnotnull flag we cannot remove, because of
either of those things, but we don't have any backing constraint for it,
which is an inconsistency with the view of the world that I described
above. I would like to manufacture one not-null constraint at that
point, or just abort the drop of the PK ... but I don't see how to do
either of those things.
thanks for your explanation.
now I understand it.
I wonder is there any incompatibility issue, or do we need to say something
about the new behavior when dropping a key column?
the comments look good to me.
only minor cosmetic issue:
+ if (unconstrained_cols)
i would like change it to
+ if (unconstrained_cols != NIL)
+ foreach(lc, unconstrained_cols)
we can change to
+ foreach_int(attnum, unconstrained_cols)
per commit
https://git.postgresql.org/cgit/postgresql.git/commit/?id=14dd0f27d7cd56ffae9ecdbe324965073d01a9ff
On 2024-Apr-13, jian he wrote:
I wonder is there any incompatibility issue, or do we need to say something
about the new behavior when dropping a key column?
Umm, yeah, maybe we should document it in ALTER TABLE DROP PRIMARY KEY
and in the release notes to note the different behavior.
only minor cosmetic issue:
+ if (unconstrained_cols)
i would like change it to
+ if (unconstrained_cols != NIL)+ foreach(lc, unconstrained_cols) we can change to + foreach_int(attnum, unconstrained_cols) per commit https://git.postgresql.org/cgit/postgresql.git/commit/?id=14dd0f27d7cd56ffae9ecdbe324965073d01a9ff
Ah, yeah. I did that, rewrote some comments and refined the tests a
little bit to ensure the pg_upgrade behavior is sane. I intend to get
this pushed tomorrow, if nothing ugly comes up.
CI run: https://cirrus-ci.com/build/5471117953990656
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"La gente vulgar sólo piensa en pasar el tiempo;
el que tiene talento, en aprovecharlo"
Attachments:
v2-0001-Better-handle-indirect-constraint-drops.patchtext/x-diff; charset=utf-8Download
From 28393502b3d11116fcf430264c10f931e948bedc Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Thu, 11 Apr 2024 12:29:34 +0200
Subject: [PATCH v2] Better handle indirect constraint drops
It is possible for certain cases to remove not-null constraints without
maintaining the attnotnull in its correct state; for example if you drop
a column that's part of the primary key, and the other columns of the PK don't
have not-null constraints, then we should reset the attnotnull flags for
those other columns; up to this commit, we didn't. Handle those cases
better by doing the attnotnull reset in RemoveConstraintById() instead
of in dropconstraint_internal().
However, there are some cases where we must not do so. For example if
those other columns are in replica identity indexes or are generated
identity columns, we must keep attnotnull set, even though it results in
the catalog inconsistency that no not-null constraint supports that.
Because the attnotnull reset now happens in more places than before, for
instance when a column of the primary key changes type, we need an
additional trick to reinstate it as necessary. Introduce a new
alter-table pass that does this, which needs simply reschedule some
AT_SetAttNotNull subcommands that were already being generated and
ignored.
Because of the exceptions in which attnotnull is not reset noted above,
we also include a pg_dump hack to include a not-null constraint when the
attnotnull flag is set even if no pg_constraint row exists. This part
is undesirable but necessary, because failing to handle the case can
result in unrestorable dumps.
Reported-by: Tender Wang <tndrwang@gmail.com>
Co-authored-by: Tender Wang <tndrwang@gmail.com>
Reviewed-by: jian he <jian.universality@gmail.com>
Discussion: https://postgr.es/m/CAHewXN=hMbNa3d43NOR=OCgdgpTt18S-1fmueCoEGesyeK4bqw@mail.gmail.com
---
src/backend/catalog/pg_constraint.c | 116 ++++++++++++++++++-
src/backend/commands/tablecmds.c | 135 +++++++++++-----------
src/bin/pg_dump/pg_dump.c | 30 +++--
src/test/regress/expected/constraints.out | 78 +++++++++++++
src/test/regress/sql/constraints.sql | 39 +++++++
5 files changed, 322 insertions(+), 76 deletions(-)
diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c
index 778b7c381d..56c083fe21 100644
--- a/src/backend/catalog/pg_constraint.c
+++ b/src/backend/catalog/pg_constraint.c
@@ -19,6 +19,7 @@
#include "access/htup_details.h"
#include "access/sysattr.h"
#include "access/table.h"
+#include "access/xact.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/heap.h"
@@ -933,6 +934,8 @@ RemoveConstraintById(Oid conId)
Relation conDesc;
HeapTuple tup;
Form_pg_constraint con;
+ bool dropping_pk = false;
+ List *unconstrained_cols = NIL;
conDesc = table_open(ConstraintRelationId, RowExclusiveLock);
@@ -957,7 +960,9 @@ RemoveConstraintById(Oid conId)
/*
* We need to update the relchecks count if it is a check constraint
* being dropped. This update will force backends to rebuild relcache
- * entries when we commit.
+ * entries when we commit. For not-null and primary key constraints,
+ * obtain the list of columns affected, so that we can reset their
+ * attnotnull flags below.
*/
if (con->contype == CONSTRAINT_CHECK)
{
@@ -984,6 +989,36 @@ RemoveConstraintById(Oid conId)
table_close(pgrel, RowExclusiveLock);
}
+ else if (con->contype == CONSTRAINT_NOTNULL)
+ {
+ unconstrained_cols = list_make1_int(extractNotNullColumn(tup));
+ }
+ else if (con->contype == CONSTRAINT_PRIMARY)
+ {
+ Datum adatum;
+ ArrayType *arr;
+ int numkeys;
+ bool isNull;
+ int16 *attnums;
+
+ dropping_pk = true;
+
+ adatum = heap_getattr(tup, Anum_pg_constraint_conkey,
+ RelationGetDescr(conDesc), &isNull);
+ if (isNull)
+ elog(ERROR, "null conkey for constraint %u", con->oid);
+ arr = DatumGetArrayTypeP(adatum); /* ensure not toasted */
+ numkeys = ARR_DIMS(arr)[0];
+ if (ARR_NDIM(arr) != 1 ||
+ numkeys < 0 ||
+ ARR_HASNULL(arr) ||
+ ARR_ELEMTYPE(arr) != INT2OID)
+ elog(ERROR, "conkey is not a 1-D smallint array");
+ attnums = (int16 *) ARR_DATA_PTR(arr);
+
+ for (int i = 0; i < numkeys; i++)
+ unconstrained_cols = lappend_int(unconstrained_cols, attnums[i]);
+ }
/* Keep lock on constraint's rel until end of xact */
table_close(rel, NoLock);
@@ -1003,6 +1038,85 @@ RemoveConstraintById(Oid conId)
/* Fry the constraint itself */
CatalogTupleDelete(conDesc, &tup->t_self);
+ /*
+ * If this was a NOT NULL or the primary key, the constrained columns must
+ * have had pg_attribute.attnotnull set. See if we need to reset it, and
+ * do so.
+ */
+ if (unconstrained_cols != NIL)
+ {
+ Relation tablerel;
+ Relation attrel;
+ Bitmapset *pkcols;
+ ListCell *lc;
+
+ /* Make the above deletion visible */
+ CommandCounterIncrement();
+
+ tablerel = table_open(con->conrelid, NoLock); /* already have lock */
+ attrel = table_open(AttributeRelationId, RowExclusiveLock);
+
+ /*
+ * We want to test columns for their presence in the primary key, but
+ * only if we're not dropping it.
+ */
+ pkcols = dropping_pk ? NULL :
+ RelationGetIndexAttrBitmap(tablerel,
+ INDEX_ATTR_BITMAP_PRIMARY_KEY);
+
+ foreach(lc, unconstrained_cols)
+ {
+ AttrNumber attnum = lfirst_int(lc);
+ HeapTuple atttup;
+ HeapTuple contup;
+ Bitmapset *ircols;
+ Form_pg_attribute attForm;
+
+ /*
+ * Obtain pg_attribute tuple and verify conditions on it. We use
+ * a copy we can scribble on.
+ */
+ atttup = SearchSysCacheCopyAttNum(con->conrelid, attnum);
+ if (!HeapTupleIsValid(atttup))
+ elog(ERROR, "cache lookup failed for attribute %d of relation %u",
+ attnum, con->conrelid);
+ attForm = (Form_pg_attribute) GETSTRUCT(atttup);
+
+ /*
+ * Since the above deletion has been made visible, we can now
+ * search for any remaining constraints on this column (or these
+ * columns, in the case we're dropping a multicol primary key.)
+ * Then, verify whether any further NOT NULL or primary key
+ * exists, and reset attnotnull if none.
+ */
+ contup = findNotNullConstraintAttnum(con->conrelid, attnum);
+ if (contup ||
+ bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber,
+ pkcols))
+ continue;
+
+ /*
+ * ... but don't do it if the column is in the replica identity or
+ * it's a generated column
+ */
+ if (attForm->attidentity != '\0')
+ continue;
+ ircols = RelationGetIndexAttrBitmap(tablerel, INDEX_ATTR_BITMAP_IDENTITY_KEY); /* XXX bms_free */
+ if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, ircols))
+ continue;
+
+ /* Reset attnotnull */
+ if (attForm->attnotnull)
+ {
+ attForm->attnotnull = false;
+ CatalogTupleUpdate(attrel, &atttup->t_self, atttup);
+ }
+ }
+
+ table_close(attrel, RowExclusiveLock);
+ table_close(tablerel, NoLock);
+ }
+
/* Clean up */
ReleaseSysCache(tup);
table_close(conDesc, RowExclusiveLock);
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index f72b2dcadf..7b01c85f6b 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -149,6 +149,7 @@ typedef enum AlterTablePass
AT_PASS_ALTER_TYPE, /* ALTER COLUMN TYPE */
AT_PASS_ADD_COL, /* ADD COLUMN */
AT_PASS_SET_EXPRESSION, /* ALTER SET EXPRESSION */
+ AT_PASS_OLD_COL_ATTRS, /* re-install attnotnull */
AT_PASS_OLD_INDEX, /* re-add existing indexes */
AT_PASS_OLD_CONSTR, /* re-add existing constraints */
/* We could support a RENAME COLUMN pass here, but not currently used */
@@ -7662,17 +7663,23 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse,
}
/*
- * Find the constraint that makes this column NOT NULL.
+ * Find the constraint that makes this column NOT NULL, and drop it if we
+ * see one. dropconstraint_internal() will do necessary consistency
+ * checking. If there isn't one, there are two possibilities: either the
+ * column is marked attnotnull because it's part of the primary key, and
+ * then we just throw an appropriate error; or it's a leftover marking
+ * that we can remove. However, before doing the latter, to avoid
+ * breaking consistency any further, prevent this if the column is part of
+ * the replica identity.
*/
conTup = findNotNullConstraint(RelationGetRelid(rel), colName);
if (conTup == NULL)
{
Bitmapset *pkcols;
+ Bitmapset *ircols;
/*
- * There's no not-null constraint, so throw an error. If the column
- * is in a primary key, we can throw a specific error. Otherwise,
- * this is unexpected.
+ * If the column is in a primary key, throw a specific error message.
*/
pkcols = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_PRIMARY_KEY);
if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber,
@@ -7681,16 +7688,27 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse,
errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("column \"%s\" is in a primary key", colName));
- /* this shouldn't happen */
- elog(ERROR, "could not find not-null constraint on column \"%s\", relation \"%s\"",
- colName, RelationGetRelationName(rel));
+ /* Also throw an error if the column is in the replica identity */
+ ircols = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_IDENTITY_KEY);
+ if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, ircols))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("column \"%s\" is in index used as replica identity",
+ get_attname(RelationGetRelid(rel), attnum, false)));
+
+ /* Otherwise, just remove the attnotnull marking and do nothing else. */
+ attTup->attnotnull = false;
+ CatalogTupleUpdate(attr_rel, &tuple->t_self, tuple);
}
+ else
+ {
+ /* The normal case: we have a pg_constraint row, remove it */
+ readyRels = NIL;
+ dropconstraint_internal(rel, conTup, DROP_RESTRICT, recurse, false,
+ false, &readyRels, lockmode);
- readyRels = NIL;
- dropconstraint_internal(rel, conTup, DROP_RESTRICT, recurse, false,
- false, &readyRels, lockmode);
-
- heap_freetuple(conTup);
+ heap_freetuple(conTup);
+ }
InvokeObjectPostAlterHook(RelationRelationId,
RelationGetRelid(rel), attnum);
@@ -12927,12 +12945,11 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
Form_pg_constraint con;
ObjectAddress conobj;
List *children;
- ListCell *child;
bool is_no_inherit_constraint = false;
- bool dropping_pk = false;
char *constrName;
List *unconstrained_cols = NIL;
- char *colname;
+ char *colname = NULL;
+ bool dropping_pk = false;
if (list_member_oid(*readyRels, RelationGetRelid(rel)))
return InvalidObjectAddress;
@@ -12989,10 +13006,12 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
*/
if (con->contype == CONSTRAINT_NOTNULL)
{
- AttrNumber colnum = extractNotNullColumn(constraintTup);
+ AttrNumber colnum;
- if (colnum != InvalidAttrNumber)
- unconstrained_cols = list_make1_int(colnum);
+ colnum = extractNotNullColumn(constraintTup);
+ unconstrained_cols = list_make1_int(colnum);
+ colname = NameStr(TupleDescAttr(RelationGetDescr(rel),
+ colnum - 1)->attname);
}
else if (con->contype == CONSTRAINT_PRIMARY)
{
@@ -13048,18 +13067,16 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
performDeletion(&conobj, behavior, 0);
/*
- * If this was a NOT NULL or the primary key, the constrained columns must
- * have had pg_attribute.attnotnull set. See if we need to reset it, and
- * do so.
+ * If this was a NOT NULL or the primary key, verify that we still have
+ * constraints to support GENERATED AS IDENTITY or the replica identity.
*/
- if (unconstrained_cols)
+ if (unconstrained_cols != NIL)
{
Relation attrel;
Bitmapset *pkcols;
Bitmapset *ircols;
- ListCell *lc;
- /* Make the above deletion visible */
+ /* Make implicit attnotnull changes visible */
CommandCounterIncrement();
attrel = table_open(AttributeRelationId, RowExclusiveLock);
@@ -13073,33 +13090,31 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
INDEX_ATTR_BITMAP_PRIMARY_KEY);
ircols = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_IDENTITY_KEY);
- foreach(lc, unconstrained_cols)
+ foreach_int(attnum, unconstrained_cols)
{
- AttrNumber attnum = lfirst_int(lc);
HeapTuple atttup;
HeapTuple contup;
Form_pg_attribute attForm;
+ char attidentity;
/*
- * Obtain pg_attribute tuple and verify conditions on it. We use
- * a copy we can scribble on.
+ * Obtain pg_attribute tuple and verify conditions on it.
*/
- atttup = SearchSysCacheCopyAttNum(RelationGetRelid(rel), attnum);
+ atttup = SearchSysCacheAttNum(RelationGetRelid(rel), attnum);
if (!HeapTupleIsValid(atttup))
elog(ERROR, "cache lookup failed for attribute %d of relation %u",
attnum, RelationGetRelid(rel));
attForm = (Form_pg_attribute) GETSTRUCT(atttup);
+ attidentity = attForm->attidentity;
+ ReleaseSysCache(atttup);
/*
* Since the above deletion has been made visible, we can now
* search for any remaining constraints on this column (or these
* columns, in the case we're dropping a multicol primary key.)
* Then, verify whether any further NOT NULL or primary key
- * exists, and reset attnotnull if none.
- *
- * However, if this is a generated identity column, abort the
- * whole thing with a specific error message, because the
- * constraint is required in that case.
+ * exists: if none and this is a generated identity column or the
+ * replica identity, abort the whole thing.
*/
contup = findNotNullConstraintAttnum(RelationGetRelid(rel), attnum);
if (contup ||
@@ -13111,7 +13126,7 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
* It's not valid to drop the not-null constraint for a GENERATED
* AS IDENTITY column.
*/
- if (attForm->attidentity)
+ if (attidentity != '\0')
ereport(ERROR,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("column \"%s\" of relation \"%s\" is an identity column",
@@ -13123,18 +13138,11 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
* It's not valid to drop the not-null constraint for a column in
* the replica identity index, either. (FULL is not affected.)
*/
- if (bms_is_member(lfirst_int(lc) - FirstLowInvalidHeapAttributeNumber, ircols))
+ if (bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber, ircols))
ereport(ERROR,
errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("column \"%s\" is in index used as replica identity",
- get_attname(RelationGetRelid(rel), lfirst_int(lc), false)));
-
- /* Reset attnotnull */
- if (attForm->attnotnull)
- {
- attForm->attnotnull = false;
- CatalogTupleUpdate(attrel, &atttup->t_self, atttup);
- }
+ get_attname(RelationGetRelid(rel), attnum, false)));
}
table_close(attrel, RowExclusiveLock);
}
@@ -13173,16 +13181,8 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
errmsg("cannot remove constraint from only the partitioned table when partitions exist"),
errhint("Do not specify the ONLY keyword.")));
- /* For not-null constraints we recurse by column name */
- if (con->contype == CONSTRAINT_NOTNULL)
- colname = NameStr(TupleDescAttr(RelationGetDescr(rel),
- linitial_int(unconstrained_cols) - 1)->attname);
- else
- colname = NULL; /* keep compiler quiet */
-
- foreach(child, children)
+ foreach_oid(childrelid, children)
{
- Oid childrelid = lfirst_oid(child);
Relation childrel;
HeapTuple tuple;
Form_pg_constraint childcon;
@@ -13195,8 +13195,8 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
CheckTableNotInUse(childrel, "ALTER TABLE");
/*
- * We search for not-null constraint by column number, and other
- * constraints by name.
+ * We search for not-null constraints by column name, and others by
+ * constraint name.
*/
if (con->contype == CONSTRAINT_NOTNULL)
{
@@ -13304,7 +13304,6 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
rel->rd_rel->relhassubclass)
{
List *colnames = NIL;
- ListCell *lc;
List *pkready = NIL;
/*
@@ -13317,15 +13316,14 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
* Find out the list of column names to process. Fortunately, we
* already have the list of column numbers.
*/
- foreach(lc, unconstrained_cols)
+ foreach_int(attnum, unconstrained_cols)
{
colnames = lappend(colnames, get_attname(RelationGetRelid(rel),
- lfirst_int(lc), false));
+ attnum, false));
}
- foreach(child, children)
+ foreach_oid(childrelid, children)
{
- Oid childrelid = lfirst_oid(child);
Relation childrel;
if (list_member_oid(pkready, childrelid))
@@ -13335,10 +13333,9 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha
childrel = table_open(childrelid, NoLock);
CheckTableNotInUse(childrel, "ALTER TABLE");
- foreach(lc, colnames)
+ foreach_ptr(char, colName, colnames)
{
HeapTuple contup;
- char *colName = lfirst(lc);
contup = findNotNullConstraint(childrelid, colName);
if (contup == NULL)
@@ -14677,12 +14674,16 @@ ATPostAlterTypeParse(Oid oldId, Oid oldRelId, Oid refRelId, char *cmd,
else if (cmd->subtype == AT_SetAttNotNull)
{
/*
- * The parser will create AT_AttSetNotNull subcommands for
- * columns of PRIMARY KEY indexes/constraints, but we need
- * not do anything with them here, because the columns'
- * NOT NULL marks will already have been propagated into
- * the new table definition.
+ * We see this subtype when a primary key is created
+ * internally, for example when it is replaced with a new
+ * constraint (say because one of the columns changes
+ * type); in this case we need to reinstate attnotnull,
+ * because it was removed because of the drop of the old
+ * PK. Schedule this subcommand to an upcoming AT pass.
*/
+ cmd->subtype = AT_SetAttNotNull;
+ tab->subcmds[AT_PASS_OLD_COL_ATTRS] =
+ lappend(tab->subcmds[AT_PASS_OLD_COL_ATTRS], cmd);
}
else
elog(ERROR, "unexpected statement subtype: %d",
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 6d2f3fdef3..9edda90469 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -8788,12 +8788,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
"), E',\n ') AS attfdwoptions,\n");
/*
- * Find out any NOT NULL markings for each column. In 17 and up we have
- * to read pg_constraint, and keep track whether it's NO INHERIT; in older
- * versions we rely on pg_attribute.attnotnull.
+ * Find out any NOT NULL markings for each column. In 17 and up we read
+ * pg_constraint to obtain the constraint name. notnull_noinherit is set
+ * according to the NO INHERIT property. For versions prior to 17, we
+ * store an empty string as the name when a constraint is marked as
+ * attnotnull (this cues dumpTableSchema to print the NOT NULL clause
+ * without a name); also, such cases are never NO INHERIT.
*
- * We also track whether the constraint was defined directly in this table
- * or via an ancestor, for binary upgrade.
+ * We track in notnull_inh whether the constraint was defined directly in
+ * this table or via an ancestor, for binary upgrade.
*
* Lastly, we need to know if the PK for the table involves each column;
* for columns that are there we need a NOT NULL marking even if there's
@@ -8801,13 +8804,24 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
* NULLs after the data is loaded when the PK is created, later in the
* dump; for this case we add throwaway constraints that are dropped once
* the PK is created.
+ *
+ * Another complication arises from columns that have attnotnull set, but
+ * for which no corresponding not-null nor PK constraint exists. This can
+ * happen if, for example, a primary key is dropped indirectly -- say,
+ * because one of its columns is dropped. This is an irregular condition,
+ * so we don't work hard to preserve it, and instead act as though an
+ * unnamed not-null constraint exists.
*/
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(q,
- "co.conname AS notnull_name,\n"
- "co.connoinherit AS notnull_noinherit,\n"
+ "CASE WHEN co.conname IS NOT NULL THEN co.conname "
+ " WHEN a.attnotnull AND copk.conname IS NULL THEN '' ELSE NULL END AS notnull_name,\n"
+ "CASE WHEN co.conname IS NOT NULL THEN co.connoinherit "
+ " WHEN a.attnotnull THEN false ELSE NULL END AS notnull_noinherit,\n"
"copk.conname IS NOT NULL as notnull_is_pk,\n"
- "coalesce(NOT co.conislocal, true) AS notnull_inh,\n");
+ "CASE WHEN co.conname IS NOT NULL THEN "
+ " coalesce(NOT co.conislocal, true) "
+ "ELSE false END as notnull_inh,\n");
else
appendPQExpBufferStr(q,
"CASE WHEN a.attnotnull THEN '' ELSE NULL END AS notnull_name,\n"
diff --git a/src/test/regress/expected/constraints.out b/src/test/regress/expected/constraints.out
index d50dd1f61a..2fc0be7925 100644
--- a/src/test/regress/expected/constraints.out
+++ b/src/test/regress/expected/constraints.out
@@ -926,9 +926,87 @@ DROP TABLE notnull_tbl1;
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
ERROR: constraint "blah" for relation "notnull_tbl2" already exists
+-- can't drop not-null in primary key
CREATE TABLE notnull_tbl2 (a INTEGER PRIMARY KEY);
ALTER TABLE notnull_tbl2 ALTER a DROP NOT NULL;
ERROR: column "a" is in a primary key
+DROP TABLE notnull_tbl2;
+-- make sure attnotnull is reset correctly when a PK is dropped indirectly,
+-- or kept if there's a reason for that
+CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1));
+ALTER TABLE notnull_tbl1 DROP c1;
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c0 | integer | | | | plain | |
+
+DROP TABLE notnull_tbl1;
+-- same, via dropping a domain
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int, PRIMARY KEY (c0, c1));
+DROP DOMAIN notnull_dom1 CASCADE;
+NOTICE: drop cascades to column c0 of table notnull_tbl1
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c1 | integer | | | | plain | |
+
+DROP TABLE notnull_tbl1;
+-- with a REPLICA IDENTITY column. Here the not-nulls must be kept
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int UNIQUE, c2 int generated by default as identity, PRIMARY KEY (c0, c1, c2));
+ALTER TABLE notnull_tbl1 DROP CONSTRAINT notnull_tbl1_c2_not_null;
+ALTER TABLE notnull_tbl1 REPLICA IDENTITY USING INDEX notnull_tbl1_c1_key;
+DROP DOMAIN notnull_dom1 CASCADE;
+NOTICE: drop cascades to column c0 of table notnull_tbl1
+ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL; -- can't be dropped
+ERROR: column "c1" is in index used as replica identity
+ALTER TABLE notnull_tbl1 ALTER c1 SET NOT NULL; -- can be set right
+\d+ notnull_tbl1
+ Table "public.notnull_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+----------------------------------+---------+--------------+-------------
+ c1 | integer | | not null | | plain | |
+ c2 | integer | | not null | generated by default as identity | plain | |
+Indexes:
+ "notnull_tbl1_c1_key" UNIQUE CONSTRAINT, btree (c1) REPLICA IDENTITY
+Not-null constraints:
+ "notnull_tbl1_c1_not_null" NOT NULL "c1"
+
+DROP TABLE notnull_tbl1;
+CREATE DOMAIN notnull_dom2 AS INTEGER;
+CREATE TABLE notnull_tbl2 (c0 notnull_dom2, c1 int UNIQUE, c2 int generated by default as identity, PRIMARY KEY (c0, c1, c2));
+ALTER TABLE notnull_tbl2 DROP CONSTRAINT notnull_tbl2_c2_not_null;
+ALTER TABLE notnull_tbl2 REPLICA IDENTITY USING INDEX notnull_tbl2_c1_key;
+DROP DOMAIN notnull_dom2 CASCADE;
+NOTICE: drop cascades to column c0 of table notnull_tbl2
+\d+ notnull_tbl2
+ Table "public.notnull_tbl2"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+----------------------------------+---------+--------------+-------------
+ c1 | integer | | not null | | plain | |
+ c2 | integer | | not null | generated by default as identity | plain | |
+Indexes:
+ "notnull_tbl2_c1_key" UNIQUE CONSTRAINT, btree (c1) REPLICA IDENTITY
+
+BEGIN;
+/* make sure the table can be put right, but roll that back */
+ALTER TABLE notnull_tbl2 REPLICA IDENTITY FULL, ALTER c2 DROP IDENTITY;
+ALTER TABLE notnull_tbl2 ALTER c1 DROP NOT NULL, ALTER c2 DROP NOT NULL;
+\d+ notnull_tbl2
+ Table "public.notnull_tbl2"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ c1 | integer | | | | plain | |
+ c2 | integer | | | | plain | |
+Indexes:
+ "notnull_tbl2_c1_key" UNIQUE CONSTRAINT, btree (c1)
+Replica Identity: FULL
+
+ROLLBACK;
+-- Leave this table around for pg_upgrade testing
CREATE TABLE notnull_tbl3 (a INTEGER NOT NULL, CHECK (a IS NOT NULL));
ALTER TABLE notnull_tbl3 ALTER A DROP NOT NULL;
ALTER TABLE notnull_tbl3 ADD b int, ADD CONSTRAINT pk PRIMARY KEY (a, b);
diff --git a/src/test/regress/sql/constraints.sql b/src/test/regress/sql/constraints.sql
index 7a39b504a3..8f85e72050 100644
--- a/src/test/regress/sql/constraints.sql
+++ b/src/test/regress/sql/constraints.sql
@@ -624,8 +624,47 @@ DROP TABLE notnull_tbl1;
-- nope
CREATE TABLE notnull_tbl2 (a INTEGER CONSTRAINT blah NOT NULL, b INTEGER CONSTRAINT blah NOT NULL);
+-- can't drop not-null in primary key
CREATE TABLE notnull_tbl2 (a INTEGER PRIMARY KEY);
ALTER TABLE notnull_tbl2 ALTER a DROP NOT NULL;
+DROP TABLE notnull_tbl2;
+
+-- make sure attnotnull is reset correctly when a PK is dropped indirectly,
+-- or kept if there's a reason for that
+CREATE TABLE notnull_tbl1 (c0 int, c1 int, PRIMARY KEY (c0, c1));
+ALTER TABLE notnull_tbl1 DROP c1;
+\d+ notnull_tbl1
+DROP TABLE notnull_tbl1;
+-- same, via dropping a domain
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int, PRIMARY KEY (c0, c1));
+DROP DOMAIN notnull_dom1 CASCADE;
+\d+ notnull_tbl1
+DROP TABLE notnull_tbl1;
+-- with a REPLICA IDENTITY column. Here the not-nulls must be kept
+CREATE DOMAIN notnull_dom1 AS INTEGER;
+CREATE TABLE notnull_tbl1 (c0 notnull_dom1, c1 int UNIQUE, c2 int generated by default as identity, PRIMARY KEY (c0, c1, c2));
+ALTER TABLE notnull_tbl1 DROP CONSTRAINT notnull_tbl1_c2_not_null;
+ALTER TABLE notnull_tbl1 REPLICA IDENTITY USING INDEX notnull_tbl1_c1_key;
+DROP DOMAIN notnull_dom1 CASCADE;
+ALTER TABLE notnull_tbl1 ALTER c1 DROP NOT NULL; -- can't be dropped
+ALTER TABLE notnull_tbl1 ALTER c1 SET NOT NULL; -- can be set right
+\d+ notnull_tbl1
+DROP TABLE notnull_tbl1;
+
+CREATE DOMAIN notnull_dom2 AS INTEGER;
+CREATE TABLE notnull_tbl2 (c0 notnull_dom2, c1 int UNIQUE, c2 int generated by default as identity, PRIMARY KEY (c0, c1, c2));
+ALTER TABLE notnull_tbl2 DROP CONSTRAINT notnull_tbl2_c2_not_null;
+ALTER TABLE notnull_tbl2 REPLICA IDENTITY USING INDEX notnull_tbl2_c1_key;
+DROP DOMAIN notnull_dom2 CASCADE;
+\d+ notnull_tbl2
+BEGIN;
+/* make sure the table can be put right, but roll that back */
+ALTER TABLE notnull_tbl2 REPLICA IDENTITY FULL, ALTER c2 DROP IDENTITY;
+ALTER TABLE notnull_tbl2 ALTER c1 DROP NOT NULL, ALTER c2 DROP NOT NULL;
+\d+ notnull_tbl2
+ROLLBACK;
+-- Leave this table around for pg_upgrade testing
CREATE TABLE notnull_tbl3 (a INTEGER NOT NULL, CHECK (a IS NOT NULL));
ALTER TABLE notnull_tbl3 ALTER A DROP NOT NULL;
--
2.39.2
Alvaro Herrera <alvherre@alvh.no-ip.org> 于2024年4月19日周五 02:49写道:
On 2024-Apr-13, jian he wrote:
I wonder is there any incompatibility issue, or do we need to say
something
about the new behavior when dropping a key column?
Umm, yeah, maybe we should document it in ALTER TABLE DROP PRIMARY KEY
and in the release notes to note the different behavior.only minor cosmetic issue:
+ if (unconstrained_cols)
i would like change it to
+ if (unconstrained_cols != NIL)+ foreach(lc, unconstrained_cols) we can change to + foreach_int(attnum, unconstrained_cols) per commithttps://git.postgresql.org/cgit/postgresql.git/commit/?id=14dd0f27d7cd56ffae9ecdbe324965073d01a9ff
Ah, yeah. I did that, rewrote some comments and refined the tests a
little bit to ensure the pg_upgrade behavior is sane. I intend to get
this pushed tomorrow, if nothing ugly comes up.
The new patch looks good to me.
--
Tender Wang
OpenPie: https://en.openpie.com/
On 2024-Apr-19, Tender Wang wrote:
The new patch looks good to me.
Thanks for looking once more. I have pushed it now. I didn't try
pg_upgrade other than running the tests, so maybe buildfarm member crake
will have more to complain about -- we'll see.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"After a quick R of TFM, all I can say is HOLY CR** THAT IS COOL! PostgreSQL was
amazing when I first started using it at 7.2, and I'm continually astounded by
learning new features and techniques made available by the continuing work of
the development team."
Berend Tober, http://archives.postgresql.org/pgsql-hackers/2007-08/msg01009.php