Dumping/restoring fails on inherited generated column

Started by Tom Laneabout 6 years ago37 messages
#1Tom Lane
tgl@sss.pgh.pa.us

Run the regression tests with "make installcheck", then:

$ pg_dump -Fc regression >r.dump
$ createdb r2
$ pg_restore -d r2 r.dump
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 6005; 2604 24821 DEFAULT gtest1_1 b postgres
pg_restore: error: could not execute query: ERROR: column "b" of relation "gtest1_1" is a generated column
Command was: ALTER TABLE ONLY public.gtest1_1 ALTER COLUMN b SET DEFAULT (a * 2);

pg_restore: warning: errors ignored on restore: 1
$

It looks like gtest1_1 inherits column "b" from gtest1, so possibly
pg_dump is just confused about the combination of inheritance and
generated columns.

I see this in v12 as well as HEAD. One interesting question is how come
the pg_upgrade test isn't failing --- maybe binary-upgrade mode handles
this case differently?

regards, tom lane

#2Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Tom Lane (#1)
1 attachment(s)
Re: Dumping/restoring fails on inherited generated column

On 2019-12-04 15:14, Tom Lane wrote:

Run the regression tests with "make installcheck", then:

$ pg_dump -Fc regression >r.dump
$ createdb r2
$ pg_restore -d r2 r.dump
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 6005; 2604 24821 DEFAULT gtest1_1 b postgres
pg_restore: error: could not execute query: ERROR: column "b" of relation "gtest1_1" is a generated column
Command was: ALTER TABLE ONLY public.gtest1_1 ALTER COLUMN b SET DEFAULT (a * 2);

pg_restore: warning: errors ignored on restore: 1
$

It looks like gtest1_1 inherits column "b" from gtest1, so possibly
pg_dump is just confused about the combination of inheritance and
generated columns.

Yeah, there was some stuff about the "separate" dumping of defaults that
I apparently forgot to address. The attached patch fixes it. I'll see
about adding a test case for it, too.

I see this in v12 as well as HEAD. One interesting question is how come
the pg_upgrade test isn't failing --- maybe binary-upgrade mode handles
this case differently?

Binary upgrade dumps out even inherited columns, so it won't run into
the "separate" case that's the issue here.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

v1-0001-Fix-dumping-of-inherited-generated-columns.patchtext/plain; charset=UTF-8; name=v1-0001-Fix-dumping-of-inherited-generated-columns.patch; x-mac-creator=0; x-mac-type=0Download
From 0ef24c98edf468ebe906c5b04e3ddf9e53bda02f Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Wed, 4 Dec 2019 21:14:19 +0100
Subject: [PATCH v1] Fix dumping of inherited generated columns

---
 src/bin/pg_dump/pg_dump.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 33e58fa287..87b3c5b09c 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -8515,6 +8515,11 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
 				{
 					attrdefs[j].separate = true;
 				}
+				else if (tbinfo->attgenerated[adnum - 1])
+				{
+					/* generated columns only go with the root table */
+					attrdefs[j].separate = false;
+				}
 				else if (!shouldPrintColumn(dopt, tbinfo, adnum - 1))
 				{
 					/* column will be suppressed, print default separately */
-- 
2.24.0

#3Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#2)
Re: Dumping/restoring fails on inherited generated column

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

On 2019-12-04 15:14, Tom Lane wrote:

It looks like gtest1_1 inherits column "b" from gtest1, so possibly
pg_dump is just confused about the combination of inheritance and
generated columns.

Yeah, there was some stuff about the "separate" dumping of defaults that
I apparently forgot to address. The attached patch fixes it. I'll see
about adding a test case for it, too.

I don't think this is right. It will probably misbehave if the
"generated" expression has any interesting dependencies:

1. You didn't duplicate the behavior of the existing separate=false
case, where it adds a dependency to try to force the default's
dependencies to exist before the table is created.

2. If that dependency turns out to create a dependency loop, the
later code will break the loop by setting separate=true anyway.
Then what?

I also find it improbable that overriding the !shouldPrintColumn
test is really the right thing. That test is what distinguishes
the is-a-parent-table from the is-a-child-table cases, and the
core of the issue here seems to be that we need to treat those
differently.

I wonder if the right fix is to not generate a DO_ATTRDEF
object at all for generated columns in child tables. Am
I right in guessing that we propagate generated-ness to
child tables automatically?

regards, tom lane

#4Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Tom Lane (#3)
1 attachment(s)
Re: Dumping/restoring fails on inherited generated column

On 2019-12-04 21:36, Tom Lane wrote:

I wonder if the right fix is to not generate a DO_ATTRDEF
object at all for generated columns in child tables. Am
I right in guessing that we propagate generated-ness to
child tables automatically?

Right. New patch using that approach attached. (Could use more
extensive comments.)

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

v2-0001-Fix-dumping-of-inherited-generated-columns.patchtext/plain; charset=UTF-8; name=v2-0001-Fix-dumping-of-inherited-generated-columns.patch; x-mac-creator=0; x-mac-type=0Download
From eeeab95adcaa298851abd3fd44d8646510e7aed2 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Mon, 3 Feb 2020 19:56:34 +0100
Subject: [PATCH v2] Fix dumping of inherited generated columns

---
 src/bin/pg_dump/pg_dump.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 33e58fa287..c9938323b1 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -8492,6 +8492,13 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
 				if (tbinfo->attisdropped[adnum - 1])
 					continue;
 
+				/*
+				 * ignore generated columns on child tables unless they have a
+				 * local definition
+				 */
+				if (tbinfo->attgenerated[adnum - 1] && !tbinfo->attislocal[adnum - 1])
+					continue;
+
 				attrdefs[j].dobj.objType = DO_ATTRDEF;
 				attrdefs[j].dobj.catId.tableoid = atooid(PQgetvalue(res, j, 0));
 				attrdefs[j].dobj.catId.oid = atooid(PQgetvalue(res, j, 1));
-- 
2.25.0

#5Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#4)
Re: Dumping/restoring fails on inherited generated column

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

On 2019-12-04 21:36, Tom Lane wrote:

I wonder if the right fix is to not generate a DO_ATTRDEF
object at all for generated columns in child tables. Am
I right in guessing that we propagate generated-ness to
child tables automatically?

Right. New patch using that approach attached. (Could use more
extensive comments.)

This looks more plausible than the previous attempt, but it's clearly
still not right, because this is what it changes in the regression
test dump:

--- r.dump.head	2020-02-03 14:16:15.774305437 -0500
+++ r.dump.patch	2020-02-03 14:18:08.599109703 -0500
@@ -15244,14 +15244,7 @@
 -- Name: gtest1_1 b; Type: DEFAULT; Schema: public; Owner: postgres
 --
-ALTER TABLE ONLY public.gtest1_1 ALTER COLUMN b SET DEFAULT (a * 2);
-
-
---
--- Name: gtest30_1 b; Type: DEFAULT; Schema: public; Owner: postgres
---
-
-ALTER TABLE ONLY public.gtest30_1 ALTER COLUMN b SET DEFAULT (a * 2);
+ALTER TABLE ONLY public.gtest1_1 ALTER COLUMN b SET DEFAULT NULL;

--

This is showing us at least two distinct problems. Now as for
"gtest30_1", what we have is that in the parent table "gtest30", column b
exists but it has no default; the generated property is only added
at the child table gtest30_1. So we need to emit ALTER COLUMN SET
GENERATED ALWAYS for gtest30_1.b. HEAD is already doing the wrong
thing there (it's emitting the expression, but as a plain default
not GENERATED). And this patch makes it emit nothing, even worse.
I think the key point here is that "attislocal" refers to whether the
column itself is locally defined, not to whether its default is.

Things are evidently also going wrong for "gtest1_1". In that case
the generated property is inherited from the parent gtest1, so we
shouldn't be emitting anything ... how come the patch fails to
make it do that?

regards, tom lane

#6Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Tom Lane (#5)
1 attachment(s)
Re: Dumping/restoring fails on inherited generated column

On 2020-02-03 20:32, Tom Lane wrote:

Things are evidently also going wrong for "gtest1_1". In that case
the generated property is inherited from the parent gtest1, so we
shouldn't be emitting anything ... how come the patch fails to
make it do that?

This is fixed by the attached new patch. It needed an additional check
in flagInhAttrs().

This is showing us at least two distinct problems. Now as for
"gtest30_1", what we have is that in the parent table "gtest30", column b
exists but it has no default; the generated property is only added
at the child table gtest30_1. So we need to emit ALTER COLUMN SET
GENERATED ALWAYS for gtest30_1.b. HEAD is already doing the wrong
thing there (it's emitting the expression, but as a plain default
not GENERATED). And this patch makes it emit nothing, even worse.
I think the key point here is that "attislocal" refers to whether the
column itself is locally defined, not to whether its default is.

This is a bit of a mess. Let me explain my thinking on generated
columns versus inheritance.

If a parent table has a generated column, then any inherited column must
also be generated and use the same expression. (Otherwise querying the
parent table would produce results that are inconsistent with the
generation expression if the rows come from the child table.)

If a parent table has a column that is not generated, then I think it
would be semantically sound if a child table had that same column but
generated. However, I think it would be very tricky to support this
correctly, and it doesn't seem useful enough, so I'd rather not do it.

That's what the gtest30_1 case above shows. It's not even clear whether
it's possible to dump this correctly in all cases because the syntax
that you allude to "turn this existing column into a generated column"
does not exist.

Note that the gtest30 test case is new in master. I'm a bit confused
why things were done that way, and I'll need to revisit this. I've also
found a few more issues with how certain combinations of DDL can create
similar situations that arguably don't make sense, and I'll continue to
look into those. Basically, my contention is that gtest30_1 should not
be allowed to exist like that.

However, the pg_dump issue is separate from those because it affects a
case that is clearly legitimate. So assuming that we end up agreeing on
a version of the attached pg_dump patch, I would like to get that into
the next minor releases and then investigate the other issues separately.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

v3-0001-pg_dump-Fix-dumping-of-inherited-generated-column.patchtext/plain; charset=UTF-8; name=v3-0001-pg_dump-Fix-dumping-of-inherited-generated-column.patch; x-mac-creator=0; x-mac-type=0Download
From 76249c9a64caaaf4c3206d5ca4ff72cc186dc416 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Thu, 6 Feb 2020 14:50:01 +0100
Subject: [PATCH v3] pg_dump: Fix dumping of inherited generated columns

Generation expressions of generated columns are always inherited, so
there is no need to set them separately in child tables, and there is
no syntax to do so either.  The code previously used the code paths
for the handling of default values, for which different rules apply;
in particular it might want to set a default value explicitly for an
inherited column.  For generated columns, just skip all that.

Discussion: https://www.postgresql.org/message-id/flat/15830.1575468847%40sss.pgh.pa.us
---
 src/bin/pg_dump/common.c  | 8 ++++++++
 src/bin/pg_dump/pg_dump.c | 9 +++++++++
 2 files changed, 17 insertions(+)

diff --git a/src/bin/pg_dump/common.c b/src/bin/pg_dump/common.c
index 3549f7bc08..170c87823d 100644
--- a/src/bin/pg_dump/common.c
+++ b/src/bin/pg_dump/common.c
@@ -476,6 +476,14 @@ flagInhAttrs(DumpOptions *dopt, TableInfo *tblinfo, int numTables)
 			if (tbinfo->attisdropped[j])
 				continue;
 
+			/*
+			 * Skip generated columns; it is not possible for an inherited
+			 * column to have a different generation expression that the
+			 * parent.
+			 */
+			if (tbinfo->attgenerated[j])
+				continue;
+
 			foundNotNull = false;
 			foundDefault = false;
 			for (k = 0; k < numParents; k++)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 33e58fa287..5fcb89093e 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -8492,6 +8492,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
 				if (tbinfo->attisdropped[adnum - 1])
 					continue;
 
+				/*
+				 * Ignore generated columns on child tables unless they have a
+				 * local definition.  Generation expressions are always
+				 * inherited, so there is no need to set them again in child
+				 * tables, and there is no syntax for it either.
+				 */
+				if (tbinfo->attgenerated[adnum - 1] && !tbinfo->attislocal[adnum - 1])
+					continue;
+
 				attrdefs[j].dobj.objType = DO_ATTRDEF;
 				attrdefs[j].dobj.catId.tableoid = atooid(PQgetvalue(res, j, 0));
 				attrdefs[j].dobj.catId.oid = atooid(PQgetvalue(res, j, 1));
-- 
2.25.0

#7Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#6)
Re: Dumping/restoring fails on inherited generated column

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

On 2020-02-03 20:32, Tom Lane wrote:

This is showing us at least two distinct problems. Now as for
"gtest30_1", what we have is that in the parent table "gtest30", column b
exists but it has no default; the generated property is only added
at the child table gtest30_1. So we need to emit ALTER COLUMN SET
GENERATED ALWAYS for gtest30_1.b. HEAD is already doing the wrong
thing there (it's emitting the expression, but as a plain default
not GENERATED). And this patch makes it emit nothing, even worse.
I think the key point here is that "attislocal" refers to whether the
column itself is locally defined, not to whether its default is.

This is a bit of a mess. Let me explain my thinking on generated
columns versus inheritance.

If a parent table has a generated column, then any inherited column must
also be generated and use the same expression. (Otherwise querying the
parent table would produce results that are inconsistent with the
generation expression if the rows come from the child table.)

Check.

If a parent table has a column that is not generated, then I think it
would be semantically sound if a child table had that same column but
generated. However, I think it would be very tricky to support this
correctly, and it doesn't seem useful enough, so I'd rather not do it.

So ... why is that so hard exactly? AFAICS, the existing regression
test cases show that it works fine. Except that pg_dump gets it wrong.
In general, we surely want to support child columns that have defaults
different from the parent column's default, so this doesn't seem quite
that huge a leap to me.

That's what the gtest30_1 case above shows. It's not even clear whether
it's possible to dump this correctly in all cases because the syntax
that you allude to "turn this existing column into a generated column"
does not exist.

I'm a little confused by that statement. What is this doing, if not
that:

regression=# create table foo (f1 int not null);
CREATE TABLE
regression=# alter table foo alter column f1 add generated always as identity;
ALTER TABLE
regression=# \d foo
Table "public.foo"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+------------------------------
f1 | integer | | not null | generated always as identity

If we didn't have things like ALTER ... SET GENERATED and
ALTER ... DROP EXPRESSION, I'd be a lot more content to accept
the position that generated-ness is an immutable column property.
But we do have those things, so the restriction you're proposing
seems mighty arbitrary.

regards, tom lane

#8Masahiko Sawada
masahiko.sawada@2ndquadrant.com
In reply to: Peter Eisentraut (#6)
1 attachment(s)
Re: Dumping/restoring fails on inherited generated column

I arrived at this thread while investigating the same issue recently
reported[1]/messages/by-id/2678bad1-048f-519a-ef24-b12962f41807@enterprisedb.com.

On Fri, 7 Feb 2020 at 04:36, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

On 2020-02-03 20:32, Tom Lane wrote:

Things are evidently also going wrong for "gtest1_1". In that case
the generated property is inherited from the parent gtest1, so we
shouldn't be emitting anything ... how come the patch fails to
make it do that?

This is fixed by the attached new patch. It needed an additional check
in flagInhAttrs().

This is showing us at least two distinct problems. Now as for
"gtest30_1", what we have is that in the parent table "gtest30", column b
exists but it has no default; the generated property is only added
at the child table gtest30_1. So we need to emit ALTER COLUMN SET
GENERATED ALWAYS for gtest30_1.b. HEAD is already doing the wrong
thing there (it's emitting the expression, but as a plain default
not GENERATED). And this patch makes it emit nothing, even worse.
I think the key point here is that "attislocal" refers to whether the
column itself is locally defined, not to whether its default is.

This is a bit of a mess. Let me explain my thinking on generated
columns versus inheritance.

If a parent table has a generated column, then any inherited column must
also be generated and use the same expression. (Otherwise querying the
parent table would produce results that are inconsistent with the
generation expression if the rows come from the child table.)

After investigating this issue, I think that current DDLs regarding
inherited tables and generated columns seem not to work fine.

We can make an inherited table have the same column having a different
generation expression as follows:

=# create table p1 (a int, b int generated always as (a + 1) stored);
=# create table c1 (a int, b int generated always as (a + 2) stored)
inherits(p1);

But the column on the inherited table has a default value, the column
will be generation expression with a const value:

=# create table p2 (a int, b int generated always as (a + 1) stored);
=# create table c2 (a int, b int default 10) inherits(p2);
=# \d c2
Table "public.c2"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------------------------------
a | integer | | |
b | integer | | | generated always as (10) stored
Inherits: p2

Also, CREATE TABLE doesn't support to create a generated column on
inherited table, which is the same name but is a normal column on the
parent table, as follows:

=# create table p3 (a int, b int);
=# create table c3 (a int, b int generated always as (a + 2) stored)
inherits(p3);
ERROR: cannot use column reference in DEFAULT expression
LINE 1: ...reate table c3 (a int, b int generated always as (a + 2) sto...

Aside from the error message seems not correct, it's actually possible
that we can have only the inherited table's column have a generation
expression by:

=# create table p4 (a int, b int);
=# create table c4 (a int);
=# alter table c4 add column b int generated always as (a * 3) stored;
=# alter table c4 inherit p4;

Because of this behavior, pg_dump generates a query for the table c4
that cannot be restored.

I think we can fix these issues with the attached patch but it seems
better discussing the desired behavior first.

Regards,

[1]: /messages/by-id/2678bad1-048f-519a-ef24-b12962f41807@enterprisedb.com

--
Masahiko Sawada http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

generated_column_fix.patchapplication/x-patch; name=generated_column_fix.patchDownload
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 5745cd648a..efa13f47d2 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -2623,6 +2623,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
 				{
 					def->raw_default = newdef->raw_default;
 					def->cooked_default = newdef->cooked_default;
+					def->generated = newdef->generated;
 				}
 			}
 			else
#9Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Masahiko Sawada (#8)
1 attachment(s)
Re: Dumping/restoring fails on inherited generated column

On 2020-04-23 08:35, Masahiko Sawada wrote:

After investigating this issue, I think that current DDLs regarding
inherited tables and generated columns seem not to work fine.

Right, there were a number of combinations that were not properly
handled. The attached patch should fix them all. It's made against
PG12 but also works on master. See contained commit message and
documentation for details.

(This does not touch the issues with pg_dump, but it helps clarify the
cases that pg_dump needs to handle.)

We can make an inherited table have the same column having a different
generation expression as follows:

=# create table p1 (a int, b int generated always as (a + 1) stored);
=# create table c1 (a int, b int generated always as (a + 2) stored)
inherits(p1);

With my patch, this becomes an error.

But the column on the inherited table has a default value, the column
will be generation expression with a const value:

=# create table p2 (a int, b int generated always as (a + 1) stored);
=# create table c2 (a int, b int default 10) inherits(p2);
=# \d c2
Table "public.c2"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------------------------------
a | integer | | |
b | integer | | | generated always as (10) stored
Inherits: p2

With my patch, this also becomes an error.

Also, CREATE TABLE doesn't support to create a generated column on
inherited table, which is the same name but is a normal column on the
parent table, as follows:

=# create table p3 (a int, b int);
=# create table c3 (a int, b int generated always as (a + 2) stored)
inherits(p3);
ERROR: cannot use column reference in DEFAULT expression
LINE 1: ...reate table c3 (a int, b int generated always as (a + 2) sto...

This is allowed with my patch (which is basically an expanded version of
your patch).

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

0001-Fix-several-DDL-issues-of-generated-columns-versus-i.patchtext/plain; charset=UTF-8; name=0001-Fix-several-DDL-issues-of-generated-columns-versus-i.patch; x-mac-creator=0; x-mac-type=0Download
From 2c82a0f77ff676634e27782b147fce9c0089fb78 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Wed, 6 May 2020 16:25:54 +0200
Subject: [PATCH] Fix several DDL issues of generated columns versus
 inheritance

Several combinations of generated columns and inheritance in CREATE
TABLE were not handled correctly.  Specifically:

- Disallow a child column specifying a generation expression if the
  parent column is a generated column.  The child column definition
  must be unadorned and the parent column's generation expression will
  be copied.

- Prohibit a child column of a generated parent column specifying
  default values or identity.

- Allow a child column of a not-generated parent column specifying
  itself as a generated column.  This previously did not work, but it
  was possible to arrive at the state via other means (involving ALTER
  TABLE), so it seems sensible to support it.

Add tests for each case.  Also add documentation about the rules
involving generated columns and inheritance.

Discussion:
    https://www.postgresql.org/message-id/flat/15830.1575468847%40sss.pgh.pa.us
    https://www.postgresql.org/message-id/flat/2678bad1-048f-519a-ef24-b12962f41807%40enterprisedb.com
    https://www.postgresql.org/message-id/flat/CAJvUf_u4h0DxkCMCeEKAWCuzGUTnDP-G5iVmSwxLQSXn0_FWNQ%40mail.gmail.com
---
 doc/src/sgml/ddl.sgml                   | 26 +++++++++++
 src/backend/commands/tablecmds.c        | 61 +++++++++++++++++++++++--
 src/test/regress/expected/generated.out | 44 +++++++++++++++++-
 src/test/regress/sql/generated.sql      | 22 ++++++++-
 4 files changed, 146 insertions(+), 7 deletions(-)

diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index a20e5fb366..bf9f0181ea 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -324,6 +324,32 @@ <title>Generated Columns</title>
       linkend="sql-createforeigntable"/> for details.
      </para>
     </listitem>
+    <listitem>
+     <para>For inheritance:</para>
+     <itemizedlist>
+      <listitem>
+       <para>
+        If a parent column is a generated column, a child column must also be
+        a generated column using the same expression.  In the definition of
+        the child column, leave off the <literal>GENERATED</literal> clause,
+        as it will be copied from the parent.
+       </para>
+      </listitem>
+      <listitem>
+       <para>
+        In case of multiple inheritance, if one parent column is a generated
+        column, then all parent columns must be generated columns and with the
+        same expression.
+       </para>
+      </listitem>
+      <listitem>
+       <para>
+        If a parent column is not a generated column, a child column may be
+        defined to be a generated column or not.
+       </para>
+      </listitem>
+     </itemizedlist>
+    </listitem>
    </itemizedlist>
   </para>
 
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 1a6ccff8f1..673166bca5 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -2577,12 +2577,55 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
 				def->is_local = true;
 				/* Merge of NOT NULL constraints = OR 'em together */
 				def->is_not_null |= newdef->is_not_null;
+
+				/*
+				 * Check for conflicts related to generated columns.
+				 *
+				 * If the parent column is generated, the child column must be
+				 * unadorned and will be made a generated column.  (We could
+				 * in theory allow the child column definition specifying the
+				 * exact same generation expression, but that's a bit
+				 * complicated to implement and doesn't seem very useful.)  We
+				 * also check that the child column doesn't specify a default
+				 * value or identity, which matches the rules for a single
+				 * column in parse_util.c.
+				 */
+				if (def->generated)
+				{
+					if (newdef->generated)
+						ereport(ERROR,
+								(errcode(ERRCODE_INVALID_COLUMN_DEFINITION),
+								 errmsg("child column \"%s\" specifies generation expression",
+										def->colname),
+								 errhint("Leave off the generation expression in the definition of the child table column to inherit the generation expression from the parent table.")));
+					if (newdef->raw_default && !newdef->generated)
+						ereport(ERROR,
+								(errcode(ERRCODE_INVALID_COLUMN_DEFINITION),
+								 errmsg("column \"%s\" inherits from generated column but specifies default",
+										def->colname)));
+					if (newdef->identity)
+						ereport(ERROR,
+								(errcode(ERRCODE_INVALID_COLUMN_DEFINITION),
+								 errmsg("column \"%s\" inherits from generated column but specifies identity",
+										def->colname)));
+				}
+				/*
+				 * If the parent column is not generated, then take whatever
+				 * the child column definition says.
+				 */
+				else
+				{
+					if (newdef->generated)
+						def->generated = newdef->generated;
+				}
+
 				/* If new def has a default, override previous default */
 				if (newdef->raw_default != NULL)
 				{
 					def->raw_default = newdef->raw_default;
 					def->cooked_default = newdef->cooked_default;
 				}
+
 			}
 			else
 			{
@@ -2668,11 +2711,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
 			ColumnDef  *def = lfirst(entry);
 
 			if (def->cooked_default == &bogus_marker)
-				ereport(ERROR,
-						(errcode(ERRCODE_INVALID_COLUMN_DEFINITION),
-						 errmsg("column \"%s\" inherits conflicting default values",
-								def->colname),
-						 errhint("To resolve the conflict, specify a default explicitly.")));
+			{
+				if (def->generated)
+					ereport(ERROR,
+							(errcode(ERRCODE_INVALID_COLUMN_DEFINITION),
+							 errmsg("column \"%s\" inherits conflicting generation expressions",
+									def->colname)));
+				else
+					ereport(ERROR,
+							(errcode(ERRCODE_INVALID_COLUMN_DEFINITION),
+							 errmsg("column \"%s\" inherits conflicting default values",
+									def->colname),
+							 errhint("To resolve the conflict, specify a default explicitly.")));
+			}
 		}
 	}
 
diff --git a/src/test/regress/expected/generated.out b/src/test/regress/expected/generated.out
index f87c86a8ce..0436773628 100644
--- a/src/test/regress/expected/generated.out
+++ b/src/test/regress/expected/generated.out
@@ -219,12 +219,54 @@ SELECT * FROM gtest1;
  4 | 8
 (2 rows)
 
--- test inheritance mismatch
+CREATE TABLE gtest_normal (a int, b int);
+CREATE TABLE gtest_normal_child (a int, b int GENERATED ALWAYS AS (a * 2) STORED) INHERITS (gtest_normal);
+NOTICE:  merging column "a" with inherited definition
+NOTICE:  merging column "b" with inherited definition
+\d gtest_normal_child
+                      Table "public.gtest_normal_child"
+ Column |  Type   | Collation | Nullable |              Default               
+--------+---------+-----------+----------+------------------------------------
+ a      | integer |           |          | 
+ b      | integer |           |          | generated always as (a * 2) stored
+Inherits: gtest_normal
+
+INSERT INTO gtest_normal (a) VALUES (1);
+INSERT INTO gtest_normal_child (a) VALUES (2);
+SELECT * FROM gtest_normal;
+ a | b 
+---+---
+ 1 |  
+ 2 | 4
+(2 rows)
+
+-- test inheritance mismatches between parent and child
+CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS (a * 22) STORED) INHERITS (gtest1);  -- error
+NOTICE:  merging column "b" with inherited definition
+ERROR:  child column "b" specifies generation expression
+HINT:  Leave off the generation expression in the definition of the child table column to inherit the generation expression from the parent table.
+CREATE TABLE gtestx (x int, b int DEFAULT 10) INHERITS (gtest1);  -- error
+NOTICE:  merging column "b" with inherited definition
+ERROR:  column "b" inherits from generated column but specifies default
+CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS IDENTITY) INHERITS (gtest1);  -- error
+NOTICE:  merging column "b" with inherited definition
+ERROR:  column "b" inherits from generated column but specifies identity
+-- test multiple inheritance mismatches
 CREATE TABLE gtesty (x int, b int);
 CREATE TABLE gtest1_2 () INHERITS (gtest1, gtesty);  -- error
 NOTICE:  merging multiple inherited definitions of column "b"
 ERROR:  inherited column "b" has a generation conflict
 DROP TABLE gtesty;
+CREATE TABLE gtesty (x int, b int GENERATED ALWAYS AS (x * 22) STORED);
+CREATE TABLE gtest1_2 () INHERITS (gtest1, gtesty);  -- error
+NOTICE:  merging multiple inherited definitions of column "b"
+ERROR:  column "b" inherits conflicting generation expressions
+DROP TABLE gtesty;
+CREATE TABLE gtesty (x int, b int DEFAULT 55);
+CREATE TABLE gtest1_2 () INHERITS (gtest0, gtesty);  -- error
+NOTICE:  merging multiple inherited definitions of column "b"
+ERROR:  inherited column "b" has a generation conflict
+DROP TABLE gtesty;
 -- test stored update
 CREATE TABLE gtest3 (a int, b int GENERATED ALWAYS AS (a * 3) STORED);
 INSERT INTO gtest3 (a) VALUES (1), (2), (3), (NULL);
diff --git a/src/test/regress/sql/generated.sql b/src/test/regress/sql/generated.sql
index bdcedbb991..f089f9a74d 100644
--- a/src/test/regress/sql/generated.sql
+++ b/src/test/regress/sql/generated.sql
@@ -89,11 +89,31 @@ CREATE TABLE gtest1_1 () INHERITS (gtest1);
 SELECT * FROM gtest1_1;
 SELECT * FROM gtest1;
 
--- test inheritance mismatch
+CREATE TABLE gtest_normal (a int, b int);
+CREATE TABLE gtest_normal_child (a int, b int GENERATED ALWAYS AS (a * 2) STORED) INHERITS (gtest_normal);
+\d gtest_normal_child
+INSERT INTO gtest_normal (a) VALUES (1);
+INSERT INTO gtest_normal_child (a) VALUES (2);
+SELECT * FROM gtest_normal;
+
+-- test inheritance mismatches between parent and child
+CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS (a * 22) STORED) INHERITS (gtest1);  -- error
+CREATE TABLE gtestx (x int, b int DEFAULT 10) INHERITS (gtest1);  -- error
+CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS IDENTITY) INHERITS (gtest1);  -- error
+
+-- test multiple inheritance mismatches
 CREATE TABLE gtesty (x int, b int);
 CREATE TABLE gtest1_2 () INHERITS (gtest1, gtesty);  -- error
 DROP TABLE gtesty;
 
+CREATE TABLE gtesty (x int, b int GENERATED ALWAYS AS (x * 22) STORED);
+CREATE TABLE gtest1_2 () INHERITS (gtest1, gtesty);  -- error
+DROP TABLE gtesty;
+
+CREATE TABLE gtesty (x int, b int DEFAULT 55);
+CREATE TABLE gtest1_2 () INHERITS (gtest0, gtesty);  -- error
+DROP TABLE gtesty;
+
 -- test stored update
 CREATE TABLE gtest3 (a int, b int GENERATED ALWAYS AS (a * 3) STORED);
 INSERT INTO gtest3 (a) VALUES (1), (2), (3), (NULL);

base-commit: 987717d7c7007055ff7fc2ecf3d40e9bdb00e071
-- 
2.26.2

#10Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Peter Eisentraut (#9)
Re: Dumping/restoring fails on inherited generated column

On 2020-05-06 16:29, Peter Eisentraut wrote:

On 2020-04-23 08:35, Masahiko Sawada wrote:

After investigating this issue, I think that current DDLs regarding
inherited tables and generated columns seem not to work fine.

Right, there were a number of combinations that were not properly
handled. The attached patch should fix them all. It's made against
PG12 but also works on master. See contained commit message and
documentation for details.

committed to master and PG12

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#11Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#10)
Re: Dumping/restoring fails on inherited generated column

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

Right, there were a number of combinations that were not properly
handled. The attached patch should fix them all. It's made against
PG12 but also works on master. See contained commit message and
documentation for details.

committed to master and PG12

So ... this did not actually fix the dump/restore problem. In fact,
it's worse, because in HEAD I see two failures not one when doing the
same test proposed at the start of this thread:

1. make installcheck
2. pg_dump -Fc regression >r.dump
3. createdb r2
4. pg_restore -d r2 r.dump

pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 6253; 2604 226187 DEFAULT gtest1_1 b postgres
pg_restore: error: could not execute query: ERROR: column "b" of relation "gtest1_1" is a generated column
Command was: ALTER TABLE ONLY public.gtest1_1 ALTER COLUMN b SET DEFAULT (a * 2);

pg_restore: from TOC entry 6279; 2604 227276 DEFAULT gtest30_1 b postgres
pg_restore: error: could not execute query: ERROR: cannot use column reference in DEFAULT expression
Command was: ALTER TABLE ONLY public.gtest30_1 ALTER COLUMN b SET DEFAULT (a * 2);

pg_restore: warning: errors ignored on restore: 2

regards, tom lane

#12Masahiko Sawada
masahiko.sawada@2ndquadrant.com
In reply to: Tom Lane (#11)
1 attachment(s)
Re: Dumping/restoring fails on inherited generated column

On Thu, 16 Jul 2020 at 04:29, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

Right, there were a number of combinations that were not properly
handled. The attached patch should fix them all. It's made against
PG12 but also works on master. See contained commit message and
documentation for details.

committed to master and PG12

So ... this did not actually fix the dump/restore problem. In fact,
it's worse, because in HEAD I see two failures not one when doing the
same test proposed at the start of this thread:

1. make installcheck
2. pg_dump -Fc regression >r.dump
3. createdb r2
4. pg_restore -d r2 r.dump

pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 6253; 2604 226187 DEFAULT gtest1_1 b postgres
pg_restore: error: could not execute query: ERROR: column "b" of relation "gtest1_1" is a generated column
Command was: ALTER TABLE ONLY public.gtest1_1 ALTER COLUMN b SET DEFAULT (a * 2);

pg_restore: from TOC entry 6279; 2604 227276 DEFAULT gtest30_1 b postgres
pg_restore: error: could not execute query: ERROR: cannot use column reference in DEFAULT expression
Command was: ALTER TABLE ONLY public.gtest30_1 ALTER COLUMN b SET DEFAULT (a * 2);

pg_restore: warning: errors ignored on restore: 2

The minimum reproducer is:

create table a (a int, b int generated always as (a * 2) stored);
create table aa () inherits (a);

pg_dump produces the following DDLs:

CREATE TABLE public.a (
a integer,
b integer GENERATED ALWAYS AS ((a * 2)) STORED
);

CREATE TABLE public.aa (
)
INHERITS (public.a);

ALTER TABLE ONLY public.aa ALTER COLUMN b SET DEFAULT (a * 2);

However, the ALTER TABLE fails.

By commit 086ffddf, the child tables must have the same generation
expression as the expression defined in the parent. So I think pg_dump
should not generate the last DDL. I've attached the patch fixing this
issue.

Apart from the fix, I wonder if we can add a test that dumps the
database where executed 'make check' and restore it to another
database.

Regards,

--
Masahiko Sawada http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

fix_gcolumn_dump.patchapplication/octet-stream; name=fix_gcolumn_dump.patchDownload
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 94459b3539..6ec5c377ed 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -16247,6 +16247,15 @@ dumpAttrDef(Archive *fout, AttrDefInfo *adinfo)
 	if (!adinfo->separate)
 		return;
 
+	/*
+	 * Skip if the column is not local and a generated column; the
+	 * inherited column must have the same generation expression as
+	 * the expression the parent has and we don't support setting the
+	 * generation expression on the column by ALTER TABLE.
+	 */
+	if (!tbinfo->attislocal[adnum - 1] && tbinfo->attgenerated[adnum - 1])
+		return;
+
 	q = createPQExpBuffer();
 	delq = createPQExpBuffer();
 
#13Masahiko Sawada
masahiko.sawada@2ndquadrant.com
In reply to: Masahiko Sawada (#12)
1 attachment(s)
Re: Dumping/restoring fails on inherited generated column

On Thu, 23 Jul 2020 at 19:55, Masahiko Sawada
<masahiko.sawada@2ndquadrant.com> wrote:

On Thu, 16 Jul 2020 at 04:29, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

Right, there were a number of combinations that were not properly
handled. The attached patch should fix them all. It's made against
PG12 but also works on master. See contained commit message and
documentation for details.

committed to master and PG12

So ... this did not actually fix the dump/restore problem. In fact,
it's worse, because in HEAD I see two failures not one when doing the
same test proposed at the start of this thread:

1. make installcheck
2. pg_dump -Fc regression >r.dump
3. createdb r2
4. pg_restore -d r2 r.dump

pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 6253; 2604 226187 DEFAULT gtest1_1 b postgres
pg_restore: error: could not execute query: ERROR: column "b" of relation "gtest1_1" is a generated column
Command was: ALTER TABLE ONLY public.gtest1_1 ALTER COLUMN b SET DEFAULT (a * 2);

pg_restore: from TOC entry 6279; 2604 227276 DEFAULT gtest30_1 b postgres
pg_restore: error: could not execute query: ERROR: cannot use column reference in DEFAULT expression
Command was: ALTER TABLE ONLY public.gtest30_1 ALTER COLUMN b SET DEFAULT (a * 2);

pg_restore: warning: errors ignored on restore: 2

The minimum reproducer is:

create table a (a int, b int generated always as (a * 2) stored);
create table aa () inherits (a);

pg_dump produces the following DDLs:

CREATE TABLE public.a (
a integer,
b integer GENERATED ALWAYS AS ((a * 2)) STORED
);

CREATE TABLE public.aa (
)
INHERITS (public.a);

ALTER TABLE ONLY public.aa ALTER COLUMN b SET DEFAULT (a * 2);

However, the ALTER TABLE fails.

By commit 086ffddf, the child tables must have the same generation
expression as the expression defined in the parent. So I think pg_dump
should not generate the last DDL. I've attached the patch fixing this
issue.

Apart from the fix, I wonder if we can add a test that dumps the
database where executed 'make check' and restore it to another
database.

This issue is not fixed yet. I've attached the updated version patch
and registered it to commit fest so as not to forget. Please review
it.

Regards,

--
Masahiko Sawada http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

fix_gcolumn_dump_v2.patchapplication/octet-stream; name=fix_gcolumn_dump_v2.patchDownload
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 2cb3f9b083..f9f752caee 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -16272,6 +16272,15 @@ dumpAttrDef(Archive *fout, AttrDefInfo *adinfo)
 	if (!adinfo->separate)
 		return;
 
+	/*
+	 * Skip if the column is an inherited column and a generated column;
+	 * inherited columns must have the same generation expression as
+	 * the parent's expression and we don't support setting the generation
+	 * expression on the column by ALTER TABLE.
+	 */
+	if (!tbinfo->attislocal[adnum - 1] && tbinfo->attgenerated[adnum - 1])
+		return;
+
 	q = createPQExpBuffer();
 	delq = createPQExpBuffer();
 
#14Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Masahiko Sawada (#13)
1 attachment(s)
Re: Dumping/restoring fails on inherited generated column

I have been analyzing this issue again. We have a few candidate patches
that do very similar things for avoiding dumping the generation
expression of table gtest1_1. We can figure out later which one of
these we like best. But there is another issue lurking nearby. The
table hierarchy of gtest30, which is created in the regression tests
like this:

CREATE TABLE gtest30 (
a int,
b int GENERATED ALWAYS AS (a * 2) STORED
);
CREATE TABLE gtest30_1 () INHERITS (gtest30);
ALTER TABLE ONLY gtest30 ALTER COLUMN b DROP EXPRESSION;

This drops the generation expression from the parent table but not the
child table. This is currently dumped like this:

CREATE TABLE public.gtest30 (
a integer,
b integer
);

CREATE TABLE public.gtest30_1 (
)
INHERITS (public.gtest30);

ALTER TABLE ONLY public.gtest30_1 ALTER COLUMN b SET DEFAULT (a * 2);

The proposed patches will cause the last statement to be omitted, but
that still won't recreate the original state. The problem is that there
is no command to make a column generated afterwards, like the SET
DEFAULT command, so we can't dump it like this. We would have to produce

CREATE TABLE public.gtest30 (
a integer,
b integer
);

CREATE TABLE public.gtest30_1 (
b integer GENERATED ALWAYS AS (a * 2) STORED
)
INHERITS (public.gtest30);

but this will create the column "b" of gtest30_1 as attlocal, which the
original command sequence does not.

We could probably fix this by having ALTER TABLE ONLY / DROP EXPRESSION
update the attlocal column of direct children to true, to make the
catalog state look like something that can be restored. However, that's
a fair amount of complicated code, so for now I propose to just prohibit
this command, meaning you can't use ONLY in this command if there are
children. This is new in PG13, so this change would have very limited
impact in practice.

Proposed patch attached.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachments:

0001-Disallow-ALTER-TABLE-ONLY-DROP-EXPRESSION.patchtext/plain; charset=UTF-8; name=0001-Disallow-ALTER-TABLE-ONLY-DROP-EXPRESSION.patch; x-mac-creator=0; x-mac-type=0Download
From 6c4209883dad527eb1140a61ed7ac6a610cf2a14 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Fri, 25 Sep 2020 14:51:59 +0200
Subject: [PATCH] Disallow ALTER TABLE ONLY / DROP EXPRESSION

The current implementation cannot handle this correct, so just forbid
it for now.

GENERATED clauses must be attached to the column definition and cannot
be added later like DEFAULT, so if a child table has a generation
expression that the parent does not have, the child column will
necessarily be an attlocal column.  So to implement ALTER TABLE ONLY /
DROP EXPRESSION, we'd need extra code to update attislocal of the
direct child tables, somewhat similar to how DROP COLUMN does it, so
that the resulting state can be properly dumped and restored.
---
 src/backend/commands/tablecmds.c        | 22 +++++++++++++++++++---
 src/test/regress/expected/generated.out | 11 ++++++-----
 src/test/regress/sql/generated.sql      |  2 +-
 3 files changed, 26 insertions(+), 9 deletions(-)

diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 16285ad09f..b1b1b1e74a 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -412,7 +412,7 @@ static ObjectAddress ATExecAddIdentity(Relation rel, const char *colName,
 static ObjectAddress ATExecSetIdentity(Relation rel, const char *colName,
 									   Node *def, LOCKMODE lockmode);
 static ObjectAddress ATExecDropIdentity(Relation rel, const char *colName, bool missing_ok, LOCKMODE lockmode);
-static void ATPrepDropExpression(Relation rel, AlterTableCmd *cmd, bool recursing);
+static void ATPrepDropExpression(Relation rel, AlterTableCmd *cmd, bool recurse, bool recursing, LOCKMODE lockmode);
 static ObjectAddress ATExecDropExpression(Relation rel, const char *colName, bool missing_ok, LOCKMODE lockmode);
 static ObjectAddress ATExecSetStatistics(Relation rel, const char *colName, int16 colNum,
 										 Node *newValue, LOCKMODE lockmode);
@@ -4142,7 +4142,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,
 		case AT_DropExpression: /* ALTER COLUMN DROP EXPRESSION */
 			ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
 			ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode, context);
-			ATPrepDropExpression(rel, cmd, recursing);
+			ATPrepDropExpression(rel, cmd, recurse, recursing, lockmode);
 			pass = AT_PASS_DROP;
 			break;
 		case AT_SetStatistics:	/* ALTER COLUMN SET STATISTICS */
@@ -7240,8 +7240,24 @@ ATExecDropIdentity(Relation rel, const char *colName, bool missing_ok, LOCKMODE
  * ALTER TABLE ALTER COLUMN DROP EXPRESSION
  */
 static void
-ATPrepDropExpression(Relation rel, AlterTableCmd *cmd, bool recursing)
+ATPrepDropExpression(Relation rel, AlterTableCmd *cmd, bool recurse, bool recursing, LOCKMODE lockmode)
 {
+	/*
+	 * Reject ONLY if there are child tables.  We could implement this, but it
+	 * is a bit complicated.  GENERATED clauses must be attached to the column
+	 * definition and cannot be added later like DEFAULT, so if a child table
+	 * has a generation expression that the parent does not have, the child
+	 * column will necessarily be an attlocal column.  So to implement ONLY
+	 * here, we'd need extra code to update attislocal of the direct child
+	 * tables, somewhat similar to how DROP COLUMN does it, so that the
+	 * resulting state can be properly dumped and restored.
+	 */
+	if (!recurse &&
+		find_inheritance_children(RelationGetRelid(rel), lockmode))
+		ereport(ERROR,
+				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+				 errmsg("ALTER TABLE / DROP EXPRESSION must be applied to child tables too")));
+
 	/*
 	 * Cannot drop generation expression from inherited columns.
 	 */
diff --git a/src/test/regress/expected/generated.out b/src/test/regress/expected/generated.out
index 7ccc3c65ed..4b06260304 100644
--- a/src/test/regress/expected/generated.out
+++ b/src/test/regress/expected/generated.out
@@ -805,13 +805,14 @@ CREATE TABLE gtest30 (
     b int GENERATED ALWAYS AS (a * 2) STORED
 );
 CREATE TABLE gtest30_1 () INHERITS (gtest30);
-ALTER TABLE ONLY gtest30 ALTER COLUMN b DROP EXPRESSION;
+ALTER TABLE ONLY gtest30 ALTER COLUMN b DROP EXPRESSION;  -- error
+ERROR:  ALTER TABLE / DROP EXPRESSION must be applied to child tables too
 \d gtest30
-              Table "public.gtest30"
- Column |  Type   | Collation | Nullable | Default 
---------+---------+-----------+----------+---------
+                            Table "public.gtest30"
+ Column |  Type   | Collation | Nullable |              Default               
+--------+---------+-----------+----------+------------------------------------
  a      | integer |           |          | 
- b      | integer |           |          | 
+ b      | integer |           |          | generated always as (a * 2) stored
 Number of child tables: 1 (Use \d+ to list them.)
 
 \d gtest30_1
diff --git a/src/test/regress/sql/generated.sql b/src/test/regress/sql/generated.sql
index 4cff1279c7..c86ad34b00 100644
--- a/src/test/regress/sql/generated.sql
+++ b/src/test/regress/sql/generated.sql
@@ -411,7 +411,7 @@ CREATE TABLE gtest30 (
     b int GENERATED ALWAYS AS (a * 2) STORED
 );
 CREATE TABLE gtest30_1 () INHERITS (gtest30);
-ALTER TABLE ONLY gtest30 ALTER COLUMN b DROP EXPRESSION;
+ALTER TABLE ONLY gtest30 ALTER COLUMN b DROP EXPRESSION;  -- error
 \d gtest30
 \d gtest30_1
 ALTER TABLE gtest30_1 ALTER COLUMN b DROP EXPRESSION;  -- error
-- 
2.28.0

#15Daniel Gustafsson
daniel@yesql.se
In reply to: Peter Eisentraut (#14)
Re: Dumping/restoring fails on inherited generated column

On 25 Sep 2020, at 15:07, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote:

We could probably fix this by having ALTER TABLE ONLY / DROP EXPRESSION update the attlocal column of direct children to true, to make the catalog state look like something that can be restored. However, that's a fair amount of complicated code, so for now I propose to just prohibit this command, meaning you can't use ONLY in this command if there are children. This is new in PG13, so this change would have very limited impact in practice.

That sounds a bit dramatic. Do you propose to do that in v13 as well or just
in HEAD? If the latter, considering that the window until the 14 freeze is
quite wide shouldn't we try to fix it first?

cheers ./daniel

#16Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#14)
Re: Dumping/restoring fails on inherited generated column

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

The proposed patches will cause the last statement to be omitted, but
that still won't recreate the original state. The problem is that there
is no command to make a column generated afterwards, like the SET
DEFAULT command, so we can't dump it like this.

Right. I'm not even sure what such a command should do ... would it run
through all existing rows and update them to be the GENERATED value?

We could probably fix this by having ALTER TABLE ONLY / DROP EXPRESSION
update the attlocal column of direct children to true, to make the
catalog state look like something that can be restored. However, that's
a fair amount of complicated code, so for now I propose to just prohibit
this command, meaning you can't use ONLY in this command if there are
children. This is new in PG13, so this change would have very limited
impact in practice.

+1. At this point we would want some fairly un-complicated fix for
the v13 branch anyhow, and this seems to fit the bill. (Also, having
child columns suddenly grow an attislocal property doesn't seem to meet
the principle of least surprise.)

regards, tom lane

#17Masahiko Sawada
masahiko.sawada@2ndquadrant.com
In reply to: Peter Eisentraut (#14)
Re: Dumping/restoring fails on inherited generated column

On Fri, 25 Sep 2020 at 22:07, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

I have been analyzing this issue again. We have a few candidate patches
that do very similar things for avoiding dumping the generation
expression of table gtest1_1. We can figure out later which one of
these we like best. But there is another issue lurking nearby. The
table hierarchy of gtest30, which is created in the regression tests
like this:

CREATE TABLE gtest30 (
a int,
b int GENERATED ALWAYS AS (a * 2) STORED
);
CREATE TABLE gtest30_1 () INHERITS (gtest30);
ALTER TABLE ONLY gtest30 ALTER COLUMN b DROP EXPRESSION;

This drops the generation expression from the parent table but not the
child table. This is currently dumped like this:

CREATE TABLE public.gtest30 (
a integer,
b integer
);

CREATE TABLE public.gtest30_1 (
)
INHERITS (public.gtest30);

ALTER TABLE ONLY public.gtest30_1 ALTER COLUMN b SET DEFAULT (a * 2);

The proposed patches will cause the last statement to be omitted, but
that still won't recreate the original state. The problem is that there
is no command to make a column generated afterwards, like the SET
DEFAULT command, so we can't dump it like this. We would have to produce

CREATE TABLE public.gtest30 (
a integer,
b integer
);

CREATE TABLE public.gtest30_1 (
b integer GENERATED ALWAYS AS (a * 2) STORED
)
INHERITS (public.gtest30);

but this will create the column "b" of gtest30_1 as attlocal, which the
original command sequence does not.

We could probably fix this by having ALTER TABLE ONLY / DROP EXPRESSION
update the attlocal column of direct children to true, to make the
catalog state look like something that can be restored. However, that's
a fair amount of complicated code, so for now I propose to just prohibit
this command, meaning you can't use ONLY in this command if there are
children. This is new in PG13, so this change would have very limited
impact in practice.

Proposed patch attached.

+1

If we have ALTER TABLE ONLY / DROP EXPRESSION update the attlocal
column of children to true to fix the issue you raised, my proposed
patch is not necessary. OTOH if we fix it by prohibiting the command,
I guess we need both patches to fix the issues.

Regards,

--
Masahiko Sawada http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#18Tom Lane
tgl@sss.pgh.pa.us
In reply to: Masahiko Sawada (#17)
Re: Dumping/restoring fails on inherited generated column

[ Pulling Daniel into this older thread seems like the cleanest way to
unify the two threads ]

Masahiko Sawada <masahiko.sawada@2ndquadrant.com> writes:

If we have ALTER TABLE ONLY / DROP EXPRESSION update the attlocal
column of children to true to fix the issue you raised, my proposed
patch is not necessary. OTOH if we fix it by prohibiting the command,
I guess we need both patches to fix the issues.

Right, Peter already mentioned that we need a further pg_dump fix on
top of prohibiting the ONLY ... DROP EXPRESSION case.

Bug #16622 [1]/messages/by-id/16622-779a79851b4e2491@postgresql.org is another report of this same issue, and in that thread,
Daniel argues that the right fix is just

+	/*
+	 * Skip if the column isn't local to the table's definition as the attrdef
+	 * will be printed with the inheritance parent table definition
+	 */
+	if (!tbinfo->attislocal[adnum - 1])
+		return;

without the attgenerated clause that Masahiko-san proposes.
I think however that that's wrong. It is possible to have
a non-attislocal column that has its own default, because
you can do

d3=# create table parent (f1 int default 2);
CREATE TABLE
d3=# create table child (f1 int default 3) inherits(parent);
NOTICE: merging column "f1" with inherited definition
CREATE TABLE
d3=# create table child2() inherits(parent);
CREATE TABLE
d3=# alter table child2 alter column f1 set default 42;
ALTER TABLE

This does not cause child2.f1's attislocal property to become
true. Maybe it should have, but it's probably too late for
that; at least, pg_dump couldn't assume it's true in older
servers. Therefore, since we can't tell whether the default
is inherited or not, we'd better dump it.

(I recall that pg_dump has a hack somewhere that checks for
textual equality of CHECK conditions to avoid dumping redundant
child copies. Maybe we could do something similar here.)

The situation is different for GENERATED columns, since we disallow
a child having a different GENERATED property than the parent.
However, I think Masahiko-san's patch is not quite right either,
because a column can be both inherited and local. An example is

d3=# create table pgen (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
CREATE TABLE
d3=# create table cgen1 (b int) inherits (pgen);
NOTICE: moving and merging column "b" with inherited definition
DETAIL: User-specified column moved to the position of the inherited column.
CREATE TABLE
d3=# select attname, attislocal, attinhcount from pg_attribute where attrelid = 'cgen1'::regclass and attnum>0;
attname | attislocal | attinhcount
---------+------------+-------------
a | f | 1
b | t | 1
(2 rows)

So it appears to me that a more correct coding is

/*
* Do not print a GENERATED default for an inherited column; it will
* be inherited from the parent, and the backend won't accept a
* command to set it separately.
*/
if (tbinfo->attinhcount[adnum - 1] > 0 && tbinfo->attgenerated[adnum - 1])
return;

Unfortunately this has still got a problem: it will mishandle the case of
a child column that is GENERATED while its parent is not. Peter opined
way upthread that we should not allow that, but according to my testing
we do.

This'd all be a lot cleaner if defaults were marked as to whether they
were inherited or locally generated. Maybe we ought to work on that.

In the meantime, maybe the best bet is for pg_dump to try to detect
whether a default is identical to a parent default, and not dump it
if so. That would fix both the GENERATED case where we must not
dump it, and the non-GENERATED case where it's merely inefficient
to do so.

regards, tom lane

[1]: /messages/by-id/16622-779a79851b4e2491@postgresql.org

#19Tom Lane
tgl@sss.pgh.pa.us
In reply to: Tom Lane (#18)
Re: Dumping/restoring fails on inherited generated column

I wrote:

The situation is different for GENERATED columns, since we disallow
a child having a different GENERATED property than the parent.

BTW, that alleged prohibition is pretty damn leaky:

d3=# create table pp1 (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
CREATE TABLE
d3=# create table cc1 (a int, b int GENERATED ALWAYS AS (a * 3) STORED);
CREATE TABLE
d3=# alter table cc1 inherit pp1;
ALTER TABLE

Maybe the *real* fix here is to give up on this idea that they
can't be different?

regards, tom lane

#20Daniel Gustafsson
daniel@yesql.se
In reply to: Tom Lane (#18)
Re: Dumping/restoring fails on inherited generated column

On 29 Sep 2020, at 18:37, Tom Lane <tgl@sss.pgh.pa.us> wrote:

[ Pulling Daniel into this older thread seems like the cleanest way to
unify the two threads ]

Masahiko Sawada <masahiko.sawada@2ndquadrant.com> writes:

If we have ALTER TABLE ONLY / DROP EXPRESSION update the attlocal
column of children to true to fix the issue you raised, my proposed
patch is not necessary. OTOH if we fix it by prohibiting the command,
I guess we need both patches to fix the issues.

Right, Peter already mentioned that we need a further pg_dump fix on
top of prohibiting the ONLY ... DROP EXPRESSION case.

Bug #16622 [1] is another report of this same issue, and in that thread,
Daniel argues that the right fix is just

+	/*
+	 * Skip if the column isn't local to the table's definition as the attrdef
+	 * will be printed with the inheritance parent table definition
+	 */
+	if (!tbinfo->attislocal[adnum - 1])
+		return;

without the attgenerated clause that Masahiko-san proposes.
I think however that that's wrong. It is possible to have
a non-attislocal column that has its own default, because
you can do

Ah, that's the sequence I didn't think of. I agree with your assessment of
this being wrong. Thanks!

This does not cause child2.f1's attislocal property to become
true. Maybe it should have, but it's probably too late for
that; at least, pg_dump couldn't assume it's true in older
servers.

Do you recall the rationale for it not being set to true? I didn't spot
anything in the commit history. Intuitively it seems a tad odd.

Therefore, since we can't tell whether the default
is inherited or not, we'd better dump it.

Agreed.

(I recall that pg_dump has a hack somewhere that checks for
textual equality of CHECK conditions to avoid dumping redundant
child copies. Maybe we could do something similar here.)

Unless someone beats me to it I will take a stab at this to see what it would
look like.

cheers ./daniel

#21Tom Lane
tgl@sss.pgh.pa.us
In reply to: Daniel Gustafsson (#20)
Re: Dumping/restoring fails on inherited generated column

Daniel Gustafsson <daniel@yesql.se> writes:

On 29 Sep 2020, at 18:37, Tom Lane <tgl@sss.pgh.pa.us> wrote:

This does not cause child2.f1's attislocal property to become
true. Maybe it should have, but it's probably too late for
that; at least, pg_dump couldn't assume it's true in older
servers.

Do you recall the rationale for it not being set to true? I didn't spot
anything in the commit history. Intuitively it seems a tad odd.

I'd bet the explanation is mostly that it didn't occur to anyone
that SET DEFAULT should do that. I'm not really proposing that it
should either. If we were to make any catalog changes in response
to this, what I'd vote for is to add an "inherited" flag to
pg_attrdef. (I'm not quite sure if a bool would be sufficient,
or if we'd need to go to the same extent as pg_attribute does,
and have a bool plus an inheritance count.)

Of course, that line of thought does not lead to anything
back-patchable. But pg_dump would have to be prepared to cope
with the situation in older servers in any case.

regards, tom lane

#22Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Tom Lane (#18)
Re: Dumping/restoring fails on inherited generated column

On 2020-09-29 18:37, Tom Lane wrote:

Unfortunately this has still got a problem: it will mishandle the case of
a child column that is GENERATED while its parent is not. Peter opined
way upthread that we should not allow that, but according to my testing
we do.

Did I opine that? Commit 086ffddf3656fb3d24d9a73ce36cb1102e42cc90
explicitly allowed that case. What we don't want is the other way around.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#23Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Peter Eisentraut (#14)
Re: Dumping/restoring fails on inherited generated column

On 2020-09-25 15:07, Peter Eisentraut wrote:

We could probably fix this by having ALTER TABLE ONLY / DROP EXPRESSION
update the attlocal column of direct children to true, to make the
catalog state look like something that can be restored. However, that's
a fair amount of complicated code, so for now I propose to just prohibit
this command, meaning you can't use ONLY in this command if there are
children. This is new in PG13, so this change would have very limited
impact in practice.

With the minor releases coming up, I have committed this patch and will
work on getting the remaining pg_dump issues fixed as well.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#24Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Masahiko Sawada (#13)
Re: Dumping/restoring fails on inherited generated column

On 2020-08-27 13:30, Masahiko Sawada wrote:

This issue is not fixed yet. I've attached the updated version patch
and registered it to commit fest so as not to forget. Please review
it.

A few fixes have been committed in this thread, basically to prevent
situations that shouldn't have been allowed.

What's left is the originally reported issue that some parts of the
regression test database are dumped incorrectly. The two proposed
patches in their most recent versions are

/messages/by-id/attachment/107447/v3-0001-pg_dump-Fix-dumping-of-inherited-generated-column.patch
(message
/messages/by-id/b1c831dd-d520-5e7f-0304-0eeed39c9996@2ndquadrant.com)

and

/messages/by-id/attachment/113487/fix_gcolumn_dump_v2.patch
(message
/messages/by-id/CA+fd4k6pLzrZDQsdsxcS06AwGRf1DgwOw84sFq9oXNw+83nB1g@mail.gmail.com)

Both of these result in the same change to the dump output. Both of
them have essentially the same idea. The first one adds the
conditionals during the information gathering phase of pg_dump, the
second one adds the conditionals during the output phase.

Any further thoughts?

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

#25Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Peter Eisentraut (#24)
Re: Dumping/restoring fails on inherited generated column

On Wed, Nov 4, 2020 at 4:23 AM Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

On 2020-08-27 13:30, Masahiko Sawada wrote:

This issue is not fixed yet. I've attached the updated version patch
and registered it to commit fest so as not to forget. Please review
it.

A few fixes have been committed in this thread, basically to prevent
situations that shouldn't have been allowed.

What's left is the originally reported issue that some parts of the
regression test database are dumped incorrectly. The two proposed
patches in their most recent versions are

/messages/by-id/attachment/107447/v3-0001-pg_dump-Fix-dumping-of-inherited-generated-column.patch
(message
/messages/by-id/b1c831dd-d520-5e7f-0304-0eeed39c9996@2ndquadrant.com)

and

/messages/by-id/attachment/113487/fix_gcolumn_dump_v2.patch
(message
/messages/by-id/CA+fd4k6pLzrZDQsdsxcS06AwGRf1DgwOw84sFq9oXNw+83nB1g@mail.gmail.com)

Both of these result in the same change to the dump output. Both of
them have essentially the same idea. The first one adds the
conditionals during the information gathering phase of pg_dump, the
second one adds the conditionals during the output phase.

Any further thoughts?

I think the first one is better than the second (mine) because it can
save the number of intermediate objects.

Regards,

--
Masahiko Sawada
EnterpriseDB: https://www.enterprisedb.com/

#26Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Masahiko Sawada (#25)
Re: Dumping/restoring fails on inherited generated column

On 2020-11-06 04:55, Masahiko Sawada wrote:

Both of these result in the same change to the dump output. Both of
them have essentially the same idea. The first one adds the
conditionals during the information gathering phase of pg_dump, the
second one adds the conditionals during the output phase.

Any further thoughts?

I think the first one is better than the second (mine) because it can
save the number of intermediate objects.

I was hoping to wrap this issue up this week, but I found more problems
with how these proposed changes interact with --binary-upgrade mode. I
think I need to formalize my findings into pg_dump test cases as a next
step. Then we can figure out what combination of tweaks will make them
all work.

--
Peter Eisentraut
2ndQuadrant, an EDB company
https://www.2ndquadrant.com/

#27Anastasia Lubennikova
a.lubennikova@postgrespro.ru
In reply to: Peter Eisentraut (#26)
Re: Dumping/restoring fails on inherited generated column

On 09.11.2020 13:43, Peter Eisentraut wrote:

On 2020-11-06 04:55, Masahiko Sawada wrote:

Both of these result in the same change to the dump output.  Both of
them have essentially the same idea.  The first one adds the
conditionals during the information gathering phase of pg_dump, the
second one adds the conditionals during the output phase.

Any further thoughts?

I think the first one is better than the second (mine) because it can
save the number of intermediate objects.

I was hoping to wrap this issue up this week, but I found more
problems with how these proposed changes interact with
--binary-upgrade mode.  I think I need to formalize my findings into
pg_dump test cases as a next step.  Then we can figure out what
combination of tweaks will make them all work.

I am moving this patch to the next CF, but it looks like the discussion
is a bit stuck.

Peter, can you please share your concerns about the interaction of the
patch with --binary-upgrade mode? If you don't have time to write tests,
you can just describe problems.

--
Anastasia Lubennikova
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

#28Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Anastasia Lubennikova (#27)
1 attachment(s)
Re: Dumping/restoring fails on inherited generated column

I've had another go at this, and I've found a solution that appears to
address all the issues I'm aware of. It's all very similar to the
previously discussed patches. The main difference is that previous
patches had attempted to use something like tbinfo->attislocal to
determine whether a column was inherited, but that's not correct. This
patch uses the existing logic in flagInhAttrs() to find whether there is
a matching parent column with a generation expression. I've added
pg_dump test cases here to check the different variations that the code
addresses.

--
Peter Eisentraut
2ndQuadrant, an EDB company
https://www.2ndquadrant.com/

Attachments:

v4-0001-pg_dump-Fix-dumping-of-inherited-generated-column.patchtext/plain; charset=UTF-8; name=v4-0001-pg_dump-Fix-dumping-of-inherited-generated-column.patch; x-mac-creator=0; x-mac-type=0Download
From 6cace6e6e7935844272fb201fc0096d8f2381d00 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Fri, 29 Jan 2021 13:20:41 +0100
Subject: [PATCH v4] pg_dump: Fix dumping of inherited generated columns

Generation expressions of generated columns are always inherited, so
there is no need to set them separately in child tables, and there is
no syntax to do so either.  The code previously used the code paths
for the handling of default values, for which different rules apply;
in particular it might want to set a default value explicitly for an
inherited column.  This resulted in unrestorable dumps.  For generated
columns, just skip them in inherited tables.

Discussion: https://www.postgresql.org/message-id/flat/15830.1575468847%40sss.pgh.pa.us
---
 src/bin/pg_dump/common.c         | 31 ++++++++++++++++-----
 src/bin/pg_dump/pg_dump.c        | 37 +++++++++++++++++++++----
 src/bin/pg_dump/t/002_pg_dump.pl | 46 ++++++++++++++++++++++++++++++++
 3 files changed, 102 insertions(+), 12 deletions(-)

diff --git a/src/bin/pg_dump/common.c b/src/bin/pg_dump/common.c
index b0f02bc1f6..1a261a5545 100644
--- a/src/bin/pg_dump/common.c
+++ b/src/bin/pg_dump/common.c
@@ -467,12 +467,22 @@ flagInhIndexes(Archive *fout, TableInfo tblinfo[], int numTables)
 /* flagInhAttrs -
  *	 for each dumpable table in tblinfo, flag its inherited attributes
  *
- * What we need to do here is detect child columns that inherit NOT NULL
- * bits from their parents (so that we needn't specify that again for the
- * child) and child columns that have DEFAULT NULL when their parents had
- * some non-null default.  In the latter case, we make up a dummy AttrDefInfo
- * object so that we'll correctly emit the necessary DEFAULT NULL clause;
- * otherwise the backend will apply an inherited default to the column.
+ * What we need to do here is:
+ *
+ * - Detect child columns that inherit NOT NULL bits from their parents, so
+ *   that we needn't specify that again for the child.
+ *
+ * - Detect child columns that have DEFAULT NULL when their parents had some
+ *   non-null default.  In this case, we make up a dummy AttrDefInfo object so
+ *   that we'll correctly emit the necessary DEFAULT NULL clause; otherwise
+ *   the backend will apply an inherited default to the column.
+ *
+ * - Detect child columns that have a generation expression when their parents
+ *   also have one.  Generation expressions are always inherited, so there is
+ *   no need to set them again in child tables, and there is no syntax for it
+ *   either.  (Exception: In binary upgrade mode we dump them because
+ *   inherited tables are recreated standalone first and then reattached to
+ *   the parent.)
  *
  * modifies tblinfo
  */
@@ -510,6 +520,7 @@ flagInhAttrs(DumpOptions *dopt, TableInfo *tblinfo, int numTables)
 		{
 			bool		foundNotNull;	/* Attr was NOT NULL in a parent */
 			bool		foundDefault;	/* Found a default in a parent */
+			bool		foundGenerated;	/* Found a generated in a parent */
 
 			/* no point in examining dropped columns */
 			if (tbinfo->attisdropped[j])
@@ -517,6 +528,7 @@ flagInhAttrs(DumpOptions *dopt, TableInfo *tblinfo, int numTables)
 
 			foundNotNull = false;
 			foundDefault = false;
+			foundGenerated = false;
 			for (k = 0; k < numParents; k++)
 			{
 				TableInfo  *parent = parents[k];
@@ -528,7 +540,8 @@ flagInhAttrs(DumpOptions *dopt, TableInfo *tblinfo, int numTables)
 				if (inhAttrInd >= 0)
 				{
 					foundNotNull |= parent->notnull[inhAttrInd];
-					foundDefault |= (parent->attrdefs[inhAttrInd] != NULL);
+					foundDefault |= (parent->attrdefs[inhAttrInd] != NULL && !parent->attgenerated[inhAttrInd]);
+					foundGenerated |= parent->attgenerated[inhAttrInd];
 				}
 			}
 
@@ -570,6 +583,10 @@ flagInhAttrs(DumpOptions *dopt, TableInfo *tblinfo, int numTables)
 
 				tbinfo->attrdefs[j] = attrDef;
 			}
+
+			/* Remove generation expression from child */
+			if (foundGenerated && !dopt->binary_upgrade)
+				tbinfo->attrdefs[j] = NULL;
 		}
 	}
 }
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 798d14580e..81a83b4a12 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -8839,13 +8839,37 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
 				attrdefs[j].dobj.dump = tbinfo->dobj.dump;
 
 				/*
-				 * Defaults on a VIEW must always be dumped as separate ALTER
-				 * TABLE commands.  Defaults on regular tables are dumped as
-				 * part of the CREATE TABLE if possible, which it won't be if
-				 * the column is not going to be emitted explicitly.
+				 * Figure out whether the default/generation expression should
+				 * be dumped as part of the main CREATE TABLE (or similar)
+				 * command or as a separate ALTER TABLE (or similar) command.
+				 * The preference is to put it into the CREATE command, but in
+				 * some cases that's not possible.
 				 */
-				if (tbinfo->relkind == RELKIND_VIEW)
+				if (tbinfo->attgenerated[adnum - 1])
 				{
+					/*
+					 * Column generation expressions cannot be dumped
+					 * separately, because there is no syntax for it.  The
+					 * !shouldPrintColumn case below will be tempted to set
+					 * them to separate if they are attached to an inherited
+					 * column without a local definition, but that would be
+					 * wrong and unnecessary, because generation expressions
+					 * are always inherited, so there is no need to set them
+					 * again in child tables, and there is no syntax for it
+					 * either.  By setting separate to false here we prevent
+					 * the "default" from being processed as its own dumpable
+					 * object, and flagInhAttrs() will remove it from the
+					 * table when it detects that it belongs to an inherited
+					 * column.
+					 */
+					attrdefs[j].separate = false;
+				}
+				else if (tbinfo->relkind == RELKIND_VIEW)
+				{
+					/*
+					 * Defaults on a VIEW must always be dumped as separate
+					 * ALTER TABLE commands.
+					 */
 					attrdefs[j].separate = true;
 				}
 				else if (!shouldPrintColumn(dopt, tbinfo, adnum - 1))
@@ -8856,7 +8880,10 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
 				else
 				{
 					attrdefs[j].separate = false;
+				}
 
+				if (!attrdefs[j].separate)
+				{
 					/*
 					 * Mark the default as needing to appear before the table,
 					 * so that any dependencies it has must be emitted before
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 798884da36..737e46464a 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2493,6 +2493,52 @@
 		unlike => { exclude_dump_test_schema => 1, },
 	},
 
+	'CREATE TABLE test_table_generated_child1 (without local columns)' => {
+		create_order => 4,
+		create_sql   => 'CREATE TABLE dump_test.test_table_generated_child1 ()
+						 INHERITS (dump_test.test_table_generated);',
+		regexp => qr/^
+			\QCREATE TABLE dump_test.test_table_generated_child1 (\E\n
+			\)\n
+			\QINHERITS (dump_test.test_table_generated);\E\n
+			/xms,
+		like =>
+		  { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+		unlike => {
+			binary_upgrade           => 1,
+			exclude_dump_test_schema => 1,
+		},
+	},
+
+	'ALTER TABLE test_table_generated_child1' => {
+		regexp =>
+		  qr/^\QALTER TABLE ONLY dump_test.test_table_generated_child1 ALTER COLUMN col2 \E/m,
+
+		# should not get emitted
+		like => {},
+	},
+
+	'CREATE TABLE test_table_generated_child2 (with local columns)' => {
+		create_order => 4,
+		create_sql   => 'CREATE TABLE dump_test.test_table_generated_child2 (
+						   col1 int,
+						   col2 int
+						 ) INHERITS (dump_test.test_table_generated);',
+		regexp => qr/^
+			\QCREATE TABLE dump_test.test_table_generated_child2 (\E\n
+			\s+\Qcol1 integer,\E\n
+			\s+\Qcol2 integer\E\n
+			\)\n
+			\QINHERITS (dump_test.test_table_generated);\E\n
+			/xms,
+		like =>
+		  { %full_runs, %dump_test_schema_runs, section_pre_data => 1, },
+		unlike => {
+			binary_upgrade           => 1,
+			exclude_dump_test_schema => 1,
+		},
+	},
+
 	'CREATE TABLE table_with_stats' => {
 		create_order => 98,
 		create_sql   => 'CREATE TABLE dump_test.table_index_stats (

base-commit: b034ef9b376dbe712caa076541d6a750f37d85ce
-- 
2.30.0

#29Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#28)
Re: Dumping/restoring fails on inherited generated column

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

I've had another go at this, and I've found a solution that appears to
address all the issues I'm aware of. It's all very similar to the
previously discussed patches. The main difference is that previous
patches had attempted to use something like tbinfo->attislocal to
determine whether a column was inherited, but that's not correct. This
patch uses the existing logic in flagInhAttrs() to find whether there is
a matching parent column with a generation expression. I've added
pg_dump test cases here to check the different variations that the code
addresses.

This is a clear improvement on the current situation, and given that
this issue is over a year old, I think we should push and back-patch
this in time for February's releases.

However ... this doesn't solve all the cases noted in this thread.
In the first example I gave at [1]/messages/by-id/660925.1601397436@sss.pgh.pa.us,

d3=# create table parent (f1 int default 2);
CREATE TABLE
d3=# create table child (f1 int default 3) inherits(parent);
NOTICE: merging column "f1" with inherited definition
CREATE TABLE
d3=# create table child2() inherits(parent);
CREATE TABLE
d3=# alter table child2 alter column f1 set default 42;
ALTER TABLE

pg_dump still fails to restore child2.f1's non-inherited default.
That's probably a pre-existing problem, since it doesn't involve
GENERATED at all, but we shouldn't forget about it.

Also, in the example from [2]/messages/by-id/661371.1601398006@sss.pgh.pa.us,

d3=# create table pp1 (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
CREATE TABLE
d3=# create table cc1 (a int, b int GENERATED ALWAYS AS (a * 3) STORED);
CREATE TABLE
d3=# alter table cc1 inherit pp1;
ALTER TABLE

pg_dump now omits to dump cc1's generation expression, which seems
strictly worse than before. Admittedly, the backend likely ought to
be rejecting this scenario, but it doesn't do so today.

Neither of these points seem like a reason to reject this patch,
they're just adjacent work that remains to be done.

regards, tom lane

[1]: /messages/by-id/660925.1601397436@sss.pgh.pa.us
[2]: /messages/by-id/661371.1601398006@sss.pgh.pa.us

#30Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Tom Lane (#29)
Re: Dumping/restoring fails on inherited generated column

On 2021-01-29 17:41, Tom Lane wrote:

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

I've had another go at this, and I've found a solution that appears to
address all the issues I'm aware of. It's all very similar to the
previously discussed patches. The main difference is that previous
patches had attempted to use something like tbinfo->attislocal to
determine whether a column was inherited, but that's not correct. This
patch uses the existing logic in flagInhAttrs() to find whether there is
a matching parent column with a generation expression. I've added
pg_dump test cases here to check the different variations that the code
addresses.

This is a clear improvement on the current situation, and given that
this issue is over a year old, I think we should push and back-patch
this in time for February's releases.

done

I will continue working on the other issues that we have been discussing.

However ... this doesn't solve all the cases noted in this thread.
In the first example I gave at [1],

d3=# create table parent (f1 int default 2);
CREATE TABLE
d3=# create table child (f1 int default 3) inherits(parent);
NOTICE: merging column "f1" with inherited definition
CREATE TABLE
d3=# create table child2() inherits(parent);
CREATE TABLE
d3=# alter table child2 alter column f1 set default 42;
ALTER TABLE

pg_dump still fails to restore child2.f1's non-inherited default.
That's probably a pre-existing problem, since it doesn't involve
GENERATED at all, but we shouldn't forget about it.

Also, in the example from [2],

d3=# create table pp1 (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
CREATE TABLE
d3=# create table cc1 (a int, b int GENERATED ALWAYS AS (a * 3) STORED);
CREATE TABLE
d3=# alter table cc1 inherit pp1;
ALTER TABLE

pg_dump now omits to dump cc1's generation expression, which seems
strictly worse than before. Admittedly, the backend likely ought to
be rejecting this scenario, but it doesn't do so today.

Neither of these points seem like a reason to reject this patch,
they're just adjacent work that remains to be done.

regards, tom lane

[1] /messages/by-id/660925.1601397436@sss.pgh.pa.us
[2] /messages/by-id/661371.1601398006@sss.pgh.pa.us

--
Peter Eisentraut
2ndQuadrant, an EDB company
https://www.2ndquadrant.com/

#31Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Tom Lane (#29)
Re: Dumping/restoring fails on inherited generated column

On 2021-01-29 17:41, Tom Lane wrote:

However ... this doesn't solve all the cases noted in this thread.
In the first example I gave at [1],

d3=# create table parent (f1 int default 2);
CREATE TABLE
d3=# create table child (f1 int default 3) inherits(parent);
NOTICE: merging column "f1" with inherited definition
CREATE TABLE
d3=# create table child2() inherits(parent);
CREATE TABLE
d3=# alter table child2 alter column f1 set default 42;
ALTER TABLE

pg_dump still fails to restore child2.f1's non-inherited default.
That's probably a pre-existing problem, since it doesn't involve
GENERATED at all, but we shouldn't forget about it.

[1] /messages/by-id/660925.1601397436@sss.pgh.pa.us

I can't tell what the problem is in this example. I tried with PG11,
12, and master, and the schema dump comes out with those same four
commands and they restore correctly AFAICT.

--
Peter Eisentraut
2ndQuadrant, an EDB company
https://www.2ndquadrant.com/

#32Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#31)
Re: Dumping/restoring fails on inherited generated column

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

On 2021-01-29 17:41, Tom Lane wrote:

However ... this doesn't solve all the cases noted in this thread.
In the first example I gave at [1],
d3=# create table parent (f1 int default 2);
CREATE TABLE
d3=# create table child (f1 int default 3) inherits(parent);
NOTICE: merging column "f1" with inherited definition
CREATE TABLE
d3=# create table child2() inherits(parent);
CREATE TABLE
d3=# alter table child2 alter column f1 set default 42;
ALTER TABLE

pg_dump still fails to restore child2.f1's non-inherited default.

I can't tell what the problem is in this example. I tried with PG11,
12, and master, and the schema dump comes out with those same four
commands and they restore correctly AFAICT.

Oh! Trying it now, I see that the child2 default does get restored
as a "separate default" object:

ALTER TABLE ONLY public.child2 ALTER COLUMN f1 SET DEFAULT 42;

This is a bit weird, because you'd think it would be handled
the same as the other child's default, but it isn't; that
one comes out as

CREATE TABLE public.child (
f1 integer DEFAULT 3
)
INHERITS (public.parent);

while child2 looks like

CREATE TABLE public.child2 (
)
INHERITS (public.parent);

I now suspect that I'd seen this dump of "child2" and missed the later
ALTER. So no bug here, just pilot error. Sorry for the noise.

regards, tom lane

#33Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Tom Lane (#29)
1 attachment(s)
Re: Dumping/restoring fails on inherited generated column

On 2021-01-29 17:41, Tom Lane wrote:

Also, in the example from [2],

d3=# create table pp1 (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
CREATE TABLE
d3=# create table cc1 (a int, b int GENERATED ALWAYS AS (a * 3) STORED);
CREATE TABLE
d3=# alter table cc1 inherit pp1;
ALTER TABLE

pg_dump now omits to dump cc1's generation expression, which seems
strictly worse than before. Admittedly, the backend likely ought to
be rejecting this scenario, but it doesn't do so today.

[2]/messages/by-id/661371.1601398006@sss.pgh.pa.us

Here is a WIP patch to address this. Probably needs another look for
column number mapping and all the usual stuff, but the basic idea should
be okay.

--
Peter Eisentraut
2ndQuadrant, an EDB company
https://www.2ndquadrant.com/

Attachments:

0001-WIP-Fix-ALTER-TABLE-INHERIT-with-generated-columns.patchtext/plain; charset=UTF-8; name=0001-WIP-Fix-ALTER-TABLE-INHERIT-with-generated-columns.patch; x-mac-creator=0; x-mac-type=0Download
From bd6ee462e5ad4a6514940e7e78d40c4f28f3dc3f Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Wed, 3 Feb 2021 20:14:05 +0100
Subject: [PATCH] WIP: Fix ALTER TABLE / INHERIT with generated columns

When running ALTER TABLE t2 INHERIT t1, we must check that columns in
t2 that correspond to a generated column in t1 are also generated and
have the same generation expression.  Otherwise, this would allow
creating setups that a normal CREATE TABLE sequence would not allow.
---
 src/backend/commands/tablecmds.c        | 49 +++++++++++++++++++++++++
 src/test/regress/expected/generated.out | 19 ++++++++++
 src/test/regress/sql/generated.sql      | 13 +++++++
 3 files changed, 81 insertions(+)

diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 420991e315..3d4fb4ce7e 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -13963,6 +13963,55 @@ MergeAttributesIntoExisting(Relation child_rel, Relation parent_rel)
 						 errmsg("column \"%s\" in child table must be marked NOT NULL",
 								attributeName)));
 
+			/*
+			 * If parent column generated, child column must be, too.
+			 */
+			if (attribute->attgenerated && !childatt->attgenerated)
+				ereport(ERROR,
+						(errcode(ERRCODE_DATATYPE_MISMATCH),
+						 errmsg("column \"%s\" in child table must be a generated column",
+								attributeName)));
+
+			/*
+			 * Check that both generation expressions match.
+			 */
+			if (attribute->attgenerated && childatt->attgenerated)
+			{
+				TupleConstr *child_constr = child_rel->rd_att->constr;
+				TupleConstr *parent_constr = parent_rel->rd_att->constr;
+				Node	   *child_expr = NULL;
+				Node	   *parent_expr = NULL;
+
+				Assert(child_constr != NULL);
+				Assert(parent_constr != NULL);
+
+				for (int i = 0; i < child_constr->num_defval; i++)
+				{
+					if (child_constr->defval[i].adnum == childatt->attnum)
+					{
+						child_expr = stringToNode(child_constr->defval[i].adbin);
+						break;
+					}
+				}
+				Assert(child_expr != NULL);
+
+				for (int i = 0; i < parent_constr->num_defval; i++)
+				{
+					if (parent_constr->defval[i].adnum == attribute->attnum)
+					{
+						parent_expr = stringToNode(parent_constr->defval[i].adbin);
+						break;
+					}
+				}
+				Assert(parent_expr != NULL);
+
+				if (!equal(child_expr, parent_expr))
+					ereport(ERROR,
+						(errcode(ERRCODE_DATATYPE_MISMATCH),
+						 errmsg("column \"%s\" in child table has a conflicting generation expression",
+								attributeName)));
+			}
+
 			/*
 			 * OK, bump the child column's inheritance count.  (If we fail
 			 * later on, this change will just roll back.)
diff --git a/src/test/regress/expected/generated.out b/src/test/regress/expected/generated.out
index ca721d38bf..f2ed3864a1 100644
--- a/src/test/regress/expected/generated.out
+++ b/src/test/regress/expected/generated.out
@@ -281,6 +281,17 @@ SELECT * FROM gtest_normal;
  2 | 4
 (2 rows)
 
+CREATE TABLE gtest_normal_child2 (a int, b int GENERATED ALWAYS AS (a * 3) STORED);
+ALTER TABLE gtest_normal_child2 INHERIT gtest_normal;
+INSERT INTO gtest_normal_child2 (a) VALUES (3);
+SELECT * FROM gtest_normal;
+ a | b 
+---+---
+ 1 |  
+ 2 | 4
+ 3 | 9
+(3 rows)
+
 -- test inheritance mismatches between parent and child
 CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS (a * 22) STORED) INHERITS (gtest1);  -- error
 NOTICE:  merging column "b" with inherited definition
@@ -292,6 +303,14 @@ ERROR:  column "b" inherits from generated column but specifies default
 CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS IDENTITY) INHERITS (gtest1);  -- error
 NOTICE:  merging column "b" with inherited definition
 ERROR:  column "b" inherits from generated column but specifies identity
+CREATE TABLE gtestxx_1 (a int NOT NULL, b int);
+ALTER TABLE gtestxx_1 INHERIT gtest1;  -- error
+ERROR:  column "b" in child table must be a generated column
+CREATE TABLE gtestxx_2 (a int NOT NULL, b int GENERATED ALWAYS AS (a * 22) STORED);
+ALTER TABLE gtestxx_2 INHERIT gtest1;  -- error
+ERROR:  column "b" in child table has a conflicting generation expression
+CREATE TABLE gtestxx_3 (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) STORED);
+ALTER TABLE gtestxx_3 INHERIT gtest1;  -- ok
 -- test multiple inheritance mismatches
 CREATE TABLE gtesty (x int, b int);
 CREATE TABLE gtest1_2 () INHERITS (gtest1, gtesty);  -- error
diff --git a/src/test/regress/sql/generated.sql b/src/test/regress/sql/generated.sql
index bd2b0bfaaa..e30dfe649d 100644
--- a/src/test/regress/sql/generated.sql
+++ b/src/test/regress/sql/generated.sql
@@ -113,11 +113,24 @@ CREATE TABLE gtest_normal_child (a int, b int GENERATED ALWAYS AS (a * 2) STORED
 INSERT INTO gtest_normal_child (a) VALUES (2);
 SELECT * FROM gtest_normal;
 
+CREATE TABLE gtest_normal_child2 (a int, b int GENERATED ALWAYS AS (a * 3) STORED);
+ALTER TABLE gtest_normal_child2 INHERIT gtest_normal;
+INSERT INTO gtest_normal_child2 (a) VALUES (3);
+SELECT * FROM gtest_normal;
+
 -- test inheritance mismatches between parent and child
 CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS (a * 22) STORED) INHERITS (gtest1);  -- error
 CREATE TABLE gtestx (x int, b int DEFAULT 10) INHERITS (gtest1);  -- error
 CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS IDENTITY) INHERITS (gtest1);  -- error
 
+CREATE TABLE gtestxx_1 (a int NOT NULL, b int);
+ALTER TABLE gtestxx_1 INHERIT gtest1;  -- error
+CREATE TABLE gtestxx_2 (a int NOT NULL, b int GENERATED ALWAYS AS (a * 22) STORED);
+ALTER TABLE gtestxx_2 INHERIT gtest1;  -- error
+CREATE TABLE gtestxx_3 (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) STORED);
+ALTER TABLE gtestxx_3 INHERIT gtest1;  -- ok
+
+
 -- test multiple inheritance mismatches
 CREATE TABLE gtesty (x int, b int);
 CREATE TABLE gtest1_2 () INHERITS (gtest1, gtesty);  -- error
-- 
2.30.0

#34Zhihong Yu
zyu@yugabyte.com
In reply to: Peter Eisentraut (#33)
Re: Dumping/restoring fails on inherited generated column
Hi,
+           if (attribute->attgenerated && !childatt->attgenerated)
+               ereport(ERROR,
...
+           if (attribute->attgenerated && childatt->attgenerated)
+           {

Looks like for the second if statement,
checking attribute->attgenerated should be enough (due to the check from
the first if statement).

Cheers

On Wed, Feb 3, 2021 at 11:18 AM Peter Eisentraut <
peter.eisentraut@2ndquadrant.com> wrote:

Show quoted text

On 2021-01-29 17:41, Tom Lane wrote:

Also, in the example from [2],

d3=# create table pp1 (a int, b int GENERATED ALWAYS AS (a * 2) STORED);
CREATE TABLE
d3=# create table cc1 (a int, b int GENERATED ALWAYS AS (a * 3) STORED);
CREATE TABLE
d3=# alter table cc1 inherit pp1;
ALTER TABLE

pg_dump now omits to dump cc1's generation expression, which seems
strictly worse than before. Admittedly, the backend likely ought to
be rejecting this scenario, but it doesn't do so today.

[2]

/messages/by-id/661371.1601398006@sss.pgh.pa.us

Here is a WIP patch to address this. Probably needs another look for
column number mapping and all the usual stuff, but the basic idea should
be okay.

--
Peter Eisentraut
2ndQuadrant, an EDB company
https://www.2ndquadrant.com/

#35Peter Eisentraut
peter.eisentraut@2ndquadrant.com
In reply to: Zhihong Yu (#34)
1 attachment(s)
Re: Dumping/restoring fails on inherited generated column

On 2021-02-04 01:17, Zhihong Yu wrote:

Hi,
+           if (attribute->attgenerated && !childatt->attgenerated)
+               ereport(ERROR,
...
+           if (attribute->attgenerated && childatt->attgenerated)
+           {

Looks like for the second if statement,
checking attribute->attgenerated should be enough (due to the check from
the first if statement).

Thanks for taking a look. I figured the way I wrote it makes it easier
to move the code around or insert other code in the future and doesn't
make it so tightly coupled.

Anyway, I figured out how to take account of generation expressions with
different column orders. I used the same approach that we use for check
constraints. The attached patch is good to go from my perspective.

--
Peter Eisentraut
2ndQuadrant, an EDB company
https://www.2ndquadrant.com/

Attachments:

v2-0001-Fix-ALTER-TABLE-INHERIT-with-generated-columns.patchtext/plain; charset=UTF-8; name=v2-0001-Fix-ALTER-TABLE-INHERIT-with-generated-columns.patch; x-mac-creator=0; x-mac-type=0Download
From 3079ddc7e1fdf663bc0e59789c9ceac4eea8fd03 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Fri, 5 Feb 2021 15:14:38 +0100
Subject: [PATCH v2] Fix ALTER TABLE / INHERIT with generated columns

When running ALTER TABLE t2 INHERIT t1, we must check that columns in
t2 that correspond to a generated column in t1 are also generated and
have the same generation expression.  Otherwise, this would allow
creating setups that a normal CREATE TABLE sequence would not allow.

Discussion: https://www.postgresql.org/message-id/22de27f6-7096-8d96-4619-7b882932ca25@2ndquadrant.com
---
 src/backend/commands/tablecmds.c        | 60 +++++++++++++++++++++++++
 src/test/regress/expected/generated.out | 21 +++++++++
 src/test/regress/sql/generated.sql      | 15 +++++++
 3 files changed, 96 insertions(+)

diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 420991e315..bff3c65582 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -13963,6 +13963,66 @@ MergeAttributesIntoExisting(Relation child_rel, Relation parent_rel)
 						 errmsg("column \"%s\" in child table must be marked NOT NULL",
 								attributeName)));
 
+			/*
+			 * If parent column generated, child column must be, too.
+			 */
+			if (attribute->attgenerated && !childatt->attgenerated)
+				ereport(ERROR,
+						(errcode(ERRCODE_DATATYPE_MISMATCH),
+						 errmsg("column \"%s\" in child table must be a generated column",
+								attributeName)));
+
+			/*
+			 * Check that both generation expressions match.
+			 *
+			 * The test we apply is to see whether they reverse-compile to the
+			 * same source string.  This insulates us from issues like whether
+			 * attributes have the same physical column numbers in parent and
+			 * child relations.  (See also constraints_equivalent().)
+			 */
+			if (attribute->attgenerated && childatt->attgenerated)
+			{
+				TupleConstr *child_constr = child_rel->rd_att->constr;
+				TupleConstr *parent_constr = parent_rel->rd_att->constr;
+				char	   *child_expr = NULL;
+				char	   *parent_expr = NULL;
+
+				Assert(child_constr != NULL);
+				Assert(parent_constr != NULL);
+
+				for (int i = 0; i < child_constr->num_defval; i++)
+				{
+					if (child_constr->defval[i].adnum == childatt->attnum)
+					{
+						child_expr =
+							TextDatumGetCString(DirectFunctionCall2(pg_get_expr,
+																	CStringGetTextDatum(child_constr->defval[i].adbin),
+																	ObjectIdGetDatum(child_rel->rd_id)));
+						break;
+					}
+				}
+				Assert(child_expr != NULL);
+
+				for (int i = 0; i < parent_constr->num_defval; i++)
+				{
+					if (parent_constr->defval[i].adnum == attribute->attnum)
+					{
+						parent_expr =
+							TextDatumGetCString(DirectFunctionCall2(pg_get_expr,
+																	CStringGetTextDatum(parent_constr->defval[i].adbin),
+																	ObjectIdGetDatum(parent_rel->rd_id)));
+						break;
+					}
+				}
+				Assert(parent_expr != NULL);
+
+				if (strcmp(child_expr, parent_expr) != 0)
+					ereport(ERROR,
+						(errcode(ERRCODE_DATATYPE_MISMATCH),
+						 errmsg("column \"%s\" in child table has a conflicting generation expression",
+								attributeName)));
+			}
+
 			/*
 			 * OK, bump the child column's inheritance count.  (If we fail
 			 * later on, this change will just roll back.)
diff --git a/src/test/regress/expected/generated.out b/src/test/regress/expected/generated.out
index ca721d38bf..675773f0c1 100644
--- a/src/test/regress/expected/generated.out
+++ b/src/test/regress/expected/generated.out
@@ -281,6 +281,17 @@ SELECT * FROM gtest_normal;
  2 | 4
 (2 rows)
 
+CREATE TABLE gtest_normal_child2 (a int, b int GENERATED ALWAYS AS (a * 3) STORED);
+ALTER TABLE gtest_normal_child2 INHERIT gtest_normal;
+INSERT INTO gtest_normal_child2 (a) VALUES (3);
+SELECT * FROM gtest_normal;
+ a | b 
+---+---
+ 1 |  
+ 2 | 4
+ 3 | 9
+(3 rows)
+
 -- test inheritance mismatches between parent and child
 CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS (a * 22) STORED) INHERITS (gtest1);  -- error
 NOTICE:  merging column "b" with inherited definition
@@ -292,6 +303,16 @@ ERROR:  column "b" inherits from generated column but specifies default
 CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS IDENTITY) INHERITS (gtest1);  -- error
 NOTICE:  merging column "b" with inherited definition
 ERROR:  column "b" inherits from generated column but specifies identity
+CREATE TABLE gtestxx_1 (a int NOT NULL, b int);
+ALTER TABLE gtestxx_1 INHERIT gtest1;  -- error
+ERROR:  column "b" in child table must be a generated column
+CREATE TABLE gtestxx_2 (a int NOT NULL, b int GENERATED ALWAYS AS (a * 22) STORED);
+ALTER TABLE gtestxx_2 INHERIT gtest1;  -- error
+ERROR:  column "b" in child table has a conflicting generation expression
+CREATE TABLE gtestxx_3 (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) STORED);
+ALTER TABLE gtestxx_3 INHERIT gtest1;  -- ok
+CREATE TABLE gtestxx_4 (b int GENERATED ALWAYS AS (a * 2) STORED, a int NOT NULL);
+ALTER TABLE gtestxx_4 INHERIT gtest1;  -- ok
 -- test multiple inheritance mismatches
 CREATE TABLE gtesty (x int, b int);
 CREATE TABLE gtest1_2 () INHERITS (gtest1, gtesty);  -- error
diff --git a/src/test/regress/sql/generated.sql b/src/test/regress/sql/generated.sql
index bd2b0bfaaa..19e6e8c39b 100644
--- a/src/test/regress/sql/generated.sql
+++ b/src/test/regress/sql/generated.sql
@@ -113,11 +113,26 @@ CREATE TABLE gtest_normal_child (a int, b int GENERATED ALWAYS AS (a * 2) STORED
 INSERT INTO gtest_normal_child (a) VALUES (2);
 SELECT * FROM gtest_normal;
 
+CREATE TABLE gtest_normal_child2 (a int, b int GENERATED ALWAYS AS (a * 3) STORED);
+ALTER TABLE gtest_normal_child2 INHERIT gtest_normal;
+INSERT INTO gtest_normal_child2 (a) VALUES (3);
+SELECT * FROM gtest_normal;
+
 -- test inheritance mismatches between parent and child
 CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS (a * 22) STORED) INHERITS (gtest1);  -- error
 CREATE TABLE gtestx (x int, b int DEFAULT 10) INHERITS (gtest1);  -- error
 CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS IDENTITY) INHERITS (gtest1);  -- error
 
+CREATE TABLE gtestxx_1 (a int NOT NULL, b int);
+ALTER TABLE gtestxx_1 INHERIT gtest1;  -- error
+CREATE TABLE gtestxx_2 (a int NOT NULL, b int GENERATED ALWAYS AS (a * 22) STORED);
+ALTER TABLE gtestxx_2 INHERIT gtest1;  -- error
+CREATE TABLE gtestxx_3 (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) STORED);
+ALTER TABLE gtestxx_3 INHERIT gtest1;  -- ok
+CREATE TABLE gtestxx_4 (b int GENERATED ALWAYS AS (a * 2) STORED, a int NOT NULL);
+ALTER TABLE gtestxx_4 INHERIT gtest1;  -- ok
+
+
 -- test multiple inheritance mismatches
 CREATE TABLE gtesty (x int, b int);
 CREATE TABLE gtest1_2 () INHERITS (gtest1, gtesty);  -- error

base-commit: c444472af5c202067a9ecb0ff8df7370fb1ea8f4
-- 
2.30.0

#36Peter Eisentraut
peter.eisentraut@enterprisedb.com
In reply to: Peter Eisentraut (#35)
1 attachment(s)
Re: Dumping/restoring fails on inherited generated column

On 05.02.21 15:18, Peter Eisentraut wrote:

Anyway, I figured out how to take account of generation expressions with
different column orders.  I used the same approach that we use for check
constraints.  The attached patch is good to go from my perspective.

Dusting this off ... this patch should go into the next minor releases.
The attached patch is for master but backpatches without manual
intervention to PG13 and PG12.

Attachments:

v3-0001-Fix-ALTER-TABLE-INHERIT-with-generated-columns.patchtext/plain; charset=UTF-8; name=v3-0001-Fix-ALTER-TABLE-INHERIT-with-generated-columns.patch; x-mac-creator=0; x-mac-type=0Download
From 5a24ea0dc89c82a84efe59ef82557b8cd017def6 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Mon, 26 Apr 2021 13:54:34 +0200
Subject: [PATCH v3] Fix ALTER TABLE / INHERIT with generated columns

When running ALTER TABLE t2 INHERIT t1, we must check that columns in
t2 that correspond to a generated column in t1 are also generated and
have the same generation expression.  Otherwise, this would allow
creating setups that a normal CREATE TABLE sequence would not allow.

Discussion: https://www.postgresql.org/message-id/22de27f6-7096-8d96-4619-7b882932ca25@2ndquadrant.com
---
 src/backend/commands/tablecmds.c        | 60 +++++++++++++++++++++++++
 src/test/regress/expected/generated.out | 21 +++++++++
 src/test/regress/sql/generated.sql      | 14 ++++++
 3 files changed, 95 insertions(+)

diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 7d00f4eb25..1b7a4282bf 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -14398,6 +14398,66 @@ MergeAttributesIntoExisting(Relation child_rel, Relation parent_rel)
 						 errmsg("column \"%s\" in child table must be marked NOT NULL",
 								attributeName)));
 
+			/*
+			 * If parent column is generated, child column must be, too.
+			 */
+			if (attribute->attgenerated && !childatt->attgenerated)
+				ereport(ERROR,
+						(errcode(ERRCODE_DATATYPE_MISMATCH),
+						 errmsg("column \"%s\" in child table must be a generated column",
+								attributeName)));
+
+			/*
+			 * Check that both generation expressions match.
+			 *
+			 * The test we apply is to see whether they reverse-compile to the
+			 * same source string.  This insulates us from issues like whether
+			 * attributes have the same physical column numbers in parent and
+			 * child relations.  (See also constraints_equivalent().)
+			 */
+			if (attribute->attgenerated && childatt->attgenerated)
+			{
+				TupleConstr *child_constr = child_rel->rd_att->constr;
+				TupleConstr *parent_constr = parent_rel->rd_att->constr;
+				char	   *child_expr = NULL;
+				char	   *parent_expr = NULL;
+
+				Assert(child_constr != NULL);
+				Assert(parent_constr != NULL);
+
+				for (int i = 0; i < child_constr->num_defval; i++)
+				{
+					if (child_constr->defval[i].adnum == childatt->attnum)
+					{
+						child_expr =
+							TextDatumGetCString(DirectFunctionCall2(pg_get_expr,
+																	CStringGetTextDatum(child_constr->defval[i].adbin),
+																	ObjectIdGetDatum(child_rel->rd_id)));
+						break;
+					}
+				}
+				Assert(child_expr != NULL);
+
+				for (int i = 0; i < parent_constr->num_defval; i++)
+				{
+					if (parent_constr->defval[i].adnum == attribute->attnum)
+					{
+						parent_expr =
+							TextDatumGetCString(DirectFunctionCall2(pg_get_expr,
+																	CStringGetTextDatum(parent_constr->defval[i].adbin),
+																	ObjectIdGetDatum(parent_rel->rd_id)));
+						break;
+					}
+				}
+				Assert(parent_expr != NULL);
+
+				if (strcmp(child_expr, parent_expr) != 0)
+					ereport(ERROR,
+						(errcode(ERRCODE_DATATYPE_MISMATCH),
+						 errmsg("column \"%s\" in child table has a conflicting generation expression",
+								attributeName)));
+			}
+
 			/*
 			 * OK, bump the child column's inheritance count.  (If we fail
 			 * later on, this change will just roll back.)
diff --git a/src/test/regress/expected/generated.out b/src/test/regress/expected/generated.out
index ca721d38bf..675773f0c1 100644
--- a/src/test/regress/expected/generated.out
+++ b/src/test/regress/expected/generated.out
@@ -281,6 +281,17 @@ SELECT * FROM gtest_normal;
  2 | 4
 (2 rows)
 
+CREATE TABLE gtest_normal_child2 (a int, b int GENERATED ALWAYS AS (a * 3) STORED);
+ALTER TABLE gtest_normal_child2 INHERIT gtest_normal;
+INSERT INTO gtest_normal_child2 (a) VALUES (3);
+SELECT * FROM gtest_normal;
+ a | b 
+---+---
+ 1 |  
+ 2 | 4
+ 3 | 9
+(3 rows)
+
 -- test inheritance mismatches between parent and child
 CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS (a * 22) STORED) INHERITS (gtest1);  -- error
 NOTICE:  merging column "b" with inherited definition
@@ -292,6 +303,16 @@ ERROR:  column "b" inherits from generated column but specifies default
 CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS IDENTITY) INHERITS (gtest1);  -- error
 NOTICE:  merging column "b" with inherited definition
 ERROR:  column "b" inherits from generated column but specifies identity
+CREATE TABLE gtestxx_1 (a int NOT NULL, b int);
+ALTER TABLE gtestxx_1 INHERIT gtest1;  -- error
+ERROR:  column "b" in child table must be a generated column
+CREATE TABLE gtestxx_2 (a int NOT NULL, b int GENERATED ALWAYS AS (a * 22) STORED);
+ALTER TABLE gtestxx_2 INHERIT gtest1;  -- error
+ERROR:  column "b" in child table has a conflicting generation expression
+CREATE TABLE gtestxx_3 (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) STORED);
+ALTER TABLE gtestxx_3 INHERIT gtest1;  -- ok
+CREATE TABLE gtestxx_4 (b int GENERATED ALWAYS AS (a * 2) STORED, a int NOT NULL);
+ALTER TABLE gtestxx_4 INHERIT gtest1;  -- ok
 -- test multiple inheritance mismatches
 CREATE TABLE gtesty (x int, b int);
 CREATE TABLE gtest1_2 () INHERITS (gtest1, gtesty);  -- error
diff --git a/src/test/regress/sql/generated.sql b/src/test/regress/sql/generated.sql
index bd2b0bfaaa..63251c443a 100644
--- a/src/test/regress/sql/generated.sql
+++ b/src/test/regress/sql/generated.sql
@@ -113,11 +113,25 @@ CREATE TABLE gtest_normal_child (a int, b int GENERATED ALWAYS AS (a * 2) STORED
 INSERT INTO gtest_normal_child (a) VALUES (2);
 SELECT * FROM gtest_normal;
 
+CREATE TABLE gtest_normal_child2 (a int, b int GENERATED ALWAYS AS (a * 3) STORED);
+ALTER TABLE gtest_normal_child2 INHERIT gtest_normal;
+INSERT INTO gtest_normal_child2 (a) VALUES (3);
+SELECT * FROM gtest_normal;
+
 -- test inheritance mismatches between parent and child
 CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS (a * 22) STORED) INHERITS (gtest1);  -- error
 CREATE TABLE gtestx (x int, b int DEFAULT 10) INHERITS (gtest1);  -- error
 CREATE TABLE gtestx (x int, b int GENERATED ALWAYS AS IDENTITY) INHERITS (gtest1);  -- error
 
+CREATE TABLE gtestxx_1 (a int NOT NULL, b int);
+ALTER TABLE gtestxx_1 INHERIT gtest1;  -- error
+CREATE TABLE gtestxx_2 (a int NOT NULL, b int GENERATED ALWAYS AS (a * 22) STORED);
+ALTER TABLE gtestxx_2 INHERIT gtest1;  -- error
+CREATE TABLE gtestxx_3 (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) STORED);
+ALTER TABLE gtestxx_3 INHERIT gtest1;  -- ok
+CREATE TABLE gtestxx_4 (b int GENERATED ALWAYS AS (a * 2) STORED, a int NOT NULL);
+ALTER TABLE gtestxx_4 INHERIT gtest1;  -- ok
+
 -- test multiple inheritance mismatches
 CREATE TABLE gtesty (x int, b int);
 CREATE TABLE gtest1_2 () INHERITS (gtest1, gtesty);  -- error

base-commit: 6d2e87a077b3c2394e4adb8eb226b3dcfe3f3346
-- 
2.31.1

#37Peter Eisentraut
peter.eisentraut@enterprisedb.com
In reply to: Peter Eisentraut (#36)
Re: Dumping/restoring fails on inherited generated column

On 26.04.21 14:10, Peter Eisentraut wrote:

On 05.02.21 15:18, Peter Eisentraut wrote:

Anyway, I figured out how to take account of generation expressions
with different column orders.  I used the same approach that we use
for check constraints.  The attached patch is good to go from my
perspective.

Dusting this off ... this patch should go into the next minor releases.
The attached patch is for master but backpatches without manual
intervention to PG13 and PG12.

committed