COPY vs. transition tables

Started by David Fetterover 8 years ago6 messages
#1David Fetter
david@fetter.org
1 attachment(s)

Folks,

Using the script attached, I'm getting this very odd result set below.

Audit records from COPY to the "foo bar" table aren't getting
recorded, but audit records from COPY to the baz table are.

Best,
David.

\i copy_oddity.sql
CREATE TABLE
COMMENT
CREATE FUNCTION
CREATE FUNCTION
CREATE EVENT TRIGGER
COMMENT
psql:logging_infra.sql:165: NOTICE: Adding log table(s) for public.foo bar
CREATE TABLE
INSERT 0 1
UPDATE 1
DELETE 1
COPY 5
timestamp | user | action | table_schema | table_name | old_row | new_row
-------------------------------+---------+-------------------------------+--------------+------------+-----------------------+-----------------------
2017-07-08 01:36:52.368228-07 | shackle | 2017-07-08 01:36:52.368228-07 | public | foo bar | | {"t": null, "id": 1}
2017-07-08 01:36:52.368228-07 | shackle | 2017-07-08 01:36:52.368228-07 | public | foo bar | {"t": null, "id": 1} | {"t": "baz", "id": 1}
2017-07-08 01:36:52.368228-07 | shackle | 2017-07-08 01:36:52.368228-07 | public | foo bar | {"t": "baz", "id": 1} |
(3 rows)

psql:logging_infra.sql:180: NOTICE: Adding log table(s) for public.baz
psql:logging_infra.sql:180: NOTICE: relation "public_log" already exists, skipping
CREATE TABLE
COPY 5
timestamp | user | action | table_schema | table_name | old_row | new_row
-------------------------------+---------+-------------------------------+--------------+------------+---------+------------
2017-07-08 01:36:52.368228-07 | shackle | 2017-07-08 01:36:52.368228-07 | public | baz | | {"t": "a"}
2017-07-08 01:36:52.368228-07 | shackle | 2017-07-08 01:36:52.368228-07 | public | baz | | {"t": "b"}
2017-07-08 01:36:52.368228-07 | shackle | 2017-07-08 01:36:52.368228-07 | public | baz | | {"t": "c"}
2017-07-08 01:36:52.368228-07 | shackle | 2017-07-08 01:36:52.368228-07 | public | baz | | {"t": "d"}
2017-07-08 01:36:52.368228-07 | shackle | 2017-07-08 01:36:52.368228-07 | public | baz | | {"t": "e"}
(5 rows)
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Attachments:

logging_infra.sqlapplication/sqlDownload
#2Thomas Munro
thomas.munro@enterprisedb.com
In reply to: David Fetter (#1)
1 attachment(s)
Re: COPY vs. transition tables

On Sat, Jul 8, 2017 at 8:42 PM, David Fetter <david@fetter.org> wrote:

Using the script attached, I'm getting this very odd result set below.

Audit records from COPY to the "foo bar" table aren't getting
recorded, but audit records from COPY to the baz table are.

Thanks for the bug report. I think it's the presence of the index on
"foo bar", not the space in its name (but thanks for that curve
ball!), that causes these tuples not to be captured.
CopyFromInsertBatch takes a different path depending on whether there
are any indexes, and mistakenly passes NULL for transition_capture.
The attached seems to fix it, but I'll look more closely and send a
version with a regression test on Monday.

--
Thomas Munro
http://www.enterprisedb.com

Attachments:

copy-with-indexes-and-transitions.patchapplication/octet-stream; name=copy-with-indexes-and-transitions.patchDownload
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index f391828e74f..fc5f4f66ead 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -2915,7 +2915,7 @@ CopyFromInsertBatch(CopyState cstate, EState *estate, CommandId mycid,
 									  estate, false, NULL, NIL);
 			ExecARInsertTriggers(estate, resultRelInfo,
 								 bufferedTuples[i],
-								 recheckIndexes, NULL);
+								 recheckIndexes, cstate->transition_capture);
 			list_free(recheckIndexes);
 		}
 	}
#3David Fetter
david@fetter.org
In reply to: Thomas Munro (#2)
Re: COPY vs. transition tables

On Sun, Jul 09, 2017 at 11:46:03AM +1200, Thomas Munro wrote:

On Sat, Jul 8, 2017 at 8:42 PM, David Fetter <david@fetter.org> wrote:

Using the script attached, I'm getting this very odd result set below.

Audit records from COPY to the "foo bar" table aren't getting
recorded, but audit records from COPY to the baz table are.

Thanks for the bug report. I think it's the presence of the index on
"foo bar", not the space in its name (but thanks for that curve
ball!), that causes these tuples not to be captured.

Thanks for getting this fixed, and apologies for not trimming this
down to a minimal repro.

Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#4Thomas Munro
thomas.munro@enterprisedb.com
In reply to: Thomas Munro (#2)
1 attachment(s)
Re: COPY vs. transition tables

On Sun, Jul 9, 2017 at 11:46 AM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:

On Sat, Jul 8, 2017 at 8:42 PM, David Fetter <david@fetter.org> wrote:

Using the script attached, I'm getting this very odd result set below.

Audit records from COPY to the "foo bar" table aren't getting
recorded, but audit records from COPY to the baz table are.

Thanks for the bug report. I think it's the presence of the index on
"foo bar", not the space in its name (but thanks for that curve
ball!), that causes these tuples not to be captured.
CopyFromInsertBatch takes a different path depending on whether there
are any indexes, and mistakenly passes NULL for transition_capture.
The attached seems to fix it, but I'll look more closely and send a
version with a regression test on Monday.

Here it is. Added to open items.

--
Thomas Munro
http://www.enterprisedb.com

Attachments:

copy-with-indexes-and-transitions-v2.patchapplication/octet-stream; name=copy-with-indexes-and-transitions-v2.patchDownload
From 214af32e9cbd942ba9b858beaa6779c2b61d352b Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@enterprisedb.com>
Date: Mon, 10 Jul 2017 10:45:10 +1200
Subject: [PATCH] Fix COPY's handling of transition tables with indexes.

Commit c46c0e5202e8cfe750c6629db7852fdb15d528f3 failed to pass the
TransitionCaptureState object to ExecARInsertTriggers() in the case
where it's using heap_multi_insert and there are indexes.  Repair.
---
 src/backend/commands/copy.c            | 2 +-
 src/test/regress/expected/triggers.out | 7 ++++++-
 src/test/regress/sql/triggers.sql      | 7 +++++++
 3 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index f391828e74f..fc5f4f66ead 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -2915,7 +2915,7 @@ CopyFromInsertBatch(CopyState cstate, EState *estate, CommandId mycid,
 									  estate, false, NULL, NIL);
 			ExecARInsertTriggers(estate, resultRelInfo,
 								 bufferedTuples[i],
-								 recheckIndexes, NULL);
+								 recheckIndexes, cstate->transition_capture);
 			list_free(recheckIndexes);
 		}
 	}
diff --git a/src/test/regress/expected/triggers.out b/src/test/regress/expected/triggers.out
index aaee30219ad..ac132b042d5 100644
--- a/src/test/regress/expected/triggers.out
+++ b/src/test/regress/expected/triggers.out
@@ -2156,6 +2156,11 @@ NOTICE:  trigger = child3_delete_trig, old table = (CCC,42,foo)
 -- are really inserted into the parent)
 copy parent (a, b) from stdin;
 NOTICE:  trigger = parent_insert_trig, new table = (AAA,42), (BBB,42), (CCC,42)
+-- same behavior for copy if there is an index (interesting because rows are
+-- captured by a different code path in copy.c if there are indexes)
+create index on parent(b);
+copy parent (a, b) from stdin;
+NOTICE:  trigger = parent_insert_trig, new table = (DDD,42)
 -- DML affecting parent sees tuples collected from children even if
 -- there is no transition table trigger on the children
 drop trigger child1_insert_trig on child1;
@@ -2168,7 +2173,7 @@ drop trigger child3_insert_trig on child3;
 drop trigger child3_update_trig on child3;
 drop trigger child3_delete_trig on child3;
 delete from parent;
-NOTICE:  trigger = parent_delete_trig, old table = (AAA,42), (BBB,42), (CCC,42)
+NOTICE:  trigger = parent_delete_trig, old table = (AAA,42), (BBB,42), (CCC,42), (DDD,42)
 drop table child1, child2, child3, parent;
 --
 -- Verify prohibition of row triggers with transition triggers on
diff --git a/src/test/regress/sql/triggers.sql b/src/test/regress/sql/triggers.sql
index 659a5a1422b..b10159a1cf2 100644
--- a/src/test/regress/sql/triggers.sql
+++ b/src/test/regress/sql/triggers.sql
@@ -1661,6 +1661,13 @@ BBB	42
 CCC	42
 \.
 
+-- same behavior for copy if there is an index (interesting because rows are
+-- captured by a different code path in copy.c if there are indexes)
+create index on parent(b);
+copy parent (a, b) from stdin;
+DDD	42
+\.
+
 -- DML affecting parent sees tuples collected from children even if
 -- there is no transition table trigger on the children
 drop trigger child1_insert_trig on child1;
-- 
2.13.2

#5Andrew Gierth
andrew@tao11.riddles.org.uk
In reply to: Thomas Munro (#4)
Re: COPY vs. transition tables

"Thomas" == Thomas Munro <thomas.munro@enterprisedb.com> writes:

Thomas> Here it is. Added to open items.

On it.

--
Andrew (irc:RhodiumToad)

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#6Andrew Gierth
andrew@tao11.riddles.org.uk
In reply to: Andrew Gierth (#5)
Re: COPY vs. transition tables

"Andrew" == Andrew Gierth <andrew@tao11.riddles.org.uk> writes:
"Thomas" == Thomas Munro <thomas.munro@enterprisedb.com> writes:

Thomas> Here it is. Added to open items.

Andrew> On it.

Committed.

--
Andrew (irc:RhodiumToad)

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers