Bug in pg_dump

Started by Gilles Daroldalmost 11 years ago26 messages
#1Gilles Darold
gilles.darold@dalibo.com
2 attachment(s)

Hello,

There's a long pending issue with pg_dump and extensions that have table
members with foreign keys. This was previously reported in this thread
/messages/by-id/CA+TgmoYVZkAdMGh_8EL7UVM472GerU0b4pnNFjQYe6ss1K9wDQ@mail.gmail.com
and discuss by Robert. All PostgreSQL users that use the PostGis
extension postgis_topology are facing the issue because the two members
tables (topology and layer) are linked by foreign keys.

If you dump a database with this extension and try to import it you will
experience this error:

pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 3345; 0 157059176
TABLE DATA layer gilles
pg_restore: [archiver (db)] COPY failed for table "layer": ERROR:
insert or update on table "layer" violates foreign key constraint
"layer_topology_id_fkey"
DETAIL: Key (topology_id)=(1) is not present in table "topology".
WARNING: errors ignored on restore: 1

The problem is that, whatever export type you choose (plain/custom and
full-export/data-only) the data of tables "topology" and "layer" are
always exported in alphabetic order. I think this is a bug because
outside extension, in data-only export, pg_dump is able to find foreign
keys dependency and dump table's data in the right order but not with
extension's members. Default is alphabetic order but that should not be
the case with extension's members because constraints are recreated
during the CREATE EXTENSION order. I hope I am clear enough.

Here we have three solutions:

1/ Inform developers of extensions to take care to alphabetical
order when they have member tables using foreign keys.
2/ Inform DBAs that they have to restore the failing table
independently. The use case above can be resumed using the following
command:

pg_restore -h localhost -n topology -t layer -Fc -d
testdb_empty testdump.dump

3/ Inform DBAs that they have to restore the schema first then the
data only using --disable-triggers
4/ Patch pg_dump to solve this issue.

I attach a patch that solves the issue in pg_dump, let me know if it
might be included in Commit Fest or if the three other solutions are a
better choice. I also join a sample extension (test_fk_in_ext) to be
able to reproduce the issue and test the patch. Note that it might
exists a simpler solution than the one I used in this patch, if this is
the case please point me on the right way, I will be pleased to rewrite
and send an other patch.

In the test extension attached, there is a file called
test_fk_in_ext/SYNOPSIS.txt that describe all actions to reproduce the
issue and test the patch. Here is the SQL part of the test extension:

CREATE TABLE IF NOT EXISTS b_test_fk_in_ext1 (
id int PRIMARY KEY
);

CREATE TABLE IF NOT EXISTS a_test_fk_in_ext1 (
id int REFERENCES b_test_fk_in_ext1(id)
);

SELECT pg_catalog.pg_extension_config_dump('b_test_fk_in_ext1', '');
SELECT pg_catalog.pg_extension_config_dump('a_test_fk_in_ext1', '');

Best regards,

--
Gilles Darold
Consultant PostgreSQL
http://dalibo.com - http://dalibo.org

Attachments:

pg_dump.c-extension-FK.patchtext/x-diff; name=pg_dump.c-extension-FK.patchDownload
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index dc062e6..49889ce 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -209,6 +209,7 @@ static void addBoundaryDependencies(DumpableObject **dobjs, int numObjs,
 
 static void getDomainConstraints(Archive *fout, TypeInfo *tyinfo);
 static void getTableData(DumpOptions *dopt, TableInfo *tblinfo, int numTables, bool oids);
+static bool hasExtensionMember(TableInfo *tblinfo, int numTables);
 static void makeTableDataInfo(DumpOptions *dopt, TableInfo *tbinfo, bool oids);
 static void buildMatViewRefreshDependencies(Archive *fout);
 static void getTableDataFKConstraints(void);
@@ -730,9 +731,17 @@ main(int argc, char **argv)
 
 	if (!dopt.schemaOnly)
 	{
+		bool has_ext_member;
+
 		getTableData(&dopt, tblinfo, numTables, dopt.oids);
+		/* Search if there is dumpable tables member of and extension */
+		has_ext_member = hasExtensionMember(tblinfo, numTables);
 		buildMatViewRefreshDependencies(fout);
-		if (dopt.dataOnly)
+		/*
+		 * Always get FK constraints even with schema+data, extension's
+		 * members can have FK so tables need to be dump-ordered.
+		 */
+		if (dopt.dataOnly || has_ext_member)
 			getTableDataFKConstraints();
 	}
 
@@ -1852,6 +1861,25 @@ getTableData(DumpOptions *dopt, TableInfo *tblinfo, int numTables, bool oids)
 }
 
 /*
+ * hasExtensionMember -
+ *	  set up dumpable objects representing the contents of tables
+ */
+static bool
+hasExtensionMember(TableInfo *tblinfo, int numTables)
+{
+	int			i;
+
+	for (i = 0; i < numTables; i++)
+	{
+		if (tblinfo[i].dobj.ext_member)
+			return true;
+	}
+
+	return false;
+}
+
+
+/*
  * Make a dumpable object for the data of this specific table
  *
  * Note: we make a TableDataInfo if and only if we are going to dump the
@@ -2024,12 +2052,14 @@ buildMatViewRefreshDependencies(Archive *fout)
 
 /*
  * getTableDataFKConstraints -
- *	  add dump-order dependencies reflecting foreign key constraints
+ *        add dump-order dependencies reflecting foreign key constraints
  *
- * This code is executed only in a data-only dump --- in schema+data dumps
- * we handle foreign key issues by not creating the FK constraints until
- * after the data is loaded.  In a data-only dump, however, we want to
- * order the table data objects in such a way that a table's referenced
+ * This code is executed only in a data-only dump or when there is extension's
+ * member -- in schema+data dumps we handle foreign key issues by not creating
+ * the FK constraints until after the data is loaded. In a data-only dump or
+ * when there is an extension member to dump (schema dumps do not concern
+ * extension's objects, they are created during the CREATE EXTENSION), we want
+ * to order the table data objects in such a way that a table's referenced
  * tables are restored first.  (In the presence of circular references or
  * self-references this may be impossible; we'll detect and complain about
  * that during the dependency sorting step.)
@@ -2929,6 +2959,10 @@ dumpPolicy(Archive *fout, DumpOptions *dopt, PolicyInfo *polinfo)
 	if (dopt->dataOnly)
 		return;
 
+	/* Do not dump if table is member of an extension */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	/*
 	 * If polname is NULL, then this record is just indicating that ROW
 	 * LEVEL SECURITY is enabled for the table. Dump as ALTER TABLE <table>
@@ -7880,6 +7914,10 @@ dumpTableComment(Archive *fout, DumpOptions *dopt, TableInfo *tbinfo,
 	if (dopt->dataOnly)
 		return;
 
+	/* Do not dump if table is member of an extension */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	/* Search for comments associated with relation, using table */
 	ncomments = findComments(fout,
 							 tbinfo->dobj.catId.tableoid,
@@ -13134,6 +13172,10 @@ dumpTableSecLabel(Archive *fout, DumpOptions *dopt, TableInfo *tbinfo, const cha
 	if (dopt->dataOnly)
 		return;
 
+	/* Do not dump if table is member of an extension */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	/* Search for comments associated with relation, using table */
 	nlabels = findSecLabels(fout,
 							tbinfo->dobj.catId.tableoid,
@@ -13341,7 +13383,7 @@ collectSecLabels(Archive *fout, SecLabelItem **items)
 static void
 dumpTable(Archive *fout, DumpOptions *dopt, TableInfo *tbinfo)
 {
-	if (tbinfo->dobj.dump && !dopt->dataOnly)
+	if (tbinfo->dobj.dump && !dopt->dataOnly && !tbinfo->dobj.ext_member)
 	{
 		char	   *namecopy;
 
@@ -13479,6 +13521,10 @@ dumpTableSchema(Archive *fout, DumpOptions *dopt, TableInfo *tbinfo)
 	int			j,
 				k;
 
+	/* Do not dump if table is member of an extension */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	/* Make sure we are in proper schema */
 	selectSourceSchema(fout, tbinfo->dobj.namespace->dobj.name);
 
@@ -14113,6 +14159,10 @@ dumpAttrDef(Archive *fout, DumpOptions *dopt, AttrDefInfo *adinfo)
 	if (!tbinfo->dobj.dump || dopt->dataOnly)
 		return;
 
+	/* Do not dump if table is member of an extension */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	/* Skip if not "separate"; it was dumped in the table's definition */
 	if (!adinfo->separate)
 		return;
@@ -14200,6 +14250,10 @@ dumpIndex(Archive *fout, DumpOptions *dopt, IndxInfo *indxinfo)
 	if (dopt->dataOnly)
 		return;
 
+	/* Do not dump if table is member of an extension */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	q = createPQExpBuffer();
 	delq = createPQExpBuffer();
 	labelq = createPQExpBuffer();
@@ -14289,6 +14343,10 @@ dumpConstraint(Archive *fout, DumpOptions *dopt, ConstraintInfo *coninfo)
 	if (!coninfo->dobj.dump || dopt->dataOnly)
 		return;
 
+	/* Do not dump if table is member of an extension */
+	if ((tbinfo != NULL) && tbinfo->dobj.ext_member)
+		return;
+
 	q = createPQExpBuffer();
 	delq = createPQExpBuffer();
 
@@ -14513,7 +14571,13 @@ static void
 dumpTableConstraintComment(Archive *fout, DumpOptions *dopt, ConstraintInfo *coninfo)
 {
 	TableInfo  *tbinfo = coninfo->contable;
-	PQExpBuffer labelq = createPQExpBuffer();
+	PQExpBuffer labelq;
+
+	/* Do not dump if table is member of an extension */
+	if (tbinfo->dobj.ext_member)
+		return;
+
+	labelq = createPQExpBuffer();
 
 	appendPQExpBuffer(labelq, "CONSTRAINT %s ",
 					  fmtId(coninfo->dobj.name));
@@ -14590,6 +14654,11 @@ dumpSequence(Archive *fout, DumpOptions *dopt, TableInfo *tbinfo)
 	char		bufm[100],
 				bufx[100];
 	bool		cycled;
+
+	/* Do not dump if table is member of an extension */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	PQExpBuffer query = createPQExpBuffer();
 	PQExpBuffer delqry = createPQExpBuffer();
 	PQExpBuffer labelq = createPQExpBuffer();
@@ -14857,6 +14926,10 @@ dumpTrigger(Archive *fout, DumpOptions *dopt, TriggerInfo *tginfo)
 	if (dopt->dataOnly)
 		return;
 
+	/* Do not dump if table is member of an extension */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	query = createPQExpBuffer();
 	delqry = createPQExpBuffer();
 	labelq = createPQExpBuffer();
@@ -15133,6 +15206,10 @@ dumpRule(Archive *fout, DumpOptions *dopt, RuleInfo *rinfo)
 	if (!rinfo->dobj.dump || dopt->dataOnly)
 		return;
 
+	/* Do not dump if table is member of an extension */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	/*
 	 * If it is an ON SELECT rule that is created implicitly by CREATE VIEW,
 	 * we do not want to dump it as a separate object.
@@ -15423,6 +15500,8 @@ getExtensionMembership(Archive *fout, DumpOptions *dopt, ExtensionInfo extinfo[]
 
 				if (dumpobj)
 				{
+					/* Mark the member table as dumpable */
+					configtbl->dobj.dump = true;
 					/*
 					 * Note: config tables are dumped without OIDs regardless
 					 * of the --oids setting.  This is because row filtering
test_fk_in_ext.tar.gzapplication/x-gzip; name=test_fk_in_ext.tar.gzDownload
#2Jim Nasby
Jim.Nasby@BlueTreble.com
In reply to: Gilles Darold (#1)
Re: Bug in pg_dump

On 1/15/15 5:26 AM, Gilles Darold wrote:

Hello,

There's a long pending issue with pg_dump and extensions that have table members with foreign keys. This was previously reported in this thread /messages/by-id/CA+TgmoYVZkAdMGh_8EL7UVM472GerU0b4pnNFjQYe6ss1K9wDQ@mail.gmail.com and discuss by Robert. All PostgreSQL users that use the PostGis extension postgis_topology are facing the issue because the two members tables (topology and layer) are linked by foreign keys.

If you dump a database with this extension and try to import it you will experience this error:

pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 3345; 0 157059176 TABLE DATA layer gilles
pg_restore: [archiver (db)] COPY failed for table "layer": ERROR: insert or update on table "layer" violates foreign key constraint "layer_topology_id_fkey"
DETAIL: Key (topology_id)=(1) is not present in table "topology".
WARNING: errors ignored on restore: 1

The problem is that, whatever export type you choose (plain/custom and full-export/data-only) the data of tables "topology" and "layer" are always exported in alphabetic order. I think this is a bug because outside extension, in data-only export, pg_dump is able to find foreign keys dependency and dump table's data in the right order but not with extension's members. Default is alphabetic order but that should not be the case with extension's members because constraints are recreated during the CREATE EXTENSION order. I hope I am clear enough.

Here we have three solutions:

1/ Inform developers of extensions to take care to alphabetical order when they have member tables using foreign keys.
2/ Inform DBAs that they have to restore the failing table independently. The use case above can be resumed using the following command:

pg_restore -h localhost -n topology -t layer -Fc -d testdb_empty testdump.dump

3/ Inform DBAs that they have to restore the schema first then the data only using --disable-triggers

I don't like 1-3, and I doubt anyone else does...

4/ Patch pg_dump to solve this issue.

5. Disable FK's during load.
This is really a bigger item than just extensions. It would have the nice benefit of doing a wholesale FK validation instead of firing per-row triggers, but it would leave the database in a weird state if a restore failed...

I attach a patch that solves the issue in pg_dump, let me know if it might be included in Commit Fest or if the three other solutions are a better choice. I also join a sample extension (test_fk_in_ext) to be able to reproduce the issue and test the patch. Note that it might exists a simpler solution than the one I used in this patch, if this is the case please point me on the right way, I will be pleased to rewrite and send an other patch.

The only problem I see with this approach is circular FK's:

decibel@decina.local=# create table a(a_id serial primary key, b_id int);
CREATE TABLE
decibel@decina.local=# create table b(b_id serial primary key, a_id int references a);
CREATE TABLE
decibel@decina.local=# alter table a add foreign key(b_id) references b;
ALTER TABLE
decibel@decina.local=#

That's esoteric enough that I think it's OK not to directly support them, but pg_dump shouldn't puke on them (and really should throw a warning). Though it looks like it doesn't handle that in the data-only case anyway...
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#3Gilles Darold
gilles.darold@dalibo.com
In reply to: Jim Nasby (#2)
Re: Bug in pg_dump

On 16/01/2015 01:06, Jim Nasby wrote:

On 1/15/15 5:26 AM, Gilles Darold wrote:

Hello,

There's a long pending issue with pg_dump and extensions that have
table members with foreign keys. This was previously reported in this
thread
/messages/by-id/CA+TgmoYVZkAdMGh_8EL7UVM472GerU0b4pnNFjQYe6ss1K9wDQ@mail.gmail.com
and discuss by Robert. All PostgreSQL users that use the PostGis
extension postgis_topology are facing the issue because the two
members tables (topology and layer) are linked by foreign keys.

If you dump a database with this extension and try to import it you
will experience this error:

pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 3345; 0
157059176 TABLE DATA layer gilles
pg_restore: [archiver (db)] COPY failed for table "layer": ERROR:
insert or update on table "layer" violates foreign key constraint
"layer_topology_id_fkey"
DETAIL: Key (topology_id)=(1) is not present in table "topology".
WARNING: errors ignored on restore: 1

The problem is that, whatever export type you choose (plain/custom
and full-export/data-only) the data of tables "topology" and "layer"
are always exported in alphabetic order. I think this is a bug
because outside extension, in data-only export, pg_dump is able to
find foreign keys dependency and dump table's data in the right order
but not with extension's members. Default is alphabetic order but
that should not be the case with extension's members because
constraints are recreated during the CREATE EXTENSION order. I hope I
am clear enough.

Here we have three solutions:

1/ Inform developers of extensions to take care to alphabetical
order when they have member tables using foreign keys.
2/ Inform DBAs that they have to restore the failing table
independently. The use case above can be resumed using the following
command:

pg_restore -h localhost -n topology -t layer -Fc -d
testdb_empty testdump.dump

3/ Inform DBAs that they have to restore the schema first then
the data only using --disable-triggers

I don't like 1-3, and I doubt anyone else does...

4/ Patch pg_dump to solve this issue.

5. Disable FK's during load.
This is really a bigger item than just extensions. It would have the
nice benefit of doing a wholesale FK validation instead of firing
per-row triggers, but it would leave the database in a weird state if
a restore failed...

I think this is an other problem. Here we just need to apply to
extension's members tables the same work than to normal tables. I guess
this is what this patch try to solve.

I attach a patch that solves the issue in pg_dump, let me know if it
might be included in Commit Fest or if the three other solutions are
a better choice. I also join a sample extension (test_fk_in_ext) to
be able to reproduce the issue and test the patch. Note that it might
exists a simpler solution than the one I used in this patch, if this
is the case please point me on the right way, I will be pleased to
rewrite and send an other patch.

The only problem I see with this approach is circular FK's:

decibel@decina.local=# create table a(a_id serial primary key, b_id int);
CREATE TABLE
decibel@decina.local=# create table b(b_id serial primary key, a_id
int references a);
CREATE TABLE
decibel@decina.local=# alter table a add foreign key(b_id) references b;
ALTER TABLE
decibel@decina.local=#

That's esoteric enough that I think it's OK not to directly support
them, but pg_dump shouldn't puke on them (and really should throw a
warning). Though it looks like it doesn't handle that in the data-only
case anyway...

The patch is taking care or circular references and you will be warn if
pg_dump found it in the extension members. That was not the case before.
If you try do dump a database with the postgis extension you will be
warn about FK defined on the edge_data table.

--
Gilles Darold
Consultant PostgreSQL
http://dalibo.com - http://dalibo.org

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#4Robert Haas
robertmhaas@gmail.com
In reply to: Gilles Darold (#1)
Re: Bug in pg_dump

On Thu, Jan 15, 2015 at 6:26 AM, Gilles Darold <gilles.darold@dalibo.com> wrote:

I attach a patch that solves the issue in pg_dump, let me know if it might
be included in Commit Fest or if the three other solutions are a better
choice.

I think a fix in pg_dump is appropriate, so I'd encourage you to add
this to the next CommitFest.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#5Gilles Darold
gilles.darold@dalibo.com
In reply to: Robert Haas (#4)
1 attachment(s)
Re: Bug in pg_dump

Le 19/01/2015 14:41, Robert Haas a écrit :

On Thu, Jan 15, 2015 at 6:26 AM, Gilles Darold <gilles.darold@dalibo.com> wrote:

I attach a patch that solves the issue in pg_dump, let me know if it might
be included in Commit Fest or if the three other solutions are a better
choice.

I think a fix in pg_dump is appropriate, so I'd encourage you to add
this to the next CommitFest.

Ok, thanks Robert. The patch have been added to next CommitFest under
the Bug Fixes section.

I've also made some review of the patch and more test. I've rewritten
some comments and added a test when TableInfo is NULL to avoid potential
pg_dump crash.

New patch attached: pg_dump.c-extension-FK-v2.patch

Regards

--
Gilles Darold
Consultant PostgreSQL
http://dalibo.com - http://dalibo.org

Attachments:

pg_dump.c-extension-FK-v2.patchtext/x-diff; name=pg_dump.c-extension-FK-v2.patchDownload
--- ../postgresql/src/bin/pg_dump/pg_dump.c	2015-01-19 19:03:45.897706879 +0100
+++ src/bin/pg_dump/pg_dump.c	2015-01-20 11:22:01.144794489 +0100
@@ -209,6 +209,7 @@
 
 static void getDomainConstraints(Archive *fout, TypeInfo *tyinfo);
 static void getTableData(DumpOptions *dopt, TableInfo *tblinfo, int numTables, bool oids);
+static bool hasExtensionMember(TableInfo *tblinfo, int numTables);
 static void makeTableDataInfo(DumpOptions *dopt, TableInfo *tbinfo, bool oids);
 static void buildMatViewRefreshDependencies(Archive *fout);
 static void getTableDataFKConstraints(void);
@@ -730,9 +731,20 @@
 
 	if (!dopt.schemaOnly)
 	{
+		bool has_ext_member;
+
 		getTableData(&dopt, tblinfo, numTables, dopt.oids);
+
+		/* Search if there's dumpable table's members in an extension */
+		has_ext_member = hasExtensionMember(tblinfo, numTables);
+
 		buildMatViewRefreshDependencies(fout);
-		if (dopt.dataOnly)
+		/*
+		 * Get FK constraints even with schema+data if extension's
+		 * members have FK because in this case tables need to be
+		 * dump-ordered too.
+		 */
+		if (dopt.dataOnly || has_ext_member)
 			getTableDataFKConstraints();
 	}
 
@@ -1852,6 +1864,26 @@
 }
 
 /*
+ * hasExtensionMember -
+ *	  return true when on of the dumpable object
+ *	  is an extension members
+ */
+static bool
+hasExtensionMember(TableInfo *tblinfo, int numTables)
+{
+	int			i;
+
+	for (i = 0; i < numTables; i++)
+	{
+		if (tblinfo[i].dobj.ext_member)
+			return true;
+	}
+
+	return false;
+}
+
+
+/*
  * Make a dumpable object for the data of this specific table
  *
  * Note: we make a TableDataInfo if and only if we are going to dump the
@@ -2026,10 +2058,12 @@
  * getTableDataFKConstraints -
  *	  add dump-order dependencies reflecting foreign key constraints
  *
- * This code is executed only in a data-only dump --- in schema+data dumps
- * we handle foreign key issues by not creating the FK constraints until
- * after the data is loaded.  In a data-only dump, however, we want to
- * order the table data objects in such a way that a table's referenced
+ * This code is executed only in a data-only dump or when there is extension's
+ * member -- in schema+data dumps we handle foreign key issues by not creating
+ * the FK constraints until after the data is loaded. In a data-only dump or
+ * when there is an extension member to dump (schema dumps do not concern
+ * extension's objects, they are created during the CREATE EXTENSION), we want
+ * to order the table data objects in such a way that a table's referenced
  * tables are restored first.  (In the presence of circular references or
  * self-references this may be impossible; we'll detect and complain about
  * that during the dependency sorting step.)
@@ -2930,9 +2964,14 @@
 	PQExpBuffer delqry;
 	const char *cmd;
 
+	/* Policy is SCHEMA only */
 	if (dopt->dataOnly)
 		return;
 
+	/* CREATE EXTENSION should take care of that */
+	if ((tbinfo != NULL) && tbinfo->dobj.ext_member)
+		return;
+
 	/*
 	 * If polname is NULL, then this record is just indicating that ROW
 	 * LEVEL SECURITY is enabled for the table. Dump as ALTER TABLE <table>
@@ -7884,6 +7923,10 @@
 	if (dopt->dataOnly)
 		return;
 
+	/* CREATE EXTENSION should take care of that */
+	if ((tbinfo != NULL) && tbinfo->dobj.ext_member)
+		return;
+
 	/* Search for comments associated with relation, using table */
 	ncomments = findComments(fout,
 							 tbinfo->dobj.catId.tableoid,
@@ -13138,6 +13181,10 @@
 	if (dopt->dataOnly)
 		return;
 
+	/* CREATE EXTENSION should take care of that */
+	if ((tbinfo != NULL) && tbinfo->dobj.ext_member)
+		return;
+
 	/* Search for comments associated with relation, using table */
 	nlabels = findSecLabels(fout,
 							tbinfo->dobj.catId.tableoid,
@@ -13345,7 +13392,7 @@
 static void
 dumpTable(Archive *fout, DumpOptions *dopt, TableInfo *tbinfo)
 {
-	if (tbinfo->dobj.dump && !dopt->dataOnly)
+	if (tbinfo->dobj.dump && !dopt->dataOnly && !tbinfo->dobj.ext_member)
 	{
 		char	   *namecopy;
 
@@ -13483,6 +13530,10 @@
 	int			j,
 				k;
 
+	/* CREATE EXTENSION should take care of that */
+	if ((tbinfo != NULL) && tbinfo->dobj.ext_member)
+		return;
+
 	/* Make sure we are in proper schema */
 	selectSourceSchema(fout, tbinfo->dobj.namespace->dobj.name);
 
@@ -14117,6 +14168,10 @@
 	if (!tbinfo->dobj.dump || dopt->dataOnly)
 		return;
 
+	/* CREATE EXTENSION should take care of that */
+	if ((tbinfo != NULL) && tbinfo->dobj.ext_member)
+		return;
+
 	/* Skip if not "separate"; it was dumped in the table's definition */
 	if (!adinfo->separate)
 		return;
@@ -14204,6 +14259,10 @@
 	if (dopt->dataOnly)
 		return;
 
+	/* CREATE EXTENSION should take care of that */
+	if ((tbinfo != NULL) && tbinfo->dobj.ext_member)
+		return;
+
 	q = createPQExpBuffer();
 	delq = createPQExpBuffer();
 	labelq = createPQExpBuffer();
@@ -14293,6 +14352,10 @@
 	if (!coninfo->dobj.dump || dopt->dataOnly)
 		return;
 
+	/* CREATE EXTENSION should take care of that */
+	if ((tbinfo != NULL) && tbinfo->dobj.ext_member)
+		return;
+
 	q = createPQExpBuffer();
 	delq = createPQExpBuffer();
 
@@ -14517,7 +14580,13 @@
 dumpTableConstraintComment(Archive *fout, DumpOptions *dopt, ConstraintInfo *coninfo)
 {
 	TableInfo  *tbinfo = coninfo->contable;
-	PQExpBuffer labelq = createPQExpBuffer();
+	PQExpBuffer labelq;
+
+	/* CREATE EXTENSION should take care of that */
+	if ((tbinfo != NULL) && tbinfo->dobj.ext_member)
+		return;
+
+	labelq = createPQExpBuffer();
 
 	appendPQExpBuffer(labelq, "CONSTRAINT %s ",
 					  fmtId(coninfo->dobj.name));
@@ -14594,9 +14663,17 @@
 	char		bufm[100],
 				bufx[100];
 	bool		cycled;
-	PQExpBuffer query = createPQExpBuffer();
-	PQExpBuffer delqry = createPQExpBuffer();
-	PQExpBuffer labelq = createPQExpBuffer();
+	PQExpBuffer query;
+	PQExpBuffer delqry;
+	PQExpBuffer labelq;
+
+	/* CREATE EXTENSION should take care of that */
+	if ((tbinfo != NULL) && tbinfo->dobj.ext_member)
+		return;
+
+	query = createPQExpBuffer();
+	delqry = createPQExpBuffer();
+	labelq = createPQExpBuffer();
 
 	/* Make sure we are in proper schema */
 	selectSourceSchema(fout, tbinfo->dobj.namespace->dobj.name);
@@ -14861,6 +14938,10 @@
 	if (dopt->dataOnly)
 		return;
 
+	/* CREATE EXTENSION should take care of that */
+	if ((tbinfo != NULL) && tbinfo->dobj.ext_member)
+		return;
+
 	query = createPQExpBuffer();
 	delqry = createPQExpBuffer();
 	labelq = createPQExpBuffer();
@@ -15137,6 +15218,10 @@
 	if (!rinfo->dobj.dump || dopt->dataOnly)
 		return;
 
+	/* CREATE EXTENSION should take care of that */
+	if ((tbinfo != NULL) && tbinfo->dobj.ext_member)
+		return;
+
 	/*
 	 * If it is an ON SELECT rule that is created implicitly by CREATE VIEW,
 	 * we do not want to dump it as a separate object.
@@ -15427,6 +15512,9 @@
 
 				if (dumpobj)
 				{
+					/* Mark member table as dumpable */
+					configtbl->dobj.dump = true;
+
 					/*
 					 * Note: config tables are dumped without OIDs regardless
 					 * of the --oids setting.  This is because row filtering
#6Michael Paquier
michael.paquier@gmail.com
In reply to: Gilles Darold (#5)
3 attachment(s)
Re: Bug in pg_dump

On Tue, Jan 20, 2015 at 8:06 PM, Gilles Darold <gilles.darold@dalibo.com> wrote:

Le 19/01/2015 14:41, Robert Haas a écrit :

On Thu, Jan 15, 2015 at 6:26 AM, Gilles Darold <gilles.darold@dalibo.com> wrote:

I attach a patch that solves the issue in pg_dump, let me know if it might
be included in Commit Fest or if the three other solutions are a better
choice.

I think a fix in pg_dump is appropriate, so I'd encourage you to add
this to the next CommitFest.

Ok, thanks Robert. The patch have been added to next CommitFest under
the Bug Fixes section.

I've also made some review of the patch and more test. I've rewritten
some comments and added a test when TableInfo is NULL to avoid potential
pg_dump crash.

New patch attached: pg_dump.c-extension-FK-v2.patch

So, I looked at your patch and I have some comments...

The approach taken by the patch looks correct to me as we cannot
create FK constraints after loading the data in the case of an
extension, something that can be done with a data-only dump.

Now, I think that the routine hasExtensionMember may impact
performance unnecessarily in the case of databases with many tables,
say thousands or more. And we can easily avoid this performance
problem by checking the content of each object dumped by doing the
legwork directly in getTableData. Also, most of the NULL pointer
checks are not needed as most of those code paths would crash if
tbinfo is NULL, and actually only keeping the one in dumpConstraint is
fine.

On top of those things, I have written a small extension able to
reproduce the problem that I have included as a test module in
src/test/modules. The dump/restore check is done using the TAP tests,
and I have hacked prove_check to install as well the contents of the
current folder to be able to use the TAP routines with an extension
easily. IMO, as we have no coverage of pg_dump with extensions I think
that it would be useful to ensure that this does not break again in
the future.

All the hacking I have done during the review results in the set of
patch attached:
- 0001 is your patch, Gilles, with some comment fixes as well as the
performance issue with hasExtensionMember fix
- 0002 is the modification of prove_check that makes TAP tests usable
with extensions
- 0003 is the test module covering the tests needed for pg_dump, at
least for the problem of this thread.

Gilles, how does that look to you?
(Btw, be sure to generate your patches directly with git next time. :) )

Regards,
--
Michael

Attachments:

0001-Fix-ordering-of-tables-part-of-extensions-linked-wit.patchtext/x-diff; charset=US-ASCII; name=0001-Fix-ordering-of-tables-part-of-extensions-linked-wit.patchDownload
From 8005cffd08c57b77564bb0294d8ebd4600bc1dcf Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Tue, 17 Feb 2015 07:39:23 +0000
Subject: [PATCH 1/3] Fix ordering of tables part of extensions linked with
 constraints

The same mechanism as data-only dumps ensuring that data is loaded
respecting foreign key constains is used if it at least one dumped
object is found as being part of an extension. This commit reinforces
as well a couple of code paths to not dump objects that are directly
part of an extension.

Patch by Gilles Darold.
---
 src/bin/pg_dump/pg_dump.c | 99 +++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 83 insertions(+), 16 deletions(-)

diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7e92b74..c95ae3d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -208,7 +208,8 @@ static void addBoundaryDependencies(DumpableObject **dobjs, int numObjs,
 						DumpableObject *boundaryObjs);
 
 static void getDomainConstraints(Archive *fout, TypeInfo *tyinfo);
-static void getTableData(DumpOptions *dopt, TableInfo *tblinfo, int numTables, bool oids);
+static void getTableData(DumpOptions *dopt, TableInfo *tblinfo, int numTables,
+						 bool oids, bool *has_ext_member);
 static void makeTableDataInfo(DumpOptions *dopt, TableInfo *tbinfo, bool oids);
 static void buildMatViewRefreshDependencies(Archive *fout);
 static void getTableDataFKConstraints(void);
@@ -730,9 +731,12 @@ main(int argc, char **argv)
 
 	if (!dopt.schemaOnly)
 	{
-		getTableData(&dopt, tblinfo, numTables, dopt.oids);
+		bool has_ext_member = false;
+
+		getTableData(&dopt, tblinfo, numTables, dopt.oids, &has_ext_member);
 		buildMatViewRefreshDependencies(fout);
-		if (dopt.dataOnly)
+
+		if (dopt.dataOnly || has_ext_member)
 			getTableDataFKConstraints();
 	}
 
@@ -1866,7 +1870,8 @@ refreshMatViewData(Archive *fout, TableDataInfo *tdinfo)
  *	  set up dumpable objects representing the contents of tables
  */
 static void
-getTableData(DumpOptions *dopt, TableInfo *tblinfo, int numTables, bool oids)
+getTableData(DumpOptions *dopt, TableInfo *tblinfo, int numTables,
+			 bool oids, bool *has_ext_member)
 {
 	int			i;
 
@@ -1874,6 +1879,8 @@ getTableData(DumpOptions *dopt, TableInfo *tblinfo, int numTables, bool oids)
 	{
 		if (tblinfo[i].dobj.dump)
 			makeTableDataInfo(dopt, &(tblinfo[i]), oids);
+		if (!(*has_ext_member) && tblinfo[i].dobj.ext_member)
+			*has_ext_member = true;
 	}
 }
 
@@ -2052,13 +2059,15 @@ buildMatViewRefreshDependencies(Archive *fout)
  * getTableDataFKConstraints -
  *	  add dump-order dependencies reflecting foreign key constraints
  *
- * This code is executed only in a data-only dump --- in schema+data dumps
- * we handle foreign key issues by not creating the FK constraints until
- * after the data is loaded.  In a data-only dump, however, we want to
- * order the table data objects in such a way that a table's referenced
- * tables are restored first.  (In the presence of circular references or
- * self-references this may be impossible; we'll detect and complain about
- * that during the dependency sorting step.)
+ * This code is executed only in a data-only dump or when an object dumped
+ * is part of an extension -- in schema+data dumps we handle foreign key
+ * issues by not creating the FK constraints until after the data is loaded.
+ * In a data-only dump or when there is an extension member to dump (schema
+ * dumps do not include extension objects, they are created with CREATE
+ * EXTENSION), we want to order the table data objects in such a way that
+ * a table's referenced tables are restored first.  (In the presence of
+ * circular references or self-references this may be impossible; we'll
+ * detect and complain about that during the dependency sorting step.)
  */
 static void
 getTableDataFKConstraints(void)
@@ -2951,9 +2960,14 @@ dumpPolicy(Archive *fout, DumpOptions *dopt, PolicyInfo *polinfo)
 	PQExpBuffer delqry;
 	const char *cmd;
 
+	/* Policies are SCHEMA not data */
 	if (dopt->dataOnly)
 		return;
 
+	/* Ignore extension members */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	/*
 	 * If polname is NULL, then this record is just indicating that ROW
 	 * LEVEL SECURITY is enabled for the table. Dump as ALTER TABLE <table>
@@ -7910,6 +7924,10 @@ dumpTableComment(Archive *fout, DumpOptions *dopt, TableInfo *tbinfo,
 	if (dopt->dataOnly)
 		return;
 
+	/* Ignore extension members */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	/* Search for comments associated with relation, using table */
 	ncomments = findComments(fout,
 							 tbinfo->dobj.catId.tableoid,
@@ -13118,6 +13136,10 @@ dumpTableSecLabel(Archive *fout, DumpOptions *dopt, TableInfo *tbinfo, const cha
 	if (dopt->dataOnly)
 		return;
 
+	/* Ignore extension members */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	/* Search for comments associated with relation, using table */
 	nlabels = findSecLabels(fout,
 							tbinfo->dobj.catId.tableoid,
@@ -13325,7 +13347,7 @@ collectSecLabels(Archive *fout, SecLabelItem **items)
 static void
 dumpTable(Archive *fout, DumpOptions *dopt, TableInfo *tbinfo)
 {
-	if (tbinfo->dobj.dump && !dopt->dataOnly)
+	if (tbinfo->dobj.dump && !dopt->dataOnly && !tbinfo->dobj.ext_member)
 	{
 		char	   *namecopy;
 
@@ -13463,6 +13485,10 @@ dumpTableSchema(Archive *fout, DumpOptions *dopt, TableInfo *tbinfo)
 	int			j,
 				k;
 
+	/* Ignore extension members */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	/* Make sure we are in proper schema */
 	selectSourceSchema(fout, tbinfo->dobj.namespace->dobj.name);
 
@@ -14097,6 +14123,14 @@ dumpAttrDef(Archive *fout, DumpOptions *dopt, AttrDefInfo *adinfo)
 	if (!tbinfo->dobj.dump || dopt->dataOnly)
 		return;
 
+	/* Ignore extension members */
+	if (tbinfo->dobj.ext_member)
+		return;
+
+	/* Ignore extension members */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	/* Skip if not "separate"; it was dumped in the table's definition */
 	if (!adinfo->separate)
 		return;
@@ -14184,6 +14218,10 @@ dumpIndex(Archive *fout, DumpOptions *dopt, IndxInfo *indxinfo)
 	if (dopt->dataOnly)
 		return;
 
+	/* Ignore extension members */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	q = createPQExpBuffer();
 	delq = createPQExpBuffer();
 	labelq = createPQExpBuffer();
@@ -14273,6 +14311,10 @@ dumpConstraint(Archive *fout, DumpOptions *dopt, ConstraintInfo *coninfo)
 	if (!coninfo->dobj.dump || dopt->dataOnly)
 		return;
 
+	/* Ignore extension members */
+	if (tbinfo != NULL && tbinfo->dobj.ext_member)
+		return;
+
 	q = createPQExpBuffer();
 	delq = createPQExpBuffer();
 
@@ -14497,7 +14539,13 @@ static void
 dumpTableConstraintComment(Archive *fout, DumpOptions *dopt, ConstraintInfo *coninfo)
 {
 	TableInfo  *tbinfo = coninfo->contable;
-	PQExpBuffer labelq = createPQExpBuffer();
+	PQExpBuffer labelq;
+
+	/* Ignore extension members */
+	if (tbinfo->dobj.ext_member)
+		return;
+
+	labelq = createPQExpBuffer();
 
 	appendPQExpBuffer(labelq, "CONSTRAINT %s ",
 					  fmtId(coninfo->dobj.name));
@@ -14574,9 +14622,17 @@ dumpSequence(Archive *fout, DumpOptions *dopt, TableInfo *tbinfo)
 	char		bufm[100],
 				bufx[100];
 	bool		cycled;
-	PQExpBuffer query = createPQExpBuffer();
-	PQExpBuffer delqry = createPQExpBuffer();
-	PQExpBuffer labelq = createPQExpBuffer();
+	PQExpBuffer query;
+	PQExpBuffer delqry;
+	PQExpBuffer labelq;
+
+	/* Ignore extension members */
+	if (tbinfo->dobj.ext_member)
+		return;
+
+	query = createPQExpBuffer();
+	delqry = createPQExpBuffer();
+	labelq = createPQExpBuffer();
 
 	/* Make sure we are in proper schema */
 	selectSourceSchema(fout, tbinfo->dobj.namespace->dobj.name);
@@ -14841,6 +14897,10 @@ dumpTrigger(Archive *fout, DumpOptions *dopt, TriggerInfo *tginfo)
 	if (dopt->dataOnly)
 		return;
 
+	/* Ignore extension members */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	query = createPQExpBuffer();
 	delqry = createPQExpBuffer();
 	labelq = createPQExpBuffer();
@@ -15117,6 +15177,10 @@ dumpRule(Archive *fout, DumpOptions *dopt, RuleInfo *rinfo)
 	if (!rinfo->dobj.dump || dopt->dataOnly)
 		return;
 
+	/* Ignore extension members */
+	if (tbinfo->dobj.ext_member)
+		return;
+
 	/*
 	 * If it is an ON SELECT rule that is created implicitly by CREATE VIEW,
 	 * we do not want to dump it as a separate object.
@@ -15407,6 +15471,9 @@ getExtensionMembership(Archive *fout, DumpOptions *dopt, ExtensionInfo extinfo[]
 
 				if (dumpobj)
 				{
+					/* Mark member table as dumpable */
+					configtbl->dobj.dump = true;
+
 					/*
 					 * Note: config tables are dumped without OIDs regardless
 					 * of the --oids setting.  This is because row filtering
-- 
2.3.0

0002-Make-prove_check-install-contents-of-current-directo.patchtext/x-diff; charset=US-ASCII; name=0002-Make-prove_check-install-contents-of-current-directo.patchDownload
From b2f34d2be7445e8e6ee74c64785d0ddae086e88c Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Tue, 17 Feb 2015 22:29:28 +0900
Subject: [PATCH 2/3] Make prove_check install contents of current directory as
 well

This is useful for example for TAP tests in extensions.
---
 src/Makefile.global.in | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 7c39d82..7c15423 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -323,6 +323,7 @@ endef
 define prove_check
 $(MKDIR_P) tmp_check/log
 $(MAKE) -C $(top_builddir) DESTDIR='$(CURDIR)'/tmp_check/install install >'$(CURDIR)'/tmp_check/log/install.log 2>&1
+$(MAKE) -C $(CURDIR) DESTDIR='$(CURDIR)'/tmp_check/install install >>'$(CURDIR)'/tmp_check/log/install.log 2>&1
 cd $(srcdir) && TESTDIR='$(CURDIR)' PATH="$(CURDIR)/tmp_check/install$(bindir):$$PATH" $(call add_to_path,$(ld_library_path_var),$(CURDIR)/tmp_check/install$(libdir)) top_builddir='$(CURDIR)/$(top_builddir)' PGPORT='6$(DEF_PGPORT)' $(PROVE) $(PG_PROVE_FLAGS) $(PROVE_FLAGS) t/*.pl
 endef
 
-- 
2.3.0

0003-Add-dump_test-test-module-to-check-pg_dump-with-exte.patchtext/x-diff; charset=US-ASCII; name=0003-Add-dump_test-test-module-to-check-pg_dump-with-exte.patchDownload
From 66862bd62a43182e86bc6fb5f732b722f3741fb0 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Tue, 17 Feb 2015 22:36:49 +0900
Subject: [PATCH 3/3] Add dump_test, test module to check pg_dump with
 extensions

It works with TAP tests.
---
 src/test/modules/Makefile                     |  1 +
 src/test/modules/dump_test/.gitignore         |  4 ++++
 src/test/modules/dump_test/Makefile           | 22 ++++++++++++++++++++++
 src/test/modules/dump_test/README             |  5 +++++
 src/test/modules/dump_test/dump_test--1.0.sql | 15 +++++++++++++++
 src/test/modules/dump_test/dump_test.control  |  5 +++++
 src/test/modules/dump_test/t/001_dump_test.pl | 22 ++++++++++++++++++++++
 7 files changed, 74 insertions(+)
 create mode 100644 src/test/modules/dump_test/.gitignore
 create mode 100644 src/test/modules/dump_test/Makefile
 create mode 100644 src/test/modules/dump_test/README
 create mode 100644 src/test/modules/dump_test/dump_test--1.0.sql
 create mode 100644 src/test/modules/dump_test/dump_test.control
 create mode 100644 src/test/modules/dump_test/t/001_dump_test.pl

diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 93d93af..6ad3b4e 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -6,6 +6,7 @@ include $(top_builddir)/src/Makefile.global
 
 SUBDIRS = \
 		  commit_ts \
+		  dump_test \
 		  worker_spi \
 		  dummy_seclabel \
 		  test_shm_mq \
diff --git a/src/test/modules/dump_test/.gitignore b/src/test/modules/dump_test/.gitignore
new file mode 100644
index 0000000..5dcb3ff
--- /dev/null
+++ b/src/test/modules/dump_test/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/dump_test/Makefile b/src/test/modules/dump_test/Makefile
new file mode 100644
index 0000000..08cd215
--- /dev/null
+++ b/src/test/modules/dump_test/Makefile
@@ -0,0 +1,22 @@
+# src/test/modules/dump_test/Makefile
+
+EXTENSION = dump_test
+DATA = dump_test--1.0.sql
+PGFILEDESC = "dump_test - test pg_dump with extensions"
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/dump_test
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check: all
+	$(prove_check)
+
+installcheck: install
+	$(prove_installcheck)
diff --git a/src/test/modules/dump_test/README b/src/test/modules/dump_test/README
new file mode 100644
index 0000000..9ebfa19
--- /dev/null
+++ b/src/test/modules/dump_test/README
@@ -0,0 +1,5 @@
+dump_test
+=========
+
+Simple module used to test consistency of pg_dump with objects in
+extensions.
diff --git a/src/test/modules/dump_test/dump_test--1.0.sql b/src/test/modules/dump_test/dump_test--1.0.sql
new file mode 100644
index 0000000..c987e28
--- /dev/null
+++ b/src/test/modules/dump_test/dump_test--1.0.sql
@@ -0,0 +1,15 @@
+/* src/test/modules/dump_test/dump_test--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION dump_test" to load this file. \quit
+
+CREATE TABLE IF NOT EXISTS bb_tab_fkey (
+	id int PRIMARY KEY
+);
+
+CREATE TABLE IF NOT EXISTS aa_tab_fkey (
+	id int REFERENCES bb_tab_fkey(id)
+);
+
+SELECT pg_catalog.pg_extension_config_dump('aa_tab_fkey', '');
+SELECT pg_catalog.pg_extension_config_dump('bb_tab_fkey', '');
diff --git a/src/test/modules/dump_test/dump_test.control b/src/test/modules/dump_test/dump_test.control
new file mode 100644
index 0000000..b32c99c
--- /dev/null
+++ b/src/test/modules/dump_test/dump_test.control
@@ -0,0 +1,5 @@
+# dump_test extension
+comment = 'dump_test - test pg_dump with extensions'
+default_version = '1.0'
+module_pathname = '$libdir/dump_test'
+relocatable = true
diff --git a/src/test/modules/dump_test/t/001_dump_test.pl b/src/test/modules/dump_test/t/001_dump_test.pl
new file mode 100644
index 0000000..e9bfdb6
--- /dev/null
+++ b/src/test/modules/dump_test/t/001_dump_test.pl
@@ -0,0 +1,22 @@
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 3;
+
+my $tempdir = tempdir;
+
+start_test_server $tempdir;
+
+psql 'postgres', 'CREATE EXTENSION dump_test';
+
+# Insert some data before running the dump, this is needed to check
+# consistent data dump of tables with foreign key dependencies
+psql 'postgres', 'INSERT INTO bb_tab_fkey VALUES (1)';
+psql 'postgres', 'INSERT INTO aa_tab_fkey VALUES (1)';
+
+# Take a dump then re-deploy it to a new database
+command_ok(['pg_dump', '-d', 'postgres', '-f', "$tempdir/dump.sql"],
+		   'taken dump with installed extension');
+command_ok(['createdb', 'foobar1'], 'created new database to redeploy dump');
+command_ok(['psql', '--set=ON_ERROR_STOP=on', '-d', 'foobar1', '-f',
+			"$tempdir/dump.sql"], 'dump restored');
-- 
2.3.0

#7Gilles Darold
gilles.darold@dalibo.com
In reply to: Michael Paquier (#6)
Re: Bug in pg_dump

Le 17/02/2015 14:44, Michael Paquier a �crit :

On Tue, Jan 20, 2015 at 8:06 PM, Gilles Darold <gilles.darold@dalibo.com> wrote:

Le 19/01/2015 14:41, Robert Haas a �crit :

On Thu, Jan 15, 2015 at 6:26 AM, Gilles Darold <gilles.darold@dalibo.com> wrote:

I attach a patch that solves the issue in pg_dump, let me know if it might
be included in Commit Fest or if the three other solutions are a better
choice.

I think a fix in pg_dump is appropriate, so I'd encourage you to add
this to the next CommitFest.

Ok, thanks Robert. The patch have been added to next CommitFest under
the Bug Fixes section.

I've also made some review of the patch and more test. I've rewritten
some comments and added a test when TableInfo is NULL to avoid potential
pg_dump crash.

New patch attached: pg_dump.c-extension-FK-v2.patch

So, I looked at your patch and I have some comments...

The approach taken by the patch looks correct to me as we cannot
create FK constraints after loading the data in the case of an
extension, something that can be done with a data-only dump.

Now, I think that the routine hasExtensionMember may impact
performance unnecessarily in the case of databases with many tables,
say thousands or more. And we can easily avoid this performance
problem by checking the content of each object dumped by doing the
legwork directly in getTableData. Also, most of the NULL pointer
checks are not needed as most of those code paths would crash if
tbinfo is NULL, and actually only keeping the one in dumpConstraint is
fine.

Yes that's exactly what we discuss at Moscow, thanks for removing the
hasExtensionMember() routine. About NULL pointer I'm was not sure that
it could not crash on some other parts so I decided to check it
everywhere. If that's ok to just check in dumpConstraint() that's fine.

On top of those things, I have written a small extension able to
reproduce the problem that I have included as a test module in
src/test/modules. The dump/restore check is done using the TAP tests,
and I have hacked prove_check to install as well the contents of the
current folder to be able to use the TAP routines with an extension
easily. IMO, as we have no coverage of pg_dump with extensions I think
that it would be useful to ensure that this does not break again in
the future.

Great ! I did not had time to do that, thank you so much.

All the hacking I have done during the review results in the set of
patch attached:
- 0001 is your patch, Gilles, with some comment fixes as well as the
performance issue with hasExtensionMember fix
- 0002 is the modification of prove_check that makes TAP tests usable
with extensions
- 0003 is the test module covering the tests needed for pg_dump, at
least for the problem of this thread.

Gilles, how does that look to you?

Looks great to me, I have tested with the postgis_topology extension
everything works fine.

(Btw, be sure to generate your patches directly with git next time. :) )

Sorry, some old reminiscence :-)

Thanks for the review.

Best regards,

--
Gilles Darold
Consultant PostgreSQL
http://dalibo.com - http://dalibo.org

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#8Michael Paquier
michael.paquier@gmail.com
In reply to: Gilles Darold (#7)
3 attachment(s)
Re: Bug in pg_dump

On Tue, Feb 24, 2015 at 2:17 AM, Gilles Darold <gilles.darold@dalibo.com>
wrote:

Looks great to me, I have tested with the postgis_topology extension
everything works fine.

Actually, after looking more in depth at the internals of pg_dump I think
that both patches are wrong (did that yesterday night for another patch).
First this patch marks a table in an extension as always dumpable:
+      /* Mark member table as dumpable */
+      configtbl->dobj.dump = true;
And then many checks on ext_member are added in many code paths to ensure
that objects in extensions have their definition never dumped.
But actually this assumption is not true all the time per this code in
getExtensionMemberShip:
        if (!dopt->binary_upgrade)
            dobj->dump = false;
        else
            dobj->dump = refdobj->dump;
So this patch would break binary upgrade where some extension objects
should be dumped (one reason why I haven't noticed that before is that
pg_upgrade tests do not include extensions).

Hence, one idea coming to my mind to fix the problem would be to add some
extra processing directly in getExtensionMembership() after building the
objects DO_TABLE_DATA with makeTableDataInfo() by checking the FK
dependencies and add a dependency link with addObjectDependency. The good
part with that is that even user tables that reference extension tables
with a FK can be restored correctly because their constraint is added
*after* loading the data. I noticed as well that with this patch the
--data-only mode was dumping tables in the correct order.

Speaking of which, patches implementing this idea are attached. The module
test case has been improved as well with a check using a table not in an
extension linked with a FK to a table in an extension.
--
Michael

Attachments:

0001-Fix-ordering-of-tables-part-of-extensions-linked-wit.patchtext/x-diff; charset=US-ASCII; name=0001-Fix-ordering-of-tables-part-of-extensions-linked-wit.patchDownload
From 7fb84d68df023d913c448f2498987ca4f0a70595 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Tue, 17 Feb 2015 07:39:23 +0000
Subject: [PATCH 1/3] Fix ordering of tables part of extensions linked with
 constraints

Additional checks on FK constraints potentially linking between them
extension objects are done and data dump ordering is ensured. Note
that this does not take into account foreign keys of tables that are
not part of an extension linking to an extension table.
---
 src/bin/pg_dump/pg_dump.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 55 insertions(+), 1 deletion(-)

diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 2b53c72..5b1b240 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -15236,7 +15236,8 @@ dumpRule(Archive *fout, DumpOptions *dopt, RuleInfo *rinfo)
 }
 
 /*
- * getExtensionMembership --- obtain extension membership data
+ * getExtensionMembership --- obtain extension membership data and check FK
+ * dependencies among extension tables.
  */
 void
 getExtensionMembership(Archive *fout, DumpOptions *dopt, ExtensionInfo extinfo[],
@@ -15423,6 +15424,59 @@ getExtensionMembership(Archive *fout, DumpOptions *dopt, ExtensionInfo extinfo[]
 					}
 				}
 			}
+
+			/*
+			 * Now that all the TableInfoData objects have been created for
+			 * all the extensions, check their FK dependencies and register
+			 * them to ensure correct data ordering.  Note that this is not
+			 * a problem for user tables not included in an extension
+			 * referencing with a FK tables in extensions as their constraint
+			 * is declared after dumping their data. In --data-only mode the
+			 * table ordering is ensured as well thanks to
+			 * getTableDataFKConstraints().
+			 */
+			for (j = 0; j < nconfigitems; j++)
+			{
+				int			i_confrelid, k;
+				PQExpBuffer query2 = createPQExpBuffer();
+				TableInfo  *configtbl;
+				Oid			configtbloid = atooid(extconfigarray[j]);
+
+				configtbl = findTableByOid(configtbloid);
+				if (configtbl == NULL || configtbl->dataObj == NULL)
+					continue;
+
+				appendPQExpBuffer(query2,
+					  "SELECT confrelid "
+					  "FROM pg_catalog.pg_constraint "
+					  "WHERE conrelid = '%u' "
+					  "AND contype = 'f'",
+					  configtbloid);
+
+				res = ExecuteSqlQuery(fout, query2->data, PGRES_TUPLES_OK);
+				ntups = PQntuples(res);
+				i_confrelid = PQfnumber(res, "confrelid");
+
+				for (k = 0; k < ntups; k++)
+				{
+					Oid			confrelid;
+					TableInfo  *reftable;
+
+					confrelid = atooid(PQgetvalue(res, k, i_confrelid));
+					reftable = findTableByOid(confrelid);
+
+					if (reftable == NULL || reftable->dataObj == NULL)
+						continue;
+
+					/*
+					 * Make referencing TABLE_DATA object depend on the
+					 * referenced table's TABLE_DATA object.
+					 */
+					addObjectDependency(&configtbl->dataObj->dobj,
+										reftable->dataObj->dobj.dumpId);
+				}
+				resetPQExpBuffer(query2);
+			}
 		}
 		if (extconfigarray)
 			free(extconfigarray);
-- 
2.3.0

0002-Make-prove_check-install-contents-of-current-directo.patchtext/x-diff; charset=US-ASCII; name=0002-Make-prove_check-install-contents-of-current-directo.patchDownload
From ad5cd360243b44e735195c5e94df3c21e8f18e07 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Tue, 17 Feb 2015 22:29:28 +0900
Subject: [PATCH 2/3] Make prove_check install contents of current directory as
 well

This is useful for example for TAP tests in extensions.
---
 src/Makefile.global.in | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 7c39d82..7c15423 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -323,6 +323,7 @@ endef
 define prove_check
 $(MKDIR_P) tmp_check/log
 $(MAKE) -C $(top_builddir) DESTDIR='$(CURDIR)'/tmp_check/install install >'$(CURDIR)'/tmp_check/log/install.log 2>&1
+$(MAKE) -C $(CURDIR) DESTDIR='$(CURDIR)'/tmp_check/install install >>'$(CURDIR)'/tmp_check/log/install.log 2>&1
 cd $(srcdir) && TESTDIR='$(CURDIR)' PATH="$(CURDIR)/tmp_check/install$(bindir):$$PATH" $(call add_to_path,$(ld_library_path_var),$(CURDIR)/tmp_check/install$(libdir)) top_builddir='$(CURDIR)/$(top_builddir)' PGPORT='6$(DEF_PGPORT)' $(PROVE) $(PG_PROVE_FLAGS) $(PROVE_FLAGS) t/*.pl
 endef
 
-- 
2.3.0

0003-Add-dump_test-test-module-to-check-pg_dump-with-exte.patchtext/x-diff; charset=US-ASCII; name=0003-Add-dump_test-test-module-to-check-pg_dump-with-exte.patchDownload
From 5c7eeb2b1ca1afcf4630a5128979d2d87af19721 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Tue, 17 Feb 2015 22:36:49 +0900
Subject: [PATCH 3/3] Add dump_test, test module to check pg_dump with
 extensions

It works with TAP tests.
---
 src/test/modules/Makefile                     |  1 +
 src/test/modules/dump_test/.gitignore         |  4 ++++
 src/test/modules/dump_test/Makefile           | 22 ++++++++++++++++++++++
 src/test/modules/dump_test/README             |  5 +++++
 src/test/modules/dump_test/dump_test--1.0.sql | 20 ++++++++++++++++++++
 src/test/modules/dump_test/dump_test.control  |  5 +++++
 src/test/modules/dump_test/t/001_dump_test.pl | 27 +++++++++++++++++++++++++++
 7 files changed, 84 insertions(+)
 create mode 100644 src/test/modules/dump_test/.gitignore
 create mode 100644 src/test/modules/dump_test/Makefile
 create mode 100644 src/test/modules/dump_test/README
 create mode 100644 src/test/modules/dump_test/dump_test--1.0.sql
 create mode 100644 src/test/modules/dump_test/dump_test.control
 create mode 100644 src/test/modules/dump_test/t/001_dump_test.pl

diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 93d93af..6ad3b4e 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -6,6 +6,7 @@ include $(top_builddir)/src/Makefile.global
 
 SUBDIRS = \
 		  commit_ts \
+		  dump_test \
 		  worker_spi \
 		  dummy_seclabel \
 		  test_shm_mq \
diff --git a/src/test/modules/dump_test/.gitignore b/src/test/modules/dump_test/.gitignore
new file mode 100644
index 0000000..5dcb3ff
--- /dev/null
+++ b/src/test/modules/dump_test/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/dump_test/Makefile b/src/test/modules/dump_test/Makefile
new file mode 100644
index 0000000..08cd215
--- /dev/null
+++ b/src/test/modules/dump_test/Makefile
@@ -0,0 +1,22 @@
+# src/test/modules/dump_test/Makefile
+
+EXTENSION = dump_test
+DATA = dump_test--1.0.sql
+PGFILEDESC = "dump_test - test pg_dump with extensions"
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/dump_test
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check: all
+	$(prove_check)
+
+installcheck: install
+	$(prove_installcheck)
diff --git a/src/test/modules/dump_test/README b/src/test/modules/dump_test/README
new file mode 100644
index 0000000..9ebfa19
--- /dev/null
+++ b/src/test/modules/dump_test/README
@@ -0,0 +1,5 @@
+dump_test
+=========
+
+Simple module used to test consistency of pg_dump with objects in
+extensions.
diff --git a/src/test/modules/dump_test/dump_test--1.0.sql b/src/test/modules/dump_test/dump_test--1.0.sql
new file mode 100644
index 0000000..d684855
--- /dev/null
+++ b/src/test/modules/dump_test/dump_test--1.0.sql
@@ -0,0 +1,20 @@
+/* src/test/modules/dump_test/dump_test--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION dump_test" to load this file. \quit
+
+CREATE TABLE IF NOT EXISTS cc_tab_fkey (
+	id int PRIMARY KEY
+);
+
+CREATE TABLE IF NOT EXISTS bb_tab_fkey (
+	id int PRIMARY KEY REFERENCES cc_tab_fkey(id)
+);
+
+CREATE TABLE IF NOT EXISTS aa_tab_fkey (
+	id int REFERENCES bb_tab_fkey(id)
+);
+
+SELECT pg_catalog.pg_extension_config_dump('aa_tab_fkey', '');
+SELECT pg_catalog.pg_extension_config_dump('bb_tab_fkey', '');
+SELECT pg_catalog.pg_extension_config_dump('cc_tab_fkey', '');
diff --git a/src/test/modules/dump_test/dump_test.control b/src/test/modules/dump_test/dump_test.control
new file mode 100644
index 0000000..b32c99c
--- /dev/null
+++ b/src/test/modules/dump_test/dump_test.control
@@ -0,0 +1,5 @@
+# dump_test extension
+comment = 'dump_test - test pg_dump with extensions'
+default_version = '1.0'
+module_pathname = '$libdir/dump_test'
+relocatable = true
diff --git a/src/test/modules/dump_test/t/001_dump_test.pl b/src/test/modules/dump_test/t/001_dump_test.pl
new file mode 100644
index 0000000..18a1135
--- /dev/null
+++ b/src/test/modules/dump_test/t/001_dump_test.pl
@@ -0,0 +1,27 @@
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 3;
+
+my $tempdir = tempdir;
+
+start_test_server $tempdir;
+
+psql 'postgres', 'CREATE EXTENSION dump_test';
+
+# Insert some data before running the dump, this is needed to check
+# consistent data dump of tables with foreign key dependencies
+psql 'postgres', 'INSERT INTO cc_tab_fkey VALUES (1)';
+psql 'postgres', 'INSERT INTO bb_tab_fkey VALUES (1)';
+psql 'postgres', 'INSERT INTO aa_tab_fkey VALUES (1)';
+
+# Create a table depending on a FK defined in the extension
+psql 'postgres', 'CREATE TABLE dd_tab_fkey (id int REFERENCES bb_tab_fkey(id))';
+psql 'postgres', 'INSERT INTO dd_tab_fkey VALUES (1)';
+
+# Take a dump then re-deploy it to a new database
+command_ok(['pg_dump', '-d', 'postgres', '-f', "$tempdir/dump.sql"],
+		   'taken dump with installed extension');
+command_ok(['createdb', 'foobar1'], 'created new database to redeploy dump');
+command_ok(['psql', '--set=ON_ERROR_STOP=on', '-d', 'foobar1', '-f',
+			"$tempdir/dump.sql"], 'dump restored');
-- 
2.3.0

#9Gilles Darold
gilles.darold@dalibo.com
In reply to: Michael Paquier (#8)
Re: Bug in pg_dump

Le 24/02/2015 05:40, Michael Paquier a �crit :

On Tue, Feb 24, 2015 at 2:17 AM, Gilles Darold
<gilles.darold@dalibo.com <mailto:gilles.darold@dalibo.com>> wrote:

Looks great to me, I have tested with the postgis_topology extension
everything works fine.

Actually, after looking more in depth at the internals of pg_dump I
think that both patches are wrong (did that yesterday night for
another patch). First this patch marks a table in an extension as
always dumpable:
+      /* Mark member table as dumpable */
+      configtbl->dobj.dump = true;
And then many checks on ext_member are added in many code paths to
ensure that objects in extensions have their definition never dumped.
But actually this assumption is not true all the time per this code in
getExtensionMemberShip:
if (!dopt->binary_upgrade)
dobj->dump = false;
else
dobj->dump = refdobj->dump;
So this patch would break binary upgrade where some extension objects
should be dumped (one reason why I haven't noticed that before is that
pg_upgrade tests do not include extensions).

Hence, one idea coming to my mind to fix the problem would be to add
some extra processing directly in getExtensionMembership() after
building the objects DO_TABLE_DATA with makeTableDataInfo() by
checking the FK dependencies and add a dependency link with
addObjectDependency. The good part with that is that even user tables
that reference extension tables with a FK can be restored correctly
because their constraint is added *after* loading the data. I noticed
as well that with this patch the --data-only mode was dumping tables
in the correct order.

Speaking of which, patches implementing this idea are attached. The
module test case has been improved as well with a check using a table
not in an extension linked with a FK to a table in an extension.
--
Michael

This is a far better patch and the test to export/import of the
postgis_topology extension works great for me.

Thanks for the work.

--
Gilles Darold
Consultant PostgreSQL
http://dalibo.com - http://dalibo.org

#10Michael Paquier
michael.paquier@gmail.com
In reply to: Gilles Darold (#9)
3 attachment(s)
Re: Bug in pg_dump

On Wed, Feb 25, 2015 at 7:46 AM, Gilles Darold <gilles.darold@dalibo.com> wrote:

This is a far better patch and the test to export/import of the
postgis_topology extension works great for me.

Thanks for the work.

Attached is a patch that uses an even better approach by querying only
once all the FK dependencies of tables in extensions whose data is
registered as dumpable by getExtensionMembership(). Could you check
that it works correctly? My test case passes but an extra check would
be a good nice. The patch footprint is pretty low so we may be able to
backport this patch easily.
--
Michael

Attachments:

0001-Fix-ordering-of-tables-part-of-extensions-linked-wit.patchtext/x-diff; charset=US-ASCII; name=0001-Fix-ordering-of-tables-part-of-extensions-linked-wit.patchDownload
From 72e369ee725301003cf065b0bf9875724455dac1 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Tue, 17 Feb 2015 07:39:23 +0000
Subject: [PATCH 1/3] Fix ordering of tables part of extensions linked with
 constraints

Additional checks on FK constraints potentially linking between them
extension objects are done and data dump ordering is ensured. Note
that this does not take into account foreign keys of tables that are
not part of an extension linking to an extension table.
---
 src/bin/pg_dump/pg_dump.c | 80 +++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 78 insertions(+), 2 deletions(-)

diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 2b53c72..bbcd600 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -15236,7 +15236,8 @@ dumpRule(Archive *fout, DumpOptions *dopt, RuleInfo *rinfo)
 }
 
 /*
- * getExtensionMembership --- obtain extension membership data
+ * getExtensionMembership --- obtain extension membership data and check FK
+ * dependencies among extension tables.
  */
 void
 getExtensionMembership(Archive *fout, DumpOptions *dopt, ExtensionInfo extinfo[],
@@ -15368,7 +15369,9 @@ getExtensionMembership(Archive *fout, DumpOptions *dopt, ExtensionInfo extinfo[]
 		  parsePGArray(extcondition, &extconditionarray, &nconditionitems) &&
 			nconfigitems == nconditionitems)
 		{
-			int			j;
+			int			j, i_conrelid, i_confrelid;
+			PQExpBuffer query2;
+			bool		first_elt = true;
 
 			for (j = 0; j < nconfigitems; j++)
 			{
@@ -15423,6 +15426,79 @@ getExtensionMembership(Archive *fout, DumpOptions *dopt, ExtensionInfo extinfo[]
 					}
 				}
 			}
+
+			/*
+			 * Now that all the TableInfoData objects have been created for
+			 * all the extensions, check their FK dependencies and register
+			 * them to ensure correct data ordering.  Note that this is not
+			 * a problem for user tables not included in an extension
+			 * referencing with a FK tables in extensions as their constraint
+			 * is declared after dumping their data. In --data-only mode the
+			 * table ordering is ensured as well thanks to
+			 * getTableDataFKConstraints().
+			 */
+			query2 = createPQExpBuffer();
+
+			/*
+			 * Query all the foreign key dependencies for all the extension
+			 * tables found previously. Only tables whose data need to be
+			 * have to be tracked.
+			 */
+			appendPQExpBuffer(query2,
+					  "SELECT conrelid, confrelid "
+					  "FROM pg_catalog.pg_constraint "
+					  "WHERE contype = 'f' AND conrelid IN (");
+
+			for (j = 0; j < nconfigitems; j++)
+			{
+				TableInfo  *configtbl;
+				Oid			configtbloid = atooid(extconfigarray[j]);
+
+				configtbl = findTableByOid(configtbloid);
+				if (configtbl == NULL || configtbl->dataObj == NULL)
+					continue;
+
+				if (first_elt)
+				{
+					appendPQExpBuffer(query2, "%u", configtbloid);
+					first_elt = false;
+				}
+				else
+					appendPQExpBuffer(query2, ", %u", configtbloid);
+			}
+			appendPQExpBuffer(query2, ");");
+
+			res = ExecuteSqlQuery(fout, query2->data, PGRES_TUPLES_OK);
+			ntups = PQntuples(res);
+
+			i_conrelid = PQfnumber(res, "conrelid");
+			i_confrelid = PQfnumber(res, "confrelid");
+
+			/* Now get the dependencies and register them */
+			for (j = 0; j < ntups; j++)
+			{
+				Oid			conrelid, confrelid;
+				TableInfo  *reftable, *contable;
+
+				conrelid = atooid(PQgetvalue(res, j, i_conrelid));
+				confrelid = atooid(PQgetvalue(res, j, i_confrelid));
+				contable = findTableByOid(conrelid);
+				reftable = findTableByOid(confrelid);
+
+				if (reftable == NULL ||
+					reftable->dataObj == NULL ||
+					contable == NULL ||
+					contable->dataObj == NULL)
+					continue;
+
+				/*
+				 * Make referencing TABLE_DATA object depend on the
+				 * referenced table's TABLE_DATA object.
+				 */
+				addObjectDependency(&contable->dataObj->dobj,
+									reftable->dataObj->dobj.dumpId);
+			}
+			resetPQExpBuffer(query2);
 		}
 		if (extconfigarray)
 			free(extconfigarray);
-- 
2.3.0

0002-Make-prove_check-install-contents-of-current-directo.patchtext/x-diff; charset=US-ASCII; name=0002-Make-prove_check-install-contents-of-current-directo.patchDownload
From 1d8fe1c39ec5049c39810f94153d347971738616 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Tue, 17 Feb 2015 22:29:28 +0900
Subject: [PATCH 2/3] Make prove_check install contents of current directory as
 well

This is useful for example for TAP tests in extensions.
---
 src/Makefile.global.in | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 7c39d82..7c15423 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -323,6 +323,7 @@ endef
 define prove_check
 $(MKDIR_P) tmp_check/log
 $(MAKE) -C $(top_builddir) DESTDIR='$(CURDIR)'/tmp_check/install install >'$(CURDIR)'/tmp_check/log/install.log 2>&1
+$(MAKE) -C $(CURDIR) DESTDIR='$(CURDIR)'/tmp_check/install install >>'$(CURDIR)'/tmp_check/log/install.log 2>&1
 cd $(srcdir) && TESTDIR='$(CURDIR)' PATH="$(CURDIR)/tmp_check/install$(bindir):$$PATH" $(call add_to_path,$(ld_library_path_var),$(CURDIR)/tmp_check/install$(libdir)) top_builddir='$(CURDIR)/$(top_builddir)' PGPORT='6$(DEF_PGPORT)' $(PROVE) $(PG_PROVE_FLAGS) $(PROVE_FLAGS) t/*.pl
 endef
 
-- 
2.3.0

0003-Add-dump_test-test-module-to-check-pg_dump-with-exte.patchtext/x-diff; charset=US-ASCII; name=0003-Add-dump_test-test-module-to-check-pg_dump-with-exte.patchDownload
From 3925dfac6745c7a3141901739d142678c45ce48e Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Tue, 17 Feb 2015 22:36:49 +0900
Subject: [PATCH 3/3] Add dump_test, test module to check pg_dump with
 extensions

It works with TAP tests.
---
 src/test/modules/Makefile                     |  1 +
 src/test/modules/dump_test/.gitignore         |  4 ++++
 src/test/modules/dump_test/Makefile           | 22 ++++++++++++++++++++++
 src/test/modules/dump_test/README             |  5 +++++
 src/test/modules/dump_test/dump_test--1.0.sql | 20 ++++++++++++++++++++
 src/test/modules/dump_test/dump_test.control  |  5 +++++
 src/test/modules/dump_test/t/001_dump_test.pl | 27 +++++++++++++++++++++++++++
 7 files changed, 84 insertions(+)
 create mode 100644 src/test/modules/dump_test/.gitignore
 create mode 100644 src/test/modules/dump_test/Makefile
 create mode 100644 src/test/modules/dump_test/README
 create mode 100644 src/test/modules/dump_test/dump_test--1.0.sql
 create mode 100644 src/test/modules/dump_test/dump_test.control
 create mode 100644 src/test/modules/dump_test/t/001_dump_test.pl

diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 93d93af..6ad3b4e 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -6,6 +6,7 @@ include $(top_builddir)/src/Makefile.global
 
 SUBDIRS = \
 		  commit_ts \
+		  dump_test \
 		  worker_spi \
 		  dummy_seclabel \
 		  test_shm_mq \
diff --git a/src/test/modules/dump_test/.gitignore b/src/test/modules/dump_test/.gitignore
new file mode 100644
index 0000000..5dcb3ff
--- /dev/null
+++ b/src/test/modules/dump_test/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/dump_test/Makefile b/src/test/modules/dump_test/Makefile
new file mode 100644
index 0000000..08cd215
--- /dev/null
+++ b/src/test/modules/dump_test/Makefile
@@ -0,0 +1,22 @@
+# src/test/modules/dump_test/Makefile
+
+EXTENSION = dump_test
+DATA = dump_test--1.0.sql
+PGFILEDESC = "dump_test - test pg_dump with extensions"
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/dump_test
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check: all
+	$(prove_check)
+
+installcheck: install
+	$(prove_installcheck)
diff --git a/src/test/modules/dump_test/README b/src/test/modules/dump_test/README
new file mode 100644
index 0000000..9ebfa19
--- /dev/null
+++ b/src/test/modules/dump_test/README
@@ -0,0 +1,5 @@
+dump_test
+=========
+
+Simple module used to test consistency of pg_dump with objects in
+extensions.
diff --git a/src/test/modules/dump_test/dump_test--1.0.sql b/src/test/modules/dump_test/dump_test--1.0.sql
new file mode 100644
index 0000000..d684855
--- /dev/null
+++ b/src/test/modules/dump_test/dump_test--1.0.sql
@@ -0,0 +1,20 @@
+/* src/test/modules/dump_test/dump_test--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION dump_test" to load this file. \quit
+
+CREATE TABLE IF NOT EXISTS cc_tab_fkey (
+	id int PRIMARY KEY
+);
+
+CREATE TABLE IF NOT EXISTS bb_tab_fkey (
+	id int PRIMARY KEY REFERENCES cc_tab_fkey(id)
+);
+
+CREATE TABLE IF NOT EXISTS aa_tab_fkey (
+	id int REFERENCES bb_tab_fkey(id)
+);
+
+SELECT pg_catalog.pg_extension_config_dump('aa_tab_fkey', '');
+SELECT pg_catalog.pg_extension_config_dump('bb_tab_fkey', '');
+SELECT pg_catalog.pg_extension_config_dump('cc_tab_fkey', '');
diff --git a/src/test/modules/dump_test/dump_test.control b/src/test/modules/dump_test/dump_test.control
new file mode 100644
index 0000000..b32c99c
--- /dev/null
+++ b/src/test/modules/dump_test/dump_test.control
@@ -0,0 +1,5 @@
+# dump_test extension
+comment = 'dump_test - test pg_dump with extensions'
+default_version = '1.0'
+module_pathname = '$libdir/dump_test'
+relocatable = true
diff --git a/src/test/modules/dump_test/t/001_dump_test.pl b/src/test/modules/dump_test/t/001_dump_test.pl
new file mode 100644
index 0000000..18a1135
--- /dev/null
+++ b/src/test/modules/dump_test/t/001_dump_test.pl
@@ -0,0 +1,27 @@
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 3;
+
+my $tempdir = tempdir;
+
+start_test_server $tempdir;
+
+psql 'postgres', 'CREATE EXTENSION dump_test';
+
+# Insert some data before running the dump, this is needed to check
+# consistent data dump of tables with foreign key dependencies
+psql 'postgres', 'INSERT INTO cc_tab_fkey VALUES (1)';
+psql 'postgres', 'INSERT INTO bb_tab_fkey VALUES (1)';
+psql 'postgres', 'INSERT INTO aa_tab_fkey VALUES (1)';
+
+# Create a table depending on a FK defined in the extension
+psql 'postgres', 'CREATE TABLE dd_tab_fkey (id int REFERENCES bb_tab_fkey(id))';
+psql 'postgres', 'INSERT INTO dd_tab_fkey VALUES (1)';
+
+# Take a dump then re-deploy it to a new database
+command_ok(['pg_dump', '-d', 'postgres', '-f', "$tempdir/dump.sql"],
+		   'taken dump with installed extension');
+command_ok(['createdb', 'foobar1'], 'created new database to redeploy dump');
+command_ok(['psql', '--set=ON_ERROR_STOP=on', '-d', 'foobar1', '-f',
+			"$tempdir/dump.sql"], 'dump restored');
-- 
2.3.0

#11Gilles Darold
gilles.darold@dalibo.com
In reply to: Michael Paquier (#10)
Re: Bug in pg_dump

Le 26/02/2015 12:41, Michael Paquier a �crit :

On Wed, Feb 25, 2015 at 7:46 AM, Gilles Darold <gilles.darold@dalibo.com> wrote:

This is a far better patch and the test to export/import of the
postgis_topology extension works great for me.

Thanks for the work.

Attached is a patch that uses an even better approach by querying only
once all the FK dependencies of tables in extensions whose data is
registered as dumpable by getExtensionMembership(). Could you check
that it works correctly? My test case passes but an extra check would
be a good nice. The patch footprint is pretty low so we may be able to
backport this patch easily.

Works great too. I'm agree that this patch should be backported but I
think we need to warn admins in the release note that their
import/restore scripts may be broken.

One of the common workaround about this issue was to not take care of
the error during data import and then reload data from the tables with
FK errors at the end of the import. If the admins are not warned, this
can conduct to duplicate entries or return an error.

--
Gilles Darold
Consultant PostgreSQL
http://dalibo.com - http://dalibo.org

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#12David Fetter
david@fetter.org
In reply to: Michael Paquier (#10)
Re: Bug in pg_dump

On Thu, Feb 26, 2015 at 08:41:46PM +0900, Michael Paquier wrote:

On Wed, Feb 25, 2015 at 7:46 AM, Gilles Darold <gilles.darold@dalibo.com> wrote:

This is a far better patch and the test to export/import of the
postgis_topology extension works great for me.

Thanks for the work.

Attached is a patch that uses an even better approach by querying
only once all the FK dependencies of tables in extensions whose data
is registered as dumpable by getExtensionMembership(). Could you
check that it works correctly? My test case passes but an extra
check would be a good nice. The patch footprint is pretty low so we
may be able to backport this patch easily.

+1 for backporting. It's a real bug, and real people get hit by it if
they're using PostGIS, one of our most popular add-ons.

Cheers,
David.
--
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#13Michael Paquier
michael.paquier@gmail.com
In reply to: Gilles Darold (#11)
Re: Bug in pg_dump

On Sat, Feb 28, 2015 at 12:29 AM, Gilles Darold
<gilles.darold@dalibo.com> wrote:

Le 26/02/2015 12:41, Michael Paquier a écrit :

On Wed, Feb 25, 2015 at 7:46 AM, Gilles Darold <gilles.darold@dalibo.com> wrote:

This is a far better patch and the test to export/import of the
postgis_topology extension works great for me.

Thanks for the work.

Attached is a patch that uses an even better approach by querying only
once all the FK dependencies of tables in extensions whose data is
registered as dumpable by getExtensionMembership(). Could you check
that it works correctly? My test case passes but an extra check would
be a good nice. The patch footprint is pretty low so we may be able to
backport this patch easily.

Works great too. I'm agree that this patch should be backported but I
think we need to warn admins in the release note that their
import/restore scripts may be broken.

Of course it makes sense to mention that in the release notes, this
behavior of pg_dump being broken since the creation of extensions.
Since it is working on your side as well, let's put it in the hands of
a committer then and I am marking it as such on the CF app. The test
case I sent on this thread can be used to reproduce the problem easily
with TAP tests...
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#14Stephen Frost
sfrost@snowman.net
In reply to: Michael Paquier (#10)
Re: Bug in pg_dump

Michael, all,

* Michael Paquier (michael.paquier@gmail.com) wrote:

On Wed, Feb 25, 2015 at 7:46 AM, Gilles Darold <gilles.darold@dalibo.com> wrote:

This is a far better patch and the test to export/import of the
postgis_topology extension works great for me.

Thanks for the work.

Attached is a patch that uses an even better approach by querying only
once all the FK dependencies of tables in extensions whose data is
registered as dumpable by getExtensionMembership(). Could you check
that it works correctly? My test case passes but an extra check would
be a good nice. The patch footprint is pretty low so we may be able to
backport this patch easily.

I've started looking at this and it looks pretty simple and definitely
something to backpatch (and mention in the release notes that existing
pg_dump exports might be broken..).

One thing that might be missing is what Jim brought up though- that this
won't be able to deal with circular dependencies. I'm not sure that we
need to care, but I *do* think we should document that in the extension
documentation as unsupported. Perhaps in the future we can improve on
this situation by setting up to drop and recreate the constraints,
though another thought I had was to require extensions with circular
dependencies to use deferrable constraints and then make sure we set
constraints to deferred. That isn't something we'd want to backpatch
though, so my plan is to push forward with this.

Thanks!

Stephen

#15Stephen Frost
sfrost@snowman.net
In reply to: Michael Paquier (#10)
Re: Bug in pg_dump

Michael,

* Michael Paquier (michael.paquier@gmail.com) wrote:

+			/*
+			 * Query all the foreign key dependencies for all the extension
+			 * tables found previously. Only tables whose data need to be
+			 * have to be tracked.
+			 */
+			appendPQExpBuffer(query2,
+					  "SELECT conrelid, confrelid "
+					  "FROM pg_catalog.pg_constraint "
+					  "WHERE contype = 'f' AND conrelid IN (");
+
+			for (j = 0; j < nconfigitems; j++)
+			{

[...]

Instead of building up a (potentially) big list like this, couldn't we
simply join against pg_extension and check if conrelid = ANY (extconfig)
instead, based on the extension we're currently processing?

eg:

appendPQExpBuffer(query2,
"SELECT conrelid, confrelid "
"FROM pg_catalog.pg_constraint, pg_extension "
"WHERE contype = 'f' AND extname = '%s' AND conrelid = ANY (extconfig)",
fmtId(curext->dobj.name));

This seemed to work just fine for me, at least, and reduces the size of
the patch a bit further since we don't need the loop that builds up the
ids.

Subject: [PATCH 2/3] Make prove_check install contents of current directory as well

This is really an independent thing, no? I don't see any particular
problem with it, for my part.

Thanks!

Stephen

#16Michael Paquier
michael.paquier@gmail.com
In reply to: Stephen Frost (#15)
4 attachment(s)
Re: Bug in pg_dump

On Sun, Mar 1, 2015 at 1:09 AM, Stephen Frost <sfrost@snowman.net> wrote:

Michael,

* Michael Paquier (michael.paquier@gmail.com) wrote:

+                     /*
+                      * Query all the foreign key dependencies for all the extension
+                      * tables found previously. Only tables whose data need to be
+                      * have to be tracked.
+                      */
+                     appendPQExpBuffer(query2,
+                                       "SELECT conrelid, confrelid "
+                                       "FROM pg_catalog.pg_constraint "
+                                       "WHERE contype = 'f' AND conrelid IN (");
+
+                     for (j = 0; j < nconfigitems; j++)
+                     {

[...]

Instead of building up a (potentially) big list like this, couldn't we
simply join against pg_extension and check if conrelid = ANY (extconfig)
instead, based on the extension we're currently processing?

eg:

appendPQExpBuffer(query2,
"SELECT conrelid, confrelid "
"FROM pg_catalog.pg_constraint, pg_extension "
"WHERE contype = 'f' AND extname = '%s' AND conrelid = ANY (extconfig)",
fmtId(curext->dobj.name));

This seemed to work just fine for me, at least, and reduces the size of
the patch a bit further since we don't need the loop that builds up the
ids.

Actually, I did a mistake in my last patch, see this comment:
+      * Now that all the TableInfoData objects have been created for
+      * all the extensions, check their FK dependencies and register
+      * them to ensure correct data ordering.
The thing is that we must absolutely wait for *all* the TableInfoData
of all the extensions to be created because we need to mark the
dependencies between them, and even my last patch, or even with what
you are proposing we may miss tracking of FK dependencies between
tables in different extensions. This has little chance to happen in
practice, but I think that we should definitely get things right.
Hence something like this query able to query all the FK dependencies
with tables in extensions is more useful, and it has no IN clause:
+       appendPQExpBuffer(query,
+                       "SELECT conrelid, confrelid "
+                       "FROM pg_constraint "
+                       "LEFT JOIN pg_depend ON (objid = confrelid) "
+                       "WHERE contype = 'f' "
+                       "AND refclassid = 'pg_extension'::regclass "
+                       "AND classid = 'pg_class'::regclass;");

Subject: [PATCH 2/3] Make prove_check install contents of current directory as well

This is really an independent thing, no? I don't see any particular
problem with it, for my part.

Yes, that's an independent thing, but my guess is that it would be
good to have a real test case this time to be sure that this does not
break again, at least on master where test/modules is an ideal place.

Attached are updated patches, the fix of pg_dump can be easily
cherry-picked down to 9.2. For 9.1 things are changed a bit because of
the way SQL queries are run, still patches are attached for all the
versions needed. I tested as well the fix down to this version 9.1
using the test case dump_test.
Thanks,
--
Michael

Attachments:

0001-Fix-ordering-of-tables-part-of-extensions-linked-wit.patchapplication/x-patch; name=0001-Fix-ordering-of-tables-part-of-extensions-linked-wit.patchDownload
From 9ef14f2f67cbec9df2a1d30978efdeda059fa12f Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Fri, 27 Feb 2015 12:23:42 +0000
Subject: [PATCH 1/3] Fix ordering of tables part of extensions linked with
 constraints

Additional checks on FK constraints potentially linking between them
extension objects are done and data dump ordering is ensured.
---
 src/bin/pg_dump/pg_dump.c | 58 +++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 56 insertions(+), 2 deletions(-)

diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 2b53c72..e210f88 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -15236,7 +15236,8 @@ dumpRule(Archive *fout, DumpOptions *dopt, RuleInfo *rinfo)
 }
 
 /*
- * getExtensionMembership --- obtain extension membership data
+ * getExtensionMembership --- obtain extension membership data and check FK
+ * dependencies among relations interacting with the ones in extensions.
  */
 void
 getExtensionMembership(Archive *fout, DumpOptions *dopt, ExtensionInfo extinfo[],
@@ -15249,7 +15250,9 @@ getExtensionMembership(Archive *fout, DumpOptions *dopt, ExtensionInfo extinfo[]
 	int			i_classid,
 				i_objid,
 				i_refclassid,
-				i_refobjid;
+				i_refobjid,
+				i_conrelid,
+				i_confrelid;
 	DumpableObject *dobj,
 			   *refdobj;
 
@@ -15431,6 +15434,57 @@ getExtensionMembership(Archive *fout, DumpOptions *dopt, ExtensionInfo extinfo[]
 	}
 
 	destroyPQExpBuffer(query);
+
+	/*
+	 * Now that all the TableInfoData objects have been created for all
+	 * the extensions, check their FK dependencies and register them to
+	 * ensure correct data ordering.  Note that this is not a problem
+	 * for user tables not included in an extension referencing with a
+	 * FK tables in extensions as their constraint is declared after
+	 * dumping their data. In --data-only mode the table ordering is
+	 * ensured as well thanks to getTableDataFKConstraints().
+	 */
+	query = createPQExpBuffer();
+
+	appendPQExpBuffer(query,
+			"SELECT conrelid, confrelid "
+			"FROM pg_constraint "
+				"LEFT JOIN pg_depend ON (objid = confrelid) "
+			"WHERE contype = 'f' "
+			"AND refclassid = 'pg_extension'::regclass "
+			"AND classid = 'pg_class'::regclass;");
+
+	res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
+	ntups = PQntuples(res);
+
+	i_conrelid = PQfnumber(res, "conrelid");
+	i_confrelid = PQfnumber(res, "confrelid");
+
+	/* Now get the dependencies and register them */
+	for (i = 0; i < ntups; i++)
+	{
+		Oid			conrelid, confrelid;
+		TableInfo  *reftable, *contable;
+
+		conrelid = atooid(PQgetvalue(res, i, i_conrelid));
+		confrelid = atooid(PQgetvalue(res, i, i_confrelid));
+		contable = findTableByOid(conrelid);
+		reftable = findTableByOid(confrelid);
+
+		if (reftable == NULL ||
+			reftable->dataObj == NULL ||
+			contable == NULL ||
+			contable->dataObj == NULL)
+			continue;
+
+		/*
+		 * Make referencing TABLE_DATA object depend on the
+		 * referenced table's TABLE_DATA object.
+		 */
+		addObjectDependency(&contable->dataObj->dobj,
+							reftable->dataObj->dobj.dumpId);
+	}
+	resetPQExpBuffer(query);
 }
 
 /*
-- 
2.3.1

0002-Make-prove_check-install-contents-of-current-directo.patchapplication/x-patch; name=0002-Make-prove_check-install-contents-of-current-directo.patchDownload
From cd429895eab802310df5b2b9a14ca236c9f3f395 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Tue, 17 Feb 2015 22:29:28 +0900
Subject: [PATCH 2/3] Make prove_check install contents of current directory as
 well

This is useful for example for TAP tests in extensions.
---
 src/Makefile.global.in | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 7c39d82..7c15423 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -323,6 +323,7 @@ endef
 define prove_check
 $(MKDIR_P) tmp_check/log
 $(MAKE) -C $(top_builddir) DESTDIR='$(CURDIR)'/tmp_check/install install >'$(CURDIR)'/tmp_check/log/install.log 2>&1
+$(MAKE) -C $(CURDIR) DESTDIR='$(CURDIR)'/tmp_check/install install >>'$(CURDIR)'/tmp_check/log/install.log 2>&1
 cd $(srcdir) && TESTDIR='$(CURDIR)' PATH="$(CURDIR)/tmp_check/install$(bindir):$$PATH" $(call add_to_path,$(ld_library_path_var),$(CURDIR)/tmp_check/install$(libdir)) top_builddir='$(CURDIR)/$(top_builddir)' PGPORT='6$(DEF_PGPORT)' $(PROVE) $(PG_PROVE_FLAGS) $(PROVE_FLAGS) t/*.pl
 endef
 
-- 
2.3.1

0003-Add-dump_test-test-module-to-check-pg_dump-with-exte.patchapplication/x-patch; name=0003-Add-dump_test-test-module-to-check-pg_dump-with-exte.patchDownload
From e0dd2f7dffe8f3c1a551130c9b44ec6e79f11190 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@otacoo.com>
Date: Tue, 17 Feb 2015 22:36:49 +0900
Subject: [PATCH 3/3] Add dump_test, test module to check pg_dump with
 extensions

It works with TAP tests.
---
 src/test/modules/Makefile                     |  1 +
 src/test/modules/dump_test/.gitignore         |  4 ++++
 src/test/modules/dump_test/Makefile           | 22 ++++++++++++++++++++++
 src/test/modules/dump_test/README             |  5 +++++
 src/test/modules/dump_test/dump_test--1.0.sql | 20 ++++++++++++++++++++
 src/test/modules/dump_test/dump_test.control  |  5 +++++
 src/test/modules/dump_test/t/001_dump_test.pl | 27 +++++++++++++++++++++++++++
 7 files changed, 84 insertions(+)
 create mode 100644 src/test/modules/dump_test/.gitignore
 create mode 100644 src/test/modules/dump_test/Makefile
 create mode 100644 src/test/modules/dump_test/README
 create mode 100644 src/test/modules/dump_test/dump_test--1.0.sql
 create mode 100644 src/test/modules/dump_test/dump_test.control
 create mode 100644 src/test/modules/dump_test/t/001_dump_test.pl

diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 93d93af..6ad3b4e 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -6,6 +6,7 @@ include $(top_builddir)/src/Makefile.global
 
 SUBDIRS = \
 		  commit_ts \
+		  dump_test \
 		  worker_spi \
 		  dummy_seclabel \
 		  test_shm_mq \
diff --git a/src/test/modules/dump_test/.gitignore b/src/test/modules/dump_test/.gitignore
new file mode 100644
index 0000000..5dcb3ff
--- /dev/null
+++ b/src/test/modules/dump_test/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/dump_test/Makefile b/src/test/modules/dump_test/Makefile
new file mode 100644
index 0000000..08cd215
--- /dev/null
+++ b/src/test/modules/dump_test/Makefile
@@ -0,0 +1,22 @@
+# src/test/modules/dump_test/Makefile
+
+EXTENSION = dump_test
+DATA = dump_test--1.0.sql
+PGFILEDESC = "dump_test - test pg_dump with extensions"
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/dump_test
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check: all
+	$(prove_check)
+
+installcheck: install
+	$(prove_installcheck)
diff --git a/src/test/modules/dump_test/README b/src/test/modules/dump_test/README
new file mode 100644
index 0000000..9ebfa19
--- /dev/null
+++ b/src/test/modules/dump_test/README
@@ -0,0 +1,5 @@
+dump_test
+=========
+
+Simple module used to test consistency of pg_dump with objects in
+extensions.
diff --git a/src/test/modules/dump_test/dump_test--1.0.sql b/src/test/modules/dump_test/dump_test--1.0.sql
new file mode 100644
index 0000000..d684855
--- /dev/null
+++ b/src/test/modules/dump_test/dump_test--1.0.sql
@@ -0,0 +1,20 @@
+/* src/test/modules/dump_test/dump_test--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION dump_test" to load this file. \quit
+
+CREATE TABLE IF NOT EXISTS cc_tab_fkey (
+	id int PRIMARY KEY
+);
+
+CREATE TABLE IF NOT EXISTS bb_tab_fkey (
+	id int PRIMARY KEY REFERENCES cc_tab_fkey(id)
+);
+
+CREATE TABLE IF NOT EXISTS aa_tab_fkey (
+	id int REFERENCES bb_tab_fkey(id)
+);
+
+SELECT pg_catalog.pg_extension_config_dump('aa_tab_fkey', '');
+SELECT pg_catalog.pg_extension_config_dump('bb_tab_fkey', '');
+SELECT pg_catalog.pg_extension_config_dump('cc_tab_fkey', '');
diff --git a/src/test/modules/dump_test/dump_test.control b/src/test/modules/dump_test/dump_test.control
new file mode 100644
index 0000000..b32c99c
--- /dev/null
+++ b/src/test/modules/dump_test/dump_test.control
@@ -0,0 +1,5 @@
+# dump_test extension
+comment = 'dump_test - test pg_dump with extensions'
+default_version = '1.0'
+module_pathname = '$libdir/dump_test'
+relocatable = true
diff --git a/src/test/modules/dump_test/t/001_dump_test.pl b/src/test/modules/dump_test/t/001_dump_test.pl
new file mode 100644
index 0000000..18a1135
--- /dev/null
+++ b/src/test/modules/dump_test/t/001_dump_test.pl
@@ -0,0 +1,27 @@
+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 3;
+
+my $tempdir = tempdir;
+
+start_test_server $tempdir;
+
+psql 'postgres', 'CREATE EXTENSION dump_test';
+
+# Insert some data before running the dump, this is needed to check
+# consistent data dump of tables with foreign key dependencies
+psql 'postgres', 'INSERT INTO cc_tab_fkey VALUES (1)';
+psql 'postgres', 'INSERT INTO bb_tab_fkey VALUES (1)';
+psql 'postgres', 'INSERT INTO aa_tab_fkey VALUES (1)';
+
+# Create a table depending on a FK defined in the extension
+psql 'postgres', 'CREATE TABLE dd_tab_fkey (id int REFERENCES bb_tab_fkey(id))';
+psql 'postgres', 'INSERT INTO dd_tab_fkey VALUES (1)';
+
+# Take a dump then re-deploy it to a new database
+command_ok(['pg_dump', '-d', 'postgres', '-f', "$tempdir/dump.sql"],
+		   'taken dump with installed extension');
+command_ok(['createdb', 'foobar1'], 'created new database to redeploy dump');
+command_ok(['psql', '--set=ON_ERROR_STOP=on', '-d', 'foobar1', '-f',
+			"$tempdir/dump.sql"], 'dump restored');
-- 
2.3.1

20150301_pgdump_extension_fk_91.patchtext/x-patch; charset=US-ASCII; name=20150301_pgdump_extension_fk_91.patchDownload
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index fd47059..7e17df0 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13738,7 +13738,8 @@ dumpRule(Archive *fout, RuleInfo *rinfo)
 }
 
 /*
- * getExtensionMembership --- obtain extension membership data
+ * getExtensionMembership --- obtain extension membership data and check FK
+ * dependencies among relations interacting with the ones in extensions.
  */
 void
 getExtensionMembership(ExtensionInfo extinfo[], int numExtensions)
@@ -13750,7 +13751,9 @@ getExtensionMembership(ExtensionInfo extinfo[], int numExtensions)
 	int			i_classid,
 				i_objid,
 				i_refclassid,
-				i_refobjid;
+				i_refobjid,
+				i_conrelid,
+				i_confrelid;
 	DumpableObject *dobj,
 			   *refdobj;
 
@@ -13930,6 +13933,59 @@ getExtensionMembership(ExtensionInfo extinfo[], int numExtensions)
 	}
 
 	destroyPQExpBuffer(query);
+
+	/*
+	 * Now that all the TableInfoData objects have been created for all
+	 * the extensions, check their FK dependencies and register them to
+	 * ensure correct data ordering.  Note that this is not a problem
+	 * for user tables not included in an extension referencing with a
+	 * FK tables in extensions as their constraint is declared after
+	 * dumping their data. In --data-only mode the table ordering is
+	 * ensured as well thanks to getTableDataFKConstraints().
+	 */
+	query = createPQExpBuffer();
+
+	appendPQExpBuffer(query,
+			"SELECT conrelid, confrelid "
+			"FROM pg_constraint "
+				"LEFT JOIN pg_depend ON (objid = confrelid) "
+			"WHERE contype = 'f' "
+			"AND refclassid = 'pg_extension'::regclass "
+			"AND classid = 'pg_class'::regclass;");
+
+	res = PQexec(g_conn, query->data);
+	check_sql_result(res, g_conn, query->data, PGRES_TUPLES_OK);
+
+	ntups = PQntuples(res);
+
+	i_conrelid = PQfnumber(res, "conrelid");
+	i_confrelid = PQfnumber(res, "confrelid");
+
+	/* Now get the dependencies and register them */
+	for (i = 0; i < ntups; i++)
+	{
+		Oid			conrelid, confrelid;
+		TableInfo  *reftable, *contable;
+
+		conrelid = atooid(PQgetvalue(res, i, i_conrelid));
+		confrelid = atooid(PQgetvalue(res, i, i_confrelid));
+		contable = findTableByOid(conrelid);
+		reftable = findTableByOid(confrelid);
+
+		if (reftable == NULL ||
+			reftable->dataObj == NULL ||
+			contable == NULL ||
+			contable->dataObj == NULL)
+			continue;
+
+		/*
+		 * Make referencing TABLE_DATA object depend on the
+		 * referenced table's TABLE_DATA object.
+		 */
+		addObjectDependency(&contable->dataObj->dobj,
+							reftable->dataObj->dobj.dumpId);
+	}
+	resetPQExpBuffer(query);
 }
 
 /*
#17Stephen Frost
sfrost@snowman.net
In reply to: Michael Paquier (#16)
Re: Bug in pg_dump

Michael,

* Michael Paquier (michael.paquier@gmail.com) wrote:

The thing is that we must absolutely wait for *all* the TableInfoData
of all the extensions to be created because we need to mark the
dependencies between them, and even my last patch, or even with what
you are proposing we may miss tracking of FK dependencies between
tables in different extensions. This has little chance to happen in
practice, but I think that we should definitely get things right.
Hence something like this query able to query all the FK dependencies
with tables in extensions is more useful, and it has no IN clause:

Ah, yes, extensions can depend on each other and so this could
definitely happen. The current issue is around postgis, which by itself
provides three different extensions.

+       appendPQExpBuffer(query,
+                       "SELECT conrelid, confrelid "
+                       "FROM pg_constraint "
+                       "LEFT JOIN pg_depend ON (objid = confrelid) "
+                       "WHERE contype = 'f' "
+                       "AND refclassid = 'pg_extension'::regclass "
+                       "AND classid = 'pg_class'::regclass;");

I'm trying to figure out why this is a left join..?

Subject: [PATCH 2/3] Make prove_check install contents of current directory as well

This is really an independent thing, no? I don't see any particular
problem with it, for my part.

Yes, that's an independent thing, but my guess is that it would be
good to have a real test case this time to be sure that this does not
break again, at least on master where test/modules is an ideal place.

No objection to it, just pointing out that it's independent.

Attached are updated patches, the fix of pg_dump can be easily
cherry-picked down to 9.2. For 9.1 things are changed a bit because of
the way SQL queries are run, still patches are attached for all the
versions needed. I tested as well the fix down to this version 9.1
using the test case dump_test.

Will review.

Thanks!

Stephen

#18Stephen Frost
sfrost@snowman.net
In reply to: Michael Paquier (#16)
Re: Bug in pg_dump

Michael,

* Michael Paquier (michael.paquier@gmail.com) wrote:

Subject: [PATCH 2/3] Make prove_check install contents of current directory as well

This is really an independent thing, no? I don't see any particular
problem with it, for my part.

Yes, that's an independent thing, but my guess is that it would be
good to have a real test case this time to be sure that this does not
break again, at least on master where test/modules is an ideal place.

I've been looking at this, but noticed the following in
src/test/Makefile:

# We want to recurse to all subdirs for all standard targets, except that
# installcheck and install should not recurse into the subdirectory "modules".

Also, adding a few more items to the Makefile makes the regression tests
pass:

+ MODULE_big = dump_test
+ REGRESS = dump_test

So I'm thinking this isn't really necessary?

I've not really looked into it further but my hunch is the difference is
a pgxs build vs. a non-pgxs build (which I think might be what the
regression suite runs..). One or both of the above might be necessary
to make non-pgxs builds work.

I've made a few other modifications to the test (basically, I wrapped
all the commands run in command_ok() since it wasn't actually failing
when I was expecting it to and plan to include it in the commit to
master.

Thanks!

Stephen

#19Stephen Frost
sfrost@snowman.net
In reply to: Stephen Frost (#18)
Re: Bug in pg_dump

Michael,

* Stephen Frost (sfrost@snowman.net) wrote:

* Michael Paquier (michael.paquier@gmail.com) wrote:

Subject: [PATCH 2/3] Make prove_check install contents of current directory as well

This is really an independent thing, no? I don't see any particular
problem with it, for my part.

Yes, that's an independent thing, but my guess is that it would be
good to have a real test case this time to be sure that this does not
break again, at least on master where test/modules is an ideal place.

I've been looking at this, but noticed the following in
src/test/Makefile:

# We want to recurse to all subdirs for all standard targets, except that
# installcheck and install should not recurse into the subdirectory "modules".

Hrmpf, that hadn't fixed it as I thought, I had another issue which was
making it appear to work.

The other modules work because they use pg_regress and pass it an
'--extra-install' option, so perhaps adding this makes sense after all,
though I'm a bit nervous that we're doing double-duty with this
approach as some things clearly do get installed by the first 'install'.

Peter, if you have a minute, could you take a look at this thread and
discussion of having TAP tests under src/test/modules which need to
install an extension? I think it's something we certainly want to
support but I'm not sure it's a good idea to just run install in every
directory that has a prove_check.

I'm going to move forward with the actual bug fix. We can certainly add
the test in later, once we've got this all sorted.

Thanks!

Stephen

#20Stephen Frost
sfrost@snowman.net
In reply to: Stephen Frost (#17)
1 attachment(s)
Re: Bug in pg_dump

Michael,

* Stephen Frost (sfrost@snowman.net) wrote:

* Michael Paquier (michael.paquier@gmail.com) wrote:

The thing is that we must absolutely wait for *all* the TableInfoData
of all the extensions to be created because we need to mark the
dependencies between them, and even my last patch, or even with what
you are proposing we may miss tracking of FK dependencies between
tables in different extensions. This has little chance to happen in
practice, but I think that we should definitely get things right.
Hence something like this query able to query all the FK dependencies
with tables in extensions is more useful, and it has no IN clause:

Ah, yes, extensions can depend on each other and so this could
definitely happen. The current issue is around postgis, which by itself
provides three different extensions.

+       appendPQExpBuffer(query,
+                       "SELECT conrelid, confrelid "
+                       "FROM pg_constraint "
+                       "LEFT JOIN pg_depend ON (objid = confrelid) "
+                       "WHERE contype = 'f' "
+                       "AND refclassid = 'pg_extension'::regclass "
+                       "AND classid = 'pg_class'::regclass;");

I'm trying to figure out why this is a left join..?

Please see the latest version of this, attached. I've removed the left
join, re-used the 'query' buffer (instead of destroying and recreating
it), and added a bit of documentation, along with a note in the commit
message for the release notes.

Would be great if you could review it and let me know. As mentioned in
my other email, I'm happy to include the TAP test in master once we've
worked out the correct way to handle installing the extension for
testing in the Makefile system.

Thanks!

Stephen

Attachments:

fix-pg_dump-extensions.patchtext/x-diff; charset=us-asciiDownload
From ffc459240a97bc4e7a4e4bc81be7a019d4f6782e Mon Sep 17 00:00:00 2001
From: Stephen Frost <sfrost@snowman.net>
Date: Sun, 1 Mar 2015 15:51:04 -0500
Subject: [PATCH] Fix pg_dump handling of extension config tables

Since 9.1, we've provided extensions with a way to denote
"configuration" tables- tables created by an extension which the user
may modify.  By marking these as "configuration" tables, the extension
is asking for the data in these tables to be pg_dump'd (tables which
are not marked in this way are assumed to be entirely handled during
CREATE EXTENSION and are not included at all in a pg_dump).

Unfortunately, pg_dump neglected to consider foreign key relationships
between extension configuration tables and therefore could end up
trying to reload the data in an order which would cause FK violations.

This patch teaches pg_dump about these dependencies, so that the data
dumped out is done so in the best order possible.  Note that there's no
way to handle circular dependencies, but those have yet to be seen in
the wild.

The release notes for this should include a caution to users that
existing pg_dump-based backups may be invalid due to this issue.  The
data is all there, but restoring from it will require extracting the
data for the configuration tables and then loading them in the correct
order by hand.

Discussed initially back in bug #6738, more recently brought up by
Gilles Darold, who provided an initial patch which was further reworked
by Michael Paquier.  Further modifications and documentation updates
by me.

Back-patch to 9.1 where we added the concept of extension configuration
tables.
---
 doc/src/sgml/extend.sgml  |  8 +++++++
 src/bin/pg_dump/pg_dump.c | 54 +++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 60 insertions(+), 2 deletions(-)

diff --git a/doc/src/sgml/extend.sgml b/doc/src/sgml/extend.sgml
index be10252..0a79f56 100644
--- a/doc/src/sgml/extend.sgml
+++ b/doc/src/sgml/extend.sgml
@@ -721,6 +721,14 @@ SELECT pg_catalog.pg_extension_config_dump('my_config', 'WHERE NOT standard_entr
      a table as no longer a configuration table is to dissociate it from the
      extension with <command>ALTER EXTENSION ... DROP TABLE</>.
     </para>
+
+    <para>
+     Note that foreign key relationships between these tables will dictate the
+     order in which the tables are dumped out by pg_dump.  Specifically, it will
+     attempt to dump the referenced-by table before the referencing table.  As
+     the foreign key relationships are set up at CREATE EXTENSION time (prior to
+     data being loaded into the tables) circular dependencies are not supported.
+    </para>
    </sect2>
 
    <sect2>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 2b53c72..4e404b4 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -15236,7 +15236,8 @@ dumpRule(Archive *fout, DumpOptions *dopt, RuleInfo *rinfo)
 }
 
 /*
- * getExtensionMembership --- obtain extension membership data
+ * getExtensionMembership --- obtain extension membership data and check FK
+ * dependencies among relations interacting with the ones in extensions.
  */
 void
 getExtensionMembership(Archive *fout, DumpOptions *dopt, ExtensionInfo extinfo[],
@@ -15249,7 +15250,9 @@ getExtensionMembership(Archive *fout, DumpOptions *dopt, ExtensionInfo extinfo[]
 	int			i_classid,
 				i_objid,
 				i_refclassid,
-				i_refobjid;
+				i_refobjid,
+				i_conrelid,
+				i_confrelid;
 	DumpableObject *dobj,
 			   *refdobj;
 
@@ -15430,6 +15433,53 @@ getExtensionMembership(Archive *fout, DumpOptions *dopt, ExtensionInfo extinfo[]
 			free(extconditionarray);
 	}
 
+	/*
+	 * Now that all the TableInfoData objects have been created for all
+	 * the extensions, check their FK dependencies and register them to
+	 * ensure correct data ordering.  Note that this is not a problem
+	 * for user tables not included in an extension referencing with a
+	 * FK tables in extensions as their constraint is declared after
+	 * dumping their data. In --data-only mode the table ordering is
+	 * ensured as well thanks to getTableDataFKConstraints().
+	 */
+	printfPQExpBuffer(query,
+			"SELECT conrelid, confrelid "
+			"FROM pg_constraint "
+				"JOIN pg_depend ON (objid = confrelid) "
+			"WHERE contype = 'f' "
+			"AND refclassid = 'pg_extension'::regclass "
+			"AND classid = 'pg_class'::regclass;");
+
+	res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
+	ntups = PQntuples(res);
+
+	i_conrelid = PQfnumber(res, "conrelid");
+	i_confrelid = PQfnumber(res, "confrelid");
+
+	/* Now get the dependencies and register them */
+	for (i = 0; i < ntups; i++)
+	{
+		Oid			conrelid, confrelid;
+		TableInfo  *reftable, *contable;
+
+		conrelid = atooid(PQgetvalue(res, i, i_conrelid));
+		confrelid = atooid(PQgetvalue(res, i, i_confrelid));
+		contable = findTableByOid(conrelid);
+		reftable = findTableByOid(confrelid);
+
+		if (reftable == NULL ||
+			reftable->dataObj == NULL ||
+			contable == NULL ||
+			contable->dataObj == NULL)
+			continue;
+
+		/*
+		 * Make referencing TABLE_DATA object depend on the
+		 * referenced table's TABLE_DATA object.
+		 */
+		addObjectDependency(&contable->dataObj->dobj,
+							reftable->dataObj->dobj.dumpId);
+	}
 	destroyPQExpBuffer(query);
 }
 
-- 
1.9.1

#21Michael Paquier
michael.paquier@gmail.com
In reply to: Stephen Frost (#20)
Re: Bug in pg_dump

On Mon, Mar 2, 2015 at 6:27 AM, Stephen Frost <sfrost@snowman.net> wrote:

Please see the latest version of this, attached. I've removed the left
join, re-used the 'query' buffer (instead of destroying and recreating
it), and added a bit of documentation, along with a note in the commit
message for the release notes.

Thanks, those modifications look good to me.

Would be great if you could review it and let me know. As mentioned in
my other email, I'm happy to include the TAP test in master once we've
worked out the correct way to handle installing the extension for
testing in the Makefile system.

Sure. As that's unrelated to this thread, perhaps we could discuss
this point on another thread with refreshed patches? I'd like to hear
some input from Peter on the matter as well.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#22Stephen Frost
sfrost@snowman.net
In reply to: Michael Paquier (#21)
Re: Bug in pg_dump

* Michael Paquier (michael.paquier@gmail.com) wrote:

On Mon, Mar 2, 2015 at 6:27 AM, Stephen Frost <sfrost@snowman.net> wrote:

Please see the latest version of this, attached. I've removed the left
join, re-used the 'query' buffer (instead of destroying and recreating
it), and added a bit of documentation, along with a note in the commit
message for the release notes.

Thanks, those modifications look good to me.

Ok, I've pushed the pg_dump fix for all the branches it applies to.
Thanks for the help!

Would be great if you could review it and let me know. As mentioned in
my other email, I'm happy to include the TAP test in master once we've
worked out the correct way to handle installing the extension for
testing in the Makefile system.

Sure. As that's unrelated to this thread, perhaps we could discuss
this point on another thread with refreshed patches? I'd like to hear
some input from Peter on the matter as well.

That's fine with me- feel free to start a new thread with new patches.

Thanks again!

Stephen

#23Michael Paquier
michael.paquier@gmail.com
In reply to: Stephen Frost (#22)
Re: Bug in pg_dump

On Tue, Mar 3, 2015 at 4:57 AM, Stephen Frost <sfrost@snowman.net> wrote:

* Michael Paquier (michael.paquier@gmail.com) wrote:

On Mon, Mar 2, 2015 at 6:27 AM, Stephen Frost <sfrost@snowman.net> wrote:

Please see the latest version of this, attached. I've removed the left
join, re-used the 'query' buffer (instead of destroying and recreating
it), and added a bit of documentation, along with a note in the commit
message for the release notes.

Thanks, those modifications look good to me.

Ok, I've pushed the pg_dump fix for all the branches it applies to.

Thanks.

Would be great if you could review it and let me know. As mentioned in
my other email, I'm happy to include the TAP test in master once we've
worked out the correct way to handle installing the extension for
testing in the Makefile system.

Sure. As that's unrelated to this thread, perhaps we could discuss
this point on another thread with refreshed patches? I'd like to hear
some input from Peter on the matter as well.

That's fine with me- feel free to start a new thread with new patches.

Will do so.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#24Peter Eisentraut
peter_e@gmx.net
In reply to: Stephen Frost (#19)
Re: Bug in pg_dump

On 3/1/15 2:17 PM, Stephen Frost wrote:

Peter, if you have a minute, could you take a look at this thread and
discussion of having TAP tests under src/test/modules which need to
install an extension? I think it's something we certainly want to
support but I'm not sure it's a good idea to just run install in every
directory that has a prove_check.

I don't think the way the tests are set up is scalable. Over time, we
will surely want to test more and different extension dumping scenarios.
We don't want to have to create a new fully built-out extension for
each of those test cases, and we're going to run out of useful names for
the extensions, too.

Moreover, I would like to have tests of pg_dump in src/bin/pg_dump/, not
in remote areas of the code.

Here's what I would do:

- set up basic scaffolding for TAP tests in src/bin/pg_dump

- write a Perl function that can create an extension on the fly, given
name, C code, SQL code

- add to the proposed t/001_dump_test.pl code to write the extension

- add that test to the pg_dump test suite

Eventually, the dump-and-restore routine could also be refactored, but
as long as we only have one test case, that can wait.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#25Michael Paquier
michael.paquier@gmail.com
In reply to: Peter Eisentraut (#24)
Re: Bug in pg_dump

On Wed, Mar 4, 2015 at 6:48 AM, Peter Eisentraut <peter_e@gmx.net> wrote:

- set up basic scaffolding for TAP tests in src/bin/pg_dump

Agreed.

- write a Perl function that can create an extension on the fly, given
name, C code, SQL code

I am perplex about that. Where would the SQL code or C code be stored?
In the pl script itself? I cannot really see the advantage to generate
automatically the skeletton of an extension based on some C or SQL
code in comparison to store the extension statically as-is. Adding
those extensions in src/test/modules is out of scope to not bloat it,
so we could for example add such test extensions in t/extensions/ or
similar, and have prove_check scan this folder, then install those
extensions in the temporary installation.

- add to the proposed t/001_dump_test.pl code to write the extension
- add that test to the pg_dump test suite
Eventually, the dump-and-restore routine could also be refactored, but
as long as we only have one test case, that can wait.

Agreed on all those points.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#26Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#25)
Re: Bug in pg_dump

On Wed, Mar 4, 2015 at 2:03 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:

On Wed, Mar 4, 2015 at 6:48 AM, Peter Eisentraut <peter_e@gmx.net> wrote:

- set up basic scaffolding for TAP tests in src/bin/pg_dump

Agreed.

- write a Perl function that can create an extension on the fly, given
name, C code, SQL code

I am perplex about that. Where would the SQL code or C code be stored?
In the pl script itself? I cannot really see the advantage to generate
automatically the skeletton of an extension based on some C or SQL
code in comparison to store the extension statically as-is. Adding
those extensions in src/test/modules is out of scope to not bloat it,
so we could for example add such test extensions in t/extensions/ or
similar, and have prove_check scan this folder, then install those
extensions in the temporary installation.

- add to the proposed t/001_dump_test.pl code to write the extension
- add that test to the pg_dump test suite
Eventually, the dump-and-restore routine could also be refactored, but
as long as we only have one test case, that can wait.

Agreed on all those points.

Please note that I have created a new thread especially for this purpose here:
/messages/by-id/CAB7nPqRx=zmBFJyjrWhGuhHqK__8M+wd+P95ceNJtMHxXR7RRg@mail.gmail.com
Perhaps we should move there this discussion as it is rather
independent of the problem that has been reported.

Regards,
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers