Proposal: IMPORT FOREIGN SCHEMA statement.

Started by Ronan Dunklaualmost 12 years ago8 messages
#1Ronan Dunklau
ronan.dunklau@dalibo.com
2 attachment(s)

Hello,

The SQL-MED specification defines the IMPORT FOREIGN SCHEMA statement.

This adds discoverability to foreign servers. The structure of the statement
as I understand it is simple enough:

IMPORT FOREIGN SCHEMA remote_schema FROM SERVER some_server [ (LIMIT TO |
EXCEPT) table_list ] INTO local_schema.

Is anyone working on this? I found a reference to this from 2010 in the
archive, stating that work should be focused on core functionality, but
nothing more recent.

This would be very useful for postgres_fdw and other RDBMS-backed fdws, but I
think even file_fdw could benefit from it if it was able to create a foreign
table for every csv-with-header file in a directory.

I can see a simple API working for that. A new function would be added to the
fdw routine, which is responsible for crafting CreateForeignTableStmt. It
could have the following signature:

typedef List *(*ImportForeignSchema_function) (ForeignServer *server,
ImportForeignSchemaStmt * parsetree);

I experimented with this idea, and came up with the attached two patches: one
for the core, and the other for actually implementing the API in postgres_fdw.

Maybe those can serve as a proof-of-concept for discussing the design?

--
Ronan Dunklau
http://dalibo.com - http://dalibo.org

Attachments:

import_foreign_schema.patchtext/x-patch; charset=UTF-8; name=import_foreign_schema.patchDownload
diff --git a/src/backend/commands/event_trigger.c b/src/backend/commands/event_trigger.c
index 024a477..40a2540 100644
--- a/src/backend/commands/event_trigger.c
+++ b/src/backend/commands/event_trigger.c
@@ -250,7 +250,8 @@ check_ddl_tag(const char *tag)
 		pg_strcasecmp(tag, "REFRESH MATERIALIZED VIEW") == 0 ||
 		pg_strcasecmp(tag, "ALTER DEFAULT PRIVILEGES") == 0 ||
 		pg_strcasecmp(tag, "ALTER LARGE OBJECT") == 0 ||
-		pg_strcasecmp(tag, "DROP OWNED") == 0)
+		pg_strcasecmp(tag, "DROP OWNED") == 0 ||
+		pg_strcasecmp(tag, "IMPORT FOREIGN SCHEMA") == 0)
 		return EVENT_TRIGGER_COMMAND_TAG_OK;
 
 	/*
diff --git a/src/backend/commands/foreigncmds.c b/src/backend/commands/foreigncmds.c
index 7f007d7..719c674 100644
--- a/src/backend/commands/foreigncmds.c
+++ b/src/backend/commands/foreigncmds.c
@@ -27,7 +27,9 @@
 #include "catalog/pg_type.h"
 #include "catalog/pg_user_mapping.h"
 #include "commands/defrem.h"
+#include "commands/tablecmds.h"
 #include "foreign/foreign.h"
+#include "foreign/fdwapi.h"
 #include "miscadmin.h"
 #include "parser/parse_func.h"
 #include "utils/acl.h"
@@ -1427,3 +1429,48 @@ CreateForeignTable(CreateForeignTableStmt *stmt, Oid relid)
 
 	heap_close(ftrel, RowExclusiveLock);
 }
+
+/*
+ * Import a foreign schema
+ */
+void
+ImportForeignSchema(ImportForeignSchemaStmt * stmt)
+{
+	Oid			ownerId;
+	ForeignDataWrapper *fdw;
+	ForeignServer *server;
+	FdwRoutine *fdw_routine;
+	AclResult	aclresult;
+	List	   *table_list = NULL;
+	ListCell   *lc;
+	char * local_schema = strdup(stmt->local_schema);
+
+	ownerId = GetUserId();
+	server = GetForeignServerByName(stmt->servername, false);
+	aclresult = pg_foreign_server_aclcheck(server->serverid, ownerId, ACL_USAGE);
+	if (aclresult != ACLCHECK_OK)
+		aclcheck_error(aclresult, ACL_KIND_FOREIGN_SERVER, server->servername);
+	
+	/* Check the permissions on the schema */
+	LookupCreationNamespace(local_schema);
+
+	fdw = GetForeignDataWrapper(server->fdwid);
+	fdw_routine = GetFdwRoutine(fdw->fdwhandler);
+	if (fdw_routine->ImportForeignSchema == NULL)
+	{
+		ereport(ERROR,
+				(errcode(ERRCODE_FDW_NO_SCHEMAS),
+				 errmsg("This FDW does not support schema importation")));
+	}
+	table_list = fdw_routine->ImportForeignSchema(server, stmt);
+	foreach(lc, table_list)
+	{
+		CreateForeignTableStmt *create_stmt = lfirst(lc);
+		Oid			relOid = DefineRelation((CreateStmt *) create_stmt,
+											RELKIND_FOREIGN_TABLE,
+											InvalidOid);
+		/* Override whatever the fdw set for the schema */
+		create_stmt->base.relation->schemaname = local_schema;
+		CreateForeignTable(create_stmt, relOid);
+	}
+}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 81169a4..80c18cc 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -235,8 +235,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
 		DropAssertStmt DropTrigStmt DropRuleStmt DropCastStmt DropRoleStmt
 		DropUserStmt DropdbStmt DropTableSpaceStmt DropFdwStmt
 		DropForeignServerStmt DropUserMappingStmt ExplainStmt FetchStmt
-		GrantStmt GrantRoleStmt IndexStmt InsertStmt ListenStmt LoadStmt
-		LockStmt NotifyStmt ExplainableStmt PreparableStmt
+		GrantStmt GrantRoleStmt IndexStmt ImportForeignSchemaStmt InsertStmt
+		ListenStmt LoadStmt LockStmt NotifyStmt ExplainableStmt PreparableStmt
 		CreateFunctionStmt AlterFunctionStmt ReindexStmt RemoveAggrStmt
 		RemoveFuncStmt RemoveOperStmt RenameStmt RevokeStmt RevokeRoleStmt
 		RuleActionStmt RuleActionStmtOrEmpty RuleStmt
@@ -319,6 +319,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
 %type <ival>	defacl_privilege_target
 %type <defelt>	DefACLOption
 %type <list>	DefACLOptionList
+%type <node>	OptImportForeignSchemaRestriction
+%type <ival>	ImportForeignSchemaRestrictionType
 
 %type <list>	stmtblock stmtmulti
 				OptTableElementList TableElementList OptInherit definition
@@ -552,7 +554,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
 
 	HANDLER HAVING HEADER_P HOLD HOUR_P
 
-	IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IN_P
+	IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IN_P IMPORT
 	INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
 	INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
 	INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -797,6 +799,7 @@ stmt :
 			| FetchStmt
 			| GrantStmt
 			| GrantRoleStmt
+			| ImportForeignSchemaStmt
 			| IndexStmt
 			| InsertStmt
 			| ListenStmt
@@ -1286,6 +1289,40 @@ schema_stmt:
 		;
 
 
+/*
+ * IMPORT FOREIGN SCHEMA statement
+ */
+ImportForeignSchemaStmt:
+			IMPORT FOREIGN SCHEMA name FROM SERVER name
+			OptImportForeignSchemaRestriction INTO name {
+				ImportForeignSchemaStmt *n = makeNode(ImportForeignSchemaStmt);
+					n->remote_schema = $4;
+					n->servername = $7;
+					n->restriction = (ImportForeignSchemaRestriction * ) $8;
+					n->local_schema = $10;
+					$$ = (Node *) n;
+			};
+
+ImportForeignSchemaRestrictionType:
+			LIMIT TO 							{ $$ = IMPORT_LIMIT_TO; }
+			| EXCEPT 						{ $$ = IMPORT_EXCEPT; };
+
+OptImportForeignSchemaRestriction:
+			ImportForeignSchemaRestrictionType '(' relation_expr_list ')' {
+				ImportForeignSchemaRestriction * restriction = makeNode(ImportForeignSchemaRestriction);
+				restriction->restriction_type = $1;
+				restriction->table_list = $3;
+				$$ = (Node *) restriction;
+			}
+			| /*EMPTY*/ { 
+				ImportForeignSchemaRestriction * restriction = makeNode(ImportForeignSchemaRestriction);
+				restriction->restriction_type = IMPORT_ALL;
+				restriction->table_list = NULL;
+				$$ = (Node *) restriction;
+			};
+				
+	
+
 /*****************************************************************************
  *
  * Set PG internal variable
@@ -2701,6 +2738,7 @@ CreateStmt:	CREATE OptTemp TABLE qualified_name '(' OptTableElementList ')'
 				}
 		;
 
+
 /*
  * Redundancy here is needed to avoid shift/reduce conflicts,
  * since TEMP is not a reserved word.  See also OptTempTableName.
@@ -12881,6 +12919,7 @@ unreserved_keyword:
 			| IMMEDIATE
 			| IMMUTABLE
 			| IMPLICIT_P
+			| IMPORT
 			| INCLUDING
 			| INCREMENT
 			| INDEX
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index d1621ad..8d8b9b9 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -202,6 +202,7 @@ check_xact_readonly(Node *parsetree)
 		case T_AlterTableSpaceOptionsStmt:
 		case T_AlterTableSpaceMoveStmt:
 		case T_CreateForeignTableStmt:
+		case T_ImportForeignSchemaStmt:
 		case T_SecLabelStmt:
 			PreventCommandIfReadOnly(CreateCommandTag(parsetree));
 			break;
@@ -1316,6 +1317,10 @@ ProcessUtilitySlow(Node *parsetree,
 				ExecAlterDefaultPrivilegesStmt((AlterDefaultPrivilegesStmt *) parsetree);
 				break;
 
+			case T_ImportForeignSchemaStmt:
+				ImportForeignSchema((ImportForeignSchemaStmt *) parsetree);
+				break;
+
 			default:
 				elog(ERROR, "unrecognized node type: %d",
 					 (int) nodeTag(parsetree));
@@ -1853,6 +1858,10 @@ CreateCommandTag(Node *parsetree)
 			tag = "CREATE FOREIGN TABLE";
 			break;
 
+		case T_ImportForeignSchemaStmt:
+			tag = "IMPORT FOREIGN SCHEMA";
+			break;
+
 		case T_DropStmt:
 			switch (((DropStmt *) parsetree)->removeType)
 			{
@@ -2518,6 +2527,7 @@ GetCommandLogLevel(Node *parsetree)
 		case T_CreateUserMappingStmt:
 		case T_AlterUserMappingStmt:
 		case T_DropUserMappingStmt:
+		case T_ImportForeignSchemaStmt:
 			lev = LOGSTMT_DDL;
 			break;
 
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index 5ec9374..ea7967f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -124,6 +124,7 @@ extern Oid	AlterUserMapping(AlterUserMappingStmt *stmt);
 extern Oid	RemoveUserMapping(DropUserMappingStmt *stmt);
 extern void RemoveUserMappingById(Oid umId);
 extern void CreateForeignTable(CreateForeignTableStmt *stmt, Oid relid);
+extern void ImportForeignSchema(ImportForeignSchemaStmt * stmt);
 extern Datum transformGenericOptions(Oid catalogId,
 						Datum oldOptions,
 						List *options,
diff --git a/src/include/foreign/fdwapi.h b/src/include/foreign/fdwapi.h
index 1b735da..adc9569 100644
--- a/src/include/foreign/fdwapi.h
+++ b/src/include/foreign/fdwapi.h
@@ -14,6 +14,7 @@
 
 #include "nodes/execnodes.h"
 #include "nodes/relation.h"
+#include "foreign/foreign.h"
 
 /* To avoid including explain.h here, reference ExplainState thus: */
 struct ExplainState;
@@ -100,6 +101,10 @@ typedef bool (*AnalyzeForeignTable_function) (Relation relation,
 												 AcquireSampleRowsFunc *func,
 													BlockNumber *totalpages);
 
+typedef List *(*ImportForeignSchema_function) (ForeignServer *server,
+										ImportForeignSchemaStmt * parsetree);
+
+
 /*
  * FdwRoutine is the struct returned by a foreign-data wrapper's handler
  * function.  It provides pointers to the callback functions needed by the
@@ -144,6 +149,9 @@ typedef struct FdwRoutine
 
 	/* Support functions for ANALYZE */
 	AnalyzeForeignTable_function AnalyzeForeignTable;
+
+	/* Support functions for IMPORT FOREIGN SCHEMA */
+	ImportForeignSchema_function ImportForeignSchema;
 } FdwRoutine;
 
 
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 5b8df59..474f25d 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -365,6 +365,7 @@ typedef enum NodeTag
 	T_RefreshMatViewStmt,
 	T_ReplicaIdentityStmt,
 	T_AlterSystemStmt,
+	T_ImportForeignSchemaStmt,
 
 	/*
 	 * TAGS FOR PARSE TREE NODES (parsenodes.h)
@@ -406,6 +407,8 @@ typedef enum NodeTag
 	T_XmlSerialize,
 	T_WithClause,
 	T_CommonTableExpr,
+	T_ImportForeignSchemaRestriction,
+	T_ImportForeignSchemaRestrictionType,
 
 	/*
 	 * TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index b5011af..39044ef 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1248,6 +1248,34 @@ typedef enum DropBehavior
 } DropBehavior;
 
 /* ----------------------
+ * Import Foreign Schema statement
+ * ----------------------
+ */
+
+typedef enum ImportForeignSchemaRestrictionType
+{
+	IMPORT_ALL,
+	IMPORT_LIMIT_TO,
+	IMPORT_EXCEPT
+}	ImportForeignSchemaRestrictionType;
+
+typedef struct ImportForeignSchemaRestriction
+{
+	ImportForeignSchemaRestrictionType restriction_type;
+	List	   *table_list;
+}	ImportForeignSchemaRestriction;
+
+typedef struct ImportForeignSchemaStmt
+{
+	NodeTag		type;
+	char	   *remote_schema;
+	char	   *servername;
+	char	   *local_schema;
+	ImportForeignSchemaRestriction * restriction;
+	List	   *table_names;
+}	ImportForeignSchemaStmt;
+
+/* ----------------------
  *	Alter Table
  * ----------------------
  */
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 61fae22..74a94e4 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -184,6 +184,7 @@ PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD)
 PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD)
 PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD)
 PG_KEYWORD("implicit", IMPLICIT_P, UNRESERVED_KEYWORD)
+PG_KEYWORD("import", IMPORT, UNRESERVED_KEYWORD)
 PG_KEYWORD("in", IN_P, RESERVED_KEYWORD)
 PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD)
 PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD)
import_foreign_schema_postgres_fdw.patchtext/x-patch; charset=UTF-8; name=import_foreign_schema_postgres_fdw.patchDownload
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index fde1ec1..6c522ae 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -288,6 +288,7 @@ static void postgresExplainForeignModify(ModifyTableState *mtstate,
 static bool postgresAnalyzeForeignTable(Relation relation,
 							AcquireSampleRowsFunc *func,
 							BlockNumber *totalpages);
+static List *postgresImportForeignSchema(ForeignServer *server, ImportForeignSchemaStmt * parsetree);
 
 /*
  * Helper functions
@@ -328,6 +329,7 @@ static HeapTuple make_tuple_from_result_row(PGresult *res,
 						   List *retrieved_attrs,
 						   MemoryContext temp_context);
 static void conversion_error_callback(void *arg);
+static PGresult *fetch_remote_tables(PGconn *conn, ImportForeignSchemaStmt * stmt);
 
 
 /*
@@ -365,6 +367,9 @@ postgres_fdw_handler(PG_FUNCTION_ARGS)
 	/* Support functions for ANALYZE */
 	routine->AnalyzeForeignTable = postgresAnalyzeForeignTable;
 
+	/* Support functions for IMPORT FOREIGN SCHEMA */
+	routine->ImportForeignSchema = postgresImportForeignSchema;
+
 	PG_RETURN_POINTER(routine);
 }
 
@@ -2300,6 +2305,170 @@ postgresAnalyzeForeignTable(Relation relation,
 	return true;
 }
 
+static PGresult *
+fetch_remote_tables(PGconn *conn, ImportForeignSchemaStmt * stmt)
+{
+	StringInfoData buf;
+	int			numparams;
+	const char **params = palloc0(sizeof(char *) * 2);
+	PGresult   *res = NULL;
+
+	initStringInfo(&buf);
+	params[0] = strdup((stmt->remote_schema));
+
+	// Check that the schema really exists
+	appendStringInfo(&buf, "SELECT 1 FROM pg_namespace WHERE nspname = $1");
+	res = PQexecParams(conn, buf.data, 1, NULL, params, NULL, NULL, 0);
+	if (PQresultStatus(res) != PGRES_TUPLES_OK)
+		pgfdw_report_error(ERROR, res, conn, true, buf.data);
+	if(PQntuples(res) != 1)
+	{
+		ereport(ERROR,
+				(errcode(ERRCODE_FDW_SCHEMA_NOT_FOUND),
+					errmsg("The schema %s does not exist on the server", params[0])));
+	}
+
+
+	// Fetch all tables from this schema
+	resetStringInfo(&buf);
+	appendStringInfo(&buf,
+		"SELECT relname, "
+			"array_agg(attname::name) as colnames, "
+			"array_agg(atttypid::regtype) as coltypes, "
+			"array_agg(atttypmod::int4) as coltypmod "
+		"FROM pg_class "
+		"INNER JOIN pg_namespace ON pg_class.relnamespace = pg_namespace.oid "
+		"LEFT  JOIN pg_attribute ON pg_class.oid = pg_attribute.attrelid "
+			 "  AND pg_attribute.attnum >= 0 AND NOT pg_attribute.attisdropped "
+		"WHERE relkind IN ('r', 'v', 'f') "
+		"AND pg_namespace.nspname = $1 ");
+	if (stmt->restriction->restriction_type != IMPORT_ALL)
+	{
+		/* Add conditions */
+		Oid			outfuncoid;
+		bool		isvarlena;
+		Datum	   *elems = palloc0(list_length(stmt->restriction->table_list) * sizeof(Datum));
+		ArrayType  *array;
+		int			i = 0;
+		ListCell   *lc;
+		FmgrInfo   *fmout = palloc0(sizeof(FmgrInfo));
+		getTypeOutputInfo(CSTRINGARRAYOID, &outfuncoid, &isvarlena);
+		fmgr_info(outfuncoid, fmout);
+		foreach(lc, stmt->restriction->table_list)
+		{
+			elems[i] = CStringGetDatum(((RangeVar *) lfirst(lc))->relname);
+			i++;
+		}
+		array = construct_array(elems, i, CSTRINGOID, -2, false, 'c');
+		params[1] = OutputFunctionCall(fmout, PointerGetDatum(array));
+		appendStringInfo(&buf, "AND ");
+		if (stmt->restriction->restriction_type == IMPORT_EXCEPT)
+		{
+			appendStringInfo(&buf, "NOT ");
+		}
+		appendStringInfo(&buf, "pg_class.relname = ANY($2)");
+		numparams = 2;
+		pfree(fmout);
+		pfree(elems);
+	}
+	else
+	{
+		numparams = 1;
+	}
+	appendStringInfo(&buf, "GROUP BY pg_class.oid, pg_class.relname");
+	res = PQexecParams(conn, buf.data, numparams, NULL, params, NULL, NULL, 0);
+	if (PQresultStatus(res) != PGRES_TUPLES_OK)
+		pgfdw_report_error(ERROR, res, conn, true, buf.data);
+	pfree(params);
+	return res;
+}
+
+/*
+ * Map a remote schema to a local one.
+ */
+static List *
+postgresImportForeignSchema(ForeignServer *server,
+							ImportForeignSchemaStmt * parsetree)
+{
+	List	   *tables = NULL;
+	Oid			userid = GetUserId();
+	UserMapping *mapping = GetUserMapping(userid, server->serverid);
+	PGconn	   *conn = GetConnection(server, mapping, false);
+	PGresult   *res;
+	int			numrows,
+				i;
+
+	/* Initialize FmgrInfo for parsing arrays */
+	FmgrInfo   *fmgr_infos = palloc0(sizeof(FmgrInfo) * 3);
+	Oid			typoid,
+			   *typioparam = palloc0(sizeof(Oid) * 3);
+	ArrayIterator *arrays = palloc0(sizeof(ArrayIterator) * 3);
+	Oid			column_types[3] = {NAMEARRAYOID, REGTYPEARRAYOID, INT4ARRAYOID};
+	Datum	   *array_item = palloc0(sizeof(Datum) * 3);
+	bool	   *isnull = palloc0(sizeof(bool) * 3);
+
+	for (i = 0; i < 3; i++)
+	{
+		getTypeInputInfo(column_types[i], &typoid, &typioparam[i]);
+		fmgr_info(typoid, &fmgr_infos[i]);
+	}
+	res = fetch_remote_tables(conn, parsetree);
+	numrows = PQntuples(res);
+	for (i = 0; i < numrows; i++)
+	{
+		CreateForeignTableStmt *stmt = makeNode(CreateForeignTableStmt);
+		char	   *tablename;
+		int			colindex;
+
+		tablename = PQgetvalue(res, i, 0);
+		/* setup the base relation information */
+		stmt->base.relation = makeRangeVar(NULL, tablename, 0);
+		stmt->base.relation->schemaname = strdup(parsetree->local_schema);
+		stmt->servername = server->servername;
+		/* Parse arrays of columns from the result */
+		for (colindex = 0; colindex < 3; colindex++)
+		{
+			Datum		array_datum = InputFunctionCall(&fmgr_infos[colindex],
+											PQgetvalue(res, i, colindex + 1),
+												   typioparam[colindex], -1);
+
+			arrays[colindex] = array_create_iterator(DatumGetArrayTypeP(array_datum), 0);
+		}
+		/* add the individual columns */
+		while (array_iterate(arrays[0], &array_item[0], &isnull[0]) &&
+			   array_iterate(arrays[1], &array_item[1], &isnull[1]) &&
+			   array_iterate(arrays[2], &array_item[2], &isnull[2]))
+		{
+			ColumnDef  *new_column = makeNode(ColumnDef);
+
+			new_column->colname = DatumGetCString(array_item[0]);
+			new_column->typeName = makeTypeNameFromOid(DatumGetObjectId(array_item[1]),
+											   DatumGetInt32(array_item[2]));
+			stmt->base.tableElts = lappend(stmt->base.tableElts, new_column);
+		}
+
+		/*
+		 * Add schema_name and table_name options table_name is added to
+		 * survive a foreign table rename.
+		 */
+		stmt->options = lappend(stmt->options,
+								makeDefElem("schema_name", (Node *) makeString(parsetree->remote_schema)));
+		stmt->options = lappend(stmt->options,
+				  makeDefElem("table_name", (Node *) makeString(tablename)));
+		tables = lappend(tables, stmt);
+	}
+	/* Cleanup */
+	PQclear(res);
+	ReleaseConnection(conn);
+	pfree(array_item);
+	pfree(fmgr_infos);
+	pfree(arrays);
+	pfree(typioparam);
+	pfree(isnull);
+
+	return tables;
+}
+
 /*
  * Acquire a random sample of rows from foreign table managed by postgres_fdw.
  *
diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h
index db18a23..47efb2f 100644
--- a/src/include/catalog/pg_type.h
+++ b/src/include/catalog/pg_type.h
@@ -445,6 +445,7 @@ DATA(insert OID = 1000 (  _bool		 PGNSP PGUID -1 f b A f t \054 0	16 0 array_in
 DATA(insert OID = 1001 (  _bytea	 PGNSP PGUID -1 f b A f t \054 0	17 0 array_in array_out array_recv array_send - - array_typanalyze i x f 0 -1 0 0 _null_ _null_ _null_ ));
 DATA(insert OID = 1002 (  _char		 PGNSP PGUID -1 f b A f t \054 0	18 0 array_in array_out array_recv array_send - - array_typanalyze i x f 0 -1 0 0 _null_ _null_ _null_ ));
 DATA(insert OID = 1003 (  _name		 PGNSP PGUID -1 f b A f t \054 0	19 0 array_in array_out array_recv array_send - - array_typanalyze i x f 0 -1 0 0 _null_ _null_ _null_ ));
+#define NAMEARRAYOID 1003
 DATA(insert OID = 1005 (  _int2		 PGNSP PGUID -1 f b A f t \054 0	21 0 array_in array_out array_recv array_send - - array_typanalyze i x f 0 -1 0 0 _null_ _null_ _null_ ));
 #define INT2ARRAYOID		1005
 DATA(insert OID = 1006 (  _int2vector PGNSP PGUID -1 f b A f t \054 0	22 0 array_in array_out array_recv array_send - - array_typanalyze i x f 0 -1 0 0 _null_ _null_ _null_ ));
#2Atri Sharma
atri.jiit@gmail.com
In reply to: Ronan Dunklau (#1)
Re: Proposal: IMPORT FOREIGN SCHEMA statement.

On Fri, Feb 21, 2014 at 4:01 PM, Ronan Dunklau <ronan.dunklau@dalibo.com>wrote:

Hello,

The SQL-MED specification defines the IMPORT FOREIGN SCHEMA statement.

This adds discoverability to foreign servers. The structure of the
statement
as I understand it is simple enough:

IMPORT FOREIGN SCHEMA remote_schema FROM SERVER some_server [ (LIMIT TO |
EXCEPT) table_list ] INTO local_schema.

Is anyone working on this? I found a reference to this from 2010 in the
archive, stating that work should be focused on core functionality, but
nothing more recent.

This would be very useful for postgres_fdw and other RDBMS-backed fdws,
but I
think even file_fdw could benefit from it if it was able to create a
foreign
table for every csv-with-header file in a directory.

I can see a simple API working for that. A new function would be added to
the
fdw routine, which is responsible for crafting CreateForeignTableStmt. It
could have the following signature:

typedef List *(*ImportForeignSchema_function) (ForeignServer *server,
ImportForeignSchemaStmt * parsetree);

I experimented with this idea, and came up with the attached two patches:
one
for the core, and the other for actually implementing the API in
postgres_fdw.

Maybe those can serve as a proof-of-concept for discussing the design?

I havent had a look at the patch yet since I dont have a nice editor right
now, but how do you handle inter operability between datatypes?
Specifically, how do you handle those datatypes which have a different name
from the PostgreSQL name for them and/or are stored in a different manner?

Regards,

Atri

#3Ronan Dunklau
ronan.dunklau@dalibo.com
In reply to: Atri Sharma (#2)
Re: Proposal: IMPORT FOREIGN SCHEMA statement.

I havent had a look at the patch yet since I dont have a nice editor right
now, but how do you handle inter operability between datatypes?
Specifically, how do you handle those datatypes which have a different name
from the PostgreSQL name for them and/or are stored in a different manner?

Do you mean in general, or for the postgres_fdw specifically ?

In general, only valid column types should be accepted in the
CreateForeignTableStmt. The CreateForeignTableStmt is passed through
DefineRelation, which takes care of looking up the actual data types.

For the postgres_fdw POC implementation, this is done by parsing the
attributes type from the query result with the regtype input functions. The
attribute typmod is injected too.

Regards,

Atri

--
Ronan Dunklau
http://dalibo.com - http://dalibo.org

#4Atri Sharma
atri.jiit@gmail.com
In reply to: Ronan Dunklau (#3)
Re: Proposal: IMPORT FOREIGN SCHEMA statement.

On Fri, Feb 21, 2014 at 4:39 PM, Ronan Dunklau <ronan.dunklau@dalibo.com>wrote:

I havent had a look at the patch yet since I dont have a nice editor

right

now, but how do you handle inter operability between datatypes?
Specifically, how do you handle those datatypes which have a different

name

from the PostgreSQL name for them and/or are stored in a different

manner?

Do you mean in general, or for the postgres_fdw specifically ?

In general, only valid column types should be accepted in the
CreateForeignTableStmt. The CreateForeignTableStmt is passed through
DefineRelation, which takes care of looking up the actual data types.

For the postgres_fdw POC implementation, this is done by parsing the
attributes type from the query result with the regtype input functions. The
attribute typmod is injected too.

I actually meant in general. Thanks for the reply.

So please help me understand here. How exactly does CreateForeignTableStmt
help in type compatibility? A statement may be valid on a foreign server
but may not be compatible.

What am I missing here naively?

Regards,

Atri

--
Regards,

Atri
*l'apprenant*

#5Ronan Dunklau
ronan.dunklau@dalibo.com
In reply to: Atri Sharma (#4)
Re: Proposal: IMPORT FOREIGN SCHEMA statement.

Le vendredi 21 février 2014 16:45:20 Atri Sharma a écrit :

On Fri, Feb 21, 2014 at 4:39 PM, Ronan Dunklau

<ronan.dunklau@dalibo.com>wrote:

I havent had a look at the patch yet since I dont have a nice editor

right

now, but how do you handle inter operability between datatypes?
Specifically, how do you handle those datatypes which have a different

name

from the PostgreSQL name for them and/or are stored in a different

manner?

Do you mean in general, or for the postgres_fdw specifically ?

In general, only valid column types should be accepted in the
CreateForeignTableStmt. The CreateForeignTableStmt is passed through
DefineRelation, which takes care of looking up the actual data types.

For the postgres_fdw POC implementation, this is done by parsing the
attributes type from the query result with the regtype input functions.
The
attribute typmod is injected too.

I actually meant in general. Thanks for the reply.

So please help me understand here. How exactly does CreateForeignTableStmt
help in type compatibility?

I'm not sure I understand your concern. It doesn't help in type compatibility,
it is still the responsibility of the FDW to convert local types to remote
ones.

The CreateForeignTableStmt defines the column, with their types. It is
executed locally to create a new foreign table according to a remote
description of the table. The only difference with a user-written
CreateForeignTable statement is that the structure is crafted by the FDW
instead of the parser.

A statement may be valid on a foreign server but may not be compatible.

Do you mean the CreateForeignTableStmt ? It has to be valid locally, or it
won't be executed. It is the FDW responsibility to build this statement in
such a way that it is valid locally.

Regards,

Atri

--
Ronan Dunklau
http://dalibo.com - http://dalibo.org

#6Atri Sharma
atri.jiit@gmail.com
In reply to: Ronan Dunklau (#5)
Re: Proposal: IMPORT FOREIGN SCHEMA statement.

On Fri, Feb 21, 2014 at 4:56 PM, Ronan Dunklau <ronan.dunklau@dalibo.com>wrote:

Le vendredi 21 février 2014 16:45:20 Atri Sharma a écrit :

On Fri, Feb 21, 2014 at 4:39 PM, Ronan Dunklau

<ronan.dunklau@dalibo.com>wrote:

I havent had a look at the patch yet since I dont have a nice editor

right

now, but how do you handle inter operability between datatypes?
Specifically, how do you handle those datatypes which have a

different

name

from the PostgreSQL name for them and/or are stored in a different

manner?

Do you mean in general, or for the postgres_fdw specifically ?

In general, only valid column types should be accepted in the
CreateForeignTableStmt. The CreateForeignTableStmt is passed through
DefineRelation, which takes care of looking up the actual data types.

For the postgres_fdw POC implementation, this is done by parsing the
attributes type from the query result with the regtype input functions.
The
attribute typmod is injected too.

I actually meant in general. Thanks for the reply.

So please help me understand here. How exactly does

CreateForeignTableStmt

help in type compatibility?

I'm not sure I understand your concern. It doesn't help in type
compatibility,
it is still the responsibility of the FDW to convert local types to remote
ones.

Yeah, thats what I wondered. Ok, now I get it. The responsibility of FDW
shall suffice for us.

The CreateForeignTableStmt defines the column, with their types. It is
executed locally to create a new foreign table according to a remote
description of the table. The only difference with a user-written
CreateForeignTable statement is that the structure is crafted by the FDW
instead of the parser.

Got that.

A statement may be valid on a foreign server but may not be compatible.

Do you mean the CreateForeignTableStmt ? It has to be valid locally, or it
won't be executed. It is the FDW responsibility to build this statement in
such a way that it is valid locally.

Yes, I understand it now. My concerns are not valid anymore.

Thanks for the detailed description.

Regards,

Atri

--
Regards,

Atri
*l'apprenant*

#7Florian Pflug
fgp@phlo.org
In reply to: Ronan Dunklau (#3)
Re: Proposal: IMPORT FOREIGN SCHEMA statement.

On Feb21, 2014, at 12:09 , Ronan Dunklau <ronan.dunklau@dalibo.com> wrote:

I havent had a look at the patch yet since I dont have a nice editor right
now, but how do you handle inter operability between datatypes?
Specifically, how do you handle those datatypes which have a different name
from the PostgreSQL name for them and/or are stored in a different manner?

For the postgres_fdw POC implementation, this is done by parsing the
attributes type from the query result with the regtype input functions. The
attribute typmod is injected too.

Who says that the OIDs are the same on the local and the remove postgres
instance? For user-defined types, that's certainly not going to be true...

Also, why do you aggregate the lists of columns, types and oids into arrays
when querying them from the remote server? Just producing a query that returns
one row per table column seems much simpler, both conceptually and implementation
wise.

Finally, I think there are a few missing pieces. For example, you quite easily
could copy over not-NULL flags, but you currently don't. Similarly, what about
inheritance relationships between remote tables? There's a patch in the current
CF, I believe, which adds support for inheritance to foreign tables, so all you'd
have to do is to make the foreign table's inheritance structure match the remote
table's.

best regards,
Florian Pflug

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#8Ronan Dunklau
ronan.dunklau@dalibo.com
In reply to: Florian Pflug (#7)
Re: Proposal: IMPORT FOREIGN SCHEMA statement.

Who says that the OIDs are the same on the local and the remove postgres
instance? For user-defined types, that's certainly not going to be true...

That's why the the result is casted as regtype[], and parsed as such. The oid
is not transmitted over the wire, but set by regtype_in.

Also, why do you aggregate the lists of columns, types and oids into arrays
when querying them from the remote server? Just producing a query that
returns one row per table column seems much simpler, both conceptually and
implementation wise.

As I said, this is a Proof-Of-Concept. It is not meant to be a fully
functional, optimal implementation, but to serve as basis for discussion of
the API and the feature themselves.

The query could indeed be replaced by what you suggest, performing the
grouping locally. I have absolutely no opinion on which implementation is
better, this one seemed the most "natural" to me.

Finally, I think there are a few missing pieces. For example, you quite
easily could copy over not-NULL flags, but you currently don't. Similarly,
what about inheritance relationships between remote tables? There's a patch
in the current CF, I believe, which adds support for inheritance to foreign
tables, so all you'd have to do is to make the foreign table's inheritance
structure match the remote table's.

Duly noted, we could probably import NOT NULL flags, and maybe even the
table's inheritance structure. I'll look into that if the feature and the API
are deemed worthy.

Thank you for the review.

--
Ronan Dunklau
http://dalibo.com - http://dalibo.org