[v9.3] writable foreign tables
Hello,
The attached patch is just a proof-of-concept of writable foreign table
feature; thus, I don't expect it getting merged at the upcoming commit
fest. The purpose of this patch is to find out the best way to support
"write stuff" in FDW.
Basic idea of this patch is to utilize "ctid" field to track an identifier of
a particular foreign-row; to be updated or deleted. It shall be moved
to the modify-stage from the scan-stage as regular table doing.
Then, newly added methods being invoked at ExecUpdate or
ExecDelete with the "pseudo ctid", so FDW shall be able to modify
the target foreign-row.
It is a responsibility of FDW extension (and DBA) to ensure each
foreign-row has a unique identifier that has 48-bits width integer
data type in maximum. In case of pgsql_fdw, "ctid" of remote table
can perform as "pseudo ctid", as is. For other RDBMS, DBA will
need to indicate which column should perform.
INSERT is simple enough; all we need to do it to carry every field
of new tuple into the remote side.
This patch adds five new methods of FdwRoutine structure.
The BeginForeignModify and EndForeignModify give a chance
to initialize / destruct a private state that can be allocated on
ResultRelInfo. As literal, ExecForeignInsert, ExecForeignDelete
and ExecForeignUpdate are invoked for each new tuple, instead
of heap_*() for regular tables. If NULL was set on them, it means
this FDW does not support these operations.
I intend FDW to set up a prepared statement that modifies
a particular remote-row being identified with pseudo-ctid,
at the BeginForeignModify(). Then, ExecForeign*() kicks
the prepared statement with given pseudo-ctid.
The patched portion at contrib/file_fdw.c does not make sense
actually. It just prints messages for each invocation.
It is just a proof-of-concept to show possibility of implementation
based on real RDBMS.
In case when file_fdw performs behalf on "ftbl".
--------------------------------
postgres=# SELECT ctid, * FROM ftbl;
ctid | a | b
--------+-----+-----
(0,1) | 101 | aaa
(0,2) | 102 | bbb
(0,3) | 103 | ccc
(0,4) | 104 | ddd
(0,5) | 105 | eee
(0,6) | 106 | fff
(0,7) | 107 | ggg
(0,8) | 108 | hhh
(0,9) | 109 | iii
(0,10) | 110 | jjj
(10 rows)
==> ctid is used to carray identifier of row; line number in this example.
postgres=# UPDATE ftbl SET a = a + 1 WHERE a > 107;
INFO: ftbl is the target relation of UPDATE
INFO: fdw_file: BeginForeignModify method
INFO: fdw_file: UPDATE (lineno = 8)
INFO: fdw_file: UPDATE (lineno = 9)
INFO: fdw_file: UPDATE (lineno = 10)
INFO: fdw_file: EndForeignModify method
UPDATE 3
postgres=# DELETE FROM ftbl WHERE a BETWEEN 103 AND 106;
INFO: ftbl is the target relation of DELETE
INFO: fdw_file: BeginForeignModify method
INFO: fdw_file: DELETE (lineno = 3)
INFO: fdw_file: DELETE (lineno = 4)
INFO: fdw_file: DELETE (lineno = 5)
INFO: fdw_file: DELETE (lineno = 6)
INFO: fdw_file: EndForeignModify method
DELETE 4
--------------------------------
This patch does not care about transaction control anyway.
According to the discussion in developer meeting at Ottawa,
I didn't include such a complex stuff in the first step.
(Probably, we can implement using XactCallback...)
Thanks,
--
KaiGai Kohei <kaigai@kaigai.gr.jp>
Attachments:
pgsql-v9.3-writable-fdw-poc.v1.patchapplication/octet-stream; name=pgsql-v9.3-writable-fdw-poc.v1.patchDownload
contrib/file_fdw/file_fdw.c | 97 ++++++++++++++++++++++++++++++++++
src/backend/executor/execMain.c | 34 ++++++++++--
src/backend/executor/nodeModifyTable.c | 65 +++++++++++++++++++++--
src/backend/rewrite/rewriteHandler.c | 3 +-
src/include/foreign/fdwapi.h | 19 +++++++
src/include/nodes/execnodes.h | 4 ++
src/include/storage/itemptr.h | 25 +++++++++
7 files changed, 237 insertions(+), 10 deletions(-)
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index 7c7fedf..ef9f4b2 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -33,6 +33,7 @@
#include "optimizer/var.h"
#include "utils/memutils.h"
#include "utils/rel.h"
+#include "utils/lsyscache.h"
PG_MODULE_MAGIC;
@@ -96,6 +97,7 @@ typedef struct FileFdwExecutionState
char *filename; /* file to read */
List *options; /* merged COPY options, excluding filename */
CopyState cstate; /* state of reading file */
+ int lineno; /* pseudo ctid of the file */
} FileFdwExecutionState;
/*
@@ -130,6 +132,19 @@ static void fileEndForeignScan(ForeignScanState *node);
static bool fileAnalyzeForeignTable(Relation relation,
AcquireSampleRowsFunc *func,
BlockNumber *totalpages);
+static void fileBeginForeignModify(CmdType operation,
+ ResultRelInfo *resultRelInfo,
+ EState *estate,
+ Plan *subplan,
+ int eflags);
+static void fileExecForeignInsert(ResultRelInfo *resultRelInfo,
+ HeapTuple tuple);
+static void fileExecForeignDelete(ResultRelInfo *resultRelInfo,
+ ItemPointer tupleid);
+static void fileExecForeignUpdate(ResultRelInfo *resultRelInfo,
+ ItemPointer tupleid,
+ HeapTuple tuple);
+static void fileEndForeignModify(ResultRelInfo *resultRelInfo);
/*
* Helper functions
@@ -169,6 +184,11 @@ file_fdw_handler(PG_FUNCTION_ARGS)
fdwroutine->ReScanForeignScan = fileReScanForeignScan;
fdwroutine->EndForeignScan = fileEndForeignScan;
fdwroutine->AnalyzeForeignTable = fileAnalyzeForeignTable;
+ fdwroutine->BeginForeignModify = fileBeginForeignModify;
+ fdwroutine->ExecForeignInsert = fileExecForeignInsert;
+ fdwroutine->ExecForeignDelete = fileExecForeignDelete;
+ fdwroutine->ExecForeignUpdate = fileExecForeignUpdate;
+ fdwroutine->EndForeignModify = fileEndForeignModify;
PG_RETURN_POINTER(fdwroutine);
}
@@ -510,6 +530,36 @@ fileGetForeignPlan(PlannerInfo *root,
List *scan_clauses)
{
Index scan_relid = baserel->relid;
+ ListCell *cell;
+
+ /*
+ * XXX - An evidence FDW module can know what kind of accesses are
+ * requires on the target relation, if UPDATE, INSERT or DELETE.
+ * It shall be also utilized to appropriate lock level on FDW
+ * extensions that performs behalf on real RDBMS.
+ */
+ if (root->parse->resultRelation == baserel->relid)
+ {
+ switch (root->parse->commandType)
+ {
+ case CMD_INSERT:
+ elog(INFO, "%s is the target relation of INSERT",
+ get_rel_name(foreigntableid));
+ break;
+ case CMD_UPDATE:
+ elog(INFO, "%s is the target relation of UPDATE",
+ get_rel_name(foreigntableid));
+ break;
+ case CMD_DELETE:
+ elog(INFO, "%s is the target relation of DELETE",
+ get_rel_name(foreigntableid));
+ break;
+ default:
+ elog(INFO, "%s is the target relation of ??????",
+ get_rel_name(foreigntableid));
+ break;
+ }
+ }
/*
* We have no native ability to evaluate restriction clauses, so we just
@@ -598,6 +648,7 @@ fileBeginForeignScan(ForeignScanState *node, int eflags)
festate->filename = filename;
festate->options = options;
festate->cstate = cstate;
+ festate->lineno = 0;
node->fdw_state = (void *) festate;
}
@@ -612,6 +663,7 @@ fileIterateForeignScan(ForeignScanState *node)
{
FileFdwExecutionState *festate = (FileFdwExecutionState *) node->fdw_state;
TupleTableSlot *slot = node->ss.ss_ScanTupleSlot;
+ HeapTuple tup;
bool found;
ErrorContextCallback errcontext;
@@ -638,7 +690,12 @@ fileIterateForeignScan(ForeignScanState *node)
slot->tts_values, slot->tts_isnull,
NULL);
if (found)
+ {
ExecStoreVirtualTuple(slot);
+ tup = ExecMaterializeSlot(slot);
+ festate->lineno++;
+ PackPseudoItemPointer(&tup->t_self, festate->lineno);
+ }
/* Remove error callback. */
error_context_stack = errcontext.previous;
@@ -661,6 +718,7 @@ fileReScanForeignScan(ForeignScanState *node)
festate->filename,
NIL,
festate->options);
+ festate->lineno = 0;
}
/*
@@ -716,6 +774,45 @@ fileAnalyzeForeignTable(Relation relation,
return true;
}
+static void
+fileBeginForeignModify(CmdType operation,
+ ResultRelInfo *resultRelInfo,
+ EState *estate,
+ Plan *subplan,
+ int eflags)
+{
+ elog(INFO, "fdw_file: BeginForeignModify method");
+}
+
+static void
+fileExecForeignInsert(ResultRelInfo *resultRelInfo, HeapTuple tuple)
+{
+ elog(INFO, "fdw_file: INSERT");
+}
+
+static void
+fileExecForeignDelete(ResultRelInfo *resultRelInfo,
+ ItemPointer tupleid)
+{
+ elog(INFO, "fdw_file: DELETE (lineno = %lu)",
+ (unsigned long) UnpackPseudoItemPointer(tupleid));
+}
+
+static void
+fileExecForeignUpdate(ResultRelInfo *resultRelInfo,
+ ItemPointer tupleid,
+ HeapTuple tuple)
+{
+ elog(INFO, "fdw_file: UPDATE (lineno = %lu)",
+ (unsigned long) UnpackPseudoItemPointer(tupleid));
+}
+
+static void
+fileEndForeignModify(ResultRelInfo *resultRelInfo)
+{
+ elog(INFO, "fdw_file: EndForeignModify method");
+}
+
/*
* check_selective_binary_conversion
*
diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c
index 440438b..7d20627 100644
--- a/src/backend/executor/execMain.c
+++ b/src/backend/executor/execMain.c
@@ -43,6 +43,7 @@
#include "catalog/namespace.h"
#include "commands/trigger.h"
#include "executor/execdebug.h"
+#include "foreign/fdwapi.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "optimizer/clauses.h"
@@ -933,6 +934,7 @@ void
CheckValidResultRel(Relation resultRel, CmdType operation)
{
TriggerDesc *trigDesc = resultRel->trigdesc;
+ FdwRoutine *fdwroutine;
switch (resultRel->rd_rel->relkind)
{
@@ -984,10 +986,34 @@ CheckValidResultRel(Relation resultRel, CmdType operation)
}
break;
case RELKIND_FOREIGN_TABLE:
- ereport(ERROR,
- (errcode(ERRCODE_WRONG_OBJECT_TYPE),
- errmsg("cannot change foreign table \"%s\"",
- RelationGetRelationName(resultRel))));
+ fdwroutine = GetFdwRoutineByRelId(RelationGetRelid(resultRel));
+ switch (operation)
+ {
+ case CMD_INSERT:
+ if (!fdwroutine || !fdwroutine->ExecForeignInsert)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot insert into foreign table \"%s\"",
+ RelationGetRelationName(resultRel))));
+ break;
+ case CMD_UPDATE:
+ if (!fdwroutine || !fdwroutine->ExecForeignUpdate)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot update foreign table \"\"",
+ RelationGetRelationName(resultRel))));
+ break;
+ case CMD_DELETE:
+ if (!fdwroutine || !fdwroutine->ExecForeignDelete)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot delete from foreign table \"\"",
+ RelationGetRelationName(resultRel))));
+ break;
+ default:
+ elog(ERROR, "unrecognized CmdType: %d", (int) operation);
+ break;
+ }
break;
default:
ereport(ERROR,
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index a7bce75..b2fc516 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -41,6 +41,7 @@
#include "commands/trigger.h"
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
+#include "foreign/fdwapi.h"
#include "miscadmin.h"
#include "nodes/nodeFuncs.h"
#include "storage/bufmgr.h"
@@ -224,6 +225,14 @@ ExecInsert(TupleTableSlot *slot,
newId = InvalidOid;
}
+ else if (resultRelInfo->ri_fdwroutine)
+ {
+ FdwRoutine *fdwroutine = resultRelInfo->ri_fdwroutine;
+
+ fdwroutine->ExecForeignInsert(resultRelInfo, tuple);
+
+ newId = InvalidOid;
+ }
else
{
/*
@@ -334,6 +343,12 @@ ExecDelete(ItemPointer tupleid,
if (!dodelete) /* "do nothing" */
return NULL;
}
+ else if (resultRelInfo->ri_fdwroutine)
+ {
+ FdwRoutine *fdwroutine = resultRelInfo->ri_fdwroutine;
+
+ fdwroutine->ExecForeignDelete(resultRelInfo, tupleid);
+ }
else
{
/*
@@ -538,6 +553,12 @@ ExecUpdate(ItemPointer tupleid,
/* trigger might have changed tuple */
tuple = ExecMaterializeSlot(slot);
}
+ else if (resultRelInfo->ri_fdwroutine)
+ {
+ FdwRoutine *fdwroutine = resultRelInfo->ri_fdwroutine;
+
+ fdwroutine->ExecForeignUpdate(resultRelInfo, tupleid, tuple);
+ }
else
{
/*
@@ -805,10 +826,12 @@ ExecModifyTable(ModifyTableState *node)
*/
if (operation == CMD_UPDATE || operation == CMD_DELETE)
{
+ Relation rel = resultRelInfo->ri_RelationDesc;
Datum datum;
bool isNull;
- if (resultRelInfo->ri_RelationDesc->rd_rel->relkind == RELKIND_RELATION)
+ if (RelationGetForm(rel)->relkind == RELKIND_RELATION ||
+ RelationGetForm(rel)->relkind == RELKIND_FOREIGN_TABLE)
{
datum = ExecGetJunkAttribute(slot,
junkfilter->jf_junkAttNo,
@@ -964,6 +987,22 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags)
estate->es_result_relation_info = resultRelInfo;
mtstate->mt_plans[i] = ExecInitNode(subplan, estate, eflags);
+ /*
+ * Also tells FDW extensions to init the plan for this result rel
+ */
+ if (RelationGetForm(resultRelInfo->ri_RelationDesc)->relkind == RELKIND_FOREIGN_TABLE)
+ {
+ Oid relid = RelationGetRelid(resultRelInfo->ri_RelationDesc);
+ FdwRoutine *fdwroutine = GetFdwRoutineByRelId(relid);
+
+ Assert(fdwroutine != NULL);
+ resultRelInfo->ri_fdwroutine = fdwroutine;
+ resultRelInfo->ri_fdw_state = NULL;
+
+ if (fdwroutine->BeginForeignModify)
+ fdwroutine->BeginForeignModify(operation, resultRelInfo,
+ estate, subplan, eflags);
+ }
resultRelInfo++;
i++;
}
@@ -1104,21 +1143,22 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags)
resultRelInfo = mtstate->resultRelInfo;
for (i = 0; i < nplans; i++)
{
+ Relation rel = resultRelInfo->ri_RelationDesc;
JunkFilter *j;
subplan = mtstate->mt_plans[i]->plan;
if (operation == CMD_INSERT || operation == CMD_UPDATE)
- ExecCheckPlanOutput(resultRelInfo->ri_RelationDesc,
- subplan->targetlist);
+ ExecCheckPlanOutput(rel, subplan->targetlist);
j = ExecInitJunkFilter(subplan->targetlist,
- resultRelInfo->ri_RelationDesc->rd_att->tdhasoid,
+ RelationGetDescr(rel)->tdhasoid,
ExecInitExtraTupleSlot(estate));
if (operation == CMD_UPDATE || operation == CMD_DELETE)
{
/* For UPDATE/DELETE, find the appropriate junk attr now */
- if (resultRelInfo->ri_RelationDesc->rd_rel->relkind == RELKIND_RELATION)
+ if (RelationGetForm(rel)->relkind == RELKIND_RELATION ||
+ RelationGetForm(rel)->relkind == RELKIND_FOREIGN_TABLE)
{
j->jf_junkAttNo = ExecFindJunkAttribute(j, "ctid");
if (!AttributeNumberIsValid(j->jf_junkAttNo))
@@ -1181,6 +1221,21 @@ ExecEndModifyTable(ModifyTableState *node)
{
int i;
+ /* Let the FDW shut dowm */
+ for (i=0; i < node->ps.state->es_num_result_relations; i++)
+ {
+ ResultRelInfo *resultRelInfo
+ = &node->ps.state->es_result_relations[i];
+
+ if (resultRelInfo->ri_fdwroutine)
+ {
+ FdwRoutine *fdwroutine = resultRelInfo->ri_fdwroutine;
+
+ if (fdwroutine->EndForeignModify)
+ fdwroutine->EndForeignModify(resultRelInfo);
+ }
+ }
+
/*
* Free the exprcontext
*/
diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c
index 8f75948..f461b87 100644
--- a/src/backend/rewrite/rewriteHandler.c
+++ b/src/backend/rewrite/rewriteHandler.c
@@ -1166,7 +1166,8 @@ rewriteTargetListUD(Query *parsetree, RangeTblEntry *target_rte,
const char *attrname;
TargetEntry *tle;
- if (target_relation->rd_rel->relkind == RELKIND_RELATION)
+ if (target_relation->rd_rel->relkind == RELKIND_RELATION ||
+ target_relation->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
{
/*
* Emit CTID so that executor can find the row to update or delete.
diff --git a/src/include/foreign/fdwapi.h b/src/include/foreign/fdwapi.h
index 721cd25..252d505 100644
--- a/src/include/foreign/fdwapi.h
+++ b/src/include/foreign/fdwapi.h
@@ -59,6 +59,20 @@ typedef bool (*AnalyzeForeignTable_function) (Relation relation,
AcquireSampleRowsFunc *func,
BlockNumber *totalpages);
+typedef void (*BeginForeignModify_function) (CmdType operation,
+ ResultRelInfo *resultRelInfo,
+ EState *estate,
+ Plan *subplan,
+ int eflags);
+typedef void (*ExecForeignInsert_function) (ResultRelInfo *resultRelInfo,
+ HeapTuple tuple);
+typedef void (*ExecForeignDelete_function) (ResultRelInfo *resultRelInfo,
+ ItemPointer tupleid);
+typedef void (*ExecForeignUpdate_function) (ResultRelInfo *resultRelInfo,
+ ItemPointer tupleid,
+ HeapTuple tuple);
+typedef void (*EndForeignModify_function) (ResultRelInfo *resultRelInfo);
+
/*
* FdwRoutine is the struct returned by a foreign-data wrapper's handler
* function. It provides pointers to the callback functions needed by the
@@ -90,6 +104,11 @@ typedef struct FdwRoutine
* not provided.
*/
AnalyzeForeignTable_function AnalyzeForeignTable;
+ BeginForeignModify_function BeginForeignModify;
+ ExecForeignInsert_function ExecForeignInsert;
+ ExecForeignDelete_function ExecForeignDelete;
+ ExecForeignUpdate_function ExecForeignUpdate;
+ EndForeignModify_function EndForeignModify;
} FdwRoutine;
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index fec07b8..db5b79b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -303,6 +303,8 @@ typedef struct JunkFilter
* ConstraintExprs array of constraint-checking expr states
* junkFilter for removing junk attributes from tuples
* projectReturning for computing a RETURNING list
+ * fdwroutine FDW callbacks if foreign table
+ * fdw_state opaque state of FDW module, or NULL
* ----------------
*/
typedef struct ResultRelInfo
@@ -320,6 +322,8 @@ typedef struct ResultRelInfo
List **ri_ConstraintExprs;
JunkFilter *ri_junkFilter;
ProjectionInfo *ri_projectReturning;
+ struct FdwRoutine *ri_fdwroutine;
+ void *ri_fdw_state;
} ResultRelInfo;
/* ----------------
diff --git a/src/include/storage/itemptr.h b/src/include/storage/itemptr.h
index 331812b..8fdee45 100644
--- a/src/include/storage/itemptr.h
+++ b/src/include/storage/itemptr.h
@@ -135,6 +135,31 @@ typedef ItemPointerData *ItemPointer;
(pointer)->ip_posid = InvalidOffsetNumber \
)
+/*
+ * PackPseudoItemPointer
+ * Pack a pseudo item pointer of foreign table
+ */
+#define PackPseudoItemPointer(pointer,value) \
+ do { \
+ AssertMacro((value) < (1UL << (8 * (sizeof(BlockIdData) + \
+ sizeof(OffsetNumber))))); \
+ BlockIdSet(&((pointer)->ip_blkid), \
+ (value) >> (8 * sizeof(OffsetNumber))); \
+ (pointer)->ip_posid = \
+ (value) & ((1 << (8 * sizeof(OffsetNumber))) - 1); \
+ } while(0)
+
+/*
+ * UnpackPseudoItemPointer
+ * Unpack a pseudo item pointer of foreign table
+ */
+#define UnpackPseudoItemPointer(pointer) \
+ ( \
+ (BlockIdGetBlockNumber(&(pointer)->ip_blkid) \
+ << (8 * sizeof(OffsetNumber))) \
+ | ((pointer)->ip_posid) \
+ )
+
/* ----------------
* externs
* ----------------
On Thu, Aug 23, 2012 at 1:10 AM, Kohei KaiGai <kaigai@kaigai.gr.jp> wrote:
It is a responsibility of FDW extension (and DBA) to ensure each
foreign-row has a unique identifier that has 48-bits width integer
data type in maximum.
It strikes me as incredibly short-sighted to decide that the row
identifier has to have the same format as what our existing heap AM
happens to have. I think we need to allow the row identifier to be of
any data type, and even compound. For example, the foreign side might
have no equivalent of CTID, and thus use primary key. And the primary
key might consist of an integer and a string, or some such.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
2012/8/25 Robert Haas <robertmhaas@gmail.com>:
On Thu, Aug 23, 2012 at 1:10 AM, Kohei KaiGai <kaigai@kaigai.gr.jp> wrote:
It is a responsibility of FDW extension (and DBA) to ensure each
foreign-row has a unique identifier that has 48-bits width integer
data type in maximum.It strikes me as incredibly short-sighted to decide that the row
identifier has to have the same format as what our existing heap AM
happens to have. I think we need to allow the row identifier to be of
any data type, and even compound. For example, the foreign side might
have no equivalent of CTID, and thus use primary key. And the primary
key might consist of an integer and a string, or some such.
I assume it is a task of FDW extension to translate between the pseudo
ctid and the primary key in remote side.
For example, if primary key of the remote table is Text data type, an idea
is to use a hash table to track the text-formed primary being associated
with a particular 48-bits integer. The pseudo ctid shall be utilized to track
the tuple to be modified on the scan-stage, then FDW can reference the
hash table to pull-out the primary key to be provided on the prepared
statement.
Do we have some other reasonable ideas?
Thanks,
--
KaiGai Kohei <kaigai@kaigai.gr.jp>
Kohei KaiGai wrote:
2012/8/25 Robert Haas <robertmhaas@gmail.com>:
On Thu, Aug 23, 2012 at 1:10 AM, Kohei KaiGai <kaigai@kaigai.gr.jp>
wrote:
It is a responsibility of FDW extension (and DBA) to ensure each
foreign-row has a unique identifier that has 48-bits width integer
data type in maximum.
It strikes me as incredibly short-sighted to decide that the row
identifier has to have the same format as what our existing heap AM
happens to have. I think we need to allow the row identifier to be
of
any data type, and even compound. For example, the foreign side
might
have no equivalent of CTID, and thus use primary key. And the
primary
key might consist of an integer and a string, or some such.
I assume it is a task of FDW extension to translate between the pseudo
ctid and the primary key in remote side.For example, if primary key of the remote table is Text data type, an
idea
is to use a hash table to track the text-formed primary being
associated
with a particular 48-bits integer. The pseudo ctid shall be utilized
to track
the tuple to be modified on the scan-stage, then FDW can reference the
hash table to pull-out the primary key to be provided on the prepared
statement.
And what if there is a hash collision? Then you would not be able to
determine which row is meant.
I agree with Robert that this should be flexible enough to cater for
all kinds of row identifiers. Oracle, for example, uses ten byte
identifiers which would give me a headache with your suggested design.
Do we have some other reasonable ideas?
Would it be too invasive to introduce a new pointer in TupleTableSlot
that is NULL for anything but virtual tuples from foreign tables?
Yours,
Laurenz Albe
Kaigai-san,
On Thu, Aug 23, 2012 at 2:10 PM, Kohei KaiGai <kaigai@kaigai.gr.jp> wrote:
The patched portion at contrib/file_fdw.c does not make sense
actually. It just prints messages for each invocation.
It is just a proof-of-concept to show possibility of implementation
based on real RDBMS.
Attached is a tar ball of pgsql_fdw. It's WIP and contains no
document, but it would be enough for your PoC purpose. Usage and
features are same as the last version posted for 9.2 cycle.
# I'll post finished patch in the CF-Sep.
Here are random comments for your PoC patch:
+ As Robert says, using CTID as virtual tuple identifier doesn't seem
nice when considering various FDWs for NoSQL or RDBMS. Having abstract
layer between FDWs and tuple sounds better, but implementing it by each
FDW seems useless effort. Do yo have any idea of generic mechanism for
tuple mapping?
+ Do you have any plan about deparsing local qualifiers into remote
query to avoid repeated query submission? This would improve
performance of big UPDATE, but its use case might be limited to
statements which consist of one foreign table. For this case, we can
consider pass-through mode as second way.
+ I have not read your patch closely yet, but I wonder how we can know
which column is actually updated. If we have only updated image of
tuple, we have to update all remote columns by "new" values?
--
Shigeru Hanada
Attachments:
pgsql_fdw_93.tar.gzapplication/gzip; name=pgsql_fdw_93.tar.gzDownload
� (;P ��y[G�8<�J���$F���1��( lM0"��87�WO#���V���2f����Yj�E���w�'1Rw�u��u��,�=M'�<���O�����'�����G[�_���������;�<�����������G�h<�g�f^"������x���z�������s/I�������������h����h���C����F��������q[����
D��V����\������8���,�?�:���2 ��3�7��N�*$����������e�y/��v��s<�v���n�Xv;��, N�k �o���y���=���b���9�;�<_�[����O��Y�E�7�������0>��������o���xY�T�.�~�e<�y�$��O?�1l��E��_�e���FgA4��wQ<������"�U/K_��7��X+�g�,�����5X�����[/7������U����YdA��?]aDi��0�L�����b��e��*�P ������������7�9������}����A :�n0����Y��"n�����w�������z�$���O��up��$���z����y����0xw�a�V;��,�7����`WM��� ���8\8�^�C�8��b�E����D3����3<M���BO_c�6&
at`�-N�����.�� 3O�ds���7���V�����i�tq -d~�^����|2��M�w�-����]�W��d�����0�f��ie�-}����zq���a8�K�^\��� �s8�J�h����}�]N�pG��]�u���GTX:v����q����=�fK��zI;I����t���5���WEN���E����Bk��M�+�~��E����OfA���S'����%���z�p�D�3 �q��4 R�zS?�tP�� tc,�a���{�i�C:�����f5�������4�a�j��H_�3�����$�����'�8���EK/����,�Q&�"�R���r�������bx@18_�,I �^�q�C�),�l���RaS�:���x
{�����l��R(6Dr�}�A."_������q���n�K����B�L� ��0�T������ '����!��9�0�/"