Construction of Plan-node by CSP (RE: Custom/Foreign-Join-APIs)
-----Original Message-----
Sent: Friday, May 15, 2015 8:44 AM
To: 'Tom Lane'; Kohei KaiGai
Cc: Robert Haas; Thom Brown; Shigeru Hanada; pgsql-hackers@postgreSQL.org
Subject: RE: Custom/Foreign-Join-APIs (Re: [HACKERS] [v9.5] Custom Plan API)A possible compromise that we could perhaps still wedge into 9.5 is to
extend CustomPath with a List of child Paths, and CustomScan with a List
of child Plans, which createplan.c would know to build from the Paths,
and other modules would then also be aware of these children. I find that
uglier than a separate join node type, but it would be tolerable I guess.The attached patch implements what you suggested as is.
It allows custom-scan providers to have child Paths without exporting
create_plan_recurse(), and enables to represent N-way join naturally.
Please add any solution, even if we don't reach the consensus of how
create_plan_recurse (and other useful static functions) are visible to
extensions.
I updated the patch to fix up this problem towards the latest master
branch.
Let me remind the problem again. (I really have a hard time of it)
When an extension tries to implement its own join logic using custom-
scan interface, it adds CustomPath on set_join_pathlist_hook with its
cost estimation.
Once the path gets chosen by planner, PlanCustomPath callback shall be
invoked, then, the custom-scan provider will construct its CustomScan
node according to the path, and I expected it recursively initializes
underlying Path nodes (that work as join input, if any) using
create_plan_recurse().
However, at this moment, we didn't get 100% consensus to export this
function to extensions. So, later commit made this function as static
one, again.
Instead of this approach, Tom suggested to add a list of child Paths
on CustomPath node, then createplan.c calls create_plan_recurse() for
each entry of the list, without this function getting exported.
I can agree with this approach as an alternative of the previous
public create_plan_recurse(), and the attached patch implements this
idea, as is. (Do I understand his suggestion correctly?)
Below is the expectation of the custom-scan provider which takes
underlying Path/Plan nodes.
1. It adds a list of underlying Path nodes on custom_children of
the CustomPath node.
2. On the PlanCustomPath, it adds adds Plan nodes (initialized by
createplan.c, and passed as argument) onto lefttree, righttree
and/or custom_children of CustomScan node
3. On the BeginCustomScan, it calls ExecInitNode() to begin execution
of the underlying plan node. Then, if it has more than 2 children,
attach these PlanState objects on the custom_children list for
EXPLAIN output.
As long as extension follows the above interface contract, it can
have underlying child Path/Plan/PlanState without direct call of
create_plan_recurse() as previously argued.
I think it is enough reasonable solution for the problem.
How about people's thought?
Thanks,
--
NEC Business Creation Division / PG-Strom Project
KaiGai Kohei <kaigai@ak.jp.nec.com>
Attachments:
custom-join-children.v2.patchapplication/octet-stream; name=custom-join-children.v2.patchDownload
doc/src/sgml/custom-scan.sgml | 12 +++++++++++-
src/backend/commands/explain.c | 22 ++++++++++++++++++++++
src/backend/optimizer/plan/createplan.c | 18 +++++++++++++++++-
src/backend/optimizer/plan/setrefs.c | 8 ++++++++
src/backend/optimizer/plan/subselect.c | 25 +++++++++++++++++++++----
src/include/nodes/execnodes.h | 1 +
src/include/nodes/plannodes.h | 1 +
src/include/nodes/relation.h | 4 +++-
8 files changed, 84 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/custom-scan.sgml b/doc/src/sgml/custom-scan.sgml
index 62a8a33..c7187c7 100644
--- a/doc/src/sgml/custom-scan.sgml
+++ b/doc/src/sgml/custom-scan.sgml
@@ -60,6 +60,7 @@ typedef struct CustomPath
{
Path path;
uint32 flags;
+ List *custom_children;
List *custom_private;
const CustomPathMethods *methods;
} CustomPath;
@@ -73,6 +74,10 @@ typedef struct CustomPath
<literal>CUSTOMPATH_SUPPORT_BACKWARD_SCAN</> if the custom path can support
a backward scan and <literal>CUSTOMPATH_SUPPORT_MARK_RESTORE</> if it
can support mark and restore. Both capabilities are optional.
+ An optional <structfield>custom_children</> is a list of underlying
+ <structname>Path</> nodes that can be executed as input data stream of
+ this custom-path node. If valid list is given, it shall be transformed
+ to the relevant <structname>Plan</> nodes by planner.
<structfield>custom_private</> can be used to store the custom path's
private data. Private data should be stored in a form that can be handled
by <literal>nodeToString</>, so that debugging routines that attempt to
@@ -112,7 +117,8 @@ Plan *(*PlanCustomPath) (PlannerInfo *root,
RelOptInfo *rel,
CustomPath *best_path,
List *tlist,
- List *clauses);
+ List *clauses,
+ List *custom_children);
</programlisting>
Convert a custom path to a finished plan. The return value will generally
be a <literal>CustomScan</> object, which the callback must allocate and
@@ -145,6 +151,7 @@ typedef struct CustomScan
{
Scan scan;
uint32 flags;
+ List *custom_children;
List *custom_exprs;
List *custom_private;
List *custom_scan_tlist;
@@ -159,6 +166,9 @@ typedef struct CustomScan
estimated costs, target lists, qualifications, and so on.
<structfield>flags</> is a bitmask with the same meaning as in
<structname>CustomPath</>.
+ <structfield>custom_children</> can be used to store child
+ <structname>Plan</> nodes, if custom-scan provider takes multiple
+ (more than two) underlying query execution plans.
<structfield>custom_exprs</> should be used to
store expression trees that will need to be fixed up by
<filename>setrefs.c</> and <filename>subselect.c</>, while
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a82c6ff..059059b 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -115,6 +115,8 @@ static void ExplainMemberNodes(List *plans, PlanState **planstates,
List *ancestors, ExplainState *es);
static void ExplainSubPlans(List *plans, List *ancestors,
const char *relationship, ExplainState *es);
+static void ExplainCustomChildren(CustomScanState *css,
+ List *ancestors, ExplainState *es);
static void ExplainProperty(const char *qlabel, const char *value,
bool numeric, ExplainState *es);
static void ExplainOpenGroup(const char *objtype, const char *labelname,
@@ -1624,6 +1626,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
IsA(plan, BitmapAnd) ||
IsA(plan, BitmapOr) ||
IsA(plan, SubqueryScan) ||
+ (IsA(planstate, CustomScanState) &&
+ ((CustomScanState *) planstate)->custom_children != NIL) ||
planstate->subPlan;
if (haschildren)
{
@@ -1678,6 +1682,10 @@ ExplainNode(PlanState *planstate, List *ancestors,
ExplainNode(((SubqueryScanState *) planstate)->subplan, ancestors,
"Subquery", NULL, es);
break;
+ case T_CustomScan:
+ ExplainCustomChildren((CustomScanState *) planstate,
+ ancestors, es);
+ break;
default:
break;
}
@@ -2648,6 +2656,20 @@ ExplainSubPlans(List *plans, List *ancestors,
}
/*
+ * Explain underlying child nodes of CustomScanState, if any
+ */
+static void
+ExplainCustomChildren(CustomScanState *css, List *ancestors, ExplainState *es)
+{
+ ListCell *cell;
+ const char *label =
+ (list_length(css->custom_children) > 1 ? "children" : "child");
+
+ foreach (cell, css->custom_children)
+ ExplainNode((PlanState *) lfirst(cell), ancestors, label, NULL, es);
+}
+
+/*
* Explain a property, such as sort keys or targets, that takes the form of
* a list of unlabeled items. "data" is a list of C strings.
*/
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index a3482de..b935e85 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -2157,6 +2157,21 @@ create_customscan_plan(PlannerInfo *root, CustomPath *best_path,
{
CustomScan *cplan;
RelOptInfo *rel = best_path->path.parent;
+ List *custom_children = NIL;
+ ListCell *lc;
+
+ /*
+ * If CustomPath takes underlying child nodes, we recursively transform
+ * these Path nodes to Plan node.
+ * Custom-scan provider will attach these plans on lefttree, righttree
+ * or custom_children list of CustomScan node.
+ */
+ foreach (lc, best_path->custom_children)
+ {
+ Plan *child = create_plan_recurse(root, (Path *) lfirst(lc));
+
+ custom_children = lappend(custom_children, child);
+ }
/*
* Sort clauses into the best execution order, although custom-scan
@@ -2172,7 +2187,8 @@ create_customscan_plan(PlannerInfo *root, CustomPath *best_path,
rel,
best_path,
tlist,
- scan_clauses);
+ scan_clauses,
+ custom_children);
Assert(IsA(cplan, CustomScan));
/*
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index a7f65dd..71efa85 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -1151,6 +1151,8 @@ set_customscan_references(PlannerInfo *root,
CustomScan *cscan,
int rtoffset)
{
+ ListCell *lc;
+
/* Adjust scanrelid if it's valid */
if (cscan->scan.scanrelid > 0)
cscan->scan.scanrelid += rtoffset;
@@ -1194,6 +1196,12 @@ set_customscan_references(PlannerInfo *root,
fix_scan_list(root, cscan->custom_exprs, rtoffset);
}
+ /* Adjust child plan-nodes recursively, if needed */
+ foreach (lc, cscan->custom_children)
+ {
+ lfirst(lc) = set_plan_refs(root, (Plan *) lfirst(lc), rtoffset);
+ }
+
/* Adjust custom_relids if needed */
if (rtoffset > 0)
{
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index f80abb4..6c05663 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -2373,10 +2373,27 @@ finalize_plan(PlannerInfo *root, Plan *plan, Bitmapset *valid_params,
break;
case T_CustomScan:
- finalize_primnode((Node *) ((CustomScan *) plan)->custom_exprs,
- &context);
- /* We assume custom_scan_tlist cannot contain Params */
- context.paramids = bms_add_members(context.paramids, scan_params);
+ {
+ CustomScan *cscan = (CustomScan *) plan;
+ ListCell *lc;
+
+ finalize_primnode((Node *) cscan->custom_exprs,
+ &context);
+ /* We assume custom_scan_tlist cannot contain Params */
+ context.paramids =
+ bms_add_members(context.paramids, scan_params);
+
+ /* child nodes if any */
+ foreach (lc, cscan->custom_children)
+ {
+ context.paramids =
+ bms_add_members(context.paramids,
+ finalize_plan(root,
+ (Plan *) lfirst(lc),
+ valid_params,
+ scan_params));
+ }
+ }
break;
case T_ModifyTable:
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index db5bd7f..ffdbbfa 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1616,6 +1616,7 @@ typedef struct CustomScanState
{
ScanState ss;
uint32 flags; /* mask of CUSTOMPATH_* flags, see relation.h */
+ List *custom_children;/* list of child PlanState nodes, if any */
const CustomExecMethods *methods;
} CustomScanState;
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index d967219..e936b1b 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -550,6 +550,7 @@ typedef struct CustomScan
{
Scan scan;
uint32 flags; /* mask of CUSTOMPATH_* flags, see relation.h */
+ List *custom_children;/* list of Plan nodes, if any */
List *custom_exprs; /* expressions that custom code may evaluate */
List *custom_private; /* private data for custom code */
List *custom_scan_tlist; /* optional tlist describing scan
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index 279051e..1d3b015 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -927,7 +927,8 @@ typedef struct CustomPathMethods
RelOptInfo *rel,
struct CustomPath *best_path,
List *tlist,
- List *clauses);
+ List *clauses,
+ List *custom_children);
/* Optional: print additional fields besides "private" */
void (*TextOutCustomPath) (StringInfo str,
const struct CustomPath *node);
@@ -937,6 +938,7 @@ typedef struct CustomPath
{
Path path;
uint32 flags; /* mask of CUSTOMPATH_* flags, see above */
+ List *custom_children;/* list of child Path nodes, if any */
List *custom_private;
const CustomPathMethods *methods;
} CustomPath;
On Mon, May 25, 2015 at 5:08 AM, Kouhei Kaigai <kaigai@ak.jp.nec.com> wrote:
I updated the patch to fix up this problem towards the latest master
branch.
[ ... ]
Instead of this approach, Tom suggested to add a list of child Paths
on CustomPath node, then createplan.c calls create_plan_recurse() for
each entry of the list, without this function getting exported.
Tom, do you want to review this patch and figure out how to solve the
underlying problem? If not, I will take care of it. But I will be
unhappy if I put time and effort into this and then you insist on
changing everything afterwards, again.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
Tom, do you want to review this patch and figure out how to solve the
underlying problem? If not, I will take care of it. But I will be
unhappy if I put time and effort into this and then you insist on
changing everything afterwards, again.
[ sorry for slow response, been busy ] I will take a look.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
-----Original Message-----
From: pgsql-hackers-owner@postgresql.org
[mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Tom Lane
Sent: Thursday, May 28, 2015 5:35 AM
To: Robert Haas
Cc: Kaigai Kouhei(海外 浩平); Thom Brown; Kohei KaiGai; Shigeru Hanada;
pgsql-hackers@postgreSQL.org
Subject: Re: [HACKERS] Construction of Plan-node by CSP (RE:
Custom/Foreign-Join-APIs)Robert Haas <robertmhaas@gmail.com> writes:
Tom, do you want to review this patch and figure out how to solve the
underlying problem? If not, I will take care of it. But I will be
unhappy if I put time and effort into this and then you insist on
changing everything afterwards, again.[ sorry for slow response, been busy ] I will take a look.
Tom, how about your availability?
From my side, I adjust my extension (PG-Strom) to fit the infrastructure you proposed,
then confirmed it is workable even if custom-scan, that replaced relations join, takes
more than two Path nodes in the custom_children list of CustomPath, with no exportiong
create_plan_recurse().
Below is an example of custom-scan (GpuJoin) that involves four relations join.
Its code base is the latest master + custom-join-children.v2.patch; unchanged from
the last post.
postgres=# explain analyze select avg(x) from t0 natural join t1 natural join t2 natural join t3 group by cat;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------
HashAggregate (cost=298513.77..298514.10 rows=26 width=12) (actual time=5622.028..5622.033 rows=26 loops=1)
Group Key: t0.cat
-> Custom Scan (GpuJoin) (cost=4702.00..249169.85 rows=9868784 width=12) (actual time=540.718..2075.566 rows=10000000 loops=1)
Bulkload: On (density: 100.00%)
Depth 1: Logic: GpuHashJoin, HashKeys: (cid), JoinQual: (cid = cid), nrows_ratio: 0.98936439
Depth 2: Logic: GpuHashJoin, HashKeys: (bid), JoinQual: (bid = bid), nrows_ratio: 0.99748135
Depth 3: Logic: GpuHashJoin, HashKeys: (aid), JoinQual: (aid = aid), nrows_ratio: 1.00000000
-> Custom Scan (BulkScan) on t0 (cost=0.00..242858.60 rows=10000060 width=24) (actual time=8.555..903.864 rows=10000000 loops=1)
-> Seq Scan on t3 (cost=0.00..734.00 rows=40000 width=4) (actual time=0.019..4.370 rows=40000 loops=1)
-> Seq Scan on t2 (cost=0.00..734.00 rows=40000 width=4) (actual time=0.004..4.182 rows=40000 loops=1)
-> Seq Scan on t1 (cost=0.00..734.00 rows=40000 width=4) (actual time=0.005..4.275 rows=40000 loops=1)
Planning time: 0.918 ms
Execution time: 6178.264 ms
(13 rows)
Thanks,
--
NEC Business Creation Division / PG-Strom Project
KaiGai Kohei <kaigai@ak.jp.nec.com>
Attachments:
custom-join-children.v2.patchapplication/octet-stream; name=custom-join-children.v2.patchDownload
doc/src/sgml/custom-scan.sgml | 12 +++++++++++-
src/backend/commands/explain.c | 22 ++++++++++++++++++++++
src/backend/optimizer/plan/createplan.c | 18 +++++++++++++++++-
src/backend/optimizer/plan/setrefs.c | 8 ++++++++
src/backend/optimizer/plan/subselect.c | 25 +++++++++++++++++++++----
src/include/nodes/execnodes.h | 1 +
src/include/nodes/plannodes.h | 1 +
src/include/nodes/relation.h | 4 +++-
8 files changed, 84 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/custom-scan.sgml b/doc/src/sgml/custom-scan.sgml
index 62a8a33..c7187c7 100644
--- a/doc/src/sgml/custom-scan.sgml
+++ b/doc/src/sgml/custom-scan.sgml
@@ -60,6 +60,7 @@ typedef struct CustomPath
{
Path path;
uint32 flags;
+ List *custom_children;
List *custom_private;
const CustomPathMethods *methods;
} CustomPath;
@@ -73,6 +74,10 @@ typedef struct CustomPath
<literal>CUSTOMPATH_SUPPORT_BACKWARD_SCAN</> if the custom path can support
a backward scan and <literal>CUSTOMPATH_SUPPORT_MARK_RESTORE</> if it
can support mark and restore. Both capabilities are optional.
+ An optional <structfield>custom_children</> is a list of underlying
+ <structname>Path</> nodes that can be executed as input data stream of
+ this custom-path node. If valid list is given, it shall be transformed
+ to the relevant <structname>Plan</> nodes by planner.
<structfield>custom_private</> can be used to store the custom path's
private data. Private data should be stored in a form that can be handled
by <literal>nodeToString</>, so that debugging routines that attempt to
@@ -112,7 +117,8 @@ Plan *(*PlanCustomPath) (PlannerInfo *root,
RelOptInfo *rel,
CustomPath *best_path,
List *tlist,
- List *clauses);
+ List *clauses,
+ List *custom_children);
</programlisting>
Convert a custom path to a finished plan. The return value will generally
be a <literal>CustomScan</> object, which the callback must allocate and
@@ -145,6 +151,7 @@ typedef struct CustomScan
{
Scan scan;
uint32 flags;
+ List *custom_children;
List *custom_exprs;
List *custom_private;
List *custom_scan_tlist;
@@ -159,6 +166,9 @@ typedef struct CustomScan
estimated costs, target lists, qualifications, and so on.
<structfield>flags</> is a bitmask with the same meaning as in
<structname>CustomPath</>.
+ <structfield>custom_children</> can be used to store child
+ <structname>Plan</> nodes, if custom-scan provider takes multiple
+ (more than two) underlying query execution plans.
<structfield>custom_exprs</> should be used to
store expression trees that will need to be fixed up by
<filename>setrefs.c</> and <filename>subselect.c</>, while
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a82c6ff..059059b 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -115,6 +115,8 @@ static void ExplainMemberNodes(List *plans, PlanState **planstates,
List *ancestors, ExplainState *es);
static void ExplainSubPlans(List *plans, List *ancestors,
const char *relationship, ExplainState *es);
+static void ExplainCustomChildren(CustomScanState *css,
+ List *ancestors, ExplainState *es);
static void ExplainProperty(const char *qlabel, const char *value,
bool numeric, ExplainState *es);
static void ExplainOpenGroup(const char *objtype, const char *labelname,
@@ -1624,6 +1626,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
IsA(plan, BitmapAnd) ||
IsA(plan, BitmapOr) ||
IsA(plan, SubqueryScan) ||
+ (IsA(planstate, CustomScanState) &&
+ ((CustomScanState *) planstate)->custom_children != NIL) ||
planstate->subPlan;
if (haschildren)
{
@@ -1678,6 +1682,10 @@ ExplainNode(PlanState *planstate, List *ancestors,
ExplainNode(((SubqueryScanState *) planstate)->subplan, ancestors,
"Subquery", NULL, es);
break;
+ case T_CustomScan:
+ ExplainCustomChildren((CustomScanState *) planstate,
+ ancestors, es);
+ break;
default:
break;
}
@@ -2648,6 +2656,20 @@ ExplainSubPlans(List *plans, List *ancestors,
}
/*
+ * Explain underlying child nodes of CustomScanState, if any
+ */
+static void
+ExplainCustomChildren(CustomScanState *css, List *ancestors, ExplainState *es)
+{
+ ListCell *cell;
+ const char *label =
+ (list_length(css->custom_children) > 1 ? "children" : "child");
+
+ foreach (cell, css->custom_children)
+ ExplainNode((PlanState *) lfirst(cell), ancestors, label, NULL, es);
+}
+
+/*
* Explain a property, such as sort keys or targets, that takes the form of
* a list of unlabeled items. "data" is a list of C strings.
*/
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index a3482de..b935e85 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -2157,6 +2157,21 @@ create_customscan_plan(PlannerInfo *root, CustomPath *best_path,
{
CustomScan *cplan;
RelOptInfo *rel = best_path->path.parent;
+ List *custom_children = NIL;
+ ListCell *lc;
+
+ /*
+ * If CustomPath takes underlying child nodes, we recursively transform
+ * these Path nodes to Plan node.
+ * Custom-scan provider will attach these plans on lefttree, righttree
+ * or custom_children list of CustomScan node.
+ */
+ foreach (lc, best_path->custom_children)
+ {
+ Plan *child = create_plan_recurse(root, (Path *) lfirst(lc));
+
+ custom_children = lappend(custom_children, child);
+ }
/*
* Sort clauses into the best execution order, although custom-scan
@@ -2172,7 +2187,8 @@ create_customscan_plan(PlannerInfo *root, CustomPath *best_path,
rel,
best_path,
tlist,
- scan_clauses);
+ scan_clauses,
+ custom_children);
Assert(IsA(cplan, CustomScan));
/*
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index a7f65dd..71efa85 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -1151,6 +1151,8 @@ set_customscan_references(PlannerInfo *root,
CustomScan *cscan,
int rtoffset)
{
+ ListCell *lc;
+
/* Adjust scanrelid if it's valid */
if (cscan->scan.scanrelid > 0)
cscan->scan.scanrelid += rtoffset;
@@ -1194,6 +1196,12 @@ set_customscan_references(PlannerInfo *root,
fix_scan_list(root, cscan->custom_exprs, rtoffset);
}
+ /* Adjust child plan-nodes recursively, if needed */
+ foreach (lc, cscan->custom_children)
+ {
+ lfirst(lc) = set_plan_refs(root, (Plan *) lfirst(lc), rtoffset);
+ }
+
/* Adjust custom_relids if needed */
if (rtoffset > 0)
{
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index f80abb4..6c05663 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -2373,10 +2373,27 @@ finalize_plan(PlannerInfo *root, Plan *plan, Bitmapset *valid_params,
break;
case T_CustomScan:
- finalize_primnode((Node *) ((CustomScan *) plan)->custom_exprs,
- &context);
- /* We assume custom_scan_tlist cannot contain Params */
- context.paramids = bms_add_members(context.paramids, scan_params);
+ {
+ CustomScan *cscan = (CustomScan *) plan;
+ ListCell *lc;
+
+ finalize_primnode((Node *) cscan->custom_exprs,
+ &context);
+ /* We assume custom_scan_tlist cannot contain Params */
+ context.paramids =
+ bms_add_members(context.paramids, scan_params);
+
+ /* child nodes if any */
+ foreach (lc, cscan->custom_children)
+ {
+ context.paramids =
+ bms_add_members(context.paramids,
+ finalize_plan(root,
+ (Plan *) lfirst(lc),
+ valid_params,
+ scan_params));
+ }
+ }
break;
case T_ModifyTable:
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index db5bd7f..ffdbbfa 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1616,6 +1616,7 @@ typedef struct CustomScanState
{
ScanState ss;
uint32 flags; /* mask of CUSTOMPATH_* flags, see relation.h */
+ List *custom_children;/* list of child PlanState nodes, if any */
const CustomExecMethods *methods;
} CustomScanState;
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index d967219..e936b1b 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -550,6 +550,7 @@ typedef struct CustomScan
{
Scan scan;
uint32 flags; /* mask of CUSTOMPATH_* flags, see relation.h */
+ List *custom_children;/* list of Plan nodes, if any */
List *custom_exprs; /* expressions that custom code may evaluate */
List *custom_private; /* private data for custom code */
List *custom_scan_tlist; /* optional tlist describing scan
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index 279051e..1d3b015 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -927,7 +927,8 @@ typedef struct CustomPathMethods
RelOptInfo *rel,
struct CustomPath *best_path,
List *tlist,
- List *clauses);
+ List *clauses,
+ List *custom_children);
/* Optional: print additional fields besides "private" */
void (*TextOutCustomPath) (StringInfo str,
const struct CustomPath *node);
@@ -937,6 +938,7 @@ typedef struct CustomPath
{
Path path;
uint32 flags; /* mask of CUSTOMPATH_* flags, see above */
+ List *custom_children;/* list of child Path nodes, if any */
List *custom_private;
const CustomPathMethods *methods;
} CustomPath;
Robert Haas <robertmhaas@gmail.com> writes:
Tom, do you want to review this patch and figure out how to solve the
underlying problem? If not, I will take care of it. But I will be
unhappy if I put time and effort into this and then you insist on
changing everything afterwards, again.[ sorry for slow response, been busy ] I will take a look.
regards, tom lane
Tom, please don't forget the problem.
It is still problematic for custom-scan provider that tries to
implement own join logic, thus we still have to apply additional
patch (or copy&paste createplan.c to module's source).
Thanks,
--
NEC Business Creation Division / PG-Strom Project
KaiGai Kohei <kaigai@ak.jp.nec.com>
Attachments:
custom-join-children.v2.patchapplication/octet-stream; name=custom-join-children.v2.patchDownload
doc/src/sgml/custom-scan.sgml | 12 +++++++++++-
src/backend/commands/explain.c | 22 ++++++++++++++++++++++
src/backend/optimizer/plan/createplan.c | 18 +++++++++++++++++-
src/backend/optimizer/plan/setrefs.c | 8 ++++++++
src/backend/optimizer/plan/subselect.c | 25 +++++++++++++++++++++----
src/include/nodes/execnodes.h | 1 +
src/include/nodes/plannodes.h | 1 +
src/include/nodes/relation.h | 4 +++-
8 files changed, 84 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/custom-scan.sgml b/doc/src/sgml/custom-scan.sgml
index 62a8a33..c7187c7 100644
--- a/doc/src/sgml/custom-scan.sgml
+++ b/doc/src/sgml/custom-scan.sgml
@@ -60,6 +60,7 @@ typedef struct CustomPath
{
Path path;
uint32 flags;
+ List *custom_children;
List *custom_private;
const CustomPathMethods *methods;
} CustomPath;
@@ -73,6 +74,10 @@ typedef struct CustomPath
<literal>CUSTOMPATH_SUPPORT_BACKWARD_SCAN</> if the custom path can support
a backward scan and <literal>CUSTOMPATH_SUPPORT_MARK_RESTORE</> if it
can support mark and restore. Both capabilities are optional.
+ An optional <structfield>custom_children</> is a list of underlying
+ <structname>Path</> nodes that can be executed as input data stream of
+ this custom-path node. If valid list is given, it shall be transformed
+ to the relevant <structname>Plan</> nodes by planner.
<structfield>custom_private</> can be used to store the custom path's
private data. Private data should be stored in a form that can be handled
by <literal>nodeToString</>, so that debugging routines that attempt to
@@ -112,7 +117,8 @@ Plan *(*PlanCustomPath) (PlannerInfo *root,
RelOptInfo *rel,
CustomPath *best_path,
List *tlist,
- List *clauses);
+ List *clauses,
+ List *custom_children);
</programlisting>
Convert a custom path to a finished plan. The return value will generally
be a <literal>CustomScan</> object, which the callback must allocate and
@@ -145,6 +151,7 @@ typedef struct CustomScan
{
Scan scan;
uint32 flags;
+ List *custom_children;
List *custom_exprs;
List *custom_private;
List *custom_scan_tlist;
@@ -159,6 +166,9 @@ typedef struct CustomScan
estimated costs, target lists, qualifications, and so on.
<structfield>flags</> is a bitmask with the same meaning as in
<structname>CustomPath</>.
+ <structfield>custom_children</> can be used to store child
+ <structname>Plan</> nodes, if custom-scan provider takes multiple
+ (more than two) underlying query execution plans.
<structfield>custom_exprs</> should be used to
store expression trees that will need to be fixed up by
<filename>setrefs.c</> and <filename>subselect.c</>, while
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a82c6ff..059059b 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -115,6 +115,8 @@ static void ExplainMemberNodes(List *plans, PlanState **planstates,
List *ancestors, ExplainState *es);
static void ExplainSubPlans(List *plans, List *ancestors,
const char *relationship, ExplainState *es);
+static void ExplainCustomChildren(CustomScanState *css,
+ List *ancestors, ExplainState *es);
static void ExplainProperty(const char *qlabel, const char *value,
bool numeric, ExplainState *es);
static void ExplainOpenGroup(const char *objtype, const char *labelname,
@@ -1624,6 +1626,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
IsA(plan, BitmapAnd) ||
IsA(plan, BitmapOr) ||
IsA(plan, SubqueryScan) ||
+ (IsA(planstate, CustomScanState) &&
+ ((CustomScanState *) planstate)->custom_children != NIL) ||
planstate->subPlan;
if (haschildren)
{
@@ -1678,6 +1682,10 @@ ExplainNode(PlanState *planstate, List *ancestors,
ExplainNode(((SubqueryScanState *) planstate)->subplan, ancestors,
"Subquery", NULL, es);
break;
+ case T_CustomScan:
+ ExplainCustomChildren((CustomScanState *) planstate,
+ ancestors, es);
+ break;
default:
break;
}
@@ -2648,6 +2656,20 @@ ExplainSubPlans(List *plans, List *ancestors,
}
/*
+ * Explain underlying child nodes of CustomScanState, if any
+ */
+static void
+ExplainCustomChildren(CustomScanState *css, List *ancestors, ExplainState *es)
+{
+ ListCell *cell;
+ const char *label =
+ (list_length(css->custom_children) > 1 ? "children" : "child");
+
+ foreach (cell, css->custom_children)
+ ExplainNode((PlanState *) lfirst(cell), ancestors, label, NULL, es);
+}
+
+/*
* Explain a property, such as sort keys or targets, that takes the form of
* a list of unlabeled items. "data" is a list of C strings.
*/
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index a3482de..b935e85 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -2157,6 +2157,21 @@ create_customscan_plan(PlannerInfo *root, CustomPath *best_path,
{
CustomScan *cplan;
RelOptInfo *rel = best_path->path.parent;
+ List *custom_children = NIL;
+ ListCell *lc;
+
+ /*
+ * If CustomPath takes underlying child nodes, we recursively transform
+ * these Path nodes to Plan node.
+ * Custom-scan provider will attach these plans on lefttree, righttree
+ * or custom_children list of CustomScan node.
+ */
+ foreach (lc, best_path->custom_children)
+ {
+ Plan *child = create_plan_recurse(root, (Path *) lfirst(lc));
+
+ custom_children = lappend(custom_children, child);
+ }
/*
* Sort clauses into the best execution order, although custom-scan
@@ -2172,7 +2187,8 @@ create_customscan_plan(PlannerInfo *root, CustomPath *best_path,
rel,
best_path,
tlist,
- scan_clauses);
+ scan_clauses,
+ custom_children);
Assert(IsA(cplan, CustomScan));
/*
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index a7f65dd..71efa85 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -1151,6 +1151,8 @@ set_customscan_references(PlannerInfo *root,
CustomScan *cscan,
int rtoffset)
{
+ ListCell *lc;
+
/* Adjust scanrelid if it's valid */
if (cscan->scan.scanrelid > 0)
cscan->scan.scanrelid += rtoffset;
@@ -1194,6 +1196,12 @@ set_customscan_references(PlannerInfo *root,
fix_scan_list(root, cscan->custom_exprs, rtoffset);
}
+ /* Adjust child plan-nodes recursively, if needed */
+ foreach (lc, cscan->custom_children)
+ {
+ lfirst(lc) = set_plan_refs(root, (Plan *) lfirst(lc), rtoffset);
+ }
+
/* Adjust custom_relids if needed */
if (rtoffset > 0)
{
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index f80abb4..6c05663 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -2373,10 +2373,27 @@ finalize_plan(PlannerInfo *root, Plan *plan, Bitmapset *valid_params,
break;
case T_CustomScan:
- finalize_primnode((Node *) ((CustomScan *) plan)->custom_exprs,
- &context);
- /* We assume custom_scan_tlist cannot contain Params */
- context.paramids = bms_add_members(context.paramids, scan_params);
+ {
+ CustomScan *cscan = (CustomScan *) plan;
+ ListCell *lc;
+
+ finalize_primnode((Node *) cscan->custom_exprs,
+ &context);
+ /* We assume custom_scan_tlist cannot contain Params */
+ context.paramids =
+ bms_add_members(context.paramids, scan_params);
+
+ /* child nodes if any */
+ foreach (lc, cscan->custom_children)
+ {
+ context.paramids =
+ bms_add_members(context.paramids,
+ finalize_plan(root,
+ (Plan *) lfirst(lc),
+ valid_params,
+ scan_params));
+ }
+ }
break;
case T_ModifyTable:
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index db5bd7f..ffdbbfa 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1616,6 +1616,7 @@ typedef struct CustomScanState
{
ScanState ss;
uint32 flags; /* mask of CUSTOMPATH_* flags, see relation.h */
+ List *custom_children;/* list of child PlanState nodes, if any */
const CustomExecMethods *methods;
} CustomScanState;
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index d967219..e936b1b 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -550,6 +550,7 @@ typedef struct CustomScan
{
Scan scan;
uint32 flags; /* mask of CUSTOMPATH_* flags, see relation.h */
+ List *custom_children;/* list of Plan nodes, if any */
List *custom_exprs; /* expressions that custom code may evaluate */
List *custom_private; /* private data for custom code */
List *custom_scan_tlist; /* optional tlist describing scan
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index 279051e..1d3b015 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -927,7 +927,8 @@ typedef struct CustomPathMethods
RelOptInfo *rel,
struct CustomPath *best_path,
List *tlist,
- List *clauses);
+ List *clauses,
+ List *custom_children);
/* Optional: print additional fields besides "private" */
void (*TextOutCustomPath) (StringInfo str,
const struct CustomPath *node);
@@ -937,6 +938,7 @@ typedef struct CustomPath
{
Path path;
uint32 flags; /* mask of CUSTOMPATH_* flags, see above */
+ List *custom_children;/* list of child Path nodes, if any */
List *custom_private;
const CustomPathMethods *methods;
} CustomPath;