Parameterized aggregate subquery (was: Pull up aggregate subquery)

Started by Hitoshi Haradaover 14 years ago26 messages
#1Hitoshi Harada
umi.tanuki@gmail.com
1 attachment(s)

Purpose & Goal
--------------
Allow planner to create NestLoop with parameterized aggregate subquery
just like inner IndexScan pattern. This helps to avoid unnecessary
aggregate process that would be filtered at the stage of upper join
filter in such case:

create table size_m as select i as id, repeat(i::text, i % 100) as val
from generate_series(1, 20000) i;
create table size_l as select i as id, m.id as m_id, repeat(i::text, i
% 100) as val from generate_series(1, 1000000) i, size_m m where i.i /
10 = m.id;
analyze size_m;
analyze size_l;
---- make this query faster
select m_id, sum_len from size_m m inner join(select m_id,
sum(length(val)) as sum_len from size_l group by m_id)l on m.id =
l.m_id where val = '10101';

In addition, it might be the first step to take account of "general
parameterized scan" design, although this proposal contains only
narrow cases of aggregate subquery.

Related information
-------------------
This work is the next step of the previously proposed "Pull up
aggregate subqery" thread.
http://archives.postgresql.org/message-id/BANLkTimW-EqS_9hk5AYj14R8U%3DuQoc6tuA%40mail.gmail.com

Design
------
In contract to the previous design, I made aggregate subquery
"parameterized" instead of pulling it up. The design is based on the
discussion with Tom Lane and Rober Haas et al. It has some benefits
compared with the pulling up appoach as, 1) parameterizing any scan
node other than index scan as nest loop's inner is our way to go, 2)
pulling up or pushing down any aggregate query has potential risk that
the aggregate results are wrong (, which may be solvable by adding
some constraints like "only when unique" etc,) 3) the decision whether
to make parameterized or not can be made by only cost without any
heuristics.

The idea is very similar to inner IndexScan. When NestLoop path is
created in match_unsorted_outer(), parameterized SubqueryScan path is
also considered, similarly to inner IndexScan paths. While IndexScan
is simple since its information like costs are well known by the base
relation, SubqueryScan should re-plan its Query to gain that, which is
expensive.

To do so, I added SubqueryPath node to store each subquery
information. Before patching, subplan, subrtable and subrowmark is
stored in RelOptInfo, but now we need to save them separately for each
parameterized one. This part might be split from the original patch
for easy review.

Result
------
Without breakage of regression test, it successfully makes subquery as
parameterized as the inner of NestLoop. Query performance is as aimed.

(without patch)
db1=# explain analyze
select m_id, sum_len from size_m m inner join(select m_id,
sum(length(val)) as sum_len from size_l group by m_id)l on m.id =
l.m_id where val = '10101';
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------
Nested Loop (cost=79393.64..82937.42 rows=100 width=12) (actual
time=1256.569..1733.903 rows=1 loops=1)
Join Filter: (m.id = size_l.m_id)
-> Seq Scan on size_m m (cost=0.00..897.00 rows=1 width=4)
(actual time=25.182..32.237 rows=1 loops=1)
Filter: (val = '10101'::text)
-> GroupAggregate (cost=79393.64..81592.65 rows=19901 width=277)
(actual time=772.791..1694.848 rows=20000 loops=1)
-> Sort (cost=79393.64..79893.64 rows=200000 width=277)
(actual time=772.754..929.225 rows=200000 loops=1)
Sort Key: size_l.m_id
Sort Method: external sort Disk: 56848kB
-> Seq Scan on size_l (cost=0.00..9830.00 rows=200000
width=277) (actual time=0.030..198.093 rows=200000 loops=1)
Total runtime: 1754.111 ms
(10 rows)

(with patch)
db1=# explain analyze
select m_id, sum_len from size_m m inner join(select m_id,
sum(length(val)) as sum_len from size_l group by m_id)l on m.id =
l.m_id where val = '10101';
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------
Nested Loop (cost=0.00..11674.86 rows=100 width=12) (actual
time=119.386..122.478 rows=1 loops=1)
Join Filter: (m.id = size_l.m_id)
-> Seq Scan on size_m m (cost=0.00..897.00 rows=1 width=4)
(actual time=9.720..12.811 rows=1 loops=1)
Filter: (val = '10101'::text)
-> GroupAggregate (cost=0.00..10330.08 rows=1 width=277) (actual
time=109.639..109.640 rows=1 loops=1)
-> Seq Scan on size_l (cost=0.00..10330.00 rows=10
width=277) (actual time=59.138..109.611 rows=10 loops=1)
Filter: (m.id = m_id)
Total runtime: 122.612 ms
(8 rows)

I tested some more cases like "where val IN ('1', '10101')", "where
val between '1' and '10101'", which shows as good as I expected.

Concern
-------
Although execution time will be shorter in many cases, planning time
will be longer. Especially re-planning subquery with pushing join
quals for each joinrel step. I didn't measure the additional cost in
planner stage.

BTW, as I changed title and design from the previous post, should I
throw away the old commit fest entry and make the new one?

Regards,

--
Hitoshi Harada

Attachments:

aggjoin-20110609.patchapplication/octet-stream; name=aggjoin-20110609.patchDownload
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 681f5f8..039fd7f 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -1557,6 +1557,17 @@ _outTidPath(StringInfo str, TidPath *node)
 }
 
 static void
+_outSubqueryPath(StringInfo str, SubqueryPath *node)
+{
+	WRITE_NODE_TYPE("SUBQUERYPATH");
+
+	_outPathInfo(str, (Path *) node);
+	WRITE_NODE_FIELD(subplan);
+	WRITE_NODE_FIELD(subrtable);
+	WRITE_NODE_FIELD(subrowmark);
+}
+
+static void
 _outForeignPath(StringInfo str, ForeignPath *node)
 {
 	WRITE_NODE_TYPE("FOREIGNPATH");
@@ -1738,9 +1749,6 @@ _outRelOptInfo(StringInfo str, RelOptInfo *node)
 	WRITE_NODE_FIELD(indexlist);
 	WRITE_UINT_FIELD(pages);
 	WRITE_FLOAT_FIELD(tuples, "%.0f");
-	WRITE_NODE_FIELD(subplan);
-	WRITE_NODE_FIELD(subrtable);
-	WRITE_NODE_FIELD(subrowmark);
 	WRITE_NODE_FIELD(baserestrictinfo);
 	WRITE_NODE_FIELD(joininfo);
 	WRITE_BOOL_FIELD(has_eclass_joins);
@@ -2948,6 +2956,9 @@ _outNode(StringInfo str, void *obj)
 			case T_TidPath:
 				_outTidPath(str, obj);
 				break;
+			case T_SubqueryPath:
+				_outSubqueryPath(str, obj);
+				break;
 			case T_ForeignPath:
 				_outForeignPath(str, obj);
 				break;
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index 47ab08e..7449487 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -31,6 +31,7 @@
 #include "optimizer/planner.h"
 #include "optimizer/prep.h"
 #include "optimizer/restrictinfo.h"
+#include "optimizer/subselect.h"
 #include "optimizer/var.h"
 #include "parser/parse_clause.h"
 #include "parser/parsetree.h"
@@ -687,7 +688,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
 	bool	   *differentTypes;
 	double		tuple_fraction;
 	PlannerInfo *subroot;
-	List	   *pathkeys;
+	Plan	    *subplan;
+	List	    *pathkeys;
 
 	/*
 	 * Must copy the Query so that planning doesn't mess up the RTE contents
@@ -749,6 +751,9 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
 
 	pfree(differentTypes);
 
+	/* save it for later use in best_inner_subqueryscan */
+	rel->preprocessed_subquery = copyObject(subquery);
+
 	/*
 	 * We can safely pass the outer tuple_fraction down to the subquery if the
 	 * outer level has no joining, aggregation, or sorting to do. Otherwise
@@ -766,27 +771,182 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
 		tuple_fraction = root->tuple_fraction;
 
 	/* Generate the plan for the subquery */
-	rel->subplan = subquery_planner(root->glob, subquery,
-									root,
-									false, tuple_fraction,
-									&subroot);
-	rel->subrtable = subroot->parse->rtable;
-	rel->subrowmark = subroot->rowMarks;
+	subplan = subquery_planner(root->glob, subquery,
+							   root,
+							   false, tuple_fraction,
+							   &subroot);
 
 	/* Mark rel with estimated output rows, width, etc */
-	set_subquery_size_estimates(root, rel, subroot);
+	set_subquery_size_estimates(root, rel, subroot, subplan);
 
 	/* Convert subquery pathkeys to outer representation */
-	pathkeys = convert_subquery_pathkeys(root, rel, subroot->query_pathkeys);
+	pathkeys = convert_subquery_pathkeys(root, rel,
+										 subroot->query_pathkeys, subplan);
 
 	/* Generate appropriate path */
-	add_path(rel, create_subqueryscan_path(rel, pathkeys));
+	add_path(rel, create_subqueryscan_path(rel, pathkeys,
+			 subplan, subroot->parse->rtable, subroot->rowMarks));
 
 	/* Select cheapest path (pretty easy in this case...) */
 	set_cheapest(rel);
 }
 
 /*
+ * best_inner_subqueryscan
+ *
+ * As the inner scan relation, try to push down some join clauses to
+ * subquery and re-plan it. If such posibility is found, return the path
+ * in *cheapest_start_path or *cheapest_total_path. See also best_inner_indexscan().
+ *
+ * It re-uses modified subquery (a.k.a. rel->preprocessed_subquery.) The basic
+ * push-down of baserestrictinfo was common among base path (done in
+ * set_subquery_pathlist()) so this routine doesn't care it. But once we find
+ * push-down-able join clause, the parsed subquery will be modified after copied.
+ */
+void
+best_inner_subqueryscan(PlannerInfo *root, RelOptInfo *rel,
+						RelOptInfo *outer_rel, List *join_clause,
+						Path **cheapest_start_path, Path **cheapest_total_path)
+{
+	Query	   *parse = root->parse;
+	Query	   *subquery;
+	Index		rti = rel->relid;
+	RangeTblEntry *rte = planner_rt_fetch(rti, root);
+	bool	   *differentTypes;
+	double		tuple_fraction;
+	PlannerInfo *subroot;
+	Plan	   *subplan;
+	List	   *pathkeys;
+	Path	   *path;
+	bool		query_modified = false;
+
+	/* initialize paths to return immediately anytime */
+	*cheapest_start_path = *cheapest_total_path = NULL;
+
+	/*
+	 * empty join condition doesn't make sense here
+	 */
+	if (join_clause == NIL)
+		return;
+
+	/*
+	 * basic push-down-qual process should have done already.
+	 * see set_subquery_pathlist
+	 */
+	Assert(rel->preprocessed_subquery);
+	/* copying the whole Query is expensive; let's copy-on-write */
+	subquery = rel->preprocessed_subquery;
+
+	/* We need a workspace for keeping track of set-op type coercions */
+	differentTypes = (bool *)
+		palloc0((list_length(subquery->targetList) + 1) * sizeof(bool));
+
+	/*
+	 * Currently we focus on only cases of aggregate subquery.
+	 * I'm not quite sure if there are other useful cases.
+	 */
+	if (subquery->hasAggs &&
+		subquery_is_pushdown_safe(subquery, subquery, differentTypes))
+	{
+		ListCell   *l;
+
+		foreach (l, join_clause)
+		{
+			RestrictInfo   *rinfo = (RestrictInfo *) lfirst(l);
+			Expr		   *lexpr, *rexpr;
+			Expr		   *clause = NULL;
+			Param		   *param;
+			OpExpr		   *rclause;
+
+			/*
+			 * For the first step, only Var = Var join clause is targeted.
+			 * This may be improved in the future.
+			 */
+			if (!IsA(rinfo->clause, OpExpr))
+				continue;
+			rclause = (OpExpr *) rinfo->clause;
+			lexpr = list_nth(rclause->args, 0);
+			rexpr = list_nth(rclause->args, 1);
+			if (!(IsA(lexpr, Var) && IsA(rexpr, Var)))
+				continue;
+
+			/* see which arg is mine and outer's */
+			if (bms_is_member(((Var *) lexpr)->varno, outer_rel->relids))
+			{
+				param = assign_nestloop_param(root, (Var *) lexpr);
+				clause = make_opclause(rclause->opno,
+									   rclause->opresulttype,
+									   false,
+									   (Expr *) param,
+									   (Expr *) copyObject(rexpr),
+									   InvalidOid,
+									   rclause->opcollid);
+			}
+			else if (bms_is_member(((Var *) rexpr)->varno, outer_rel->relids))
+			{
+				param = assign_nestloop_param(root, (Var *) rexpr);
+				clause = make_opclause(rclause->opno,
+									   rclause->opresulttype,
+									   false,
+									   (Expr *) copyObject(lexpr),
+									   (Expr *) param,
+									   InvalidOid,
+									   rclause->opcollid);
+			}
+
+			/* TODO:no check isouter or not? */
+			if (clause)
+			{
+				/* copy on write */
+				if (rel->preprocessed_subquery == subquery)
+					subquery = copyObject(subquery);
+				subquery_push_qual(subquery, rte, rti, (Node *) clause);
+				query_modified = true;
+			}
+		}
+	}
+
+	pfree(differentTypes);
+
+	/* return immediately if such subquery isn't found. Use original one */
+	if (!query_modified)
+		return;
+
+	/*
+	 * We can safely pass the outer tuple_fraction down to the subquery if the
+	 * outer level has no joining, aggregation, or sorting to do. Otherwise
+	 * we'd better tell the subquery to plan for full retrieval. (XXX This
+	 * could probably be made more intelligent ...)
+	 */
+	if (parse->hasAggs ||
+		parse->groupClause ||
+		parse->havingQual ||
+		parse->distinctClause ||
+		parse->sortClause ||
+		has_multiple_baserels(root))
+		tuple_fraction = 0.0;	/* default case */
+	else
+		tuple_fraction = root->tuple_fraction;
+
+	/* Generate the plan for the subquery */
+	subplan = subquery_planner(root->glob, subquery,
+							   root,
+							   false, tuple_fraction,
+							   &subroot);
+
+	/* Convert subquery pathkeys to outer representation */
+	pathkeys = convert_subquery_pathkeys(root, rel,
+										 subroot->query_pathkeys, subplan);
+
+	/* Generate appropriate path */
+	path = create_subqueryscan_path(rel, pathkeys,
+			 subplan, subroot->parse->rtable, subroot->rowMarks);
+
+	/* return the same path for now */
+	*cheapest_start_path = *cheapest_total_path = path;
+}
+
+/*
  * set_function_pathlist
  *		Build the (single) access path for a function RTE
  */
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index c4404b1..e352b4b 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -892,7 +892,7 @@ cost_tidscan(Path *path, PlannerInfo *root,
  *	  Determines and returns the cost of scanning a subquery RTE.
  */
 void
-cost_subqueryscan(Path *path, RelOptInfo *baserel)
+cost_subqueryscan(SubqueryPath *path, RelOptInfo *baserel)
 {
 	Cost		startup_cost;
 	Cost		run_cost;
@@ -907,15 +907,15 @@ cost_subqueryscan(Path *path, RelOptInfo *baserel)
 	 * any restriction clauses that will be attached to the SubqueryScan node,
 	 * plus cpu_tuple_cost to account for selection and projection overhead.
 	 */
-	path->startup_cost = baserel->subplan->startup_cost;
-	path->total_cost = baserel->subplan->total_cost;
+	path->path.startup_cost = path->subplan->startup_cost;
+	path->path.total_cost = path->subplan->total_cost;
 
 	startup_cost = baserel->baserestrictcost.startup;
 	cpu_per_tuple = cpu_tuple_cost + baserel->baserestrictcost.per_tuple;
 	run_cost = cpu_per_tuple * baserel->tuples;
 
-	path->startup_cost += startup_cost;
-	path->total_cost += startup_cost + run_cost;
+	path->path.startup_cost += startup_cost;
+	path->path.total_cost += startup_cost + run_cost;
 }
 
 /*
@@ -3222,7 +3222,7 @@ set_joinrel_size_estimates(PlannerInfo *root, RelOptInfo *rel,
  */
 void
 set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
-							PlannerInfo *subroot)
+							PlannerInfo *subroot, Plan *subplan)
 {
 	RangeTblEntry *rte;
 	ListCell   *lc;
@@ -3233,7 +3233,7 @@ set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
 	Assert(rte->rtekind == RTE_SUBQUERY);
 
 	/* Copy raw number of output rows from subplan */
-	rel->tuples = rel->subplan->plan_rows;
+	rel->tuples = subplan->plan_rows;
 
 	/*
 	 * Compute per-output-column width estimates by examining the subquery's
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index 7d3cf42..c79fd43 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -329,7 +329,9 @@ sort_inner_and_outer(PlannerInfo *root,
  * inner path, one on the same with materialization, one on the
  * cheapest-startup-cost inner path (if different), one on the
  * cheapest-total inner-indexscan path (if any), and one on the
- * cheapest-startup inner-indexscan path (if different).
+ * cheapest-startup inner-indexscan path (if different) or on the
+ * cheapest-total inner-subqueryscan path (if any).
+ * Note inner-indexpaths and inner-subquerypaths are not made at the same time.
  *
  * We also consider mergejoins if mergejoin clauses are available.	We have
  * two ways to generate the inner path for a mergejoin: sort the cheapest
@@ -369,6 +371,8 @@ match_unsorted_outer(PlannerInfo *root,
 	Path	   *matpath = NULL;
 	Path	   *index_cheapest_startup = NULL;
 	Path	   *index_cheapest_total = NULL;
+	Path	   *sub_cheapest_startup = NULL;
+	Path	   *sub_cheapest_total = NULL;
 	ListCell   *l;
 
 	/*
@@ -430,8 +434,8 @@ match_unsorted_outer(PlannerInfo *root,
 				create_material_path(innerrel, inner_cheapest_total);
 
 		/*
-		 * Get the best innerjoin indexpaths (if any) for this outer rel.
-		 * They're the same for all outer paths.
+		 * Get the best innerjoin indexpaths or subquerypaths (if any)
+		 * for this outer rel. They're the same for all outer paths.
 		 */
 		if (innerrel->reloptkind != RELOPT_JOINREL)
 		{
@@ -444,6 +448,10 @@ match_unsorted_outer(PlannerInfo *root,
 				best_inner_indexscan(root, innerrel, outerrel, jointype,
 									 &index_cheapest_startup,
 									 &index_cheapest_total);
+			else if (innerrel->rtekind == RTE_SUBQUERY)
+				best_inner_subqueryscan(root, innerrel, outerrel, restrictlist,
+										&sub_cheapest_startup,
+										&sub_cheapest_total);
 		}
 	}
 
@@ -487,7 +495,7 @@ match_unsorted_outer(PlannerInfo *root,
 			 * cheapest-total-cost inner.  When appropriate, also consider
 			 * using the materialized form of the cheapest inner, the
 			 * cheapest-startup-cost inner path, and the cheapest innerjoin
-			 * indexpaths.
+			 * indexpaths or subquerypaths.
 			 */
 			add_path(joinrel, (Path *)
 					 create_nestloop_path(root,
@@ -539,6 +547,16 @@ match_unsorted_outer(PlannerInfo *root,
 											  index_cheapest_startup,
 											  restrictlist,
 											  merge_pathkeys));
+			if (sub_cheapest_total != NULL)
+				add_path(joinrel, (Path *)
+						 create_nestloop_path(root,
+											  joinrel,
+											  jointype,
+											  sjinfo,
+											  outerpath,
+											  sub_cheapest_startup,
+											  restrictlist,
+											  merge_pathkeys));
 		}
 
 		/* Can't do anything else if outer path needs to be unique'd */
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 42427d3..e3db45c 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -745,7 +745,6 @@ make_join_rel(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2)
 	return joinrel;
 }
 
-
 /*
  * have_join_order_restriction
  *		Detect whether the two relations should be joined to satisfy
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index e5228a8..cee5c71 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -623,12 +623,12 @@ find_indexkey_var(PlannerInfo *root, RelOptInfo *rel, AttrNumber varattno)
  */
 List *
 convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel,
-						  List *subquery_pathkeys)
+						  List *subquery_pathkeys, Plan *subplan)
 {
 	List	   *retval = NIL;
 	int			retvallen = 0;
 	int			outer_query_keys = list_length(root->query_pathkeys);
-	List	   *sub_tlist = rel->subplan->targetlist;
+	List	   *sub_tlist = subplan->targetlist;
 	ListCell   *i;
 
 	foreach(i, subquery_pathkeys)
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index ac80511..f1002f3 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -62,8 +62,8 @@ static Plan *create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual,
 					  List **qual, List **indexqual);
 static TidScan *create_tidscan_plan(PlannerInfo *root, TidPath *best_path,
 					List *tlist, List *scan_clauses);
-static SubqueryScan *create_subqueryscan_plan(PlannerInfo *root, Path *best_path,
-						 List *tlist, List *scan_clauses);
+static SubqueryScan *create_subqueryscan_plan(PlannerInfo *root,
+					SubqueryPath *best_path, List *tlist, List *scan_clauses);
 static FunctionScan *create_functionscan_plan(PlannerInfo *root, Path *best_path,
 						 List *tlist, List *scan_clauses);
 static ValuesScan *create_valuesscan_plan(PlannerInfo *root, Path *best_path,
@@ -321,7 +321,7 @@ create_scan_plan(PlannerInfo *root, Path *best_path)
 
 		case T_SubqueryScan:
 			plan = (Plan *) create_subqueryscan_plan(root,
-													 best_path,
+													 (SubqueryPath *) best_path,
 													 tlist,
 													 scan_clauses);
 			break;
@@ -1538,15 +1538,15 @@ create_tidscan_plan(PlannerInfo *root, TidPath *best_path,
  *	 with restriction clauses 'scan_clauses' and targetlist 'tlist'.
  */
 static SubqueryScan *
-create_subqueryscan_plan(PlannerInfo *root, Path *best_path,
+create_subqueryscan_plan(PlannerInfo *root, SubqueryPath *best_path,
 						 List *tlist, List *scan_clauses)
 {
 	SubqueryScan *scan_plan;
-	Index		scan_relid = best_path->parent->relid;
+	Index		scan_relid = best_path->path.parent->relid;
 
 	/* it should be a subquery base rel... */
 	Assert(scan_relid > 0);
-	Assert(best_path->parent->rtekind == RTE_SUBQUERY);
+	Assert(best_path->path.parent->rtekind == RTE_SUBQUERY);
 
 	/* Sort clauses into best execution order */
 	scan_clauses = order_qual_clauses(root, scan_clauses);
@@ -1557,11 +1557,63 @@ create_subqueryscan_plan(PlannerInfo *root, Path *best_path,
 	scan_plan = make_subqueryscan(tlist,
 								  scan_clauses,
 								  scan_relid,
-								  best_path->parent->subplan,
-								  best_path->parent->subrtable,
-								  best_path->parent->subrowmark);
+								  best_path->subplan,
+								  best_path->subrtable,
+								  best_path->subrowmark);
 
-	copy_path_costsize(&scan_plan->scan.plan, best_path);
+	copy_path_costsize(&scan_plan->scan.plan, &best_path->path);
+
+	/*
+	 * If this plan is inner of nestloop and push-down-qual case,
+	 * params are extracted and regsitered for the upper nestloop.
+	 * As Plan has only bitmap extParam field, we cannot take them
+	 * directly; instead take them from PlannerGlobal.
+	 */
+	if (scan_plan->subplan->extParam)
+	{
+		int		i;
+		ListCell   *ppl;
+		PlannerParamItem   *pitem;
+
+		i = 0;
+		foreach(ppl, root->glob->paramlist)
+		{
+			pitem = (PlannerParamItem *) lfirst(ppl);
+			/* take the ones only in this query level */
+			if (pitem->abslevel == root->query_level &&
+				IsA(pitem->item, Var) &&
+				((Var *) pitem->item)->varlevelsup == 0)
+			{
+				Var	   *var = (Var *) pitem->item;
+				NestLoopParam	   *nlp;
+				ListCell   *lc;
+				bool		found;
+
+				/* Is the param already listed in root->curOuterParams? */
+				foreach(lc, root->curOuterParams)
+				{
+					nlp = (NestLoopParam *) lfirst(lc);
+					if (equal(nlp->paramval, var))
+					{
+						/* Present, so we can just skip */
+						found = true;
+						break;
+					}
+				}
+				if (found)
+				{
+					i++;
+					continue;
+				}
+				/* No, so add it */
+				nlp = makeNode(NestLoopParam);
+				nlp->paramno = i;
+				nlp->paramval = var;
+				root->curOuterParams = lappend(root->curOuterParams, nlp);
+			}
+			i++;
+		}
+	}
 
 	return scan_plan;
 }
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 161d5ab..2e711d8 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -1331,17 +1331,21 @@ distinct_col_search(int colno, List *colnos, List *opids)
  *	  returning the pathnode.
  */
 Path *
-create_subqueryscan_path(RelOptInfo *rel, List *pathkeys)
+create_subqueryscan_path(RelOptInfo *rel, List *pathkeys, Plan *subplan,
+						 List *subrtable, List *subrowmark)
 {
-	Path	   *pathnode = makeNode(Path);
+	SubqueryPath   *pathnode = makeNode(SubqueryPath);
 
-	pathnode->pathtype = T_SubqueryScan;
-	pathnode->parent = rel;
-	pathnode->pathkeys = pathkeys;
+	pathnode->path.pathtype = T_SubqueryScan;
+	pathnode->path.parent = rel;
+	pathnode->path.pathkeys = pathkeys;
+	pathnode->subplan = subplan;
+	pathnode->subrtable = subrtable;
+	pathnode->subrowmark = subrowmark;
 
 	cost_subqueryscan(pathnode, rel);
 
-	return pathnode;
+	return (Path *) pathnode;
 }
 
 /*
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index b7a5845..ab2338f 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -82,9 +82,7 @@ build_simple_rel(PlannerInfo *root, int relid, RelOptKind reloptkind)
 	rel->indexlist = NIL;
 	rel->pages = 0;
 	rel->tuples = 0;
-	rel->subplan = NULL;
-	rel->subrtable = NIL;
-	rel->subrowmark = NIL;
+	rel->preprocessed_subquery = NULL;
 	rel->baserestrictinfo = NIL;
 	rel->baserestrictcost.startup = 0;
 	rel->baserestrictcost.per_tuple = 0;
@@ -336,9 +334,7 @@ build_join_rel(PlannerInfo *root,
 	joinrel->indexlist = NIL;
 	joinrel->pages = 0;
 	joinrel->tuples = 0;
-	joinrel->subplan = NULL;
-	joinrel->subrtable = NIL;
-	joinrel->subrowmark = NIL;
+	joinrel->preprocessed_subquery = NULL;
 	joinrel->baserestrictinfo = NIL;
 	joinrel->baserestrictcost.startup = 0;
 	joinrel->baserestrictcost.per_tuple = 0;
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index d8bc6b8..e951fd1 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -220,6 +220,7 @@ typedef enum NodeTag
 	T_MergePath,
 	T_HashPath,
 	T_TidPath,
+	T_SubqueryPath,
 	T_ForeignPath,
 	T_AppendPath,
 	T_MergeAppendPath,
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index f659269..a6983d6 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -408,9 +408,8 @@ typedef struct RelOptInfo
 	List	   *indexlist;		/* list of IndexOptInfo */
 	BlockNumber pages;
 	double		tuples;
-	struct Plan *subplan;		/* if subquery */
-	List	   *subrtable;		/* if subquery */
-	List	   *subrowmark;		/* if subquery */
+
+	Query	   *preprocessed_subquery; /* if subquery */
 
 	/* used by various scans and joins: */
 	List	   *baserestrictinfo;		/* RestrictInfo structures (if base
@@ -769,6 +768,16 @@ typedef struct TidPath
 	List	   *tidquals;		/* qual(s) involving CTID = something */
 } TidPath;
 
+typedef struct SubqueryPath
+{
+	Path		path;
+	struct Plan *subplan;
+	List	   *subrtable;
+	List	   *subrowmark;
+	Relids		paramvars;
+	double		rows;
+} SubqueryPath;
+
 /*
  * ForeignPath represents a scan of a foreign table
  */
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 2763863..7ec2c7b 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -75,7 +75,7 @@ extern void cost_bitmap_or_node(BitmapOrPath *path, PlannerInfo *root);
 extern void cost_bitmap_tree_node(Path *path, Cost *cost, Selectivity *selec);
 extern void cost_tidscan(Path *path, PlannerInfo *root,
 			 RelOptInfo *baserel, List *tidquals);
-extern void cost_subqueryscan(Path *path, RelOptInfo *baserel);
+extern void cost_subqueryscan(SubqueryPath *path, RelOptInfo *baserel);
 extern void cost_functionscan(Path *path, PlannerInfo *root,
 				  RelOptInfo *baserel);
 extern void cost_valuesscan(Path *path, PlannerInfo *root,
@@ -122,7 +122,7 @@ extern void set_joinrel_size_estimates(PlannerInfo *root, RelOptInfo *rel,
 						   SpecialJoinInfo *sjinfo,
 						   List *restrictlist);
 extern void set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
-							PlannerInfo *subroot);
+							PlannerInfo *subroot, Plan *subplan);
 extern void set_function_size_estimates(PlannerInfo *root, RelOptInfo *rel);
 extern void set_values_size_estimates(PlannerInfo *root, RelOptInfo *rel);
 extern void set_cte_size_estimates(PlannerInfo *root, RelOptInfo *rel,
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 1da2131..6de2816 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -14,6 +14,7 @@
 #ifndef PATHNODE_H
 #define PATHNODE_H
 
+#include "nodes/plannodes.h"
 #include "nodes/relation.h"
 
 
@@ -56,7 +57,8 @@ extern ResultPath *create_result_path(List *quals);
 extern MaterialPath *create_material_path(RelOptInfo *rel, Path *subpath);
 extern UniquePath *create_unique_path(PlannerInfo *root, RelOptInfo *rel,
 				   Path *subpath, SpecialJoinInfo *sjinfo);
-extern Path *create_subqueryscan_path(RelOptInfo *rel, List *pathkeys);
+extern Path *create_subqueryscan_path(RelOptInfo *rel, List *pathkeys,
+					Plan *subplan, List *subrtable, List *subrowmark);
 extern Path *create_functionscan_path(PlannerInfo *root, RelOptInfo *rel);
 extern Path *create_valuesscan_path(PlannerInfo *root, RelOptInfo *rel);
 extern Path *create_ctescan_path(PlannerInfo *root, RelOptInfo *rel);
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 7f1353a..54d0b01 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -33,6 +33,9 @@ extern PGDLLIMPORT join_search_hook_type join_search_hook;
 extern RelOptInfo *make_one_rel(PlannerInfo *root, List *joinlist);
 extern RelOptInfo *standard_join_search(PlannerInfo *root, int levels_needed,
 					 List *initial_rels);
+extern void best_inner_subqueryscan(PlannerInfo *root, RelOptInfo *rel,
+						RelOptInfo *outer_rel, List *join_clause,
+						Path **cheapest_start_path, Path **cheapest_total_path);
 
 #ifdef OPTIMIZER_DEBUG
 extern void debug_print_rel(PlannerInfo *root, RelOptInfo *rel);
@@ -154,7 +157,7 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
 extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
 					 ScanDirection scandir);
 extern List *convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel,
-						  List *subquery_pathkeys);
+						  List *subquery_pathkeys, Plan *subplan);
 extern List *build_join_pathkeys(PlannerInfo *root,
 					RelOptInfo *joinrel,
 					JoinType jointype,
#2Robert Haas
robertmhaas@gmail.com
In reply to: Hitoshi Harada (#1)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

On Thu, Jun 9, 2011 at 2:28 AM, Hitoshi Harada <umi.tanuki@gmail.com> wrote:

BTW, as I changed title and design from the previous post, should I
throw away the old commit fest entry and make the new one?

Nah, just edit the existing entry and change the title.

Also add a link to the new patch, of course.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#3Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Robert Haas (#2)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

2011/6/9 Robert Haas <robertmhaas@gmail.com>:

On Thu, Jun 9, 2011 at 2:28 AM, Hitoshi Harada <umi.tanuki@gmail.com> wrote:

BTW, as I changed title and design from the previous post, should I
throw away the old commit fest entry and make the new one?

Nah, just edit the existing entry and change the title.

Also add a link to the new patch, of course.

Ok, done.

--
Hitoshi Harada

#4Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Hitoshi Harada (#3)
1 attachment(s)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

2011/6/10 Hitoshi Harada <umi.tanuki@gmail.com>:

2011/6/9 Robert Haas <robertmhaas@gmail.com>:

On Thu, Jun 9, 2011 at 2:28 AM, Hitoshi Harada <umi.tanuki@gmail.com> wrote:

BTW, as I changed title and design from the previous post, should I
throw away the old commit fest entry and make the new one?

Nah, just edit the existing entry and change the title.

Also add a link to the new patch, of course.

Ok, done.

While reviewing the gist/box patch, I found some planner APIs that can
replace parts in my patch. Also, comments in includes wasn't updated
appropriately. Revised patch attached.

Regards,

--
Hitoshi Harada

Attachments:

aggjoin-20110617.patchapplication/octet-stream; name=aggjoin-20110617.patchDownload
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 681f5f8..039fd7f 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -1557,6 +1557,17 @@ _outTidPath(StringInfo str, TidPath *node)
 }
 
 static void
+_outSubqueryPath(StringInfo str, SubqueryPath *node)
+{
+	WRITE_NODE_TYPE("SUBQUERYPATH");
+
+	_outPathInfo(str, (Path *) node);
+	WRITE_NODE_FIELD(subplan);
+	WRITE_NODE_FIELD(subrtable);
+	WRITE_NODE_FIELD(subrowmark);
+}
+
+static void
 _outForeignPath(StringInfo str, ForeignPath *node)
 {
 	WRITE_NODE_TYPE("FOREIGNPATH");
@@ -1738,9 +1749,6 @@ _outRelOptInfo(StringInfo str, RelOptInfo *node)
 	WRITE_NODE_FIELD(indexlist);
 	WRITE_UINT_FIELD(pages);
 	WRITE_FLOAT_FIELD(tuples, "%.0f");
-	WRITE_NODE_FIELD(subplan);
-	WRITE_NODE_FIELD(subrtable);
-	WRITE_NODE_FIELD(subrowmark);
 	WRITE_NODE_FIELD(baserestrictinfo);
 	WRITE_NODE_FIELD(joininfo);
 	WRITE_BOOL_FIELD(has_eclass_joins);
@@ -2948,6 +2956,9 @@ _outNode(StringInfo str, void *obj)
 			case T_TidPath:
 				_outTidPath(str, obj);
 				break;
+			case T_SubqueryPath:
+				_outSubqueryPath(str, obj);
+				break;
 			case T_ForeignPath:
 				_outForeignPath(str, obj);
 				break;
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index 47ab08e..37578b9 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -31,6 +31,7 @@
 #include "optimizer/planner.h"
 #include "optimizer/prep.h"
 #include "optimizer/restrictinfo.h"
+#include "optimizer/subselect.h"
 #include "optimizer/var.h"
 #include "parser/parse_clause.h"
 #include "parser/parsetree.h"
@@ -687,7 +688,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
 	bool	   *differentTypes;
 	double		tuple_fraction;
 	PlannerInfo *subroot;
-	List	   *pathkeys;
+	Plan	    *subplan;
+	List	    *pathkeys;
 
 	/*
 	 * Must copy the Query so that planning doesn't mess up the RTE contents
@@ -749,6 +751,9 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
 
 	pfree(differentTypes);
 
+	/* save it for later use in best_inner_subqueryscan */
+	rel->preprocessed_subquery = copyObject(subquery);
+
 	/*
 	 * We can safely pass the outer tuple_fraction down to the subquery if the
 	 * outer level has no joining, aggregation, or sorting to do. Otherwise
@@ -766,27 +771,182 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
 		tuple_fraction = root->tuple_fraction;
 
 	/* Generate the plan for the subquery */
-	rel->subplan = subquery_planner(root->glob, subquery,
-									root,
-									false, tuple_fraction,
-									&subroot);
-	rel->subrtable = subroot->parse->rtable;
-	rel->subrowmark = subroot->rowMarks;
+	subplan = subquery_planner(root->glob, subquery,
+							   root,
+							   false, tuple_fraction,
+							   &subroot);
 
 	/* Mark rel with estimated output rows, width, etc */
-	set_subquery_size_estimates(root, rel, subroot);
+	set_subquery_size_estimates(root, rel, subroot, subplan);
 
 	/* Convert subquery pathkeys to outer representation */
-	pathkeys = convert_subquery_pathkeys(root, rel, subroot->query_pathkeys);
+	pathkeys = convert_subquery_pathkeys(root, rel,
+										 subroot->query_pathkeys, subplan);
 
 	/* Generate appropriate path */
-	add_path(rel, create_subqueryscan_path(rel, pathkeys));
+	add_path(rel, create_subqueryscan_path(rel, pathkeys,
+			 subplan, subroot->parse->rtable, subroot->rowMarks));
 
 	/* Select cheapest path (pretty easy in this case...) */
 	set_cheapest(rel);
 }
 
 /*
+ * best_inner_subqueryscan
+ *
+ * As the inner scan relation, try to push down some join clauses to
+ * subquery and re-plan it. If such posibility is found, return the path
+ * in *cheapest_start_path or *cheapest_total_path. See also best_inner_indexscan().
+ *
+ * It re-uses modified subquery (a.k.a. rel->preprocessed_subquery.) The basic
+ * push-down of baserestrictinfo was common among base path (done in
+ * set_subquery_pathlist()) so this routine doesn't care it. But once we find
+ * push-down-able join clause, the parsed subquery will be modified after copied.
+ */
+void
+best_inner_subqueryscan(PlannerInfo *root, RelOptInfo *rel,
+						RelOptInfo *outer_rel, List *join_clause,
+						Path **cheapest_start_path, Path **cheapest_total_path)
+{
+	Query	   *parse = root->parse;
+	Query	   *subquery;
+	Index		rti = rel->relid;
+	RangeTblEntry *rte = planner_rt_fetch(rti, root);
+	bool	   *differentTypes;
+	double		tuple_fraction;
+	PlannerInfo *subroot;
+	Plan	   *subplan;
+	List	   *pathkeys;
+	Path	   *path;
+	bool		query_modified = false;
+
+	/* initialize paths to return immediately anytime */
+	*cheapest_start_path = *cheapest_total_path = NULL;
+
+	/*
+	 * empty join condition doesn't make sense here
+	 */
+	if (join_clause == NIL)
+		return;
+
+	/*
+	 * basic push-down-qual process should have done already.
+	 * see set_subquery_pathlist
+	 */
+	Assert(rel->preprocessed_subquery);
+	/* copying the whole Query is expensive; let's copy-on-write */
+	subquery = rel->preprocessed_subquery;
+
+	/* We need a workspace for keeping track of set-op type coercions */
+	differentTypes = (bool *)
+		palloc0((list_length(subquery->targetList) + 1) * sizeof(bool));
+
+	/*
+	 * Currently we focus on only cases of aggregate subquery.
+	 * I'm not quite sure if there are other useful cases.
+	 */
+	if (subquery->hasAggs &&
+		subquery_is_pushdown_safe(subquery, subquery, differentTypes))
+	{
+		ListCell   *l;
+
+		foreach (l, join_clause)
+		{
+			RestrictInfo   *rinfo = (RestrictInfo *) lfirst(l);
+			Node		   *lexpr, *rexpr;
+			Expr		   *clause = NULL;
+			Param		   *param;
+			OpExpr		   *rclause;
+
+			/*
+			 * For the first step, only Var = Var join clause is targeted.
+			 * This may be improved in the future.
+			 */
+			if (!is_opclause(rinfo->clause))
+				continue;
+			rclause = (OpExpr *) rinfo->clause;
+			lexpr = get_leftop((Expr *) rclause);
+			rexpr = get_rightop((Expr *) rclause);
+			if (!(IsA(lexpr, Var) && IsA(rexpr, Var)))
+				continue;
+
+			/* see which arg is mine and outer's */
+			if (bms_is_member(((Var *) lexpr)->varno, outer_rel->relids))
+			{
+				param = assign_nestloop_param(root, (Var *) lexpr);
+				clause = make_opclause(rclause->opno,
+									   rclause->opresulttype,
+									   false,
+									   (Expr *) param,
+									   (Expr *) copyObject(rexpr),
+									   InvalidOid,
+									   rclause->opcollid);
+			}
+			else if (bms_is_member(((Var *) rexpr)->varno, outer_rel->relids))
+			{
+				param = assign_nestloop_param(root, (Var *) rexpr);
+				clause = make_opclause(rclause->opno,
+									   rclause->opresulttype,
+									   false,
+									   (Expr *) copyObject(lexpr),
+									   (Expr *) param,
+									   InvalidOid,
+									   rclause->opcollid);
+			}
+
+			/* TODO:no check isouter or not? */
+			if (clause)
+			{
+				/* copy on write */
+				if (rel->preprocessed_subquery == subquery)
+					subquery = copyObject(subquery);
+				subquery_push_qual(subquery, rte, rti, (Node *) clause);
+				query_modified = true;
+			}
+		}
+	}
+
+	pfree(differentTypes);
+
+	/* return immediately if such subquery isn't found. Use original one */
+	if (!query_modified)
+		return;
+
+	/*
+	 * We can safely pass the outer tuple_fraction down to the subquery if the
+	 * outer level has no joining, aggregation, or sorting to do. Otherwise
+	 * we'd better tell the subquery to plan for full retrieval. (XXX This
+	 * could probably be made more intelligent ...)
+	 */
+	if (parse->hasAggs ||
+		parse->groupClause ||
+		parse->havingQual ||
+		parse->distinctClause ||
+		parse->sortClause ||
+		has_multiple_baserels(root))
+		tuple_fraction = 0.0;	/* default case */
+	else
+		tuple_fraction = root->tuple_fraction;
+
+	/* Generate the plan for the subquery */
+	subplan = subquery_planner(root->glob, subquery,
+							   root,
+							   false, tuple_fraction,
+							   &subroot);
+
+	/* Convert subquery pathkeys to outer representation */
+	pathkeys = convert_subquery_pathkeys(root, rel,
+										 subroot->query_pathkeys, subplan);
+
+	/* Generate appropriate path */
+	path = create_subqueryscan_path(rel, pathkeys,
+			 subplan, subroot->parse->rtable, subroot->rowMarks);
+
+	/* return the same path for now */
+	*cheapest_start_path = *cheapest_total_path = path;
+}
+
+/*
  * set_function_pathlist
  *		Build the (single) access path for a function RTE
  */
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index c4404b1..e352b4b 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -892,7 +892,7 @@ cost_tidscan(Path *path, PlannerInfo *root,
  *	  Determines and returns the cost of scanning a subquery RTE.
  */
 void
-cost_subqueryscan(Path *path, RelOptInfo *baserel)
+cost_subqueryscan(SubqueryPath *path, RelOptInfo *baserel)
 {
 	Cost		startup_cost;
 	Cost		run_cost;
@@ -907,15 +907,15 @@ cost_subqueryscan(Path *path, RelOptInfo *baserel)
 	 * any restriction clauses that will be attached to the SubqueryScan node,
 	 * plus cpu_tuple_cost to account for selection and projection overhead.
 	 */
-	path->startup_cost = baserel->subplan->startup_cost;
-	path->total_cost = baserel->subplan->total_cost;
+	path->path.startup_cost = path->subplan->startup_cost;
+	path->path.total_cost = path->subplan->total_cost;
 
 	startup_cost = baserel->baserestrictcost.startup;
 	cpu_per_tuple = cpu_tuple_cost + baserel->baserestrictcost.per_tuple;
 	run_cost = cpu_per_tuple * baserel->tuples;
 
-	path->startup_cost += startup_cost;
-	path->total_cost += startup_cost + run_cost;
+	path->path.startup_cost += startup_cost;
+	path->path.total_cost += startup_cost + run_cost;
 }
 
 /*
@@ -3222,7 +3222,7 @@ set_joinrel_size_estimates(PlannerInfo *root, RelOptInfo *rel,
  */
 void
 set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
-							PlannerInfo *subroot)
+							PlannerInfo *subroot, Plan *subplan)
 {
 	RangeTblEntry *rte;
 	ListCell   *lc;
@@ -3233,7 +3233,7 @@ set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
 	Assert(rte->rtekind == RTE_SUBQUERY);
 
 	/* Copy raw number of output rows from subplan */
-	rel->tuples = rel->subplan->plan_rows;
+	rel->tuples = subplan->plan_rows;
 
 	/*
 	 * Compute per-output-column width estimates by examining the subquery's
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index 7d3cf42..c79fd43 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -329,7 +329,9 @@ sort_inner_and_outer(PlannerInfo *root,
  * inner path, one on the same with materialization, one on the
  * cheapest-startup-cost inner path (if different), one on the
  * cheapest-total inner-indexscan path (if any), and one on the
- * cheapest-startup inner-indexscan path (if different).
+ * cheapest-startup inner-indexscan path (if different) or on the
+ * cheapest-total inner-subqueryscan path (if any).
+ * Note inner-indexpaths and inner-subquerypaths are not made at the same time.
  *
  * We also consider mergejoins if mergejoin clauses are available.	We have
  * two ways to generate the inner path for a mergejoin: sort the cheapest
@@ -369,6 +371,8 @@ match_unsorted_outer(PlannerInfo *root,
 	Path	   *matpath = NULL;
 	Path	   *index_cheapest_startup = NULL;
 	Path	   *index_cheapest_total = NULL;
+	Path	   *sub_cheapest_startup = NULL;
+	Path	   *sub_cheapest_total = NULL;
 	ListCell   *l;
 
 	/*
@@ -430,8 +434,8 @@ match_unsorted_outer(PlannerInfo *root,
 				create_material_path(innerrel, inner_cheapest_total);
 
 		/*
-		 * Get the best innerjoin indexpaths (if any) for this outer rel.
-		 * They're the same for all outer paths.
+		 * Get the best innerjoin indexpaths or subquerypaths (if any)
+		 * for this outer rel. They're the same for all outer paths.
 		 */
 		if (innerrel->reloptkind != RELOPT_JOINREL)
 		{
@@ -444,6 +448,10 @@ match_unsorted_outer(PlannerInfo *root,
 				best_inner_indexscan(root, innerrel, outerrel, jointype,
 									 &index_cheapest_startup,
 									 &index_cheapest_total);
+			else if (innerrel->rtekind == RTE_SUBQUERY)
+				best_inner_subqueryscan(root, innerrel, outerrel, restrictlist,
+										&sub_cheapest_startup,
+										&sub_cheapest_total);
 		}
 	}
 
@@ -487,7 +495,7 @@ match_unsorted_outer(PlannerInfo *root,
 			 * cheapest-total-cost inner.  When appropriate, also consider
 			 * using the materialized form of the cheapest inner, the
 			 * cheapest-startup-cost inner path, and the cheapest innerjoin
-			 * indexpaths.
+			 * indexpaths or subquerypaths.
 			 */
 			add_path(joinrel, (Path *)
 					 create_nestloop_path(root,
@@ -539,6 +547,16 @@ match_unsorted_outer(PlannerInfo *root,
 											  index_cheapest_startup,
 											  restrictlist,
 											  merge_pathkeys));
+			if (sub_cheapest_total != NULL)
+				add_path(joinrel, (Path *)
+						 create_nestloop_path(root,
+											  joinrel,
+											  jointype,
+											  sjinfo,
+											  outerpath,
+											  sub_cheapest_startup,
+											  restrictlist,
+											  merge_pathkeys));
 		}
 
 		/* Can't do anything else if outer path needs to be unique'd */
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 42427d3..e3db45c 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -745,7 +745,6 @@ make_join_rel(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2)
 	return joinrel;
 }
 
-
 /*
  * have_join_order_restriction
  *		Detect whether the two relations should be joined to satisfy
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index e5228a8..cee5c71 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -623,12 +623,12 @@ find_indexkey_var(PlannerInfo *root, RelOptInfo *rel, AttrNumber varattno)
  */
 List *
 convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel,
-						  List *subquery_pathkeys)
+						  List *subquery_pathkeys, Plan *subplan)
 {
 	List	   *retval = NIL;
 	int			retvallen = 0;
 	int			outer_query_keys = list_length(root->query_pathkeys);
-	List	   *sub_tlist = rel->subplan->targetlist;
+	List	   *sub_tlist = subplan->targetlist;
 	ListCell   *i;
 
 	foreach(i, subquery_pathkeys)
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index ac80511..f1002f3 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -62,8 +62,8 @@ static Plan *create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual,
 					  List **qual, List **indexqual);
 static TidScan *create_tidscan_plan(PlannerInfo *root, TidPath *best_path,
 					List *tlist, List *scan_clauses);
-static SubqueryScan *create_subqueryscan_plan(PlannerInfo *root, Path *best_path,
-						 List *tlist, List *scan_clauses);
+static SubqueryScan *create_subqueryscan_plan(PlannerInfo *root,
+					SubqueryPath *best_path, List *tlist, List *scan_clauses);
 static FunctionScan *create_functionscan_plan(PlannerInfo *root, Path *best_path,
 						 List *tlist, List *scan_clauses);
 static ValuesScan *create_valuesscan_plan(PlannerInfo *root, Path *best_path,
@@ -321,7 +321,7 @@ create_scan_plan(PlannerInfo *root, Path *best_path)
 
 		case T_SubqueryScan:
 			plan = (Plan *) create_subqueryscan_plan(root,
-													 best_path,
+													 (SubqueryPath *) best_path,
 													 tlist,
 													 scan_clauses);
 			break;
@@ -1538,15 +1538,15 @@ create_tidscan_plan(PlannerInfo *root, TidPath *best_path,
  *	 with restriction clauses 'scan_clauses' and targetlist 'tlist'.
  */
 static SubqueryScan *
-create_subqueryscan_plan(PlannerInfo *root, Path *best_path,
+create_subqueryscan_plan(PlannerInfo *root, SubqueryPath *best_path,
 						 List *tlist, List *scan_clauses)
 {
 	SubqueryScan *scan_plan;
-	Index		scan_relid = best_path->parent->relid;
+	Index		scan_relid = best_path->path.parent->relid;
 
 	/* it should be a subquery base rel... */
 	Assert(scan_relid > 0);
-	Assert(best_path->parent->rtekind == RTE_SUBQUERY);
+	Assert(best_path->path.parent->rtekind == RTE_SUBQUERY);
 
 	/* Sort clauses into best execution order */
 	scan_clauses = order_qual_clauses(root, scan_clauses);
@@ -1557,11 +1557,63 @@ create_subqueryscan_plan(PlannerInfo *root, Path *best_path,
 	scan_plan = make_subqueryscan(tlist,
 								  scan_clauses,
 								  scan_relid,
-								  best_path->parent->subplan,
-								  best_path->parent->subrtable,
-								  best_path->parent->subrowmark);
+								  best_path->subplan,
+								  best_path->subrtable,
+								  best_path->subrowmark);
 
-	copy_path_costsize(&scan_plan->scan.plan, best_path);
+	copy_path_costsize(&scan_plan->scan.plan, &best_path->path);
+
+	/*
+	 * If this plan is inner of nestloop and push-down-qual case,
+	 * params are extracted and regsitered for the upper nestloop.
+	 * As Plan has only bitmap extParam field, we cannot take them
+	 * directly; instead take them from PlannerGlobal.
+	 */
+	if (scan_plan->subplan->extParam)
+	{
+		int		i;
+		ListCell   *ppl;
+		PlannerParamItem   *pitem;
+
+		i = 0;
+		foreach(ppl, root->glob->paramlist)
+		{
+			pitem = (PlannerParamItem *) lfirst(ppl);
+			/* take the ones only in this query level */
+			if (pitem->abslevel == root->query_level &&
+				IsA(pitem->item, Var) &&
+				((Var *) pitem->item)->varlevelsup == 0)
+			{
+				Var	   *var = (Var *) pitem->item;
+				NestLoopParam	   *nlp;
+				ListCell   *lc;
+				bool		found;
+
+				/* Is the param already listed in root->curOuterParams? */
+				foreach(lc, root->curOuterParams)
+				{
+					nlp = (NestLoopParam *) lfirst(lc);
+					if (equal(nlp->paramval, var))
+					{
+						/* Present, so we can just skip */
+						found = true;
+						break;
+					}
+				}
+				if (found)
+				{
+					i++;
+					continue;
+				}
+				/* No, so add it */
+				nlp = makeNode(NestLoopParam);
+				nlp->paramno = i;
+				nlp->paramval = var;
+				root->curOuterParams = lappend(root->curOuterParams, nlp);
+			}
+			i++;
+		}
+	}
 
 	return scan_plan;
 }
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 161d5ab..2e711d8 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -1331,17 +1331,21 @@ distinct_col_search(int colno, List *colnos, List *opids)
  *	  returning the pathnode.
  */
 Path *
-create_subqueryscan_path(RelOptInfo *rel, List *pathkeys)
+create_subqueryscan_path(RelOptInfo *rel, List *pathkeys, Plan *subplan,
+						 List *subrtable, List *subrowmark)
 {
-	Path	   *pathnode = makeNode(Path);
+	SubqueryPath   *pathnode = makeNode(SubqueryPath);
 
-	pathnode->pathtype = T_SubqueryScan;
-	pathnode->parent = rel;
-	pathnode->pathkeys = pathkeys;
+	pathnode->path.pathtype = T_SubqueryScan;
+	pathnode->path.parent = rel;
+	pathnode->path.pathkeys = pathkeys;
+	pathnode->subplan = subplan;
+	pathnode->subrtable = subrtable;
+	pathnode->subrowmark = subrowmark;
 
 	cost_subqueryscan(pathnode, rel);
 
-	return pathnode;
+	return (Path *) pathnode;
 }
 
 /*
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index b7a5845..ab2338f 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -82,9 +82,7 @@ build_simple_rel(PlannerInfo *root, int relid, RelOptKind reloptkind)
 	rel->indexlist = NIL;
 	rel->pages = 0;
 	rel->tuples = 0;
-	rel->subplan = NULL;
-	rel->subrtable = NIL;
-	rel->subrowmark = NIL;
+	rel->preprocessed_subquery = NULL;
 	rel->baserestrictinfo = NIL;
 	rel->baserestrictcost.startup = 0;
 	rel->baserestrictcost.per_tuple = 0;
@@ -336,9 +334,7 @@ build_join_rel(PlannerInfo *root,
 	joinrel->indexlist = NIL;
 	joinrel->pages = 0;
 	joinrel->tuples = 0;
-	joinrel->subplan = NULL;
-	joinrel->subrtable = NIL;
-	joinrel->subrowmark = NIL;
+	joinrel->preprocessed_subquery = NULL;
 	joinrel->baserestrictinfo = NIL;
 	joinrel->baserestrictcost.startup = 0;
 	joinrel->baserestrictcost.per_tuple = 0;
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index d8bc6b8..e951fd1 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -220,6 +220,7 @@ typedef enum NodeTag
 	T_MergePath,
 	T_HashPath,
 	T_TidPath,
+	T_SubqueryPath,
 	T_ForeignPath,
 	T_AppendPath,
 	T_MergeAppendPath,
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index f659269..98e2744 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -322,13 +322,6 @@ typedef struct PlannerInfo
  *					(always NIL if it's not a table)
  *		pages - number of disk pages in relation (zero if not a table)
  *		tuples - number of tuples in relation (not considering restrictions)
- *		subplan - plan for subquery (NULL if it's not a subquery)
- *		subrtable - rangetable for subquery (NIL if it's not a subquery)
- *		subrowmark - rowmarks for subquery (NIL if it's not a subquery)
- *
- *		Note: for a subquery, tuples and subplan are not set immediately
- *		upon creation of the RelOptInfo object; they are filled in when
- *		set_base_rel_pathlist processes the object.
  *
  *		For otherrels that are appendrel members, these fields are filled
  *		in just as for a baserel.
@@ -408,9 +401,8 @@ typedef struct RelOptInfo
 	List	   *indexlist;		/* list of IndexOptInfo */
 	BlockNumber pages;
 	double		tuples;
-	struct Plan *subplan;		/* if subquery */
-	List	   *subrtable;		/* if subquery */
-	List	   *subrowmark;		/* if subquery */
+
+	Query	   *preprocessed_subquery; /* if subquery */
 
 	/* used by various scans and joins: */
 	List	   *baserestrictinfo;		/* RestrictInfo structures (if base
@@ -770,6 +762,23 @@ typedef struct TidPath
 } TidPath;
 
 /*
+ * SubqueryPath represents a scan of a subquery scan
+ *
+ * The struct holds subplan, subrtable and subrowmark, which are
+ * the temporary spaces to transport to the final stage of planner.
+ * Note each SubqueryPath may have different plan, depending on
+ * the pushed down qual clauses (as parameterized inner subquery).
+ */
+typedef struct SubqueryPath
+{
+	Path		path;
+	struct Plan *subplan;
+	List	   *subrtable;
+	List	   *subrowmark;
+	double		rows;
+} SubqueryPath;
+
+/*
  * ForeignPath represents a scan of a foreign table
  */
 typedef struct ForeignPath
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 2763863..7ec2c7b 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -75,7 +75,7 @@ extern void cost_bitmap_or_node(BitmapOrPath *path, PlannerInfo *root);
 extern void cost_bitmap_tree_node(Path *path, Cost *cost, Selectivity *selec);
 extern void cost_tidscan(Path *path, PlannerInfo *root,
 			 RelOptInfo *baserel, List *tidquals);
-extern void cost_subqueryscan(Path *path, RelOptInfo *baserel);
+extern void cost_subqueryscan(SubqueryPath *path, RelOptInfo *baserel);
 extern void cost_functionscan(Path *path, PlannerInfo *root,
 				  RelOptInfo *baserel);
 extern void cost_valuesscan(Path *path, PlannerInfo *root,
@@ -122,7 +122,7 @@ extern void set_joinrel_size_estimates(PlannerInfo *root, RelOptInfo *rel,
 						   SpecialJoinInfo *sjinfo,
 						   List *restrictlist);
 extern void set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
-							PlannerInfo *subroot);
+							PlannerInfo *subroot, Plan *subplan);
 extern void set_function_size_estimates(PlannerInfo *root, RelOptInfo *rel);
 extern void set_values_size_estimates(PlannerInfo *root, RelOptInfo *rel);
 extern void set_cte_size_estimates(PlannerInfo *root, RelOptInfo *rel,
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 1da2131..6de2816 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -14,6 +14,7 @@
 #ifndef PATHNODE_H
 #define PATHNODE_H
 
+#include "nodes/plannodes.h"
 #include "nodes/relation.h"
 
 
@@ -56,7 +57,8 @@ extern ResultPath *create_result_path(List *quals);
 extern MaterialPath *create_material_path(RelOptInfo *rel, Path *subpath);
 extern UniquePath *create_unique_path(PlannerInfo *root, RelOptInfo *rel,
 				   Path *subpath, SpecialJoinInfo *sjinfo);
-extern Path *create_subqueryscan_path(RelOptInfo *rel, List *pathkeys);
+extern Path *create_subqueryscan_path(RelOptInfo *rel, List *pathkeys,
+					Plan *subplan, List *subrtable, List *subrowmark);
 extern Path *create_functionscan_path(PlannerInfo *root, RelOptInfo *rel);
 extern Path *create_valuesscan_path(PlannerInfo *root, RelOptInfo *rel);
 extern Path *create_ctescan_path(PlannerInfo *root, RelOptInfo *rel);
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 7f1353a..54d0b01 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -33,6 +33,9 @@ extern PGDLLIMPORT join_search_hook_type join_search_hook;
 extern RelOptInfo *make_one_rel(PlannerInfo *root, List *joinlist);
 extern RelOptInfo *standard_join_search(PlannerInfo *root, int levels_needed,
 					 List *initial_rels);
+extern void best_inner_subqueryscan(PlannerInfo *root, RelOptInfo *rel,
+						RelOptInfo *outer_rel, List *join_clause,
+						Path **cheapest_start_path, Path **cheapest_total_path);
 
 #ifdef OPTIMIZER_DEBUG
 extern void debug_print_rel(PlannerInfo *root, RelOptInfo *rel);
@@ -154,7 +157,7 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
 extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
 					 ScanDirection scandir);
 extern List *convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel,
-						  List *subquery_pathkeys);
+						  List *subquery_pathkeys, Plan *subplan);
 extern List *build_join_pathkeys(PlannerInfo *root,
 					RelOptInfo *joinrel,
 					JoinType jointype,
#5Yeb Havinga
yebhavinga@gmail.com
In reply to: Hitoshi Harada (#4)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

On 2011-06-17 09:54, Hitoshi Harada wrote:

While reviewing the gist/box patch, I found some planner APIs that can
replace parts in my patch. Also, comments in includes wasn't updated
appropriately. Revised patch attached.

Hello Hitoshi-san,

I read your latest patch implementing parameterizing subquery scans.

1)
In the email from june 9 with the patch You wrote: "While IndexScan
is simple since its information like costs are well known by the base
relation, SubqueryScan should re-plan its Query to gain that, which is
expensive."

Initial concerns I had were caused by misinterpreting 'replanning' as:
for each outer tuple, replan the subquery (it sounds a bit like
'ReScan'). Though the general comments in the patch are helpful, I think
it would be an improvement to describe why subqueries are planned more
than once, i.e. something like
"best_inner_subqueryscan
Try to find a better subqueryscan path and its associated plan for
each join clause that can be pushed down, in addition to the path that
is already calculated by set_subquery_pathlist."

The consequences of having multiple subquery paths and plans for each
new subquery path makes the bulk of the remainder of the patch.

2)
Since 'subquery_is_pushdown_safe' is invariant under join clause push
down, it might be possible to have it called only once in
set_subquery_pathlist, i.e. by only attaching rel->preprocessed_subquery
if the subquery_is_pushdown_safe.

3)
/*
* set_subquery_pathlist
* Build the (single) access path for a subquery RTE
*/
This unchanged comment is still correct, but 'the (single) access path'
might have become a bit misleading now there are multiple paths
possible, though not by set_subquery_pathlist.

4) somewhere in the patch
s/regsitered/registered/

5) Regression tests are missing; I think at this point they'd aid in
speeding up development/test cycles.

6) Before patching Postgres, I could execute the test with the size_l
and size_m tables, after patching against current git HEAD (patch
without errors), I get the following error when running the example query:
ERROR: plan should not reference subplan's variable

I get the same error with the version from june 9 with current git HEAD.

7) Since both set_subquery_pathlist and best_inner_subqueryscan push
down clauses, as well as add a path and subplan, could a generalized
function be made to support both, to reduce duplicate code?

regards,
Yeb Havinga

--
Yeb Havinga
http://www.mgrid.net/
Mastering Medical Data

#6Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Yeb Havinga (#5)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

2011/6/29 Yeb Havinga <yebhavinga@gmail.com>:

On 2011-06-17 09:54, Hitoshi Harada wrote:

While reviewing the gist/box patch, I found some planner APIs that can
replace parts in my patch. Also, comments in includes wasn't updated
appropriately. Revised patch attached.

Hello Hitoshi-san,

Hi Yeb,

I read your latest patch implementing parameterizing subquery scans.

6) Before patching Postgres, I could execute the test with the size_l and
size_m tables, after patching against current git HEAD (patch without
errors), I get the following error when running the example query:
ERROR:  plan should not reference subplan's variable

I get the same error with the version from june 9 with current git HEAD.

It seems that something has changed since I developed the patch at
first. The last one and the one before are not so different with each
other, especially in terms of runtime but might not be tested enough.
I tried time-slip of the local git branch to around june 10, but the
same error occurs. The error message itself is not in parts changed
recently, so I guess some surrounding change would affect it.

7) Since both set_subquery_pathlist and best_inner_subqueryscan push down
clauses, as well as add a path and subplan, could a generalized function be
made to support both, to reduce duplicate code?

I tried it but decided as it is, for the future extensibility. The
subquery parameterizing will (can) be extended more complicatedly. I'm
not sure if we'd better gathering some small parts into one by
throwing future capability.

Other things are all good points. Thanks for elaborate review!
More than anything, I'm going to fix the 6) issue, at least to find the cause.

Regards,
--
Hitoshi Harada

#7Yeb Havinga
yebhavinga@gmail.com
In reply to: Hitoshi Harada (#6)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

On 2011-06-29 19:22, Hitoshi Harada wrote:

Other things are all good points. Thanks for elaborate review!
More than anything, I'm going to fix the 6) issue, at least to find the cause.

Some more questions:
8) why are cheapest start path and cheapest total path in
best_inner_subqueryscan the same?

9) as remarked up a different thread, more than one clause could be
pushed down a subquery. The current patch only considers alternative
paths that have at most one clause pushed down. Is this because of the
call site of best_inner_subqueryscan, i.e. under one join rel? Would it
be an idea to consider, for each subquery, only a single alternative
plan that tries to have all suitable clauses pushed down?

10) I have a hard time imagining use cases that will actually result in
a alternative plan, especially since not all subqueries are allowed to
have quals pushed down into, and like Simon Riggs pointed out that many
users will write queries like this with the subqueries pulled up. If it
is the case that the subqueries that can't be pulled up have a large
overlap with the ones that are not pushdown safe (limit, set operations
etc), there might be little actual use cases for this patch.

I think the most important thing for this patch to go forward is to have
a few examples, from which it's clear that the patch is beneficial.

regards,

--

Yeb Havinga
http://www.mgrid.net/
Mastering Medical Data

#8Yeb Havinga
yebhavinga@gmail.com
In reply to: Yeb Havinga (#7)
Re: Parameterized aggregate subquery

On 2011-06-30 09:39, Yeb Havinga wrote:

9) as remarked up a different thread, more than one clause could be
pushed down a subquery. The current patch only considers alternative
paths that have at most one clause pushed down. Is this because of the
call site of best_inner_subqueryscan, i.e. under one join rel? Would
it be an idea to consider, for each subquery, only a single
alternative plan that tries to have all suitable clauses pushed down?

Ehm, please forget this remark, I've mistaken join rel for base rel.

-- Yeb

#9Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Yeb Havinga (#7)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

2011/6/30 Yeb Havinga <yebhavinga@gmail.com>:

On 2011-06-29 19:22, Hitoshi Harada wrote:

Other things are all good points. Thanks for elaborate review!
More than anything, I'm going to fix the 6) issue, at least to find the
cause.

Some more questions:
8) why are cheapest start path and cheapest total path in
best_inner_subqueryscan the same?

Because best_inner_indexscan has the two. Actually one of them is
enough so far. I aligned it as the existing interface but they might
be one.

10) I have a hard time imagining use cases that will actually result in a
alternative plan, especially since not all subqueries are allowed to have
quals pushed down into, and like Simon Riggs pointed out that many users
will write queries like this with the subqueries pulled up. If it is the
case that the subqueries that can't be pulled up have a large overlap with
the ones that are not pushdown safe (limit, set operations etc), there might
be little actual use cases for this patch.

I have seen many cases that this planner hack would help
significantly, which were difficult to rewrite. Why were they
difficult to write? Because, quals on size_m (and they have quals on
size_l in fact) are usually very complicated (5-10 op clauses) and the
join+agg part itself is kind of subquery in other big query. Of course
there were workaround like split the statement to two, filtering
size_m then aggregate size_l by the result of the first statement.
However, it's against instinct. The reason why planner is in RDBMS is
to let users to write simple (as needed) statements. I don't know if
the example I raise here is common or not, but I believe the example
represents "one to many" relation simply, therefore there should be
many users who just don't find themselves currently in the slow query
performance.

I think the most important thing for this patch to go forward is to have a
few examples, from which it's clear that the patch is beneficial.

What will be good examples to show benefit of the patch? I guess the
test case of size_m/size_l shows it. What lacks on the case, do you
think?

Regards,

--
Hitoshi Harada

#10Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Yeb Havinga (#5)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

2011/6/29 Yeb Havinga <yebhavinga@gmail.com>:

On 2011-06-17 09:54, Hitoshi Harada wrote:

While reviewing the gist/box patch, I found some planner APIs that can
replace parts in my patch. Also, comments in includes wasn't updated
appropriately. Revised patch attached.

Hello Hitoshi-san,

I read your latest patch implementing parameterizing subquery scans.

Attached is revised version.

1)
In the email from june 9 with the patch You wrote: "While IndexScan
is simple since its information like costs are well known by the base
relation, SubqueryScan should re-plan its Query to gain that, which is
expensive."

Initial concerns I had were caused by misinterpreting 'replanning' as: for
each outer tuple, replan the subquery (it sounds a bit like 'ReScan').
Though the general comments in the patch are helpful, I think it would be an
improvement to describe why subqueries are planned more than once, i.e.
something like
"best_inner_subqueryscan
  Try to find a better subqueryscan path and its associated plan for each
join clause that can be pushed down, in addition to the path that is already
calculated by set_subquery_pathlist."

I changed comments around set_subquery_pathlist and best_inner_subqueryscan.

2)
Since 'subquery_is_pushdown_safe' is invariant under join clause push down,
it might be possible to have it called only once in set_subquery_pathlist,
i.e. by only attaching rel->preprocessed_subquery if the
subquery_is_pushdown_safe.

I modified as you suggested.

3)
/*
 * set_subquery_pathlist
 *        Build the (single) access path for a subquery RTE
 */
This unchanged comment is still correct, but 'the (single) access path'
might have become a bit misleading now there are multiple paths possible,
though not by set_subquery_pathlist.

As noted #1.

4) somewhere in the patch
s/regsitered/registered/

Fixed.

5) Regression tests are missing; I think at this point they'd aid in
speeding up development/test cycles.

I'm still thinking about it. I can add complex test but the concept of
regression test focuses on small pieces of simple cases. I don't want
take pg_regress much more than before.

6) Before patching Postgres, I could execute the test with the size_l and
size_m tables, after patching against current git HEAD (patch without
errors), I get the following error when running the example query:
ERROR:  plan should not reference subplan's variable

I get the same error with the version from june 9 with current git HEAD.

Fixed. Some variable initializing was wrong.

7) Since both set_subquery_pathlist and best_inner_subqueryscan push down
clauses, as well as add a path and subplan, could a generalized function be
made to support both, to reduce duplicate code?

No touch as answered before.

Although I still need to think about suitable regression test case,
the patch itself can be reviewed again. You may want to try some
additional tests as you imagine after finding my test case gets
quicker.

Thanks,

--
Hitoshi Harada

#11Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Hitoshi Harada (#10)
1 attachment(s)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

2011/7/2 Hitoshi Harada <umi.tanuki@gmail.com>:

2011/6/29 Yeb Havinga <yebhavinga@gmail.com>:

On 2011-06-17 09:54, Hitoshi Harada wrote:

While reviewing the gist/box patch, I found some planner APIs that can
replace parts in my patch. Also, comments in includes wasn't updated
appropriately. Revised patch attached.

Hello Hitoshi-san,

I read your latest patch implementing parameterizing subquery scans.

Attached is revised version.

I failed to attached the patch. I'm trying again.

1)
In the email from june 9 with the patch You wrote: "While IndexScan
is simple since its information like costs are well known by the base
relation, SubqueryScan should re-plan its Query to gain that, which is
expensive."

Initial concerns I had were caused by misinterpreting 'replanning' as: for
each outer tuple, replan the subquery (it sounds a bit like 'ReScan').
Though the general comments in the patch are helpful, I think it would be an
improvement to describe why subqueries are planned more than once, i.e.
something like
"best_inner_subqueryscan
  Try to find a better subqueryscan path and its associated plan for each
join clause that can be pushed down, in addition to the path that is already
calculated by set_subquery_pathlist."

I changed comments around set_subquery_pathlist and best_inner_subqueryscan.

2)
Since 'subquery_is_pushdown_safe' is invariant under join clause push down,
it might be possible to have it called only once in set_subquery_pathlist,
i.e. by only attaching rel->preprocessed_subquery if the
subquery_is_pushdown_safe.

I modified as you suggested.

3)
/*
 * set_subquery_pathlist
 *        Build the (single) access path for a subquery RTE
 */
This unchanged comment is still correct, but 'the (single) access path'
might have become a bit misleading now there are multiple paths possible,
though not by set_subquery_pathlist.

As noted #1.

4) somewhere in the patch
s/regsitered/registered/

Fixed.

5) Regression tests are missing; I think at this point they'd aid in
speeding up development/test cycles.

I'm still thinking about it. I can add complex test but the concept of
regression test focuses on small pieces of simple cases. I don't want
take pg_regress much more than before.

6) Before patching Postgres, I could execute the test with the size_l and
size_m tables, after patching against current git HEAD (patch without
errors), I get the following error when running the example query:
ERROR:  plan should not reference subplan's variable

I get the same error with the version from june 9 with current git HEAD.

Fixed. Some variable initializing was wrong.

7) Since both set_subquery_pathlist and best_inner_subqueryscan push down
clauses, as well as add a path and subplan, could a generalized function be
made to support both, to reduce duplicate code?

No touch as answered before.

Although I still need to think about suitable regression test case,
the patch itself can be reviewed again. You may want to try some
additional tests as you imagine after finding my test case gets
quicker.

Thanks,

--
Hitoshi Harada

--
Hitoshi Harada

Attachments:

aggjoin-20110702.patchtext/x-patch; charset=US-ASCII; name=aggjoin-20110702.patchDownload
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 681f5f8..039fd7f 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -1557,6 +1557,17 @@ _outTidPath(StringInfo str, TidPath *node)
 }
 
 static void
+_outSubqueryPath(StringInfo str, SubqueryPath *node)
+{
+	WRITE_NODE_TYPE("SUBQUERYPATH");
+
+	_outPathInfo(str, (Path *) node);
+	WRITE_NODE_FIELD(subplan);
+	WRITE_NODE_FIELD(subrtable);
+	WRITE_NODE_FIELD(subrowmark);
+}
+
+static void
 _outForeignPath(StringInfo str, ForeignPath *node)
 {
 	WRITE_NODE_TYPE("FOREIGNPATH");
@@ -1738,9 +1749,6 @@ _outRelOptInfo(StringInfo str, RelOptInfo *node)
 	WRITE_NODE_FIELD(indexlist);
 	WRITE_UINT_FIELD(pages);
 	WRITE_FLOAT_FIELD(tuples, "%.0f");
-	WRITE_NODE_FIELD(subplan);
-	WRITE_NODE_FIELD(subrtable);
-	WRITE_NODE_FIELD(subrowmark);
 	WRITE_NODE_FIELD(baserestrictinfo);
 	WRITE_NODE_FIELD(joininfo);
 	WRITE_BOOL_FIELD(has_eclass_joins);
@@ -2948,6 +2956,9 @@ _outNode(StringInfo str, void *obj)
 			case T_TidPath:
 				_outTidPath(str, obj);
 				break;
+			case T_SubqueryPath:
+				_outSubqueryPath(str, obj);
+				break;
 			case T_ForeignPath:
 				_outForeignPath(str, obj);
 				break;
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index 47ab08e..354a3e5 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -31,6 +31,7 @@
 #include "optimizer/planner.h"
 #include "optimizer/prep.h"
 #include "optimizer/restrictinfo.h"
+#include "optimizer/subselect.h"
 #include "optimizer/var.h"
 #include "parser/parse_clause.h"
 #include "parser/parsetree.h"
@@ -677,6 +678,12 @@ has_multiple_baserels(PlannerInfo *root)
 /*
  * set_subquery_pathlist
  *		Build the (single) access path for a subquery RTE
+ *
+ * Although we build only one access path for the subquery,
+ * join search process may find another path by pushing down
+ * the nestloop param into subquery. The finding process will
+ * be done much later than here, but some common operation like
+ * preprocessing subquery are shared.
  */
 static void
 set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
@@ -687,7 +694,9 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
 	bool	   *differentTypes;
 	double		tuple_fraction;
 	PlannerInfo *subroot;
-	List	   *pathkeys;
+	Plan	    *subplan;
+	List	    *pathkeys;
+	bool		issafe = false;
 
 	/*
 	 * Must copy the Query so that planning doesn't mess up the RTE contents
@@ -721,7 +730,7 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
 	 * push down a pushable qual, because it'd result in a worse plan?
 	 */
 	if (rel->baserestrictinfo != NIL &&
-		subquery_is_pushdown_safe(subquery, subquery, differentTypes))
+		(issafe = subquery_is_pushdown_safe(subquery, subquery, differentTypes)))
 	{
 		/* OK to consider pushing down individual quals */
 		List	   *upperrestrictlist = NIL;
@@ -749,6 +758,10 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
 
 	pfree(differentTypes);
 
+	/* save it for later use in best_inner_subqueryscan */
+	if (issafe)
+		rel->preprocessed_subquery = copyObject(subquery);
+
 	/*
 	 * We can safely pass the outer tuple_fraction down to the subquery if the
 	 * outer level has no joining, aggregation, or sorting to do. Otherwise
@@ -766,27 +779,187 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
 		tuple_fraction = root->tuple_fraction;
 
 	/* Generate the plan for the subquery */
-	rel->subplan = subquery_planner(root->glob, subquery,
-									root,
-									false, tuple_fraction,
-									&subroot);
-	rel->subrtable = subroot->parse->rtable;
-	rel->subrowmark = subroot->rowMarks;
+	subplan = subquery_planner(root->glob, subquery,
+							   root,
+							   false, tuple_fraction,
+							   &subroot);
 
 	/* Mark rel with estimated output rows, width, etc */
-	set_subquery_size_estimates(root, rel, subroot);
+	set_subquery_size_estimates(root, rel, subroot, subplan);
 
 	/* Convert subquery pathkeys to outer representation */
-	pathkeys = convert_subquery_pathkeys(root, rel, subroot->query_pathkeys);
+	pathkeys = convert_subquery_pathkeys(root, rel,
+										 subroot->query_pathkeys, subplan);
 
 	/* Generate appropriate path */
-	add_path(rel, create_subqueryscan_path(rel, pathkeys));
+	add_path(rel, create_subqueryscan_path(rel, pathkeys,
+			 subplan, subroot->parse->rtable, subroot->rowMarks));
 
 	/* Select cheapest path (pretty easy in this case...) */
 	set_cheapest(rel);
 }
 
 /*
+ * best_inner_subqueryscan
+ *
+ * As the inner scan relation, try to find another subquer path which may
+ * be better by pushing down some join clauses. If such possibility is found,
+ * return the path in *cheapest_start_path or *cheapest_total_path.
+ * Currently both paths will be resulted in the same, whereas further
+ * improvement might find cases they'd be different.
+ *
+ * It re-uses modified subquery (a.k.a. rel->preprocessed_subquery.) The basic
+ * push-down of baserestrictinfo was common among base path (done in
+ * set_subquery_pathlist()) so this routine doesn't care it. But once we find
+ * push-down-able join clause, the parsed subquery will be modified after copied.
+ */
+void
+best_inner_subqueryscan(PlannerInfo *root, RelOptInfo *rel,
+						RelOptInfo *outer_rel, List *join_clause,
+						Path **cheapest_start_path, Path **cheapest_total_path)
+{
+	Query	   *parse = root->parse;
+	Query	   *subquery;
+	Index		rti = rel->relid;
+	RangeTblEntry *rte = planner_rt_fetch(rti, root);
+	bool	   *differentTypes;
+	double		tuple_fraction;
+	PlannerInfo *subroot;
+	Plan	   *subplan;
+	List	   *pathkeys;
+	Path	   *path;
+	bool		query_modified = false;
+
+	/* initialize paths to return immediately anytime */
+	*cheapest_start_path = *cheapest_total_path = NULL;
+
+	/*
+	 * empty join condition doesn't make sense here
+	 */
+	if (join_clause == NIL)
+		return;
+
+	/*
+	 * If the subquery is not safe for pushdown, preprocessed
+	 * subquery was not saved. Simply skip this process.
+	 */
+	if (!rel->preprocessed_subquery)
+		return;
+
+	/* copying the whole Query is expensive; let's copy-on-write */
+	subquery = rel->preprocessed_subquery;
+
+	/* We need a workspace for keeping track of set-op type coercions */
+	differentTypes = (bool *)
+		palloc0((list_length(subquery->targetList) + 1) * sizeof(bool));
+
+	/*
+	 * Currently we focus on only cases of aggregate subquery.
+	 * I'm not quite sure if there are other useful cases.
+	 */
+	if (subquery->hasAggs)
+	{
+		ListCell   *l;
+
+		foreach (l, join_clause)
+		{
+			RestrictInfo   *rinfo = (RestrictInfo *) lfirst(l);
+			Node		   *lexpr, *rexpr;
+			Expr		   *clause = NULL;
+			Param		   *param;
+			OpExpr		   *rclause;
+
+			/*
+			 * For the first step, only Var = Var join clause is targeted.
+			 * This may be improved in the future.
+			 */
+			if (!is_opclause(rinfo->clause))
+				continue;
+			rclause = (OpExpr *) rinfo->clause;
+			lexpr = get_leftop((Expr *) rclause);
+			rexpr = get_rightop((Expr *) rclause);
+			if (!(IsA(lexpr, Var) && IsA(rexpr, Var)))
+				continue;
+
+			/* see which arg is mine and outer's */
+			if (bms_is_member(((Var *) lexpr)->varno, outer_rel->relids))
+			{
+				param = assign_nestloop_param(root, (Var *) lexpr);
+				clause = make_opclause(rclause->opno,
+									   rclause->opresulttype,
+									   false,
+									   (Expr *) param,
+									   (Expr *) copyObject(rexpr),
+									   InvalidOid,
+									   rclause->opcollid);
+			}
+			else if (bms_is_member(((Var *) rexpr)->varno, outer_rel->relids))
+			{
+				param = assign_nestloop_param(root, (Var *) rexpr);
+				clause = make_opclause(rclause->opno,
+									   rclause->opresulttype,
+									   false,
+									   (Expr *) copyObject(lexpr),
+									   (Expr *) param,
+									   InvalidOid,
+									   rclause->opcollid);
+			}
+
+			/* TODO:no check isouter or not? */
+			if (clause &&
+				qual_is_pushdown_safe(subquery, rti, (Node *) clause,
+									  differentTypes))
+			{
+				/* copy on write */
+				if (rel->preprocessed_subquery == subquery)
+					subquery = copyObject(subquery);
+				subquery_push_qual(subquery, rte, rti, (Node *) clause);
+				query_modified = true;
+			}
+		}
+	}
+
+	pfree(differentTypes);
+
+	/* return immediately if such subquery isn't found. Use original one */
+	if (!query_modified)
+		return;
+
+	/*
+	 * We can safely pass the outer tuple_fraction down to the subquery if the
+	 * outer level has no joining, aggregation, or sorting to do. Otherwise
+	 * we'd better tell the subquery to plan for full retrieval. (XXX This
+	 * could probably be made more intelligent ...)
+	 */
+	if (parse->hasAggs ||
+		parse->groupClause ||
+		parse->havingQual ||
+		parse->distinctClause ||
+		parse->sortClause ||
+		has_multiple_baserels(root))
+		tuple_fraction = 0.0;	/* default case */
+	else
+		tuple_fraction = root->tuple_fraction;
+
+	/* Generate the plan for the subquery */
+	subplan = subquery_planner(root->glob, subquery,
+							   root,
+							   false, tuple_fraction,
+							   &subroot);
+
+	/* Convert subquery pathkeys to outer representation */
+	pathkeys = convert_subquery_pathkeys(root, rel,
+										 subroot->query_pathkeys, subplan);
+
+	/* Generate appropriate path */
+	path = create_subqueryscan_path(rel, pathkeys,
+			 subplan, subroot->parse->rtable, subroot->rowMarks);
+
+	/* return the same path for now */
+	*cheapest_start_path = *cheapest_total_path = path;
+}
+
+/*
  * set_function_pathlist
  *		Build the (single) access path for a function RTE
  */
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index bb38768..b7271c3 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -892,7 +892,7 @@ cost_tidscan(Path *path, PlannerInfo *root,
  *	  Determines and returns the cost of scanning a subquery RTE.
  */
 void
-cost_subqueryscan(Path *path, RelOptInfo *baserel)
+cost_subqueryscan(SubqueryPath *path, RelOptInfo *baserel)
 {
 	Cost		startup_cost;
 	Cost		run_cost;
@@ -907,15 +907,15 @@ cost_subqueryscan(Path *path, RelOptInfo *baserel)
 	 * any restriction clauses that will be attached to the SubqueryScan node,
 	 * plus cpu_tuple_cost to account for selection and projection overhead.
 	 */
-	path->startup_cost = baserel->subplan->startup_cost;
-	path->total_cost = baserel->subplan->total_cost;
+	path->path.startup_cost = path->subplan->startup_cost;
+	path->path.total_cost = path->subplan->total_cost;
 
 	startup_cost = baserel->baserestrictcost.startup;
 	cpu_per_tuple = cpu_tuple_cost + baserel->baserestrictcost.per_tuple;
 	run_cost = cpu_per_tuple * baserel->tuples;
 
-	path->startup_cost += startup_cost;
-	path->total_cost += startup_cost + run_cost;
+	path->path.startup_cost += startup_cost;
+	path->path.total_cost += startup_cost + run_cost;
 }
 
 /*
@@ -3222,7 +3222,7 @@ set_joinrel_size_estimates(PlannerInfo *root, RelOptInfo *rel,
  */
 void
 set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
-							PlannerInfo *subroot)
+							PlannerInfo *subroot, Plan *subplan)
 {
 	RangeTblEntry *rte;
 	ListCell   *lc;
@@ -3233,7 +3233,7 @@ set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
 	Assert(rte->rtekind == RTE_SUBQUERY);
 
 	/* Copy raw number of output rows from subplan */
-	rel->tuples = rel->subplan->plan_rows;
+	rel->tuples = subplan->plan_rows;
 
 	/*
 	 * Compute per-output-column width estimates by examining the subquery's
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index 7d3cf42..c79fd43 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -329,7 +329,9 @@ sort_inner_and_outer(PlannerInfo *root,
  * inner path, one on the same with materialization, one on the
  * cheapest-startup-cost inner path (if different), one on the
  * cheapest-total inner-indexscan path (if any), and one on the
- * cheapest-startup inner-indexscan path (if different).
+ * cheapest-startup inner-indexscan path (if different) or on the
+ * cheapest-total inner-subqueryscan path (if any).
+ * Note inner-indexpaths and inner-subquerypaths are not made at the same time.
  *
  * We also consider mergejoins if mergejoin clauses are available.	We have
  * two ways to generate the inner path for a mergejoin: sort the cheapest
@@ -369,6 +371,8 @@ match_unsorted_outer(PlannerInfo *root,
 	Path	   *matpath = NULL;
 	Path	   *index_cheapest_startup = NULL;
 	Path	   *index_cheapest_total = NULL;
+	Path	   *sub_cheapest_startup = NULL;
+	Path	   *sub_cheapest_total = NULL;
 	ListCell   *l;
 
 	/*
@@ -430,8 +434,8 @@ match_unsorted_outer(PlannerInfo *root,
 				create_material_path(innerrel, inner_cheapest_total);
 
 		/*
-		 * Get the best innerjoin indexpaths (if any) for this outer rel.
-		 * They're the same for all outer paths.
+		 * Get the best innerjoin indexpaths or subquerypaths (if any)
+		 * for this outer rel. They're the same for all outer paths.
 		 */
 		if (innerrel->reloptkind != RELOPT_JOINREL)
 		{
@@ -444,6 +448,10 @@ match_unsorted_outer(PlannerInfo *root,
 				best_inner_indexscan(root, innerrel, outerrel, jointype,
 									 &index_cheapest_startup,
 									 &index_cheapest_total);
+			else if (innerrel->rtekind == RTE_SUBQUERY)
+				best_inner_subqueryscan(root, innerrel, outerrel, restrictlist,
+										&sub_cheapest_startup,
+										&sub_cheapest_total);
 		}
 	}
 
@@ -487,7 +495,7 @@ match_unsorted_outer(PlannerInfo *root,
 			 * cheapest-total-cost inner.  When appropriate, also consider
 			 * using the materialized form of the cheapest inner, the
 			 * cheapest-startup-cost inner path, and the cheapest innerjoin
-			 * indexpaths.
+			 * indexpaths or subquerypaths.
 			 */
 			add_path(joinrel, (Path *)
 					 create_nestloop_path(root,
@@ -539,6 +547,16 @@ match_unsorted_outer(PlannerInfo *root,
 											  index_cheapest_startup,
 											  restrictlist,
 											  merge_pathkeys));
+			if (sub_cheapest_total != NULL)
+				add_path(joinrel, (Path *)
+						 create_nestloop_path(root,
+											  joinrel,
+											  jointype,
+											  sjinfo,
+											  outerpath,
+											  sub_cheapest_startup,
+											  restrictlist,
+											  merge_pathkeys));
 		}
 
 		/* Can't do anything else if outer path needs to be unique'd */
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 24e4e59..c3f7ca0 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -745,7 +745,6 @@ make_join_rel(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2)
 	return joinrel;
 }
 
-
 /*
  * have_join_order_restriction
  *		Detect whether the two relations should be joined to satisfy
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index e5228a8..cee5c71 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -623,12 +623,12 @@ find_indexkey_var(PlannerInfo *root, RelOptInfo *rel, AttrNumber varattno)
  */
 List *
 convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel,
-						  List *subquery_pathkeys)
+						  List *subquery_pathkeys, Plan *subplan)
 {
 	List	   *retval = NIL;
 	int			retvallen = 0;
 	int			outer_query_keys = list_length(root->query_pathkeys);
-	List	   *sub_tlist = rel->subplan->targetlist;
+	List	   *sub_tlist = subplan->targetlist;
 	ListCell   *i;
 
 	foreach(i, subquery_pathkeys)
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index e4ccf5c..417c9e3 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -62,8 +62,8 @@ static Plan *create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual,
 					  List **qual, List **indexqual);
 static TidScan *create_tidscan_plan(PlannerInfo *root, TidPath *best_path,
 					List *tlist, List *scan_clauses);
-static SubqueryScan *create_subqueryscan_plan(PlannerInfo *root, Path *best_path,
-						 List *tlist, List *scan_clauses);
+static SubqueryScan *create_subqueryscan_plan(PlannerInfo *root,
+					SubqueryPath *best_path, List *tlist, List *scan_clauses);
 static FunctionScan *create_functionscan_plan(PlannerInfo *root, Path *best_path,
 						 List *tlist, List *scan_clauses);
 static ValuesScan *create_valuesscan_plan(PlannerInfo *root, Path *best_path,
@@ -321,7 +321,7 @@ create_scan_plan(PlannerInfo *root, Path *best_path)
 
 		case T_SubqueryScan:
 			plan = (Plan *) create_subqueryscan_plan(root,
-													 best_path,
+													 (SubqueryPath *) best_path,
 													 tlist,
 													 scan_clauses);
 			break;
@@ -1538,15 +1538,15 @@ create_tidscan_plan(PlannerInfo *root, TidPath *best_path,
  *	 with restriction clauses 'scan_clauses' and targetlist 'tlist'.
  */
 static SubqueryScan *
-create_subqueryscan_plan(PlannerInfo *root, Path *best_path,
+create_subqueryscan_plan(PlannerInfo *root, SubqueryPath *best_path,
 						 List *tlist, List *scan_clauses)
 {
 	SubqueryScan *scan_plan;
-	Index		scan_relid = best_path->parent->relid;
+	Index		scan_relid = best_path->path.parent->relid;
 
 	/* it should be a subquery base rel... */
 	Assert(scan_relid > 0);
-	Assert(best_path->parent->rtekind == RTE_SUBQUERY);
+	Assert(best_path->path.parent->rtekind == RTE_SUBQUERY);
 
 	/* Sort clauses into best execution order */
 	scan_clauses = order_qual_clauses(root, scan_clauses);
@@ -1557,11 +1557,63 @@ create_subqueryscan_plan(PlannerInfo *root, Path *best_path,
 	scan_plan = make_subqueryscan(tlist,
 								  scan_clauses,
 								  scan_relid,
-								  best_path->parent->subplan,
-								  best_path->parent->subrtable,
-								  best_path->parent->subrowmark);
+								  best_path->subplan,
+								  best_path->subrtable,
+								  best_path->subrowmark);
 
-	copy_path_costsize(&scan_plan->scan.plan, best_path);
+	copy_path_costsize(&scan_plan->scan.plan, &best_path->path);
+
+	/*
+	 * If this plan is inner of nestloop and push-down-qual case,
+	 * params are extracted and registered for the upper nestloop.
+	 * As Plan has only bitmap extParam field, we cannot take them
+	 * directly; instead take them from PlannerGlobal.
+	 */
+	if (scan_plan->subplan->extParam)
+	{
+		int		i;
+		ListCell   *ppl;
+		PlannerParamItem   *pitem;
+
+		i = 0;
+		foreach(ppl, root->glob->paramlist)
+		{
+			pitem = (PlannerParamItem *) lfirst(ppl);
+			/* take the ones only in this query level */
+			if (pitem->abslevel == root->query_level &&
+				IsA(pitem->item, Var) &&
+				((Var *) pitem->item)->varlevelsup == 0)
+			{
+				Var	   *var = (Var *) pitem->item;
+				NestLoopParam	   *nlp;
+				ListCell   *lc;
+				bool		found = false;
+
+				/* Is the param already listed in root->curOuterParams? */
+				foreach(lc, root->curOuterParams)
+				{
+					nlp = (NestLoopParam *) lfirst(lc);
+					if (equal(nlp->paramval, var))
+					{
+						/* Present, so we can just skip */
+						found = true;
+						break;
+					}
+				}
+				if (found)
+				{
+					i++;
+					continue;
+				}
+				/* No, so add it */
+				nlp = makeNode(NestLoopParam);
+				nlp->paramno = i;
+				nlp->paramval = var;
+				root->curOuterParams = lappend(root->curOuterParams, nlp);
+			}
+			i++;
+		}
+	}
 
 	return scan_plan;
 }
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 161d5ab..2e711d8 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -1331,17 +1331,21 @@ distinct_col_search(int colno, List *colnos, List *opids)
  *	  returning the pathnode.
  */
 Path *
-create_subqueryscan_path(RelOptInfo *rel, List *pathkeys)
+create_subqueryscan_path(RelOptInfo *rel, List *pathkeys, Plan *subplan,
+						 List *subrtable, List *subrowmark)
 {
-	Path	   *pathnode = makeNode(Path);
+	SubqueryPath   *pathnode = makeNode(SubqueryPath);
 
-	pathnode->pathtype = T_SubqueryScan;
-	pathnode->parent = rel;
-	pathnode->pathkeys = pathkeys;
+	pathnode->path.pathtype = T_SubqueryScan;
+	pathnode->path.parent = rel;
+	pathnode->path.pathkeys = pathkeys;
+	pathnode->subplan = subplan;
+	pathnode->subrtable = subrtable;
+	pathnode->subrowmark = subrowmark;
 
 	cost_subqueryscan(pathnode, rel);
 
-	return pathnode;
+	return (Path *) pathnode;
 }
 
 /*
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index b7a5845..ab2338f 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -82,9 +82,7 @@ build_simple_rel(PlannerInfo *root, int relid, RelOptKind reloptkind)
 	rel->indexlist = NIL;
 	rel->pages = 0;
 	rel->tuples = 0;
-	rel->subplan = NULL;
-	rel->subrtable = NIL;
-	rel->subrowmark = NIL;
+	rel->preprocessed_subquery = NULL;
 	rel->baserestrictinfo = NIL;
 	rel->baserestrictcost.startup = 0;
 	rel->baserestrictcost.per_tuple = 0;
@@ -336,9 +334,7 @@ build_join_rel(PlannerInfo *root,
 	joinrel->indexlist = NIL;
 	joinrel->pages = 0;
 	joinrel->tuples = 0;
-	joinrel->subplan = NULL;
-	joinrel->subrtable = NIL;
-	joinrel->subrowmark = NIL;
+	joinrel->preprocessed_subquery = NULL;
 	joinrel->baserestrictinfo = NIL;
 	joinrel->baserestrictcost.startup = 0;
 	joinrel->baserestrictcost.per_tuple = 0;
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index d8bc6b8..e951fd1 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -220,6 +220,7 @@ typedef enum NodeTag
 	T_MergePath,
 	T_HashPath,
 	T_TidPath,
+	T_SubqueryPath,
 	T_ForeignPath,
 	T_AppendPath,
 	T_MergeAppendPath,
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index f659269..98e2744 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -322,13 +322,6 @@ typedef struct PlannerInfo
  *					(always NIL if it's not a table)
  *		pages - number of disk pages in relation (zero if not a table)
  *		tuples - number of tuples in relation (not considering restrictions)
- *		subplan - plan for subquery (NULL if it's not a subquery)
- *		subrtable - rangetable for subquery (NIL if it's not a subquery)
- *		subrowmark - rowmarks for subquery (NIL if it's not a subquery)
- *
- *		Note: for a subquery, tuples and subplan are not set immediately
- *		upon creation of the RelOptInfo object; they are filled in when
- *		set_base_rel_pathlist processes the object.
  *
  *		For otherrels that are appendrel members, these fields are filled
  *		in just as for a baserel.
@@ -408,9 +401,8 @@ typedef struct RelOptInfo
 	List	   *indexlist;		/* list of IndexOptInfo */
 	BlockNumber pages;
 	double		tuples;
-	struct Plan *subplan;		/* if subquery */
-	List	   *subrtable;		/* if subquery */
-	List	   *subrowmark;		/* if subquery */
+
+	Query	   *preprocessed_subquery; /* if subquery */
 
 	/* used by various scans and joins: */
 	List	   *baserestrictinfo;		/* RestrictInfo structures (if base
@@ -770,6 +762,23 @@ typedef struct TidPath
 } TidPath;
 
 /*
+ * SubqueryPath represents a scan of a subquery scan
+ *
+ * The struct holds subplan, subrtable and subrowmark, which are
+ * the temporary spaces to transport to the final stage of planner.
+ * Note each SubqueryPath may have different plan, depending on
+ * the pushed down qual clauses (as parameterized inner subquery).
+ */
+typedef struct SubqueryPath
+{
+	Path		path;
+	struct Plan *subplan;
+	List	   *subrtable;
+	List	   *subrowmark;
+	double		rows;
+} SubqueryPath;
+
+/*
  * ForeignPath represents a scan of a foreign table
  */
 typedef struct ForeignPath
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 2763863..7ec2c7b 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -75,7 +75,7 @@ extern void cost_bitmap_or_node(BitmapOrPath *path, PlannerInfo *root);
 extern void cost_bitmap_tree_node(Path *path, Cost *cost, Selectivity *selec);
 extern void cost_tidscan(Path *path, PlannerInfo *root,
 			 RelOptInfo *baserel, List *tidquals);
-extern void cost_subqueryscan(Path *path, RelOptInfo *baserel);
+extern void cost_subqueryscan(SubqueryPath *path, RelOptInfo *baserel);
 extern void cost_functionscan(Path *path, PlannerInfo *root,
 				  RelOptInfo *baserel);
 extern void cost_valuesscan(Path *path, PlannerInfo *root,
@@ -122,7 +122,7 @@ extern void set_joinrel_size_estimates(PlannerInfo *root, RelOptInfo *rel,
 						   SpecialJoinInfo *sjinfo,
 						   List *restrictlist);
 extern void set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
-							PlannerInfo *subroot);
+							PlannerInfo *subroot, Plan *subplan);
 extern void set_function_size_estimates(PlannerInfo *root, RelOptInfo *rel);
 extern void set_values_size_estimates(PlannerInfo *root, RelOptInfo *rel);
 extern void set_cte_size_estimates(PlannerInfo *root, RelOptInfo *rel,
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 1da2131..6de2816 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -14,6 +14,7 @@
 #ifndef PATHNODE_H
 #define PATHNODE_H
 
+#include "nodes/plannodes.h"
 #include "nodes/relation.h"
 
 
@@ -56,7 +57,8 @@ extern ResultPath *create_result_path(List *quals);
 extern MaterialPath *create_material_path(RelOptInfo *rel, Path *subpath);
 extern UniquePath *create_unique_path(PlannerInfo *root, RelOptInfo *rel,
 				   Path *subpath, SpecialJoinInfo *sjinfo);
-extern Path *create_subqueryscan_path(RelOptInfo *rel, List *pathkeys);
+extern Path *create_subqueryscan_path(RelOptInfo *rel, List *pathkeys,
+					Plan *subplan, List *subrtable, List *subrowmark);
 extern Path *create_functionscan_path(PlannerInfo *root, RelOptInfo *rel);
 extern Path *create_valuesscan_path(PlannerInfo *root, RelOptInfo *rel);
 extern Path *create_ctescan_path(PlannerInfo *root, RelOptInfo *rel);
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 7f1353a..54d0b01 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -33,6 +33,9 @@ extern PGDLLIMPORT join_search_hook_type join_search_hook;
 extern RelOptInfo *make_one_rel(PlannerInfo *root, List *joinlist);
 extern RelOptInfo *standard_join_search(PlannerInfo *root, int levels_needed,
 					 List *initial_rels);
+extern void best_inner_subqueryscan(PlannerInfo *root, RelOptInfo *rel,
+						RelOptInfo *outer_rel, List *join_clause,
+						Path **cheapest_start_path, Path **cheapest_total_path);
 
 #ifdef OPTIMIZER_DEBUG
 extern void debug_print_rel(PlannerInfo *root, RelOptInfo *rel);
@@ -154,7 +157,7 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
 extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
 					 ScanDirection scandir);
 extern List *convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel,
-						  List *subquery_pathkeys);
+						  List *subquery_pathkeys, Plan *subplan);
 extern List *build_join_pathkeys(PlannerInfo *root,
 					RelOptInfo *joinrel,
 					JoinType jointype,
#12Yeb Havinga
yebhavinga@gmail.com
In reply to: Hitoshi Harada (#11)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

Hello Hitosh, list,

Attached is revised version.

I failed to attached the patch. I'm trying again.

I'm currently unable to test, since on holiday. I'm happy to continue

testing once returned but it may not be within the bounds of the current
commitfest, sorry.

5) Regression tests are missing; I think at this point they'd aid in
speeding up development/test cycles.

I'm still thinking about it. I can add complex test but the concept of
regression test focuses on small pieces of simple cases. I don't want
take pg_regress much more than before.

IMHO, at least one query where the new code is touched is a good idea.

I get the same error with the version from june 9 with current git HEAD.

Fixed. Some variable initializing was wrong.

Thanks - will test when I am back from holiday and other duties.

regards,
Yeb

#13Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Yeb Havinga (#12)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

2011/7/5 Yeb Havinga <yebhavinga@gmail.com>:

Hello Hitosh, list,

Attached is revised version.

I failed to attached the patch. I'm trying again.

I'm currently unable to test, since on holiday. I'm happy to continue
testing once returned but it may not be within the bounds of the current
commitfest, sorry.

Thanks for replying. Regarding the time remained until the end of this
commitfest, I'd think we should mark this item "Returned with
Feedback" if no objection appears. I will be very happy if Yeb tests
my latest patch after he comes back. I'll make up my mind around
regression test.

Regards,

--
Hitoshi Harada

#14Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Hitoshi Harada (#13)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

2011/7/5 Hitoshi Harada <umi.tanuki@gmail.com>:

2011/7/5 Yeb Havinga <yebhavinga@gmail.com>:

Hello Hitosh, list,

Attached is revised version.

I failed to attached the patch. I'm trying again.

I'm currently unable to test, since on holiday. I'm happy to continue
testing once returned but it may not be within the bounds of the current
commitfest, sorry.

Thanks for replying. Regarding the time remained until the end of this
commitfest, I'd think we should mark this item "Returned with
Feedback" if no objection appears. I will be very happy if Yeb tests
my latest patch after he comes back. I'll make up my mind around
regression test.

It seems that Yeb marked "Returned with Feedback". Thanks for the
review again. Still, I'd appreciate if further review is done on my
latest patch.

Thanks,
--
Hitoshi Harada

#15Yeb Havinga
yebhavinga@gmail.com
In reply to: Hitoshi Harada (#14)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

On 2011-07-09 16:23, Hitoshi Harada wrote:

2011/7/5 Hitoshi Harada<umi.tanuki@gmail.com>:

2011/7/5 Yeb Havinga<yebhavinga@gmail.com>:

Hello Hitosh, list,

Attached is revised version.

I failed to attached the patch. I'm trying again.

I'm currently unable to test, since on holiday. I'm happy to continue
testing once returned but it may not be within the bounds of the current
commitfest, sorry.

Thanks for replying. Regarding the time remained until the end of this
commitfest, I'd think we should mark this item "Returned with
Feedback" if no objection appears. I will be very happy if Yeb tests
my latest patch after he comes back. I'll make up my mind around
regression test.

It seems that Yeb marked "Returned with Feedback". Thanks for the
review again. Still, I'd appreciate if further review is done on my
latest patch.

Yes, I did so after you suggested to return it to feedback. I'll review
your latest patch as soon as there is enough time to do a proper review,
which is probably after next week.

regards,
Yeb Havinga

#16Yeb Havinga
yebhavinga@gmail.com
In reply to: Hitoshi Harada (#11)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

On 2011-07-02 10:02, Hitoshi Harada wrote:

Although I still need to think about suitable regression test case,
the patch itself can be reviewed again. You may want to try some
additional tests as you imagine after finding my test case gets
quicker.

Hello Hitoshi-san,

I took a look at your latest patch and it looks good, no comments.
However I also tried it against current 9.2 HEAD and the test query of
the start of this thread.

Before and after applying the patch, I get the same result for the test
query.

postgres=# explain select m_id, sum_len from size_m m inner join(select
m_id,
sum(length(val)) as sum_len from size_l group by m_id)l on m.id =
l.m_id where val = '10101';
QUERY PLAN
----------------------------------------------------------------------------------
Nested Loop (cost=79392.64..82938.05 rows=100 width=12)
Join Filter: (m.id = size_l.m_id)
-> Seq Scan on size_m m (cost=0.00..897.00 rows=1 width=4)
Filter: (val = '10101'::text)
-> GroupAggregate (cost=79392.64..81592.15 rows=19951 width=277)
-> Sort (cost=79392.64..79892.64 rows=200000 width=277)
Sort Key: size_l.m_id
-> Seq Scan on size_l (cost=0.00..9829.00 rows=200000
width=277)

I double checked that I had applied the patch (git diff shows the
patch), installed and restarted postgres. The database is a fresh
created database with no edits in postgresql.conf.

regards,

--
Yeb Havinga
http://www.mgrid.net/
Mastering Medical Data

#17Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Yeb Havinga (#16)
1 attachment(s)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

2011/7/22 Yeb Havinga <yebhavinga@gmail.com>:

On 2011-07-02 10:02, Hitoshi Harada wrote:

Although I still need to think about suitable regression test case,
the patch itself can be reviewed again. You may want to try some
additional tests as you imagine after finding my test case gets
quicker.

Hello Hitoshi-san,

Hi,

I double checked that I had applied the patch (git diff shows the patch),
installed and restarted postgres. The database is a fresh created database
with no edits in postgresql.conf.

:(
I updated the patch. Could you try attached once more? The "issafe"
switch seems wrong.

Thanks,
--
Hitoshi Harada

Attachments:

aggjoin-20110722.patchapplication/octet-stream; name=aggjoin-20110722.patchDownload
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 681f5f8..039fd7f 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -1557,6 +1557,17 @@ _outTidPath(StringInfo str, TidPath *node)
 }
 
 static void
+_outSubqueryPath(StringInfo str, SubqueryPath *node)
+{
+	WRITE_NODE_TYPE("SUBQUERYPATH");
+
+	_outPathInfo(str, (Path *) node);
+	WRITE_NODE_FIELD(subplan);
+	WRITE_NODE_FIELD(subrtable);
+	WRITE_NODE_FIELD(subrowmark);
+}
+
+static void
 _outForeignPath(StringInfo str, ForeignPath *node)
 {
 	WRITE_NODE_TYPE("FOREIGNPATH");
@@ -1738,9 +1749,6 @@ _outRelOptInfo(StringInfo str, RelOptInfo *node)
 	WRITE_NODE_FIELD(indexlist);
 	WRITE_UINT_FIELD(pages);
 	WRITE_FLOAT_FIELD(tuples, "%.0f");
-	WRITE_NODE_FIELD(subplan);
-	WRITE_NODE_FIELD(subrtable);
-	WRITE_NODE_FIELD(subrowmark);
 	WRITE_NODE_FIELD(baserestrictinfo);
 	WRITE_NODE_FIELD(joininfo);
 	WRITE_BOOL_FIELD(has_eclass_joins);
@@ -2948,6 +2956,9 @@ _outNode(StringInfo str, void *obj)
 			case T_TidPath:
 				_outTidPath(str, obj);
 				break;
+			case T_SubqueryPath:
+				_outSubqueryPath(str, obj);
+				break;
 			case T_ForeignPath:
 				_outForeignPath(str, obj);
 				break;
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index 6b43aee..dbea2f1 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -31,6 +31,7 @@
 #include "optimizer/planner.h"
 #include "optimizer/prep.h"
 #include "optimizer/restrictinfo.h"
+#include "optimizer/subselect.h"
 #include "optimizer/var.h"
 #include "parser/parse_clause.h"
 #include "parser/parsetree.h"
@@ -677,6 +678,12 @@ has_multiple_baserels(PlannerInfo *root)
 /*
  * set_subquery_pathlist
  *		Build the (single) access path for a subquery RTE
+ *
+ * Although we build only one access path for the subquery,
+ * join search process may find another path by pushing down
+ * the nestloop param into subquery. The finding process will
+ * be done much later than here, but some common operation like
+ * preprocessing subquery are shared.
  */
 static void
 set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
@@ -687,7 +694,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
 	bool	   *differentTypes;
 	double		tuple_fraction;
 	PlannerInfo *subroot;
-	List	   *pathkeys;
+	Plan	    *subplan;
+	List	    *pathkeys;
 
 	/*
 	 * Must copy the Query so that planning doesn't mess up the RTE contents
@@ -749,6 +757,9 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
 
 	pfree(differentTypes);
 
+	/* save it for later use in best_inner_subqueryscan */
+	rel->preprocessed_subquery = copyObject(subquery);
+
 	/*
 	 * We can safely pass the outer tuple_fraction down to the subquery if the
 	 * outer level has no joining, aggregation, or sorting to do. Otherwise
@@ -766,27 +777,181 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
 		tuple_fraction = root->tuple_fraction;
 
 	/* Generate the plan for the subquery */
-	rel->subplan = subquery_planner(root->glob, subquery,
-									root,
-									false, tuple_fraction,
-									&subroot);
-	rel->subrtable = subroot->parse->rtable;
-	rel->subrowmark = subroot->rowMarks;
+	subplan = subquery_planner(root->glob, subquery,
+							   root,
+							   false, tuple_fraction,
+							   &subroot);
 
 	/* Mark rel with estimated output rows, width, etc */
-	set_subquery_size_estimates(root, rel, subroot);
+	set_subquery_size_estimates(root, rel, subroot, subplan);
 
 	/* Convert subquery pathkeys to outer representation */
-	pathkeys = convert_subquery_pathkeys(root, rel, subroot->query_pathkeys);
+	pathkeys = convert_subquery_pathkeys(root, rel,
+										 subroot->query_pathkeys, subplan);
 
 	/* Generate appropriate path */
-	add_path(rel, create_subqueryscan_path(rel, pathkeys));
+	add_path(rel, create_subqueryscan_path(rel, pathkeys,
+			 subplan, subroot->parse->rtable, subroot->rowMarks));
 
 	/* Select cheapest path (pretty easy in this case...) */
 	set_cheapest(rel);
 }
 
 /*
+ * best_inner_subqueryscan
+ *
+ * As the inner scan relation, try to find another subquer path which may
+ * be better by pushing down some join clauses. If such possibility is found,
+ * return the path in *cheapest_start_path or *cheapest_total_path.
+ * Currently both paths will be resulted in the same, whereas further
+ * improvement might find cases they'd be different.
+ *
+ * It re-uses modified subquery (a.k.a. rel->preprocessed_subquery.) The basic
+ * push-down of baserestrictinfo was common among base path (done in
+ * set_subquery_pathlist()) so this routine doesn't care it. But once we find
+ * push-down-able join clause, the parsed subquery will be modified after copied.
+ */
+void
+best_inner_subqueryscan(PlannerInfo *root, RelOptInfo *rel,
+						RelOptInfo *outer_rel, List *join_clause,
+						Path **cheapest_start_path, Path **cheapest_total_path)
+{
+	Query	   *parse = root->parse;
+	Query	   *subquery;
+	Index		rti = rel->relid;
+	RangeTblEntry *rte = planner_rt_fetch(rti, root);
+	bool	   *differentTypes;
+	double		tuple_fraction;
+	PlannerInfo *subroot;
+	Plan	   *subplan;
+	List	   *pathkeys;
+	Path	   *path;
+	bool		query_modified = false;
+
+	/* initialize paths to return immediately anytime */
+	*cheapest_start_path = *cheapest_total_path = NULL;
+
+	/*
+	 * empty join condition doesn't make sense here
+	 */
+	if (join_clause == NIL)
+		return;
+
+	/* copying the whole Query is expensive; let's copy-on-write */
+	subquery = rel->preprocessed_subquery;
+
+	/* We need a workspace for keeping track of set-op type coercions */
+	differentTypes = (bool *)
+		palloc0((list_length(subquery->targetList) + 1) * sizeof(bool));
+
+	/*
+	 * Currently we focus on only cases of aggregate subquery.
+	 * I'm not quite sure if there are other useful cases.
+	 */
+	if (subquery->hasAggs &&
+		subquery_is_pushdown_safe(subquery, subquery, differentTypes))
+	{
+		ListCell   *l;
+
+		foreach (l, join_clause)
+		{
+			RestrictInfo   *rinfo = (RestrictInfo *) lfirst(l);
+			Node		   *lexpr, *rexpr;
+			Expr		   *clause = NULL;
+			Param		   *param;
+			OpExpr		   *rclause;
+
+			/*
+			 * For the first step, only Var = Var join clause is targeted.
+			 * This may be improved in the future.
+			 */
+			if (!is_opclause(rinfo->clause))
+				continue;
+			rclause = (OpExpr *) rinfo->clause;
+			lexpr = get_leftop((Expr *) rclause);
+			rexpr = get_rightop((Expr *) rclause);
+			if (!(IsA(lexpr, Var) && IsA(rexpr, Var)))
+				continue;
+
+			/* see which arg is mine and outer's */
+			if (bms_is_member(((Var *) lexpr)->varno, outer_rel->relids))
+			{
+				param = assign_nestloop_param(root, (Var *) lexpr);
+				clause = make_opclause(rclause->opno,
+									   rclause->opresulttype,
+									   false,
+									   (Expr *) param,
+									   (Expr *) copyObject(rexpr),
+									   InvalidOid,
+									   rclause->opcollid);
+			}
+			else if (bms_is_member(((Var *) rexpr)->varno, outer_rel->relids))
+			{
+				param = assign_nestloop_param(root, (Var *) rexpr);
+				clause = make_opclause(rclause->opno,
+									   rclause->opresulttype,
+									   false,
+									   (Expr *) copyObject(lexpr),
+									   (Expr *) param,
+									   InvalidOid,
+									   rclause->opcollid);
+			}
+
+			/* TODO:no check isouter or not? */
+			if (clause &&
+				qual_is_pushdown_safe(subquery, rti, (Node *) clause,
+									  differentTypes))
+			{
+				/* copy on write */
+				if (rel->preprocessed_subquery == subquery)
+					subquery = copyObject(subquery);
+				subquery_push_qual(subquery, rte, rti, (Node *) clause);
+				query_modified = true;
+			}
+		}
+	}
+
+	pfree(differentTypes);
+
+	/* return immediately if such subquery isn't found. Use original one */
+	if (!query_modified)
+		return;
+
+	/*
+	 * We can safely pass the outer tuple_fraction down to the subquery if the
+	 * outer level has no joining, aggregation, or sorting to do. Otherwise
+	 * we'd better tell the subquery to plan for full retrieval. (XXX This
+	 * could probably be made more intelligent ...)
+	 */
+	if (parse->hasAggs ||
+		parse->groupClause ||
+		parse->havingQual ||
+		parse->distinctClause ||
+		parse->sortClause ||
+		has_multiple_baserels(root))
+		tuple_fraction = 0.0;	/* default case */
+	else
+		tuple_fraction = root->tuple_fraction;
+
+	/* Generate the plan for the subquery */
+	subplan = subquery_planner(root->glob, subquery,
+							   root,
+							   false, tuple_fraction,
+							   &subroot);
+
+	/* Convert subquery pathkeys to outer representation */
+	pathkeys = convert_subquery_pathkeys(root, rel,
+										 subroot->query_pathkeys, subplan);
+
+	/* Generate appropriate path */
+	path = create_subqueryscan_path(rel, pathkeys,
+			 subplan, subroot->parse->rtable, subroot->rowMarks);
+
+	/* return the same path for now */
+	*cheapest_start_path = *cheapest_total_path = path;
+}
+
+/*
  * set_function_pathlist
  *		Build the (single) access path for a function RTE
  */
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index bb38768..b7271c3 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -892,7 +892,7 @@ cost_tidscan(Path *path, PlannerInfo *root,
  *	  Determines and returns the cost of scanning a subquery RTE.
  */
 void
-cost_subqueryscan(Path *path, RelOptInfo *baserel)
+cost_subqueryscan(SubqueryPath *path, RelOptInfo *baserel)
 {
 	Cost		startup_cost;
 	Cost		run_cost;
@@ -907,15 +907,15 @@ cost_subqueryscan(Path *path, RelOptInfo *baserel)
 	 * any restriction clauses that will be attached to the SubqueryScan node,
 	 * plus cpu_tuple_cost to account for selection and projection overhead.
 	 */
-	path->startup_cost = baserel->subplan->startup_cost;
-	path->total_cost = baserel->subplan->total_cost;
+	path->path.startup_cost = path->subplan->startup_cost;
+	path->path.total_cost = path->subplan->total_cost;
 
 	startup_cost = baserel->baserestrictcost.startup;
 	cpu_per_tuple = cpu_tuple_cost + baserel->baserestrictcost.per_tuple;
 	run_cost = cpu_per_tuple * baserel->tuples;
 
-	path->startup_cost += startup_cost;
-	path->total_cost += startup_cost + run_cost;
+	path->path.startup_cost += startup_cost;
+	path->path.total_cost += startup_cost + run_cost;
 }
 
 /*
@@ -3222,7 +3222,7 @@ set_joinrel_size_estimates(PlannerInfo *root, RelOptInfo *rel,
  */
 void
 set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
-							PlannerInfo *subroot)
+							PlannerInfo *subroot, Plan *subplan)
 {
 	RangeTblEntry *rte;
 	ListCell   *lc;
@@ -3233,7 +3233,7 @@ set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
 	Assert(rte->rtekind == RTE_SUBQUERY);
 
 	/* Copy raw number of output rows from subplan */
-	rel->tuples = rel->subplan->plan_rows;
+	rel->tuples = subplan->plan_rows;
 
 	/*
 	 * Compute per-output-column width estimates by examining the subquery's
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index 7d3cf42..c79fd43 100644
--- a/src/backend/optimizer/path/joinpath.c
+++ b/src/backend/optimizer/path/joinpath.c
@@ -329,7 +329,9 @@ sort_inner_and_outer(PlannerInfo *root,
  * inner path, one on the same with materialization, one on the
  * cheapest-startup-cost inner path (if different), one on the
  * cheapest-total inner-indexscan path (if any), and one on the
- * cheapest-startup inner-indexscan path (if different).
+ * cheapest-startup inner-indexscan path (if different) or on the
+ * cheapest-total inner-subqueryscan path (if any).
+ * Note inner-indexpaths and inner-subquerypaths are not made at the same time.
  *
  * We also consider mergejoins if mergejoin clauses are available.	We have
  * two ways to generate the inner path for a mergejoin: sort the cheapest
@@ -369,6 +371,8 @@ match_unsorted_outer(PlannerInfo *root,
 	Path	   *matpath = NULL;
 	Path	   *index_cheapest_startup = NULL;
 	Path	   *index_cheapest_total = NULL;
+	Path	   *sub_cheapest_startup = NULL;
+	Path	   *sub_cheapest_total = NULL;
 	ListCell   *l;
 
 	/*
@@ -430,8 +434,8 @@ match_unsorted_outer(PlannerInfo *root,
 				create_material_path(innerrel, inner_cheapest_total);
 
 		/*
-		 * Get the best innerjoin indexpaths (if any) for this outer rel.
-		 * They're the same for all outer paths.
+		 * Get the best innerjoin indexpaths or subquerypaths (if any)
+		 * for this outer rel. They're the same for all outer paths.
 		 */
 		if (innerrel->reloptkind != RELOPT_JOINREL)
 		{
@@ -444,6 +448,10 @@ match_unsorted_outer(PlannerInfo *root,
 				best_inner_indexscan(root, innerrel, outerrel, jointype,
 									 &index_cheapest_startup,
 									 &index_cheapest_total);
+			else if (innerrel->rtekind == RTE_SUBQUERY)
+				best_inner_subqueryscan(root, innerrel, outerrel, restrictlist,
+										&sub_cheapest_startup,
+										&sub_cheapest_total);
 		}
 	}
 
@@ -487,7 +495,7 @@ match_unsorted_outer(PlannerInfo *root,
 			 * cheapest-total-cost inner.  When appropriate, also consider
 			 * using the materialized form of the cheapest inner, the
 			 * cheapest-startup-cost inner path, and the cheapest innerjoin
-			 * indexpaths.
+			 * indexpaths or subquerypaths.
 			 */
 			add_path(joinrel, (Path *)
 					 create_nestloop_path(root,
@@ -539,6 +547,16 @@ match_unsorted_outer(PlannerInfo *root,
 											  index_cheapest_startup,
 											  restrictlist,
 											  merge_pathkeys));
+			if (sub_cheapest_total != NULL)
+				add_path(joinrel, (Path *)
+						 create_nestloop_path(root,
+											  joinrel,
+											  jointype,
+											  sjinfo,
+											  outerpath,
+											  sub_cheapest_startup,
+											  restrictlist,
+											  merge_pathkeys));
 		}
 
 		/* Can't do anything else if outer path needs to be unique'd */
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 24e4e59..c3f7ca0 100644
--- a/src/backend/optimizer/path/joinrels.c
+++ b/src/backend/optimizer/path/joinrels.c
@@ -745,7 +745,6 @@ make_join_rel(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2)
 	return joinrel;
 }
 
-
 /*
  * have_join_order_restriction
  *		Detect whether the two relations should be joined to satisfy
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index e5228a8..cee5c71 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -623,12 +623,12 @@ find_indexkey_var(PlannerInfo *root, RelOptInfo *rel, AttrNumber varattno)
  */
 List *
 convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel,
-						  List *subquery_pathkeys)
+						  List *subquery_pathkeys, Plan *subplan)
 {
 	List	   *retval = NIL;
 	int			retvallen = 0;
 	int			outer_query_keys = list_length(root->query_pathkeys);
-	List	   *sub_tlist = rel->subplan->targetlist;
+	List	   *sub_tlist = subplan->targetlist;
 	ListCell   *i;
 
 	foreach(i, subquery_pathkeys)
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index a3a82ec..f0c1a6c 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -62,8 +62,8 @@ static Plan *create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual,
 					  List **qual, List **indexqual);
 static TidScan *create_tidscan_plan(PlannerInfo *root, TidPath *best_path,
 					List *tlist, List *scan_clauses);
-static SubqueryScan *create_subqueryscan_plan(PlannerInfo *root, Path *best_path,
-						 List *tlist, List *scan_clauses);
+static SubqueryScan *create_subqueryscan_plan(PlannerInfo *root,
+					SubqueryPath *best_path, List *tlist, List *scan_clauses);
 static FunctionScan *create_functionscan_plan(PlannerInfo *root, Path *best_path,
 						 List *tlist, List *scan_clauses);
 static ValuesScan *create_valuesscan_plan(PlannerInfo *root, Path *best_path,
@@ -321,7 +321,7 @@ create_scan_plan(PlannerInfo *root, Path *best_path)
 
 		case T_SubqueryScan:
 			plan = (Plan *) create_subqueryscan_plan(root,
-													 best_path,
+													 (SubqueryPath *) best_path,
 													 tlist,
 													 scan_clauses);
 			break;
@@ -1538,15 +1538,15 @@ create_tidscan_plan(PlannerInfo *root, TidPath *best_path,
  *	 with restriction clauses 'scan_clauses' and targetlist 'tlist'.
  */
 static SubqueryScan *
-create_subqueryscan_plan(PlannerInfo *root, Path *best_path,
+create_subqueryscan_plan(PlannerInfo *root, SubqueryPath *best_path,
 						 List *tlist, List *scan_clauses)
 {
 	SubqueryScan *scan_plan;
-	Index		scan_relid = best_path->parent->relid;
+	Index		scan_relid = best_path->path.parent->relid;
 
 	/* it should be a subquery base rel... */
 	Assert(scan_relid > 0);
-	Assert(best_path->parent->rtekind == RTE_SUBQUERY);
+	Assert(best_path->path.parent->rtekind == RTE_SUBQUERY);
 
 	/* Sort clauses into best execution order */
 	scan_clauses = order_qual_clauses(root, scan_clauses);
@@ -1557,11 +1557,63 @@ create_subqueryscan_plan(PlannerInfo *root, Path *best_path,
 	scan_plan = make_subqueryscan(tlist,
 								  scan_clauses,
 								  scan_relid,
-								  best_path->parent->subplan,
-								  best_path->parent->subrtable,
-								  best_path->parent->subrowmark);
+								  best_path->subplan,
+								  best_path->subrtable,
+								  best_path->subrowmark);
 
-	copy_path_costsize(&scan_plan->scan.plan, best_path);
+	copy_path_costsize(&scan_plan->scan.plan, &best_path->path);
+
+	/*
+	 * If this plan is inner of nestloop and push-down-qual case,
+	 * params are extracted and registered for the upper nestloop.
+	 * As Plan has only bitmap extParam field, we cannot take them
+	 * directly; instead take them from PlannerGlobal.
+	 */
+	if (scan_plan->subplan->extParam)
+	{
+		int		i;
+		ListCell   *ppl;
+		PlannerParamItem   *pitem;
+
+		i = 0;
+		foreach(ppl, root->glob->paramlist)
+		{
+			pitem = (PlannerParamItem *) lfirst(ppl);
+			/* take the ones only in this query level */
+			if (pitem->abslevel == root->query_level &&
+				IsA(pitem->item, Var) &&
+				((Var *) pitem->item)->varlevelsup == 0)
+			{
+				Var	   *var = (Var *) pitem->item;
+				NestLoopParam	   *nlp;
+				ListCell   *lc;
+				bool		found = false;
+
+				/* Is the param already listed in root->curOuterParams? */
+				foreach(lc, root->curOuterParams)
+				{
+					nlp = (NestLoopParam *) lfirst(lc);
+					if (equal(nlp->paramval, var))
+					{
+						/* Present, so we can just skip */
+						found = true;
+						break;
+					}
+				}
+				if (found)
+				{
+					i++;
+					continue;
+				}
+				/* No, so add it */
+				nlp = makeNode(NestLoopParam);
+				nlp->paramno = i;
+				nlp->paramval = var;
+				root->curOuterParams = lappend(root->curOuterParams, nlp);
+			}
+			i++;
+		}
+	}
 
 	return scan_plan;
 }
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 161d5ab..2e711d8 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -1331,17 +1331,21 @@ distinct_col_search(int colno, List *colnos, List *opids)
  *	  returning the pathnode.
  */
 Path *
-create_subqueryscan_path(RelOptInfo *rel, List *pathkeys)
+create_subqueryscan_path(RelOptInfo *rel, List *pathkeys, Plan *subplan,
+						 List *subrtable, List *subrowmark)
 {
-	Path	   *pathnode = makeNode(Path);
+	SubqueryPath   *pathnode = makeNode(SubqueryPath);
 
-	pathnode->pathtype = T_SubqueryScan;
-	pathnode->parent = rel;
-	pathnode->pathkeys = pathkeys;
+	pathnode->path.pathtype = T_SubqueryScan;
+	pathnode->path.parent = rel;
+	pathnode->path.pathkeys = pathkeys;
+	pathnode->subplan = subplan;
+	pathnode->subrtable = subrtable;
+	pathnode->subrowmark = subrowmark;
 
 	cost_subqueryscan(pathnode, rel);
 
-	return pathnode;
+	return (Path *) pathnode;
 }
 
 /*
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index b7a5845..ab2338f 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -82,9 +82,7 @@ build_simple_rel(PlannerInfo *root, int relid, RelOptKind reloptkind)
 	rel->indexlist = NIL;
 	rel->pages = 0;
 	rel->tuples = 0;
-	rel->subplan = NULL;
-	rel->subrtable = NIL;
-	rel->subrowmark = NIL;
+	rel->preprocessed_subquery = NULL;
 	rel->baserestrictinfo = NIL;
 	rel->baserestrictcost.startup = 0;
 	rel->baserestrictcost.per_tuple = 0;
@@ -336,9 +334,7 @@ build_join_rel(PlannerInfo *root,
 	joinrel->indexlist = NIL;
 	joinrel->pages = 0;
 	joinrel->tuples = 0;
-	joinrel->subplan = NULL;
-	joinrel->subrtable = NIL;
-	joinrel->subrowmark = NIL;
+	joinrel->preprocessed_subquery = NULL;
 	joinrel->baserestrictinfo = NIL;
 	joinrel->baserestrictcost.startup = 0;
 	joinrel->baserestrictcost.per_tuple = 0;
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index d8bc6b8..e951fd1 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -220,6 +220,7 @@ typedef enum NodeTag
 	T_MergePath,
 	T_HashPath,
 	T_TidPath,
+	T_SubqueryPath,
 	T_ForeignPath,
 	T_AppendPath,
 	T_MergeAppendPath,
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index f659269..98e2744 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -322,13 +322,6 @@ typedef struct PlannerInfo
  *					(always NIL if it's not a table)
  *		pages - number of disk pages in relation (zero if not a table)
  *		tuples - number of tuples in relation (not considering restrictions)
- *		subplan - plan for subquery (NULL if it's not a subquery)
- *		subrtable - rangetable for subquery (NIL if it's not a subquery)
- *		subrowmark - rowmarks for subquery (NIL if it's not a subquery)
- *
- *		Note: for a subquery, tuples and subplan are not set immediately
- *		upon creation of the RelOptInfo object; they are filled in when
- *		set_base_rel_pathlist processes the object.
  *
  *		For otherrels that are appendrel members, these fields are filled
  *		in just as for a baserel.
@@ -408,9 +401,8 @@ typedef struct RelOptInfo
 	List	   *indexlist;		/* list of IndexOptInfo */
 	BlockNumber pages;
 	double		tuples;
-	struct Plan *subplan;		/* if subquery */
-	List	   *subrtable;		/* if subquery */
-	List	   *subrowmark;		/* if subquery */
+
+	Query	   *preprocessed_subquery; /* if subquery */
 
 	/* used by various scans and joins: */
 	List	   *baserestrictinfo;		/* RestrictInfo structures (if base
@@ -770,6 +762,23 @@ typedef struct TidPath
 } TidPath;
 
 /*
+ * SubqueryPath represents a scan of a subquery scan
+ *
+ * The struct holds subplan, subrtable and subrowmark, which are
+ * the temporary spaces to transport to the final stage of planner.
+ * Note each SubqueryPath may have different plan, depending on
+ * the pushed down qual clauses (as parameterized inner subquery).
+ */
+typedef struct SubqueryPath
+{
+	Path		path;
+	struct Plan *subplan;
+	List	   *subrtable;
+	List	   *subrowmark;
+	double		rows;
+} SubqueryPath;
+
+/*
  * ForeignPath represents a scan of a foreign table
  */
 typedef struct ForeignPath
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 2763863..7ec2c7b 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -75,7 +75,7 @@ extern void cost_bitmap_or_node(BitmapOrPath *path, PlannerInfo *root);
 extern void cost_bitmap_tree_node(Path *path, Cost *cost, Selectivity *selec);
 extern void cost_tidscan(Path *path, PlannerInfo *root,
 			 RelOptInfo *baserel, List *tidquals);
-extern void cost_subqueryscan(Path *path, RelOptInfo *baserel);
+extern void cost_subqueryscan(SubqueryPath *path, RelOptInfo *baserel);
 extern void cost_functionscan(Path *path, PlannerInfo *root,
 				  RelOptInfo *baserel);
 extern void cost_valuesscan(Path *path, PlannerInfo *root,
@@ -122,7 +122,7 @@ extern void set_joinrel_size_estimates(PlannerInfo *root, RelOptInfo *rel,
 						   SpecialJoinInfo *sjinfo,
 						   List *restrictlist);
 extern void set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
-							PlannerInfo *subroot);
+							PlannerInfo *subroot, Plan *subplan);
 extern void set_function_size_estimates(PlannerInfo *root, RelOptInfo *rel);
 extern void set_values_size_estimates(PlannerInfo *root, RelOptInfo *rel);
 extern void set_cte_size_estimates(PlannerInfo *root, RelOptInfo *rel,
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 1da2131..6de2816 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -14,6 +14,7 @@
 #ifndef PATHNODE_H
 #define PATHNODE_H
 
+#include "nodes/plannodes.h"
 #include "nodes/relation.h"
 
 
@@ -56,7 +57,8 @@ extern ResultPath *create_result_path(List *quals);
 extern MaterialPath *create_material_path(RelOptInfo *rel, Path *subpath);
 extern UniquePath *create_unique_path(PlannerInfo *root, RelOptInfo *rel,
 				   Path *subpath, SpecialJoinInfo *sjinfo);
-extern Path *create_subqueryscan_path(RelOptInfo *rel, List *pathkeys);
+extern Path *create_subqueryscan_path(RelOptInfo *rel, List *pathkeys,
+					Plan *subplan, List *subrtable, List *subrowmark);
 extern Path *create_functionscan_path(PlannerInfo *root, RelOptInfo *rel);
 extern Path *create_valuesscan_path(PlannerInfo *root, RelOptInfo *rel);
 extern Path *create_ctescan_path(PlannerInfo *root, RelOptInfo *rel);
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 7f1353a..54d0b01 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -33,6 +33,9 @@ extern PGDLLIMPORT join_search_hook_type join_search_hook;
 extern RelOptInfo *make_one_rel(PlannerInfo *root, List *joinlist);
 extern RelOptInfo *standard_join_search(PlannerInfo *root, int levels_needed,
 					 List *initial_rels);
+extern void best_inner_subqueryscan(PlannerInfo *root, RelOptInfo *rel,
+						RelOptInfo *outer_rel, List *join_clause,
+						Path **cheapest_start_path, Path **cheapest_total_path);
 
 #ifdef OPTIMIZER_DEBUG
 extern void debug_print_rel(PlannerInfo *root, RelOptInfo *rel);
@@ -154,7 +157,7 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
 extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
 					 ScanDirection scandir);
 extern List *convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel,
-						  List *subquery_pathkeys);
+						  List *subquery_pathkeys, Plan *subplan);
 extern List *build_join_pathkeys(PlannerInfo *root,
 					RelOptInfo *joinrel,
 					JoinType jointype,
#18Yeb Havinga
yebhavinga@gmail.com
In reply to: Hitoshi Harada (#17)
1 attachment(s)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

On 2011-07-22 16:17, Hitoshi Harada wrote:

:(
I updated the patch. Could you try attached once more? The "issafe"
switch seems wrong.

Works like a charm :-). However, now there is always a copyObject of a
subquery even when the subquery is not safe for qual pushdown. The
problem with the previous issafe was that it was only assigned for
rel->baserestrictinfo != NIL. If it is assigned before the if statement,
it still works. See attached patch that avoids subquery copy for unsafe
subqueries, and also exits best_inner_subqueryscan before palloc of
differenttypes in case of unsafe queries.

regards,
--

Yeb Havinga
http://www.mgrid.net/
Mastering Medical Data

Attachments:

aggjoin-v2.patchtext/x-patch; name=aggjoin-v2.patchDownload
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
new file mode 100644
index b5be09a..6596a1c
*** a/src/backend/nodes/outfuncs.c
--- b/src/backend/nodes/outfuncs.c
*************** _outTidPath(StringInfo str, TidPath *nod
*** 1557,1562 ****
--- 1557,1573 ----
  }
  
  static void
+ _outSubqueryPath(StringInfo str, SubqueryPath *node)
+ {
+ 	WRITE_NODE_TYPE("SUBQUERYPATH");
+ 
+ 	_outPathInfo(str, (Path *) node);
+ 	WRITE_NODE_FIELD(subplan);
+ 	WRITE_NODE_FIELD(subrtable);
+ 	WRITE_NODE_FIELD(subrowmark);
+ }
+ 
+ static void
  _outForeignPath(StringInfo str, ForeignPath *node)
  {
  	WRITE_NODE_TYPE("FOREIGNPATH");
*************** _outRelOptInfo(StringInfo str, RelOptInf
*** 1738,1746 ****
  	WRITE_NODE_FIELD(indexlist);
  	WRITE_UINT_FIELD(pages);
  	WRITE_FLOAT_FIELD(tuples, "%.0f");
- 	WRITE_NODE_FIELD(subplan);
- 	WRITE_NODE_FIELD(subrtable);
- 	WRITE_NODE_FIELD(subrowmark);
  	WRITE_NODE_FIELD(baserestrictinfo);
  	WRITE_NODE_FIELD(joininfo);
  	WRITE_BOOL_FIELD(has_eclass_joins);
--- 1749,1754 ----
*************** _outNode(StringInfo str, void *obj)
*** 2949,2954 ****
--- 2957,2965 ----
  			case T_TidPath:
  				_outTidPath(str, obj);
  				break;
+ 			case T_SubqueryPath:
+ 				_outSubqueryPath(str, obj);
+ 				break;
  			case T_ForeignPath:
  				_outForeignPath(str, obj);
  				break;
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
new file mode 100644
index 6b43aee..bd3a53f
*** a/src/backend/optimizer/path/allpaths.c
--- b/src/backend/optimizer/path/allpaths.c
***************
*** 31,36 ****
--- 31,37 ----
  #include "optimizer/planner.h"
  #include "optimizer/prep.h"
  #include "optimizer/restrictinfo.h"
+ #include "optimizer/subselect.h"
  #include "optimizer/var.h"
  #include "parser/parse_clause.h"
  #include "parser/parsetree.h"
*************** has_multiple_baserels(PlannerInfo *root)
*** 677,682 ****
--- 678,689 ----
  /*
   * set_subquery_pathlist
   *		Build the (single) access path for a subquery RTE
+  *
+  * Although we build only one access path for the subquery,
+  * join search process may find another path by pushing down
+  * the nestloop param into subquery. The finding process will
+  * be done much later than here, but some common operation like
+  * preprocessing subquery are shared.
   */
  static void
  set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel,
*************** set_subquery_pathlist(PlannerInfo *root,
*** 687,693 ****
  	bool	   *differentTypes;
  	double		tuple_fraction;
  	PlannerInfo *subroot;
! 	List	   *pathkeys;
  
  	/*
  	 * Must copy the Query so that planning doesn't mess up the RTE contents
--- 694,702 ----
  	bool	   *differentTypes;
  	double		tuple_fraction;
  	PlannerInfo *subroot;
! 	Plan	    *subplan;
! 	List	    *pathkeys;
! 	bool issafe;
  
  	/*
  	 * Must copy the Query so that planning doesn't mess up the RTE contents
*************** set_subquery_pathlist(PlannerInfo *root,
*** 720,727 ****
  	 * XXX Are there any cases where we want to make a policy decision not to
  	 * push down a pushable qual, because it'd result in a worse plan?
  	 */
! 	if (rel->baserestrictinfo != NIL &&
! 		subquery_is_pushdown_safe(subquery, subquery, differentTypes))
  	{
  		/* OK to consider pushing down individual quals */
  		List	   *upperrestrictlist = NIL;
--- 729,738 ----
  	 * XXX Are there any cases where we want to make a policy decision not to
  	 * push down a pushable qual, because it'd result in a worse plan?
  	 */
! 
! 	issafe = subquery_is_pushdown_safe(subquery, subquery, differentTypes);
! 
! 	if (rel->baserestrictinfo != NIL && issafe)
  	{
  		/* OK to consider pushing down individual quals */
  		List	   *upperrestrictlist = NIL;
*************** set_subquery_pathlist(PlannerInfo *root,
*** 749,754 ****
--- 760,769 ----
  
  	pfree(differentTypes);
  
+ 	/* save it for later use in best_inner_subqueryscan */
+ 	if (issafe)
+ 		rel->preprocessed_subquery = copyObject(subquery);
+ 
  	/*
  	 * We can safely pass the outer tuple_fraction down to the subquery if the
  	 * outer level has no joining, aggregation, or sorting to do. Otherwise
*************** set_subquery_pathlist(PlannerInfo *root,
*** 766,792 ****
  		tuple_fraction = root->tuple_fraction;
  
  	/* Generate the plan for the subquery */
! 	rel->subplan = subquery_planner(root->glob, subquery,
! 									root,
! 									false, tuple_fraction,
! 									&subroot);
! 	rel->subrtable = subroot->parse->rtable;
! 	rel->subrowmark = subroot->rowMarks;
  
  	/* Mark rel with estimated output rows, width, etc */
! 	set_subquery_size_estimates(root, rel, subroot);
  
  	/* Convert subquery pathkeys to outer representation */
! 	pathkeys = convert_subquery_pathkeys(root, rel, subroot->query_pathkeys);
  
  	/* Generate appropriate path */
! 	add_path(rel, create_subqueryscan_path(rel, pathkeys));
  
  	/* Select cheapest path (pretty easy in this case...) */
  	set_cheapest(rel);
  }
  
  /*
   * set_function_pathlist
   *		Build the (single) access path for a function RTE
   */
--- 781,959 ----
  		tuple_fraction = root->tuple_fraction;
  
  	/* Generate the plan for the subquery */
! 	subplan = subquery_planner(root->glob, subquery,
! 							   root,
! 							   false, tuple_fraction,
! 							   &subroot);
  
  	/* Mark rel with estimated output rows, width, etc */
! 	set_subquery_size_estimates(root, rel, subroot, subplan);
  
  	/* Convert subquery pathkeys to outer representation */
! 	pathkeys = convert_subquery_pathkeys(root, rel,
! 										 subroot->query_pathkeys, subplan);
  
  	/* Generate appropriate path */
! 	add_path(rel, create_subqueryscan_path(rel, pathkeys,
! 			 subplan, subroot->parse->rtable, subroot->rowMarks));
  
  	/* Select cheapest path (pretty easy in this case...) */
  	set_cheapest(rel);
  }
  
  /*
+  * best_inner_subqueryscan
+  *
+  * As the inner scan relation, try to find another subquer path which may
+  * be better by pushing down some join clauses. If such possibility is found,
+  * return the path in *cheapest_start_path or *cheapest_total_path.
+  * Currently both paths will be resulted in the same, whereas further
+  * improvement might find cases they'd be different.
+  *
+  * It re-uses modified subquery (a.k.a. rel->preprocessed_subquery.) The basic
+  * push-down of baserestrictinfo was common among base path (done in
+  * set_subquery_pathlist()) so this routine doesn't care it. But once we find
+  * push-down-able join clause, the parsed subquery will be modified after copied.
+  */
+ void
+ best_inner_subqueryscan(PlannerInfo *root, RelOptInfo *rel,
+ 						RelOptInfo *outer_rel, List *join_clause,
+ 						Path **cheapest_start_path, Path **cheapest_total_path)
+ {
+ 	Query	   *parse = root->parse;
+ 	Query	   *subquery;
+ 	Index		rti = rel->relid;
+ 	RangeTblEntry *rte = planner_rt_fetch(rti, root);
+ 	bool	   *differentTypes;
+ 	double		tuple_fraction;
+ 	PlannerInfo *subroot;
+ 	Plan	   *subplan;
+ 	List	   *pathkeys;
+ 	Path	   *path;
+ 	bool		query_modified = false;
+ 
+ 	/* initialize paths to return immediately anytime */
+ 	*cheapest_start_path = *cheapest_total_path = NULL;
+ 
+ 	/* empty join condition doesn't make sense here */
+ 	if (join_clause == NIL)
+ 		return;
+ 
+ 	/* rel->preprocessed_subquery was only saved for pushdown save subqueries */
+ 	if ((subquery = rel->preprocessed_subquery) == NULL)
+ 		return;
+ 
+ 	/* We need a workspace for keeping track of set-op type coercions */
+ 	differentTypes = (bool *)
+ 		palloc0((list_length(subquery->targetList) + 1) * sizeof(bool));
+ 
+ 	/*
+ 	 * Currently we focus on only cases of aggregate subquery.
+ 	 * I'm not quite sure if there are other useful cases.
+ 	 */
+ 	if (subquery->hasAggs)
+ 	{
+ 		ListCell   *l;
+ 
+ 		foreach (l, join_clause)
+ 		{
+ 			RestrictInfo   *rinfo = (RestrictInfo *) lfirst(l);
+ 			Node		   *lexpr, *rexpr;
+ 			Expr		   *clause = NULL;
+ 			Param		   *param;
+ 			OpExpr		   *rclause;
+ 
+ 			/*
+ 			 * For the first step, only Var = Var join clause is targeted.
+ 			 * This may be improved in the future.
+ 			 */
+ 			if (!is_opclause(rinfo->clause))
+ 				continue;
+ 			rclause = (OpExpr *) rinfo->clause;
+ 			lexpr = get_leftop((Expr *) rclause);
+ 			rexpr = get_rightop((Expr *) rclause);
+ 			if (!(IsA(lexpr, Var) && IsA(rexpr, Var)))
+ 				continue;
+ 
+ 			/* see which arg is mine and outer's */
+ 			if (bms_is_member(((Var *) lexpr)->varno, outer_rel->relids))
+ 			{
+ 				param = assign_nestloop_param(root, (Var *) lexpr);
+ 				clause = make_opclause(rclause->opno,
+ 									   rclause->opresulttype,
+ 									   false,
+ 									   (Expr *) param,
+ 									   (Expr *) copyObject(rexpr),
+ 									   InvalidOid,
+ 									   rclause->opcollid);
+ 			}
+ 			else if (bms_is_member(((Var *) rexpr)->varno, outer_rel->relids))
+ 			{
+ 				param = assign_nestloop_param(root, (Var *) rexpr);
+ 				clause = make_opclause(rclause->opno,
+ 									   rclause->opresulttype,
+ 									   false,
+ 									   (Expr *) copyObject(lexpr),
+ 									   (Expr *) param,
+ 									   InvalidOid,
+ 									   rclause->opcollid);
+ 			}
+ 
+ 			/* TODO:no check isouter or not? */
+ 			if (clause &&
+ 				qual_is_pushdown_safe(subquery, rti, (Node *) clause,
+ 									  differentTypes))
+ 			{
+ 				/* copying the whole Query is expensive; let's copy-on-write */
+ 				if (rel->preprocessed_subquery == subquery)
+ 					subquery = copyObject(subquery);
+ 				subquery_push_qual(subquery, rte, rti, (Node *) clause);
+ 				query_modified = true;
+ 			}
+ 		}
+ 	}
+ 
+ 	pfree(differentTypes);
+ 
+ 	/* return immediately if such subquery isn't found. Use original one */
+ 	if (!query_modified)
+ 		return;
+ 
+ 	/*
+ 	 * We can safely pass the outer tuple_fraction down to the subquery if the
+ 	 * outer level has no joining, aggregation, or sorting to do. Otherwise
+ 	 * we'd better tell the subquery to plan for full retrieval. (XXX This
+ 	 * could probably be made more intelligent ...)
+ 	 */
+ 	if (parse->hasAggs ||
+ 		parse->groupClause ||
+ 		parse->havingQual ||
+ 		parse->distinctClause ||
+ 		parse->sortClause ||
+ 		has_multiple_baserels(root))
+ 		tuple_fraction = 0.0;	/* default case */
+ 	else
+ 		tuple_fraction = root->tuple_fraction;
+ 
+ 	/* Generate the plan for the subquery */
+ 	subplan = subquery_planner(root->glob, subquery,
+ 							   root,
+ 							   false, tuple_fraction,
+ 							   &subroot);
+ 
+ 	/* Convert subquery pathkeys to outer representation */
+ 	pathkeys = convert_subquery_pathkeys(root, rel,
+ 										 subroot->query_pathkeys, subplan);
+ 
+ 	/* Generate appropriate path */
+ 	path = create_subqueryscan_path(rel, pathkeys,
+ 			 subplan, subroot->parse->rtable, subroot->rowMarks);
+ 
+ 	/* return the same path for now */
+ 	*cheapest_start_path = *cheapest_total_path = path;
+ }
+ 
+ /*
   * set_function_pathlist
   *		Build the (single) access path for a function RTE
   */
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
new file mode 100644
index bb38768..b7271c3
*** a/src/backend/optimizer/path/costsize.c
--- b/src/backend/optimizer/path/costsize.c
*************** cost_tidscan(Path *path, PlannerInfo *ro
*** 892,898 ****
   *	  Determines and returns the cost of scanning a subquery RTE.
   */
  void
! cost_subqueryscan(Path *path, RelOptInfo *baserel)
  {
  	Cost		startup_cost;
  	Cost		run_cost;
--- 892,898 ----
   *	  Determines and returns the cost of scanning a subquery RTE.
   */
  void
! cost_subqueryscan(SubqueryPath *path, RelOptInfo *baserel)
  {
  	Cost		startup_cost;
  	Cost		run_cost;
*************** cost_subqueryscan(Path *path, RelOptInfo
*** 907,921 ****
  	 * any restriction clauses that will be attached to the SubqueryScan node,
  	 * plus cpu_tuple_cost to account for selection and projection overhead.
  	 */
! 	path->startup_cost = baserel->subplan->startup_cost;
! 	path->total_cost = baserel->subplan->total_cost;
  
  	startup_cost = baserel->baserestrictcost.startup;
  	cpu_per_tuple = cpu_tuple_cost + baserel->baserestrictcost.per_tuple;
  	run_cost = cpu_per_tuple * baserel->tuples;
  
! 	path->startup_cost += startup_cost;
! 	path->total_cost += startup_cost + run_cost;
  }
  
  /*
--- 907,921 ----
  	 * any restriction clauses that will be attached to the SubqueryScan node,
  	 * plus cpu_tuple_cost to account for selection and projection overhead.
  	 */
! 	path->path.startup_cost = path->subplan->startup_cost;
! 	path->path.total_cost = path->subplan->total_cost;
  
  	startup_cost = baserel->baserestrictcost.startup;
  	cpu_per_tuple = cpu_tuple_cost + baserel->baserestrictcost.per_tuple;
  	run_cost = cpu_per_tuple * baserel->tuples;
  
! 	path->path.startup_cost += startup_cost;
! 	path->path.total_cost += startup_cost + run_cost;
  }
  
  /*
*************** set_joinrel_size_estimates(PlannerInfo *
*** 3222,3228 ****
   */
  void
  set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
! 							PlannerInfo *subroot)
  {
  	RangeTblEntry *rte;
  	ListCell   *lc;
--- 3222,3228 ----
   */
  void
  set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
! 							PlannerInfo *subroot, Plan *subplan)
  {
  	RangeTblEntry *rte;
  	ListCell   *lc;
*************** set_subquery_size_estimates(PlannerInfo 
*** 3233,3239 ****
  	Assert(rte->rtekind == RTE_SUBQUERY);
  
  	/* Copy raw number of output rows from subplan */
! 	rel->tuples = rel->subplan->plan_rows;
  
  	/*
  	 * Compute per-output-column width estimates by examining the subquery's
--- 3233,3239 ----
  	Assert(rte->rtekind == RTE_SUBQUERY);
  
  	/* Copy raw number of output rows from subplan */
! 	rel->tuples = subplan->plan_rows;
  
  	/*
  	 * Compute per-output-column width estimates by examining the subquery's
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
new file mode 100644
index 7d3cf42..c79fd43
*** a/src/backend/optimizer/path/joinpath.c
--- b/src/backend/optimizer/path/joinpath.c
*************** sort_inner_and_outer(PlannerInfo *root,
*** 329,335 ****
   * inner path, one on the same with materialization, one on the
   * cheapest-startup-cost inner path (if different), one on the
   * cheapest-total inner-indexscan path (if any), and one on the
!  * cheapest-startup inner-indexscan path (if different).
   *
   * We also consider mergejoins if mergejoin clauses are available.	We have
   * two ways to generate the inner path for a mergejoin: sort the cheapest
--- 329,337 ----
   * inner path, one on the same with materialization, one on the
   * cheapest-startup-cost inner path (if different), one on the
   * cheapest-total inner-indexscan path (if any), and one on the
!  * cheapest-startup inner-indexscan path (if different) or on the
!  * cheapest-total inner-subqueryscan path (if any).
!  * Note inner-indexpaths and inner-subquerypaths are not made at the same time.
   *
   * We also consider mergejoins if mergejoin clauses are available.	We have
   * two ways to generate the inner path for a mergejoin: sort the cheapest
*************** match_unsorted_outer(PlannerInfo *root,
*** 369,374 ****
--- 371,378 ----
  	Path	   *matpath = NULL;
  	Path	   *index_cheapest_startup = NULL;
  	Path	   *index_cheapest_total = NULL;
+ 	Path	   *sub_cheapest_startup = NULL;
+ 	Path	   *sub_cheapest_total = NULL;
  	ListCell   *l;
  
  	/*
*************** match_unsorted_outer(PlannerInfo *root,
*** 430,437 ****
  				create_material_path(innerrel, inner_cheapest_total);
  
  		/*
! 		 * Get the best innerjoin indexpaths (if any) for this outer rel.
! 		 * They're the same for all outer paths.
  		 */
  		if (innerrel->reloptkind != RELOPT_JOINREL)
  		{
--- 434,441 ----
  				create_material_path(innerrel, inner_cheapest_total);
  
  		/*
! 		 * Get the best innerjoin indexpaths or subquerypaths (if any)
! 		 * for this outer rel. They're the same for all outer paths.
  		 */
  		if (innerrel->reloptkind != RELOPT_JOINREL)
  		{
*************** match_unsorted_outer(PlannerInfo *root,
*** 444,449 ****
--- 448,457 ----
  				best_inner_indexscan(root, innerrel, outerrel, jointype,
  									 &index_cheapest_startup,
  									 &index_cheapest_total);
+ 			else if (innerrel->rtekind == RTE_SUBQUERY)
+ 				best_inner_subqueryscan(root, innerrel, outerrel, restrictlist,
+ 										&sub_cheapest_startup,
+ 										&sub_cheapest_total);
  		}
  	}
  
*************** match_unsorted_outer(PlannerInfo *root,
*** 487,493 ****
  			 * cheapest-total-cost inner.  When appropriate, also consider
  			 * using the materialized form of the cheapest inner, the
  			 * cheapest-startup-cost inner path, and the cheapest innerjoin
! 			 * indexpaths.
  			 */
  			add_path(joinrel, (Path *)
  					 create_nestloop_path(root,
--- 495,501 ----
  			 * cheapest-total-cost inner.  When appropriate, also consider
  			 * using the materialized form of the cheapest inner, the
  			 * cheapest-startup-cost inner path, and the cheapest innerjoin
! 			 * indexpaths or subquerypaths.
  			 */
  			add_path(joinrel, (Path *)
  					 create_nestloop_path(root,
*************** match_unsorted_outer(PlannerInfo *root,
*** 539,544 ****
--- 547,562 ----
  											  index_cheapest_startup,
  											  restrictlist,
  											  merge_pathkeys));
+ 			if (sub_cheapest_total != NULL)
+ 				add_path(joinrel, (Path *)
+ 						 create_nestloop_path(root,
+ 											  joinrel,
+ 											  jointype,
+ 											  sjinfo,
+ 											  outerpath,
+ 											  sub_cheapest_startup,
+ 											  restrictlist,
+ 											  merge_pathkeys));
  		}
  
  		/* Can't do anything else if outer path needs to be unique'd */
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
new file mode 100644
index 24e4e59..c3f7ca0
*** a/src/backend/optimizer/path/joinrels.c
--- b/src/backend/optimizer/path/joinrels.c
*************** make_join_rel(PlannerInfo *root, RelOptI
*** 745,751 ****
  	return joinrel;
  }
  
- 
  /*
   * have_join_order_restriction
   *		Detect whether the two relations should be joined to satisfy
--- 745,750 ----
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
new file mode 100644
index e5228a8..cee5c71
*** a/src/backend/optimizer/path/pathkeys.c
--- b/src/backend/optimizer/path/pathkeys.c
*************** find_indexkey_var(PlannerInfo *root, Rel
*** 623,634 ****
   */
  List *
  convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel,
! 						  List *subquery_pathkeys)
  {
  	List	   *retval = NIL;
  	int			retvallen = 0;
  	int			outer_query_keys = list_length(root->query_pathkeys);
! 	List	   *sub_tlist = rel->subplan->targetlist;
  	ListCell   *i;
  
  	foreach(i, subquery_pathkeys)
--- 623,634 ----
   */
  List *
  convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel,
! 						  List *subquery_pathkeys, Plan *subplan)
  {
  	List	   *retval = NIL;
  	int			retvallen = 0;
  	int			outer_query_keys = list_length(root->query_pathkeys);
! 	List	   *sub_tlist = subplan->targetlist;
  	ListCell   *i;
  
  	foreach(i, subquery_pathkeys)
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
new file mode 100644
index a3a82ec..f0c1a6c
*** a/src/backend/optimizer/plan/createplan.c
--- b/src/backend/optimizer/plan/createplan.c
*************** static Plan *create_bitmap_subplan(Plann
*** 62,69 ****
  					  List **qual, List **indexqual);
  static TidScan *create_tidscan_plan(PlannerInfo *root, TidPath *best_path,
  					List *tlist, List *scan_clauses);
! static SubqueryScan *create_subqueryscan_plan(PlannerInfo *root, Path *best_path,
! 						 List *tlist, List *scan_clauses);
  static FunctionScan *create_functionscan_plan(PlannerInfo *root, Path *best_path,
  						 List *tlist, List *scan_clauses);
  static ValuesScan *create_valuesscan_plan(PlannerInfo *root, Path *best_path,
--- 62,69 ----
  					  List **qual, List **indexqual);
  static TidScan *create_tidscan_plan(PlannerInfo *root, TidPath *best_path,
  					List *tlist, List *scan_clauses);
! static SubqueryScan *create_subqueryscan_plan(PlannerInfo *root,
! 					SubqueryPath *best_path, List *tlist, List *scan_clauses);
  static FunctionScan *create_functionscan_plan(PlannerInfo *root, Path *best_path,
  						 List *tlist, List *scan_clauses);
  static ValuesScan *create_valuesscan_plan(PlannerInfo *root, Path *best_path,
*************** create_scan_plan(PlannerInfo *root, Path
*** 321,327 ****
  
  		case T_SubqueryScan:
  			plan = (Plan *) create_subqueryscan_plan(root,
! 													 best_path,
  													 tlist,
  													 scan_clauses);
  			break;
--- 321,327 ----
  
  		case T_SubqueryScan:
  			plan = (Plan *) create_subqueryscan_plan(root,
! 													 (SubqueryPath *) best_path,
  													 tlist,
  													 scan_clauses);
  			break;
*************** create_tidscan_plan(PlannerInfo *root, T
*** 1538,1552 ****
   *	 with restriction clauses 'scan_clauses' and targetlist 'tlist'.
   */
  static SubqueryScan *
! create_subqueryscan_plan(PlannerInfo *root, Path *best_path,
  						 List *tlist, List *scan_clauses)
  {
  	SubqueryScan *scan_plan;
! 	Index		scan_relid = best_path->parent->relid;
  
  	/* it should be a subquery base rel... */
  	Assert(scan_relid > 0);
! 	Assert(best_path->parent->rtekind == RTE_SUBQUERY);
  
  	/* Sort clauses into best execution order */
  	scan_clauses = order_qual_clauses(root, scan_clauses);
--- 1538,1552 ----
   *	 with restriction clauses 'scan_clauses' and targetlist 'tlist'.
   */
  static SubqueryScan *
! create_subqueryscan_plan(PlannerInfo *root, SubqueryPath *best_path,
  						 List *tlist, List *scan_clauses)
  {
  	SubqueryScan *scan_plan;
! 	Index		scan_relid = best_path->path.parent->relid;
  
  	/* it should be a subquery base rel... */
  	Assert(scan_relid > 0);
! 	Assert(best_path->path.parent->rtekind == RTE_SUBQUERY);
  
  	/* Sort clauses into best execution order */
  	scan_clauses = order_qual_clauses(root, scan_clauses);
*************** create_subqueryscan_plan(PlannerInfo *ro
*** 1557,1567 ****
  	scan_plan = make_subqueryscan(tlist,
  								  scan_clauses,
  								  scan_relid,
! 								  best_path->parent->subplan,
! 								  best_path->parent->subrtable,
! 								  best_path->parent->subrowmark);
  
! 	copy_path_costsize(&scan_plan->scan.plan, best_path);
  
  	return scan_plan;
  }
--- 1557,1619 ----
  	scan_plan = make_subqueryscan(tlist,
  								  scan_clauses,
  								  scan_relid,
! 								  best_path->subplan,
! 								  best_path->subrtable,
! 								  best_path->subrowmark);
  
! 	copy_path_costsize(&scan_plan->scan.plan, &best_path->path);
! 
! 	/*
! 	 * If this plan is inner of nestloop and push-down-qual case,
! 	 * params are extracted and registered for the upper nestloop.
! 	 * As Plan has only bitmap extParam field, we cannot take them
! 	 * directly; instead take them from PlannerGlobal.
! 	 */
! 	if (scan_plan->subplan->extParam)
! 	{
! 		int		i;
! 		ListCell   *ppl;
! 		PlannerParamItem   *pitem;
! 
! 		i = 0;
! 		foreach(ppl, root->glob->paramlist)
! 		{
! 			pitem = (PlannerParamItem *) lfirst(ppl);
! 			/* take the ones only in this query level */
! 			if (pitem->abslevel == root->query_level &&
! 				IsA(pitem->item, Var) &&
! 				((Var *) pitem->item)->varlevelsup == 0)
! 			{
! 				Var	   *var = (Var *) pitem->item;
! 				NestLoopParam	   *nlp;
! 				ListCell   *lc;
! 				bool		found = false;
! 
! 				/* Is the param already listed in root->curOuterParams? */
! 				foreach(lc, root->curOuterParams)
! 				{
! 					nlp = (NestLoopParam *) lfirst(lc);
! 					if (equal(nlp->paramval, var))
! 					{
! 						/* Present, so we can just skip */
! 						found = true;
! 						break;
! 					}
! 				}
! 				if (found)
! 				{
! 					i++;
! 					continue;
! 				}
! 				/* No, so add it */
! 				nlp = makeNode(NestLoopParam);
! 				nlp->paramno = i;
! 				nlp->paramval = var;
! 				root->curOuterParams = lappend(root->curOuterParams, nlp);
! 			}
! 			i++;
! 		}
! 	}
  
  	return scan_plan;
  }
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
new file mode 100644
index 161d5ab..2e711d8
*** a/src/backend/optimizer/util/pathnode.c
--- b/src/backend/optimizer/util/pathnode.c
*************** distinct_col_search(int colno, List *col
*** 1331,1347 ****
   *	  returning the pathnode.
   */
  Path *
! create_subqueryscan_path(RelOptInfo *rel, List *pathkeys)
  {
! 	Path	   *pathnode = makeNode(Path);
  
! 	pathnode->pathtype = T_SubqueryScan;
! 	pathnode->parent = rel;
! 	pathnode->pathkeys = pathkeys;
  
  	cost_subqueryscan(pathnode, rel);
  
! 	return pathnode;
  }
  
  /*
--- 1331,1351 ----
   *	  returning the pathnode.
   */
  Path *
! create_subqueryscan_path(RelOptInfo *rel, List *pathkeys, Plan *subplan,
! 						 List *subrtable, List *subrowmark)
  {
! 	SubqueryPath   *pathnode = makeNode(SubqueryPath);
  
! 	pathnode->path.pathtype = T_SubqueryScan;
! 	pathnode->path.parent = rel;
! 	pathnode->path.pathkeys = pathkeys;
! 	pathnode->subplan = subplan;
! 	pathnode->subrtable = subrtable;
! 	pathnode->subrowmark = subrowmark;
  
  	cost_subqueryscan(pathnode, rel);
  
! 	return (Path *) pathnode;
  }
  
  /*
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
new file mode 100644
index b7a5845..ab2338f
*** a/src/backend/optimizer/util/relnode.c
--- b/src/backend/optimizer/util/relnode.c
*************** build_simple_rel(PlannerInfo *root, int 
*** 82,90 ****
  	rel->indexlist = NIL;
  	rel->pages = 0;
  	rel->tuples = 0;
! 	rel->subplan = NULL;
! 	rel->subrtable = NIL;
! 	rel->subrowmark = NIL;
  	rel->baserestrictinfo = NIL;
  	rel->baserestrictcost.startup = 0;
  	rel->baserestrictcost.per_tuple = 0;
--- 82,88 ----
  	rel->indexlist = NIL;
  	rel->pages = 0;
  	rel->tuples = 0;
! 	rel->preprocessed_subquery = NULL;
  	rel->baserestrictinfo = NIL;
  	rel->baserestrictcost.startup = 0;
  	rel->baserestrictcost.per_tuple = 0;
*************** build_join_rel(PlannerInfo *root,
*** 336,344 ****
  	joinrel->indexlist = NIL;
  	joinrel->pages = 0;
  	joinrel->tuples = 0;
! 	joinrel->subplan = NULL;
! 	joinrel->subrtable = NIL;
! 	joinrel->subrowmark = NIL;
  	joinrel->baserestrictinfo = NIL;
  	joinrel->baserestrictcost.startup = 0;
  	joinrel->baserestrictcost.per_tuple = 0;
--- 334,340 ----
  	joinrel->indexlist = NIL;
  	joinrel->pages = 0;
  	joinrel->tuples = 0;
! 	joinrel->preprocessed_subquery = NULL;
  	joinrel->baserestrictinfo = NIL;
  	joinrel->baserestrictcost.startup = 0;
  	joinrel->baserestrictcost.per_tuple = 0;
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
new file mode 100644
index d8bc6b8..e951fd1
*** a/src/include/nodes/nodes.h
--- b/src/include/nodes/nodes.h
*************** typedef enum NodeTag
*** 220,225 ****
--- 220,226 ----
  	T_MergePath,
  	T_HashPath,
  	T_TidPath,
+ 	T_SubqueryPath,
  	T_ForeignPath,
  	T_AppendPath,
  	T_MergeAppendPath,
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
new file mode 100644
index f659269..98e2744
*** a/src/include/nodes/relation.h
--- b/src/include/nodes/relation.h
*************** typedef struct PlannerInfo
*** 322,334 ****
   *					(always NIL if it's not a table)
   *		pages - number of disk pages in relation (zero if not a table)
   *		tuples - number of tuples in relation (not considering restrictions)
-  *		subplan - plan for subquery (NULL if it's not a subquery)
-  *		subrtable - rangetable for subquery (NIL if it's not a subquery)
-  *		subrowmark - rowmarks for subquery (NIL if it's not a subquery)
-  *
-  *		Note: for a subquery, tuples and subplan are not set immediately
-  *		upon creation of the RelOptInfo object; they are filled in when
-  *		set_base_rel_pathlist processes the object.
   *
   *		For otherrels that are appendrel members, these fields are filled
   *		in just as for a baserel.
--- 322,327 ----
*************** typedef struct RelOptInfo
*** 408,416 ****
  	List	   *indexlist;		/* list of IndexOptInfo */
  	BlockNumber pages;
  	double		tuples;
! 	struct Plan *subplan;		/* if subquery */
! 	List	   *subrtable;		/* if subquery */
! 	List	   *subrowmark;		/* if subquery */
  
  	/* used by various scans and joins: */
  	List	   *baserestrictinfo;		/* RestrictInfo structures (if base
--- 401,408 ----
  	List	   *indexlist;		/* list of IndexOptInfo */
  	BlockNumber pages;
  	double		tuples;
! 
! 	Query	   *preprocessed_subquery; /* if subquery */
  
  	/* used by various scans and joins: */
  	List	   *baserestrictinfo;		/* RestrictInfo structures (if base
*************** typedef struct TidPath
*** 770,775 ****
--- 762,784 ----
  } TidPath;
  
  /*
+  * SubqueryPath represents a scan of a subquery scan
+  *
+  * The struct holds subplan, subrtable and subrowmark, which are
+  * the temporary spaces to transport to the final stage of planner.
+  * Note each SubqueryPath may have different plan, depending on
+  * the pushed down qual clauses (as parameterized inner subquery).
+  */
+ typedef struct SubqueryPath
+ {
+ 	Path		path;
+ 	struct Plan *subplan;
+ 	List	   *subrtable;
+ 	List	   *subrowmark;
+ 	double		rows;
+ } SubqueryPath;
+ 
+ /*
   * ForeignPath represents a scan of a foreign table
   */
  typedef struct ForeignPath
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
new file mode 100644
index 2763863..7ec2c7b
*** a/src/include/optimizer/cost.h
--- b/src/include/optimizer/cost.h
*************** extern void cost_bitmap_or_node(BitmapOr
*** 75,81 ****
  extern void cost_bitmap_tree_node(Path *path, Cost *cost, Selectivity *selec);
  extern void cost_tidscan(Path *path, PlannerInfo *root,
  			 RelOptInfo *baserel, List *tidquals);
! extern void cost_subqueryscan(Path *path, RelOptInfo *baserel);
  extern void cost_functionscan(Path *path, PlannerInfo *root,
  				  RelOptInfo *baserel);
  extern void cost_valuesscan(Path *path, PlannerInfo *root,
--- 75,81 ----
  extern void cost_bitmap_tree_node(Path *path, Cost *cost, Selectivity *selec);
  extern void cost_tidscan(Path *path, PlannerInfo *root,
  			 RelOptInfo *baserel, List *tidquals);
! extern void cost_subqueryscan(SubqueryPath *path, RelOptInfo *baserel);
  extern void cost_functionscan(Path *path, PlannerInfo *root,
  				  RelOptInfo *baserel);
  extern void cost_valuesscan(Path *path, PlannerInfo *root,
*************** extern void set_joinrel_size_estimates(P
*** 122,128 ****
  						   SpecialJoinInfo *sjinfo,
  						   List *restrictlist);
  extern void set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
! 							PlannerInfo *subroot);
  extern void set_function_size_estimates(PlannerInfo *root, RelOptInfo *rel);
  extern void set_values_size_estimates(PlannerInfo *root, RelOptInfo *rel);
  extern void set_cte_size_estimates(PlannerInfo *root, RelOptInfo *rel,
--- 122,128 ----
  						   SpecialJoinInfo *sjinfo,
  						   List *restrictlist);
  extern void set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel,
! 							PlannerInfo *subroot, Plan *subplan);
  extern void set_function_size_estimates(PlannerInfo *root, RelOptInfo *rel);
  extern void set_values_size_estimates(PlannerInfo *root, RelOptInfo *rel);
  extern void set_cte_size_estimates(PlannerInfo *root, RelOptInfo *rel,
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
new file mode 100644
index 1da2131..6de2816
*** a/src/include/optimizer/pathnode.h
--- b/src/include/optimizer/pathnode.h
***************
*** 14,19 ****
--- 14,20 ----
  #ifndef PATHNODE_H
  #define PATHNODE_H
  
+ #include "nodes/plannodes.h"
  #include "nodes/relation.h"
  
  
*************** extern ResultPath *create_result_path(Li
*** 56,62 ****
  extern MaterialPath *create_material_path(RelOptInfo *rel, Path *subpath);
  extern UniquePath *create_unique_path(PlannerInfo *root, RelOptInfo *rel,
  				   Path *subpath, SpecialJoinInfo *sjinfo);
! extern Path *create_subqueryscan_path(RelOptInfo *rel, List *pathkeys);
  extern Path *create_functionscan_path(PlannerInfo *root, RelOptInfo *rel);
  extern Path *create_valuesscan_path(PlannerInfo *root, RelOptInfo *rel);
  extern Path *create_ctescan_path(PlannerInfo *root, RelOptInfo *rel);
--- 57,64 ----
  extern MaterialPath *create_material_path(RelOptInfo *rel, Path *subpath);
  extern UniquePath *create_unique_path(PlannerInfo *root, RelOptInfo *rel,
  				   Path *subpath, SpecialJoinInfo *sjinfo);
! extern Path *create_subqueryscan_path(RelOptInfo *rel, List *pathkeys,
! 					Plan *subplan, List *subrtable, List *subrowmark);
  extern Path *create_functionscan_path(PlannerInfo *root, RelOptInfo *rel);
  extern Path *create_valuesscan_path(PlannerInfo *root, RelOptInfo *rel);
  extern Path *create_ctescan_path(PlannerInfo *root, RelOptInfo *rel);
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
new file mode 100644
index 7f1353a..54d0b01
*** a/src/include/optimizer/paths.h
--- b/src/include/optimizer/paths.h
*************** extern PGDLLIMPORT join_search_hook_type
*** 33,38 ****
--- 33,41 ----
  extern RelOptInfo *make_one_rel(PlannerInfo *root, List *joinlist);
  extern RelOptInfo *standard_join_search(PlannerInfo *root, int levels_needed,
  					 List *initial_rels);
+ extern void best_inner_subqueryscan(PlannerInfo *root, RelOptInfo *rel,
+ 						RelOptInfo *outer_rel, List *join_clause,
+ 						Path **cheapest_start_path, Path **cheapest_total_path);
  
  #ifdef OPTIMIZER_DEBUG
  extern void debug_print_rel(PlannerInfo *root, RelOptInfo *rel);
*************** extern Path *get_cheapest_fractional_pat
*** 154,160 ****
  extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
  					 ScanDirection scandir);
  extern List *convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel,
! 						  List *subquery_pathkeys);
  extern List *build_join_pathkeys(PlannerInfo *root,
  					RelOptInfo *joinrel,
  					JoinType jointype,
--- 157,163 ----
  extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
  					 ScanDirection scandir);
  extern List *convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel,
! 						  List *subquery_pathkeys, Plan *subplan);
  extern List *build_join_pathkeys(PlannerInfo *root,
  					RelOptInfo *joinrel,
  					JoinType jointype,
#19Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Yeb Havinga (#18)
Re: Parameterized aggregate subquery (was: Pull up aggregate subquery)

2011/7/23 Yeb Havinga <yebhavinga@gmail.com>:

On 2011-07-22 16:17, Hitoshi Harada wrote:

:(
I updated the patch. Could you try attached once more? The "issafe"
switch seems wrong.

Works like a charm :-). However, now there is always a copyObject of a
subquery even when the subquery is not safe for qual pushdown. The problem
with the previous issafe was that it was only assigned for
rel->baserestrictinfo != NIL. If it is assigned before the if statement, it
still works. See attached patch that avoids subquery copy for unsafe
subqueries, and also exits best_inner_subqueryscan before palloc of
differenttypes in case of unsafe queries.

Ah, yeah, right. Too quick fix bloated my brain :P Thanks for testing!
I'll check it more.

--
Hitoshi Harada

#20Yeb Havinga
yebhavinga@gmail.com
In reply to: Hitoshi Harada (#19)
Pull up aggregate sublink (was: Parameterized aggregate subquery (was: Pull up aggregate subquery))

On 2011-07-22 17:35, Hitoshi Harada wrote:

2011/7/23 Yeb Havinga<yebhavinga@gmail.com>:

Works like a charm :-). However, now there is always a copyObject of a
subquery even when the subquery is not safe for qual pushdown. The problem
with the previous issafe was that it was only assigned for
rel->baserestrictinfo != NIL. If it is assigned before the if statement, it
still works. See attached patch that avoids subquery copy for unsafe
subqueries, and also exits best_inner_subqueryscan before palloc of
differenttypes in case of unsafe queries.

Ah, yeah, right. Too quick fix bloated my brain :P Thanks for testing!
I'll check it more.

A few days ago I read Tomas Vondra's blog post about dss tpc-h queries
on PostgreSQL at
http://fuzzy.cz/en/articles/dss-tpc-h-benchmark-with-postgresql/ - in
which he showed how to manually pull up a dss subquery to get a large
speed up. Initially I thought: cool, this is probably now handled by
Hitoshi's patch, but it turns out the subquery type in the dss query is
different.

The original and rewritten queries are below. The debug_print_plan
output shows the subquery is called from a opexpr (< l_quantity,
subquery output) and the sublink type is EXPR_SUBLINK. Looking at the
source code; pull_up_sublink only considers ANY and EXISTS sublinks. I'm
wondering if this could be expanded to deal with EXPR sublinks. Clearly
in the example Tomas has given this can be done. I'm wondering if there
are show stoppers that prevent this to be possible in the general case,
but can't think of any, other than the case of a sublink returning NULL
and the opexpr is part of a larger OR expression or IS NULL; in which
case it should not be pulled op, or perhaps it could be pulled up as
outer join.

Thoughts?

regards,
Yeb

The original query:

tpch=# explain select
sum(l_extendedprice) / 7.0 as avg_yearly
from
lineitem,
part
where
p_partkey = l_partkey
and p_brand = 'Brand#13'
and p_container = 'JUMBO PKG'
and l_quantity < (
select
0.2 * avg(l_quantity)
from
lineitem
where
l_partkey = p_partkey
)
LIMIT 1;
QUERY PLAN
------------------------------------------------------------------------------------------------------------
Limit (cost=183345309.79..183345309.81 rows=1 width=8)
-> Aggregate (cost=183345309.79..183345309.81 rows=1 width=8)
-> Hash Join (cost=2839.99..183345307.76 rows=815 width=8)
Hash Cond: (public.lineitem.l_partkey = part.p_partkey)
Join Filter: (public.lineitem.l_quantity < (SubPlan 1))
-> Seq Scan on lineitem (cost=0.00..68985.69
rows=2399869 width=17)
-> Hash (cost=2839.00..2839.00 rows=79 width=4)
-> Seq Scan on part (cost=0.00..2839.00 rows=79
width=4)
Filter: ((p_brand = 'Brand#13'::bpchar) AND
(p_container = 'JUMBO PKG'::bpchar))
SubPlan 1
-> Aggregate (cost=74985.44..74985.46 rows=1 width=5)
-> Seq Scan on lineitem (cost=0.00..74985.36
rows=31 width=5)
Filter: (l_partkey = part.p_partkey)

manually rewritten variant:

tpch=# explain select
sum(l_extendedprice) / 7.0 as avg_yearly
from
lineitem,
part,
(SELECT l_partkey AS agg_partkey,
0.2 * avg(l_quantity) AS avg_quantity
FROM lineitem GROUP BY l_partkey) part_agg
where
p_partkey = l_partkey
and agg_partkey = l_partkey
and p_brand = 'Brand#13'
and p_container = 'JUMBO PKG'
and l_quantity < avg_quantity
LIMIT 1;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------
Limit (cost=179643.88..179643.89 rows=1 width=8)
-> Aggregate (cost=179643.88..179643.89 rows=1 width=8)
-> Hash Join (cost=161865.21..178853.91 rows=315985 width=8)
Hash Cond: (public.lineitem.l_partkey = part.p_partkey)
Join Filter: (public.lineitem.l_quantity < ((0.2 *
avg(public.lineitem.l_quantity))))
-> HashAggregate (cost=80985.04..82148.65 rows=77574
width=9)
-> Seq Scan on lineitem (cost=0.00..68985.69
rows=2399869 width=9)
-> Hash (cost=80849.63..80849.63 rows=2444 width=21)
-> Hash Join (cost=2839.99..80849.63 rows=2444
width=21)
Hash Cond: (public.lineitem.l_partkey =
part.p_partkey)
-> Seq Scan on lineitem
(cost=0.00..68985.69 rows=2399869 width=17)
-> Hash (cost=2839.00..2839.00 rows=79
width=4)
-> Seq Scan on part
(cost=0.00..2839.00 rows=79 width=4)
Filter: ((p_brand =
'Brand#13'::bpchar) AND (p_container = 'JUMBO PKG'::bpchar))

#21Tom Lane
tgl@sss.pgh.pa.us
In reply to: Yeb Havinga (#20)
Re: Pull up aggregate sublink (was: Parameterized aggregate subquery (was: Pull up aggregate subquery))

Yeb Havinga <yebhavinga@gmail.com> writes:

A few days ago I read Tomas Vondra's blog post about dss tpc-h queries
on PostgreSQL at
http://fuzzy.cz/en/articles/dss-tpc-h-benchmark-with-postgresql/ - in
which he showed how to manually pull up a dss subquery to get a large
speed up. Initially I thought: cool, this is probably now handled by
Hitoshi's patch, but it turns out the subquery type in the dss query is
different.

Actually, I believe this example is the exact opposite of the
transformation Hitoshi proposes. Tomas was manually replacing an
aggregated subquery by a reference to a grouped table, which can be
a win if the subquery would be executed enough times to amortize
calculation of the grouped table over all the groups (some of which
might never be demanded by the outer query). Hitoshi was talking about
avoiding calculations of grouped-table elements that we don't need,
which would be a win in different cases. Or at least that was the
thrust of his original proposal; I'm not sure where the patch went since
then.

This leads me to think that we need to represent both cases as the same
sort of query and make a cost-based decision as to which way to go.
Thinking of it as a pull-up or push-down transformation is the wrong
approach because those sorts of transformations are done too early to
be able to use cost comparisons.

regards, tom lane

#22Robert Haas
robertmhaas@gmail.com
In reply to: Tom Lane (#21)
Re: Pull up aggregate sublink (was: Parameterized aggregate subquery (was: Pull up aggregate subquery))

On Tue, Jul 26, 2011 at 5:37 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Yeb Havinga <yebhavinga@gmail.com> writes:

A few days ago I read Tomas Vondra's blog post about dss tpc-h queries
on PostgreSQL at
http://fuzzy.cz/en/articles/dss-tpc-h-benchmark-with-postgresql/ - in
which he showed how to manually pull up a dss subquery to get a large
speed up. Initially I thought: cool, this is probably now handled by
Hitoshi's patch, but it turns out the subquery type in the dss query is
different.

Actually, I believe this example is the exact opposite of the
transformation Hitoshi proposes.  Tomas was manually replacing an
aggregated subquery by a reference to a grouped table, which can be
a win if the subquery would be executed enough times to amortize
calculation of the grouped table over all the groups (some of which
might never be demanded by the outer query).  Hitoshi was talking about
avoiding calculations of grouped-table elements that we don't need,
which would be a win in different cases.  Or at least that was the
thrust of his original proposal; I'm not sure where the patch went since
then.

This leads me to think that we need to represent both cases as the same
sort of query and make a cost-based decision as to which way to go.
Thinking of it as a pull-up or push-down transformation is the wrong
approach because those sorts of transformations are done too early to
be able to use cost comparisons.

I think you're right. OTOH, our estimates of what will pop out of an
aggregate are so poor that denying the user to control the plan on the
basis of how they write the query might be a net negative. :-(

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#23Yeb Havinga
yebhavinga@gmail.com
In reply to: Robert Haas (#22)
Re: Pull up aggregate sublink (was: Parameterized aggregate subquery (was: Pull up aggregate subquery))

On 2011-07-27 16:16, Robert Haas wrote:

On Tue, Jul 26, 2011 at 5:37 PM, Tom Lane<tgl@sss.pgh.pa.us> wrote:

Yeb Havinga<yebhavinga@gmail.com> writes:

A few days ago I read Tomas Vondra's blog post about dss tpc-h queries
on PostgreSQL at
http://fuzzy.cz/en/articles/dss-tpc-h-benchmark-with-postgresql/ - in
which he showed how to manually pull up a dss subquery to get a large
speed up. Initially I thought: cool, this is probably now handled by
Hitoshi's patch, but it turns out the subquery type in the dss query is
different.

Actually, I believe this example is the exact opposite of the
transformation Hitoshi proposes. Tomas was manually replacing an
aggregated subquery by a reference to a grouped table, which can be
a win if the subquery would be executed enough times to amortize
calculation of the grouped table over all the groups (some of which
might never be demanded by the outer query). Hitoshi was talking about
avoiding calculations of grouped-table elements that we don't need,
which would be a win in different cases. Or at least that was the
thrust of his original proposal; I'm not sure where the patch went since
then.

This leads me to think that we need to represent both cases as the same
sort of query and make a cost-based decision as to which way to go.
Thinking of it as a pull-up or push-down transformation is the wrong
approach because those sorts of transformations are done too early to
be able to use cost comparisons.

I think you're right. OTOH, our estimates of what will pop out of an
aggregate are so poor that denying the user to control the plan on the
basis of how they write the query might be a net negative. :-(

Tom and Robert, thank you both for your replies. I think I'm having some
blind spots and maybe false assumptions regarding the overal work in the
optimizer, as it is not clear to me what 'the same sort of query' would
look like. I was under the impression that using cost to select the best
paths is only done per simple query, and fail to see how a total
combined plan with pulled up subquery could be compared on cost with a
total plan where the subquery is still a separate subplan, since the
range tables / simple-queries to compare are different.

regards,
Yeb

#24Tom Lane
tgl@sss.pgh.pa.us
In reply to: Yeb Havinga (#23)
Re: Pull up aggregate sublink (was: Parameterized aggregate subquery (was: Pull up aggregate subquery))

Yeb Havinga <yebhavinga@gmail.com> writes:

Tom and Robert, thank you both for your replies. I think I'm having some
blind spots and maybe false assumptions regarding the overal work in the
optimizer, as it is not clear to me what 'the same sort of query' would
look like. I was under the impression that using cost to select the best
paths is only done per simple query, and fail to see how a total
combined plan with pulled up subquery could be compared on cost with a
total plan where the subquery is still a separate subplan, since the
range tables / simple-queries to compare are different.

Well, you could look at what planagg.c does to decide whether an
indexscan optimization of MIN/MAX is worthwhile, or at the calculations
in planner.c that decide which way to do grouping/aggregation/ordering.

It could be fairly expensive to handle this type of problem because of
the need to cost out two fundamentally different scan/join trees, but
we're assuming the queries are expensive enough to make that worthwhile
...

regards, tom lane

#25Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Yeb Havinga (#20)
Re: Pull up aggregate sublink (was: Parameterized aggregate subquery (was: Pull up aggregate subquery))

2011/7/27 Yeb Havinga <yebhavinga@gmail.com>:

On 2011-07-22 17:35, Hitoshi Harada wrote:

2011/7/23 Yeb Havinga<yebhavinga@gmail.com>:

A few days ago I read Tomas Vondra's blog post about dss tpc-h queries on
PostgreSQL at
http://fuzzy.cz/en/articles/dss-tpc-h-benchmark-with-postgresql/ - in which
he showed how to manually pull up a dss subquery to get a large speed up.
Initially I thought: cool, this is probably now handled by Hitoshi's patch,
but it turns out the subquery type in the dss query is different.

The original and rewritten queries are below. The debug_print_plan output
shows the subquery is called from a opexpr (< l_quantity, subquery output)
and the sublink type is EXPR_SUBLINK. Looking at the source code;
pull_up_sublink only considers ANY and EXISTS sublinks. I'm wondering if
this could be expanded to deal with EXPR sublinks. Clearly in the example
Tomas has given this can be done. I'm wondering if there are show stoppers
that prevent this to be possible in the general case, but can't think of
any, other than the case of a sublink returning NULL and the opexpr is part
of a larger OR expression or IS NULL; in which case it should not be pulled
op, or perhaps it could be pulled up as outer join.

Thoughts?

Good catch. I was not aware of the sublink case so I'm not sure if it
is possible, but I believe it will be worth modifying the optimizer to
handle them in the same way. Since my latest proposal is based on
parameterized NestLoop, the first step is how to transform the sublink
expression into join. I bet there are chances in simple cases since we
have Semi/Anti Join technique. On the other hand, those pseudo-join
types are easily failing to be transformed to join, in such cases
above like it have another filter clause than join qual expression.

If tpc bechmark can be speed up that's a good use case which you
pointed out I'm missing.

Regards,

--
Hitoshi Harada

#26Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Tom Lane (#21)
Re: Pull up aggregate sublink (was: Parameterized aggregate subquery (was: Pull up aggregate subquery))

2011/7/27 Tom Lane <tgl@sss.pgh.pa.us>:

Yeb Havinga <yebhavinga@gmail.com> writes:

A few days ago I read Tomas Vondra's blog post about dss tpc-h queries
on PostgreSQL at
http://fuzzy.cz/en/articles/dss-tpc-h-benchmark-with-postgresql/ - in
which he showed how to manually pull up a dss subquery to get a large
speed up. Initially I thought: cool, this is probably now handled by
Hitoshi's patch, but it turns out the subquery type in the dss query is
different.

Actually, I believe this example is the exact opposite of the
transformation Hitoshi proposes. Tomas was manually replacing an
aggregated subquery by a reference to a grouped table, which can be
a win if the subquery would be executed enough times to amortize
calculation of the grouped table over all the groups (some of which
might never be demanded by the outer query).  Hitoshi was talking about
avoiding calculations of grouped-table elements that we don't need,
which would be a win in different cases.  Or at least that was the
thrust of his original proposal; I'm not sure where the patch went since
then.

My first proposal which is about pulling up aggregate like sublink
expression is exact opposite of this (Tomas pushed down the sublink
expression to join subquery). But the latest proposal is upon
parameterized NestLoop, so I think my latest patch might help
something for the *second* query. Actually the problem is the same; We
want to reduce grouping operation which is not interesting to the
final output, by filtering other relations expression. In this case,
if the joined lineitem-part relation has very few rows by WHERE
conditions (p_brand, p_container), we don't want calculate avg of huge
lineitem because we know almost all of the avg result is not in the
upper result. However, the current optimizer cannot pass the upper
query's condition (like "it will have only few rows") down to the
lower aggregate query.

This leads me to think that we need to represent both cases as the same
sort of query and make a cost-based decision as to which way to go.
Thinking of it as a pull-up or push-down transformation is the wrong
approach because those sorts of transformations are done too early to
be able to use cost comparisons.

Wrapping up my mind above and reading this paragraph, it might be
another work to make sublink expression look like the same as join.
But what we want to solve is the same goal, I think.

Regards,

--
Hitoshi Harada