LATERAL quals revisited
I've been studying the bug reported at
/messages/by-id/20130617235236.GA1636@jeremyevans.local
that the planner can do the wrong thing with queries like
SELECT * FROM
i LEFT JOIN LATERAL (SELECT * FROM j WHERE i.n = j.n) j ON true;
I think the fundamental problem is that, because the "i.n = j.n" clause
appears syntactically in WHERE, the planner is treating it as if it were
an inner-join clause; but really it ought to be considered a clause of
the upper LEFT JOIN. That is, semantically this query ought to be
equivalent to
SELECT * FROM
i LEFT JOIN LATERAL (SELECT * FROM j) j ON i.n = j.n;
However, because distribute_qual_to_rels doesn't see the clause as being
attached to the outer join, it's not marked with the correct properties
and ends up getting evaluated in the wrong place (as a "filter" clause
not a "join filter" clause). The bug is masked in the test cases we've
used so far because those cases are designed to let the clause get
pushed down into the scan of the inner relation --- but if it doesn't
get pushed down, it's evaluated the wrong way.
After some contemplation, I think that the most practical way to fix
this is for deconstruct_recurse and distribute_qual_to_rels to
effectively move such a qual to the place where it logically belongs;
that is, rather than processing it when we look at the lower WHERE
clause, set it aside for a moment and then add it back when looking at
the ON clause of the appropriate outer join. This should be reasonably
easy to do by keeping a list of "postponed lateral clauses" while we're
scanning the join tree.
For there to *be* a unique "appropriate outer join", we need to require
that a LATERAL-using qual clause that's under an outer join contain
lateral references only to the outer side of the nearest enclosing outer
join. There's no such restriction in the spec of course, but we can
make it so by refusing to flatten a sub-select if pulling it up would
result in having a clause in the outer query that violates this rule.
There's already some code in prepjointree.c (around line 1300) that
attempts to enforce this, though now that I look at it again I'm not
sure it's covering all the bases. We may need to extend that check.
I'm inclined to process all LATERAL-using qual clauses this way, ie
postpone them till we recurse back up to a place where they can
logically be evaluated. That won't make any real difference when no
outer joins are present, but it will eliminate the ugliness that right
now distribute_qual_to_rels is prevented from sanity-checking the scope
of the references in a qual when LATERAL is present. If we do it like
this, we can resurrect full enforcement of that sanity check, and then
throw an error if any "postponed" quals are left over when we're done
recursing.
Thoughts, better ideas?
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
(Please excuse me if my proposal sounds silly, i'm still not too
advanced in this area...)
On 06/25/2013 10:00 PM, Tom Lane wrote:
After some contemplation, I think that the most practical way to fix
this is for deconstruct_recurse and distribute_qual_to_rels to
effectively move such a qual to the place where it logically belongs;
that is, rather than processing it when we look at the lower WHERE
clause, set it aside for a moment and then add it back when looking at
the ON clause of the appropriate outer join. This should be reasonably
easy to do by keeping a list of "postponed lateral clauses" while we're
scanning the join tree.
Instead of setting it aside, can we (mis)use placeholder var (PHV), to
ensure
that the WHERE clause is evaluated below the OJ; instead of combining it
with
the ON clause? That would be a special PHV(s) in that it's not actually
referenced from outside the subquery.
Whether I'm right or not, I seem to have found another problem while
trying to enforce such a PHV:
postgres=# SELECT i.*, j.* FROM i LEFT JOIN LATERAL (SELECT COALESCE(i)
FROM j WHERE (i.n = j.n)) j ON true;
The connection to the server was lost. Attempting reset: Failed.
TRAP: FailedAssertion("!(!bms_overlap(min_lefthand, min_righthand))",
File: "initsplan.c", Line: 1043)
LOG: server process (PID 24938) was terminated by signal 6: Aborted
DETAIL: Failed process was running: SELECT i.*, j.* FROM i LEFT JOIN
LATERAL (SELECT COALESCE(i) FROM j WHERE (i.n = j.n)) j ON true;
LOG: terminating any other active server processes
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back
the current transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and
repeat your command.
FATAL: the database system is in recovery mode
I'm not able to judge right now whether the Assert() statement is the
problem itself or anything
else. You'll probably know better.
(4f14c86d7434376b95477aeeb07fcc7272f4c47d is the last commit in my
environment)
Regards,
Antonin Houska (Tony)
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Antonin Houska <antonin.houska@gmail.com> writes:
On 06/25/2013 10:00 PM, Tom Lane wrote:
After some contemplation, I think that the most practical way to fix
this is for deconstruct_recurse and distribute_qual_to_rels to
effectively move such a qual to the place where it logically belongs;
that is, rather than processing it when we look at the lower WHERE
clause, set it aside for a moment and then add it back when looking at
the ON clause of the appropriate outer join.
Instead of setting it aside, can we (mis)use placeholder var (PHV), to
ensure that the WHERE clause is evaluated below the OJ; instead of
combining it with the ON clause?
No, that doesn't help; it has to be part of the joinquals at the join
node, or you don't get the right execution semantics. Plus you could
lose some optimization opportunities, for example if we fail to see
that there's a strict join clause associated with the outer join
(cf lhs_strict). Worse, I think wrapping a PHV around an otherwise
indexable clause would prevent using it for an indexscan.
Whether I'm right or not, I seem to have found another problem while
trying to enforce such a PHV:
postgres=# SELECT i.*, j.* FROM i LEFT JOIN LATERAL (SELECT COALESCE(i)
FROM j WHERE (i.n = j.n)) j ON true;
The connection to the server was lost. Attempting reset: Failed.
[ pokes at that ... ] Hm, right offhand this seems like an independent
issue --- the ph_eval_at for the PHV is wrong AFAICS. Thanks for
reporting it!
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 06/26/2013 12:52 AM, Tom Lane wrote:
Instead of setting it aside, can we (mis)use placeholder var (PHV), to
ensure that the WHERE clause is evaluated below the OJ; instead of
combining it with the ON clause?No, that doesn't help; it has to be part of the joinquals at the join
node, or you don't get the right execution semantics.
When I wrote 'below the OJ' I meant to retain something of the original
semantics (just like the subquery applies the WHERE clause below the OJ).
However that's probably too restrictive and your next arguments
Plus you could
lose some optimization opportunities, for example if we fail to see
that there's a strict join clause associated with the outer join
(cf lhs_strict). Worse, I think wrapping a PHV around an otherwise
indexable clause would prevent using it for an indexscan.
also confirm the restrictiveness. So I can forget.
One more concern anyway: doesn't your proposal make subquery pull-up a
little bit risky in terms of cost of the resulting plan?
IMO the subquery in the original query may filter out many rows and thus
decrease the number of pairs to be evaluated by the join the ON clause
belongs to.
If the WHERE clause moves up, then the resulting plan might be less
efficient than the one we'd get if the subquery hadn't been pulled-up at
all.
However at the time of cost evaluation there's no way to get back (not
even to notice the higher cost) because the original subquery has gone
at earlier stage of the planning.
Regards,
Antonin Houska (Tony)
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Antonin Houska <antonin.houska@gmail.com> writes:
If the WHERE clause moves up, then the resulting plan might be less
efficient than the one we'd get if the subquery hadn't been pulled-up at
all.
No, because we can push the qual back down again (using a parameterized
path) if that's appropriate. The problem at this stage is to understand
the semantics of the outer join correctly, not to make a choice of what
the plan will be.
In fact, the reason we'd not noticed this bug before is exactly that
all the test cases in the regression tests do end up pushing the qual
back down.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Antonin Houska <antonin.houska@gmail.com> writes:
Whether I'm right or not, I seem to have found another problem while
trying to enforce such a PHV:
postgres=# SELECT i.*, j.* FROM i LEFT JOIN LATERAL (SELECT COALESCE(i)
FROM j WHERE (i.n = j.n)) j ON true;
The connection to the server was lost. Attempting reset: Failed.
I've been poking at this problem, and have found out that there are
several other multi-legged creatures underneath this rock. LATERAL
references turn out to have many more interactions with PlaceHolderVars
than I'd previously thought. I think the existing code was okay in
the initial cut at LATERAL, when we never tried to flatten any LATERAL
subqueries into the parent query --- but now that we allow such flattening
to happen, it's possible that a PlaceHolderVar that's been wrapped
around a pulled-up subquery output expression will contain a lateral
reference. There was a previous report of problems with that sort of
thing, which I tried to fix in a quick-hack way in commit
4da6439bd8553059766011e2a42c6e39df08717f, but that was totally wrong and
in fact caused the Assert you show above. The right way to think about
it is that a PlaceHolderVar should be evaluated at its syntactic
location, but if it contains a lateral reference then that creates an
outer-reference requirement for the scan or join level at which it gets
evaluated.
So attached is a draft patch for this. It's not complete yet because
there are various comments that are now wrong and need to be updated;
but I think the code is functioning correctly. Also the
lateral_vars/lateral_relids stuff seems a bit crude and Rube Goldbergish
now, because it considers *only* lateral references occurring at relation
scan level, which I now see is just part of the problem. I'm not sure
if there's a good way to generalize that or if it's best left alone.
Note that the original join-qual-misplacement problem reported by Jeremy
Evans is not fixed yet; this is just addressing PlaceHolderVar issues.
Comments?
regards, tom lane
Attachments:
placeholdervar-fixes-1.patchtext/x-patch; charset=us-asciiDownload
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 1c93e0c5ac30aa70694308d8bc799c01b0e88360..e911e067948674afd78bac753d229ac08f22eeae 100644
*** a/contrib/postgres_fdw/postgres_fdw.c
--- b/contrib/postgres_fdw/postgres_fdw.c
*************** postgresGetForeignPaths(PlannerInfo *roo
*** 592,599 ****
LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(lc);
if (bms_is_member(baserel->relid, ljinfo->lateral_lhs))
! lateral_referencers = bms_add_member(lateral_referencers,
! ljinfo->lateral_rhs);
}
/* Scan the rel's join clauses */
--- 592,599 ----
LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(lc);
if (bms_is_member(baserel->relid, ljinfo->lateral_lhs))
! lateral_referencers = bms_add_members(lateral_referencers,
! ljinfo->lateral_rhs);
}
/* Scan the rel's join clauses */
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index b5b8d63cff79247ed1340b228fd5a10887a5a1bf..705f14a3884f85d132e5b233027ceaaf1c4c7bcc 100644
*** a/src/backend/nodes/copyfuncs.c
--- b/src/backend/nodes/copyfuncs.c
*************** _copyLateralJoinInfo(const LateralJoinIn
*** 1917,1924 ****
{
LateralJoinInfo *newnode = makeNode(LateralJoinInfo);
- COPY_SCALAR_FIELD(lateral_rhs);
COPY_BITMAPSET_FIELD(lateral_lhs);
return newnode;
}
--- 1917,1924 ----
{
LateralJoinInfo *newnode = makeNode(LateralJoinInfo);
COPY_BITMAPSET_FIELD(lateral_lhs);
+ COPY_BITMAPSET_FIELD(lateral_rhs);
return newnode;
}
*************** _copyPlaceHolderInfo(const PlaceHolderIn
*** 1952,1957 ****
--- 1952,1958 ----
COPY_SCALAR_FIELD(phid);
COPY_NODE_FIELD(ph_var);
COPY_BITMAPSET_FIELD(ph_eval_at);
+ COPY_BITMAPSET_FIELD(ph_lateral);
COPY_BITMAPSET_FIELD(ph_needed);
COPY_BITMAPSET_FIELD(ph_may_need);
COPY_SCALAR_FIELD(ph_width);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 3f96595e8eb041bcebe38639e075db7eb8a77fd2..41b75044a001f8affa557984fc672b7b310f1934 100644
*** a/src/backend/nodes/equalfuncs.c
--- b/src/backend/nodes/equalfuncs.c
*************** _equalPlaceHolderVar(const PlaceHolderVa
*** 761,775 ****
/*
* We intentionally do not compare phexpr. Two PlaceHolderVars with the
* same ID and levelsup should be considered equal even if the contained
! * expressions have managed to mutate to different states. One way in
! * which that can happen is that initplan sublinks would get replaced by
! * differently-numbered Params when sublink folding is done. (The end
! * result of such a situation would be some unreferenced initplans, which
! * is annoying but not really a problem.)
*
* COMPARE_NODE_FIELD(phexpr);
*/
- COMPARE_BITMAPSET_FIELD(phrels);
COMPARE_SCALAR_FIELD(phid);
COMPARE_SCALAR_FIELD(phlevelsup);
--- 761,779 ----
/*
* We intentionally do not compare phexpr. Two PlaceHolderVars with the
* same ID and levelsup should be considered equal even if the contained
! * expressions have managed to mutate to different states. This will
! * happen during final plan construction when there are nested PHVs, since
! * the inner PHV will get replaced by a Param in some copies of the outer
! * PHV. Another way in which it can happen is that initplan sublinks
! * could get replaced by differently-numbered Params when sublink folding
! * is done. (The end result of such a situation would be some
! * unreferenced initplans, which is annoying but not really a problem.) On
! * the same reasoning, there is no need to examine phrels.
*
* COMPARE_NODE_FIELD(phexpr);
+ *
+ * COMPARE_BITMAPSET_FIELD(phrels);
*/
COMPARE_SCALAR_FIELD(phid);
COMPARE_SCALAR_FIELD(phlevelsup);
*************** _equalSpecialJoinInfo(const SpecialJoinI
*** 794,801 ****
static bool
_equalLateralJoinInfo(const LateralJoinInfo *a, const LateralJoinInfo *b)
{
- COMPARE_SCALAR_FIELD(lateral_rhs);
COMPARE_BITMAPSET_FIELD(lateral_lhs);
return true;
}
--- 798,805 ----
static bool
_equalLateralJoinInfo(const LateralJoinInfo *a, const LateralJoinInfo *b)
{
COMPARE_BITMAPSET_FIELD(lateral_lhs);
+ COMPARE_BITMAPSET_FIELD(lateral_rhs);
return true;
}
*************** static bool
*** 817,824 ****
_equalPlaceHolderInfo(const PlaceHolderInfo *a, const PlaceHolderInfo *b)
{
COMPARE_SCALAR_FIELD(phid);
! COMPARE_NODE_FIELD(ph_var);
COMPARE_BITMAPSET_FIELD(ph_eval_at);
COMPARE_BITMAPSET_FIELD(ph_needed);
COMPARE_BITMAPSET_FIELD(ph_may_need);
COMPARE_SCALAR_FIELD(ph_width);
--- 821,829 ----
_equalPlaceHolderInfo(const PlaceHolderInfo *a, const PlaceHolderInfo *b)
{
COMPARE_SCALAR_FIELD(phid);
! COMPARE_NODE_FIELD(ph_var); /* should be redundant */
COMPARE_BITMAPSET_FIELD(ph_eval_at);
+ COMPARE_BITMAPSET_FIELD(ph_lateral);
COMPARE_BITMAPSET_FIELD(ph_needed);
COMPARE_BITMAPSET_FIELD(ph_may_need);
COMPARE_SCALAR_FIELD(ph_width);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index b2183f42137bd474925df18d96349ac99d5f0a80..752d2ca3444cd1a9e24121e4e8559d4acbc211c2 100644
*** a/src/backend/nodes/outfuncs.c
--- b/src/backend/nodes/outfuncs.c
*************** _outLateralJoinInfo(StringInfo str, cons
*** 1909,1916 ****
{
WRITE_NODE_TYPE("LATERALJOININFO");
- WRITE_UINT_FIELD(lateral_rhs);
WRITE_BITMAPSET_FIELD(lateral_lhs);
}
static void
--- 1909,1916 ----
{
WRITE_NODE_TYPE("LATERALJOININFO");
WRITE_BITMAPSET_FIELD(lateral_lhs);
+ WRITE_BITMAPSET_FIELD(lateral_rhs);
}
static void
*************** _outPlaceHolderInfo(StringInfo str, cons
*** 1934,1939 ****
--- 1934,1940 ----
WRITE_UINT_FIELD(phid);
WRITE_NODE_FIELD(ph_var);
WRITE_BITMAPSET_FIELD(ph_eval_at);
+ WRITE_BITMAPSET_FIELD(ph_lateral);
WRITE_BITMAPSET_FIELD(ph_needed);
WRITE_BITMAPSET_FIELD(ph_may_need);
WRITE_INT_FIELD(ph_width);
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index 65eb344cde449b9cfeab3da87672ff06734b7d1b..109e14cd7368ed453cd311a197642a9cee228095 100644
*** a/src/backend/optimizer/path/indxpath.c
--- b/src/backend/optimizer/path/indxpath.c
*************** create_index_paths(PlannerInfo *root, Re
*** 259,266 ****
LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(lc);
if (bms_is_member(rel->relid, ljinfo->lateral_lhs))
! lateral_referencers = bms_add_member(lateral_referencers,
! ljinfo->lateral_rhs);
}
/* Bitmap paths are collected and then dealt with at the end */
--- 259,266 ----
LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(lc);
if (bms_is_member(rel->relid, ljinfo->lateral_lhs))
! lateral_referencers = bms_add_members(lateral_referencers,
! ljinfo->lateral_rhs);
}
/* Bitmap paths are collected and then dealt with at the end */
diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c
index d6050a616c73093b0b10c9c454ddd85829353b6d..5b477e52d3fc72bcffe0d7cfb4f62f03610756c9 100644
*** a/src/backend/optimizer/path/joinpath.c
--- b/src/backend/optimizer/path/joinpath.c
*************** static void sort_inner_and_outer(Planner
*** 29,47 ****
RelOptInfo *outerrel, RelOptInfo *innerrel,
List *restrictlist, List *mergeclause_list,
JoinType jointype, SpecialJoinInfo *sjinfo,
! Relids param_source_rels);
static void match_unsorted_outer(PlannerInfo *root, RelOptInfo *joinrel,
RelOptInfo *outerrel, RelOptInfo *innerrel,
List *restrictlist, List *mergeclause_list,
JoinType jointype, SpecialJoinInfo *sjinfo,
SemiAntiJoinFactors *semifactors,
! Relids param_source_rels);
static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel,
RelOptInfo *outerrel, RelOptInfo *innerrel,
List *restrictlist,
JoinType jointype, SpecialJoinInfo *sjinfo,
SemiAntiJoinFactors *semifactors,
! Relids param_source_rels);
static List *select_mergejoin_clauses(PlannerInfo *root,
RelOptInfo *joinrel,
RelOptInfo *outerrel,
--- 29,47 ----
RelOptInfo *outerrel, RelOptInfo *innerrel,
List *restrictlist, List *mergeclause_list,
JoinType jointype, SpecialJoinInfo *sjinfo,
! Relids param_source_rels, Relids extra_lateral_rels);
static void match_unsorted_outer(PlannerInfo *root, RelOptInfo *joinrel,
RelOptInfo *outerrel, RelOptInfo *innerrel,
List *restrictlist, List *mergeclause_list,
JoinType jointype, SpecialJoinInfo *sjinfo,
SemiAntiJoinFactors *semifactors,
! Relids param_source_rels, Relids extra_lateral_rels);
static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel,
RelOptInfo *outerrel, RelOptInfo *innerrel,
List *restrictlist,
JoinType jointype, SpecialJoinInfo *sjinfo,
SemiAntiJoinFactors *semifactors,
! Relids param_source_rels, Relids extra_lateral_rels);
static List *select_mergejoin_clauses(PlannerInfo *root,
RelOptInfo *joinrel,
RelOptInfo *outerrel,
*************** add_paths_to_joinrel(PlannerInfo *root,
*** 87,92 ****
--- 87,93 ----
bool mergejoin_allowed = true;
SemiAntiJoinFactors semifactors;
Relids param_source_rels = NULL;
+ Relids extra_lateral_rels = NULL;
ListCell *lc;
/*
*************** add_paths_to_joinrel(PlannerInfo *root,
*** 162,181 ****
{
LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(lc);
! if (bms_is_member(ljinfo->lateral_rhs, joinrel->relids))
param_source_rels = bms_join(param_source_rels,
bms_difference(ljinfo->lateral_lhs,
joinrel->relids));
}
/*
* 1. Consider mergejoin paths where both relations must be explicitly
* sorted. Skip this if we can't mergejoin.
*/
if (mergejoin_allowed)
sort_inner_and_outer(root, joinrel, outerrel, innerrel,
restrictlist, mergeclause_list, jointype,
! sjinfo, param_source_rels);
/*
* 2. Consider paths where the outer relation need not be explicitly
--- 163,220 ----
{
LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(lc);
! if (bms_is_subset(ljinfo->lateral_rhs, joinrel->relids))
param_source_rels = bms_join(param_source_rels,
bms_difference(ljinfo->lateral_lhs,
joinrel->relids));
}
/*
+ * Another issue created by LATERAL references is that PlaceHolderVars
+ * that need to be computed at this join level might contain lateral
+ * references to rels not in the join, meaning that the paths for the join
+ * would need to be marked as parameterized by those rels, independently
+ * of all other considerations. Set extra_lateral_rels to the set of such
+ * rels. This will not affect our decisions as to which paths to
+ * generate; we merely add these rels to their required_outer sets.
+ */
+ foreach(lc, root->placeholder_list)
+ {
+ PlaceHolderInfo *phinfo = (PlaceHolderInfo *) lfirst(lc);
+
+ /* PHVs without lateral refs can be skipped over quickly */
+ if (phinfo->ph_lateral == NULL)
+ continue;
+ /* Is it due to be evaluated at this join, and not in either input? */
+ if (bms_is_subset(phinfo->ph_eval_at, joinrel->relids) &&
+ !bms_is_subset(phinfo->ph_eval_at, outerrel->relids) &&
+ !bms_is_subset(phinfo->ph_eval_at, innerrel->relids))
+ {
+ /* Yes, remember its lateral rels */
+ extra_lateral_rels = bms_add_members(extra_lateral_rels,
+ phinfo->ph_lateral);
+ }
+ }
+
+ /*
+ * Make sure extra_lateral_rels doesn't list anything within the join, and
+ * that it's NULL if empty. (This allows us to use bms_add_members to add
+ * it to required_outer below, while preserving the property that
+ * required_outer is exactly NULL if empty.)
+ */
+ extra_lateral_rels = bms_del_members(extra_lateral_rels, joinrel->relids);
+ if (bms_is_empty(extra_lateral_rels))
+ extra_lateral_rels = NULL;
+
+ /*
* 1. Consider mergejoin paths where both relations must be explicitly
* sorted. Skip this if we can't mergejoin.
*/
if (mergejoin_allowed)
sort_inner_and_outer(root, joinrel, outerrel, innerrel,
restrictlist, mergeclause_list, jointype,
! sjinfo,
! param_source_rels, extra_lateral_rels);
/*
* 2. Consider paths where the outer relation need not be explicitly
*************** add_paths_to_joinrel(PlannerInfo *root,
*** 187,193 ****
if (mergejoin_allowed)
match_unsorted_outer(root, joinrel, outerrel, innerrel,
restrictlist, mergeclause_list, jointype,
! sjinfo, &semifactors, param_source_rels);
#ifdef NOT_USED
--- 226,233 ----
if (mergejoin_allowed)
match_unsorted_outer(root, joinrel, outerrel, innerrel,
restrictlist, mergeclause_list, jointype,
! sjinfo, &semifactors,
! param_source_rels, extra_lateral_rels);
#ifdef NOT_USED
*************** add_paths_to_joinrel(PlannerInfo *root,
*** 205,211 ****
if (mergejoin_allowed)
match_unsorted_inner(root, joinrel, outerrel, innerrel,
restrictlist, mergeclause_list, jointype,
! sjinfo, &semifactors, param_source_rels);
#endif
/*
--- 245,252 ----
if (mergejoin_allowed)
match_unsorted_inner(root, joinrel, outerrel, innerrel,
restrictlist, mergeclause_list, jointype,
! sjinfo, &semifactors,
! param_source_rels, extra_lateral_rels);
#endif
/*
*************** add_paths_to_joinrel(PlannerInfo *root,
*** 216,222 ****
if (enable_hashjoin || jointype == JOIN_FULL)
hash_inner_and_outer(root, joinrel, outerrel, innerrel,
restrictlist, jointype,
! sjinfo, &semifactors, param_source_rels);
}
/*
--- 257,264 ----
if (enable_hashjoin || jointype == JOIN_FULL)
hash_inner_and_outer(root, joinrel, outerrel, innerrel,
restrictlist, jointype,
! sjinfo, &semifactors,
! param_source_rels, extra_lateral_rels);
}
/*
*************** try_nestloop_path(PlannerInfo *root,
*** 231,236 ****
--- 273,279 ----
SpecialJoinInfo *sjinfo,
SemiAntiJoinFactors *semifactors,
Relids param_source_rels,
+ Relids extra_lateral_rels,
Path *outer_path,
Path *inner_path,
List *restrict_clauses,
*************** try_nestloop_path(PlannerInfo *root,
*** 254,259 ****
--- 297,308 ----
}
/*
+ * Independently of that, add parameterization needed for any
+ * PlaceHolderVars that need to be computed at the join.
+ */
+ required_outer = bms_add_members(required_outer, extra_lateral_rels);
+
+ /*
* Do a precheck to quickly eliminate obviously-inferior paths. We
* calculate a cheap lower bound on the path's cost and then use
* add_path_precheck() to see if the path is clearly going to be dominated
*************** try_mergejoin_path(PlannerInfo *root,
*** 301,306 ****
--- 350,356 ----
JoinType jointype,
SpecialJoinInfo *sjinfo,
Relids param_source_rels,
+ Relids extra_lateral_rels,
Path *outer_path,
Path *inner_path,
List *restrict_clauses,
*************** try_mergejoin_path(PlannerInfo *root,
*** 327,332 ****
--- 377,388 ----
}
/*
+ * Independently of that, add parameterization needed for any
+ * PlaceHolderVars that need to be computed at the join.
+ */
+ required_outer = bms_add_members(required_outer, extra_lateral_rels);
+
+ /*
* If the given paths are already well enough ordered, we can skip doing
* an explicit sort.
*/
*************** try_hashjoin_path(PlannerInfo *root,
*** 383,388 ****
--- 439,445 ----
SpecialJoinInfo *sjinfo,
SemiAntiJoinFactors *semifactors,
Relids param_source_rels,
+ Relids extra_lateral_rels,
Path *outer_path,
Path *inner_path,
List *restrict_clauses,
*************** try_hashjoin_path(PlannerInfo *root,
*** 406,411 ****
--- 463,474 ----
}
/*
+ * Independently of that, add parameterization needed for any
+ * PlaceHolderVars that need to be computed at the join.
+ */
+ required_outer = bms_add_members(required_outer, extra_lateral_rels);
+
+ /*
* See comments in try_nestloop_path(). Also note that hashjoin paths
* never have any output pathkeys, per comments in create_hashjoin_path.
*/
*************** clause_sides_match_join(RestrictInfo *ri
*** 483,488 ****
--- 546,552 ----
* 'jointype' is the type of join to do
* 'sjinfo' is extra info about the join for selectivity estimation
* 'param_source_rels' are OK targets for parameterization of result paths
+ * 'extra_lateral_rels' are additional parameterization for result paths
*/
static void
sort_inner_and_outer(PlannerInfo *root,
*************** sort_inner_and_outer(PlannerInfo *root,
*** 493,499 ****
List *mergeclause_list,
JoinType jointype,
SpecialJoinInfo *sjinfo,
! Relids param_source_rels)
{
Path *outer_path;
Path *inner_path;
--- 557,564 ----
List *mergeclause_list,
JoinType jointype,
SpecialJoinInfo *sjinfo,
! Relids param_source_rels,
! Relids extra_lateral_rels)
{
Path *outer_path;
Path *inner_path;
*************** sort_inner_and_outer(PlannerInfo *root,
*** 623,628 ****
--- 688,694 ----
jointype,
sjinfo,
param_source_rels,
+ extra_lateral_rels,
outer_path,
inner_path,
restrictlist,
*************** sort_inner_and_outer(PlannerInfo *root,
*** 668,673 ****
--- 734,740 ----
* 'sjinfo' is extra info about the join for selectivity estimation
* 'semifactors' contains valid data if jointype is SEMI or ANTI
* 'param_source_rels' are OK targets for parameterization of result paths
+ * 'extra_lateral_rels' are additional parameterization for result paths
*/
static void
match_unsorted_outer(PlannerInfo *root,
*************** match_unsorted_outer(PlannerInfo *root,
*** 679,685 ****
JoinType jointype,
SpecialJoinInfo *sjinfo,
SemiAntiJoinFactors *semifactors,
! Relids param_source_rels)
{
JoinType save_jointype = jointype;
bool nestjoinOK;
--- 746,753 ----
JoinType jointype,
SpecialJoinInfo *sjinfo,
SemiAntiJoinFactors *semifactors,
! Relids param_source_rels,
! Relids extra_lateral_rels)
{
JoinType save_jointype = jointype;
bool nestjoinOK;
*************** match_unsorted_outer(PlannerInfo *root,
*** 809,814 ****
--- 877,883 ----
sjinfo,
semifactors,
param_source_rels,
+ extra_lateral_rels,
outerpath,
inner_cheapest_total,
restrictlist,
*************** match_unsorted_outer(PlannerInfo *root,
*** 834,839 ****
--- 903,909 ----
sjinfo,
semifactors,
param_source_rels,
+ extra_lateral_rels,
outerpath,
innerpath,
restrictlist,
*************** match_unsorted_outer(PlannerInfo *root,
*** 848,853 ****
--- 918,924 ----
sjinfo,
semifactors,
param_source_rels,
+ extra_lateral_rels,
outerpath,
matpath,
restrictlist,
*************** match_unsorted_outer(PlannerInfo *root,
*** 903,908 ****
--- 974,980 ----
jointype,
sjinfo,
param_source_rels,
+ extra_lateral_rels,
outerpath,
inner_cheapest_total,
restrictlist,
*************** match_unsorted_outer(PlannerInfo *root,
*** 1001,1006 ****
--- 1073,1079 ----
jointype,
sjinfo,
param_source_rels,
+ extra_lateral_rels,
outerpath,
innerpath,
restrictlist,
*************** match_unsorted_outer(PlannerInfo *root,
*** 1046,1051 ****
--- 1119,1125 ----
jointype,
sjinfo,
param_source_rels,
+ extra_lateral_rels,
outerpath,
innerpath,
restrictlist,
*************** match_unsorted_outer(PlannerInfo *root,
*** 1080,1085 ****
--- 1154,1160 ----
* 'sjinfo' is extra info about the join for selectivity estimation
* 'semifactors' contains valid data if jointype is SEMI or ANTI
* 'param_source_rels' are OK targets for parameterization of result paths
+ * 'extra_lateral_rels' are additional parameterization for result paths
*/
static void
hash_inner_and_outer(PlannerInfo *root,
*************** hash_inner_and_outer(PlannerInfo *root,
*** 1090,1096 ****
JoinType jointype,
SpecialJoinInfo *sjinfo,
SemiAntiJoinFactors *semifactors,
! Relids param_source_rels)
{
bool isouterjoin = IS_OUTER_JOIN(jointype);
List *hashclauses;
--- 1165,1172 ----
JoinType jointype,
SpecialJoinInfo *sjinfo,
SemiAntiJoinFactors *semifactors,
! Relids param_source_rels,
! Relids extra_lateral_rels)
{
bool isouterjoin = IS_OUTER_JOIN(jointype);
List *hashclauses;
*************** hash_inner_and_outer(PlannerInfo *root,
*** 1164,1169 ****
--- 1240,1246 ----
sjinfo,
semifactors,
param_source_rels,
+ extra_lateral_rels,
cheapest_total_outer,
cheapest_total_inner,
restrictlist,
*************** hash_inner_and_outer(PlannerInfo *root,
*** 1183,1188 ****
--- 1260,1266 ----
sjinfo,
semifactors,
param_source_rels,
+ extra_lateral_rels,
cheapest_total_outer,
cheapest_total_inner,
restrictlist,
*************** hash_inner_and_outer(PlannerInfo *root,
*** 1195,1200 ****
--- 1273,1279 ----
sjinfo,
semifactors,
param_source_rels,
+ extra_lateral_rels,
cheapest_startup_outer,
cheapest_total_inner,
restrictlist,
*************** hash_inner_and_outer(PlannerInfo *root,
*** 1219,1224 ****
--- 1298,1304 ----
sjinfo,
semifactors,
param_source_rels,
+ extra_lateral_rels,
cheapest_startup_outer,
cheapest_total_inner,
restrictlist,
*************** hash_inner_and_outer(PlannerInfo *root,
*** 1256,1261 ****
--- 1336,1342 ----
sjinfo,
semifactors,
param_source_rels,
+ extra_lateral_rels,
outerpath,
innerpath,
restrictlist,
diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c
index 819498a4281d191f4ecb927fe821e027cc55e8b0..d627f9e130c00e1a40e88294ccfb11d70fbeb380 100644
*** a/src/backend/optimizer/path/joinrels.c
--- b/src/backend/optimizer/path/joinrels.c
*************** join_is_legal(PlannerInfo *root, RelOptI
*** 526,532 ****
{
LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(l);
! if (bms_is_member(ljinfo->lateral_rhs, rel2->relids) &&
bms_overlap(ljinfo->lateral_lhs, rel1->relids))
{
/* has to be implemented as nestloop with rel1 on left */
--- 526,532 ----
{
LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(l);
! if (bms_is_subset(ljinfo->lateral_rhs, rel2->relids) &&
bms_overlap(ljinfo->lateral_lhs, rel1->relids))
{
/* has to be implemented as nestloop with rel1 on left */
*************** join_is_legal(PlannerInfo *root, RelOptI
*** 539,545 ****
(reversed || match_sjinfo->jointype == JOIN_FULL))
return false; /* not implementable as nestloop */
}
! if (bms_is_member(ljinfo->lateral_rhs, rel1->relids) &&
bms_overlap(ljinfo->lateral_lhs, rel2->relids))
{
/* has to be implemented as nestloop with rel2 on left */
--- 539,545 ----
(reversed || match_sjinfo->jointype == JOIN_FULL))
return false; /* not implementable as nestloop */
}
! if (bms_is_subset(ljinfo->lateral_rhs, rel1->relids) &&
bms_overlap(ljinfo->lateral_lhs, rel2->relids))
{
/* has to be implemented as nestloop with rel2 on left */
*************** have_join_order_restriction(PlannerInfo
*** 829,838 ****
{
LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(l);
! if (bms_is_member(ljinfo->lateral_rhs, rel2->relids) &&
bms_overlap(ljinfo->lateral_lhs, rel1->relids))
return true;
! if (bms_is_member(ljinfo->lateral_rhs, rel1->relids) &&
bms_overlap(ljinfo->lateral_lhs, rel2->relids))
return true;
}
--- 829,838 ----
{
LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(l);
! if (bms_is_subset(ljinfo->lateral_rhs, rel2->relids) &&
bms_overlap(ljinfo->lateral_lhs, rel1->relids))
return true;
! if (bms_is_subset(ljinfo->lateral_rhs, rel1->relids) &&
bms_overlap(ljinfo->lateral_lhs, rel2->relids))
return true;
}
*************** has_join_restriction(PlannerInfo *root,
*** 928,934 ****
{
LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(l);
! if (bms_is_member(ljinfo->lateral_rhs, rel->relids) ||
bms_overlap(ljinfo->lateral_lhs, rel->relids))
return true;
}
--- 928,934 ----
{
LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(l);
! if (bms_is_subset(ljinfo->lateral_rhs, rel->relids) ||
bms_overlap(ljinfo->lateral_lhs, rel->relids))
return true;
}
diff --git a/src/backend/optimizer/plan/analyzejoins.c b/src/backend/optimizer/plan/analyzejoins.c
index a7db69c85bfabc95e37196398e4f18c57969fbe3..795f2286b1ebf768c95c25706309fadbcc8246ff 100644
*** a/src/backend/optimizer/plan/analyzejoins.c
--- b/src/backend/optimizer/plan/analyzejoins.c
*************** join_is_removable(PlannerInfo *root, Spe
*** 202,208 ****
* that will be used above the join. We only need to fail if such a PHV
* actually references some inner-rel attributes; but the correct check
* for that is relatively expensive, so we first check against ph_eval_at,
! * which must mention the inner rel if the PHV uses any inner-rel attrs.
*/
foreach(l, root->placeholder_list)
{
--- 202,210 ----
* that will be used above the join. We only need to fail if such a PHV
* actually references some inner-rel attributes; but the correct check
* for that is relatively expensive, so we first check against ph_eval_at,
! * which must mention the inner rel if the PHV uses any inner-rel attrs as
! * non-lateral references. Note also that if the PHV's syntactic scope is
! * just the inner rel, we can't drop it even if it's variable-free.
*/
foreach(l, root->placeholder_list)
{
*************** join_is_removable(PlannerInfo *root, Spe
*** 210,218 ****
if (bms_is_subset(phinfo->ph_needed, joinrelids))
continue; /* PHV is not used above the join */
if (!bms_overlap(phinfo->ph_eval_at, innerrel->relids))
continue; /* it definitely doesn't reference innerrel */
! if (bms_overlap(pull_varnos((Node *) phinfo->ph_var),
innerrel->relids))
return false; /* it does reference innerrel */
}
--- 212,224 ----
if (bms_is_subset(phinfo->ph_needed, joinrelids))
continue; /* PHV is not used above the join */
+ if (bms_overlap(phinfo->ph_lateral, innerrel->relids))
+ return false; /* it references innerrel laterally */
if (!bms_overlap(phinfo->ph_eval_at, innerrel->relids))
continue; /* it definitely doesn't reference innerrel */
! if (bms_is_subset(phinfo->ph_eval_at, innerrel->relids))
! return false; /* there isn't any other place to eval PHV */
! if (bms_overlap(pull_varnos((Node *) phinfo->ph_var->phexpr),
innerrel->relids))
return false; /* it does reference innerrel */
}
*************** remove_rel_from_query(PlannerInfo *root,
*** 355,361 ****
* Likewise remove references from LateralJoinInfo data structures.
*
* If we are deleting a LATERAL subquery, we can forget its
! * LateralJoinInfo altogether. Otherwise, make sure the target is not
* included in any lateral_lhs set. (It probably can't be, since that
* should have precluded deciding to remove it; but let's cope anyway.)
*/
--- 361,367 ----
* Likewise remove references from LateralJoinInfo data structures.
*
* If we are deleting a LATERAL subquery, we can forget its
! * LateralJoinInfos altogether. Otherwise, make sure the target is not
* included in any lateral_lhs set. (It probably can't be, since that
* should have precluded deciding to remove it; but let's cope anyway.)
*/
*************** remove_rel_from_query(PlannerInfo *root,
*** 364,392 ****
LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(l);
nextl = lnext(l);
! if (ljinfo->lateral_rhs == relid)
root->lateral_info_list = list_delete_ptr(root->lateral_info_list,
ljinfo);
else
ljinfo->lateral_lhs = bms_del_member(ljinfo->lateral_lhs, relid);
}
/*
* Likewise remove references from PlaceHolderVar data structures.
- *
- * Here we have a special case: if a PHV's eval_at set is just the target
- * relid, we want to leave it that way instead of reducing it to the empty
- * set. An empty eval_at set would confuse later processing since it
- * would match every possible eval placement.
*/
foreach(l, root->placeholder_list)
{
PlaceHolderInfo *phinfo = (PlaceHolderInfo *) lfirst(l);
phinfo->ph_eval_at = bms_del_member(phinfo->ph_eval_at, relid);
! if (bms_is_empty(phinfo->ph_eval_at)) /* oops, belay that */
! phinfo->ph_eval_at = bms_add_member(phinfo->ph_eval_at, relid);
!
phinfo->ph_needed = bms_del_member(phinfo->ph_needed, relid);
/* ph_may_need probably isn't used after this, but fix it anyway */
phinfo->ph_may_need = bms_del_member(phinfo->ph_may_need, relid);
--- 370,396 ----
LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(l);
nextl = lnext(l);
! ljinfo->lateral_rhs = bms_del_member(ljinfo->lateral_rhs, relid);
! if (bms_is_empty(ljinfo->lateral_rhs))
root->lateral_info_list = list_delete_ptr(root->lateral_info_list,
ljinfo);
else
+ {
ljinfo->lateral_lhs = bms_del_member(ljinfo->lateral_lhs, relid);
+ Assert(!bms_is_empty(ljinfo->lateral_lhs));
+ }
}
/*
* Likewise remove references from PlaceHolderVar data structures.
*/
foreach(l, root->placeholder_list)
{
PlaceHolderInfo *phinfo = (PlaceHolderInfo *) lfirst(l);
phinfo->ph_eval_at = bms_del_member(phinfo->ph_eval_at, relid);
! Assert(!bms_is_empty(phinfo->ph_eval_at));
! Assert(!bms_is_member(relid, phinfo->ph_lateral));
phinfo->ph_needed = bms_del_member(phinfo->ph_needed, relid);
/* ph_may_need probably isn't used after this, but fix it anyway */
phinfo->ph_may_need = bms_del_member(phinfo->ph_may_need, relid);
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 52bab79007e66d80205a07da08bc4437c424232c..c501737a2671e8c0f6a0fc21157e7e125a7b1a34 100644
*** a/src/backend/optimizer/plan/createplan.c
--- b/src/backend/optimizer/plan/createplan.c
***************
*** 44,52 ****
static Plan *create_plan_recurse(PlannerInfo *root, Path *best_path);
static Plan *create_scan_plan(PlannerInfo *root, Path *best_path);
! static List *build_relation_tlist(RelOptInfo *rel);
static bool use_physical_tlist(PlannerInfo *root, RelOptInfo *rel);
! static void disuse_physical_tlist(Plan *plan, Path *path);
static Plan *create_gating_plan(PlannerInfo *root, Plan *plan, List *quals);
static Plan *create_join_plan(PlannerInfo *root, JoinPath *best_path);
static Plan *create_append_plan(PlannerInfo *root, AppendPath *best_path);
--- 44,52 ----
static Plan *create_plan_recurse(PlannerInfo *root, Path *best_path);
static Plan *create_scan_plan(PlannerInfo *root, Path *best_path);
! static List *build_path_tlist(PlannerInfo *root, Path *path);
static bool use_physical_tlist(PlannerInfo *root, RelOptInfo *rel);
! static void disuse_physical_tlist(PlannerInfo *root, Plan *plan, Path *path);
static Plan *create_gating_plan(PlannerInfo *root, Plan *plan, List *quals);
static Plan *create_join_plan(PlannerInfo *root, JoinPath *best_path);
static Plan *create_append_plan(PlannerInfo *root, AppendPath *best_path);
*************** create_scan_plan(PlannerInfo *root, Path
*** 305,325 ****
tlist = build_physical_tlist(root, rel);
/* if fail because of dropped cols, use regular method */
if (tlist == NIL)
! tlist = build_relation_tlist(rel);
}
}
else
{
! tlist = build_relation_tlist(rel);
!
! /*
! * If it's a parameterized otherrel, there might be lateral references
! * in the tlist, which need to be replaced with Params. This cannot
! * happen for regular baserels, though. Note use_physical_tlist()
! * always fails for otherrels, so we don't need to check this above.
! */
! if (rel->reloptkind != RELOPT_BASEREL && best_path->param_info)
! tlist = (List *) replace_nestloop_params(root, (Node *) tlist);
}
/*
--- 305,316 ----
tlist = build_physical_tlist(root, rel);
/* if fail because of dropped cols, use regular method */
if (tlist == NIL)
! tlist = build_path_tlist(root, best_path);
}
}
else
{
! tlist = build_path_tlist(root, best_path);
}
/*
*************** create_scan_plan(PlannerInfo *root, Path
*** 439,449 ****
}
/*
! * Build a target list (ie, a list of TargetEntry) for a relation.
*/
static List *
! build_relation_tlist(RelOptInfo *rel)
{
List *tlist = NIL;
int resno = 1;
ListCell *v;
--- 430,441 ----
}
/*
! * Build a target list (ie, a list of TargetEntry) for the Path's output.
*/
static List *
! build_path_tlist(PlannerInfo *root, Path *path)
{
+ RelOptInfo *rel = path->parent;
List *tlist = NIL;
int resno = 1;
ListCell *v;
*************** build_relation_tlist(RelOptInfo *rel)
*** 453,458 ****
--- 445,459 ----
/* Do we really need to copy here? Not sure */
Node *node = (Node *) copyObject(lfirst(v));
+ /*
+ * If it's a parameterized path, there might be lateral references in
+ * the tlist, which need to be replaced with Params. There's no need
+ * to remake the TargetEntry nodes, so apply this to each list item
+ * separately.
+ */
+ if (path->param_info)
+ node = replace_nestloop_params(root, node);
+
tlist = lappend(tlist, makeTargetEntry((Expr *) node,
resno,
NULL,
*************** use_physical_tlist(PlannerInfo *root, Re
*** 528,534 ****
* and Material nodes want this, so they don't have to store useless columns.
*/
static void
! disuse_physical_tlist(Plan *plan, Path *path)
{
/* Only need to undo it for path types handled by create_scan_plan() */
switch (path->pathtype)
--- 529,535 ----
* and Material nodes want this, so they don't have to store useless columns.
*/
static void
! disuse_physical_tlist(PlannerInfo *root, Plan *plan, Path *path)
{
/* Only need to undo it for path types handled by create_scan_plan() */
switch (path->pathtype)
*************** disuse_physical_tlist(Plan *plan, Path *
*** 544,550 ****
case T_CteScan:
case T_WorkTableScan:
case T_ForeignScan:
! plan->targetlist = build_relation_tlist(path->parent);
break;
default:
break;
--- 545,551 ----
case T_CteScan:
case T_WorkTableScan:
case T_ForeignScan:
! plan->targetlist = build_path_tlist(root, path);
break;
default:
break;
*************** static Plan *
*** 678,684 ****
create_append_plan(PlannerInfo *root, AppendPath *best_path)
{
Append *plan;
! List *tlist = build_relation_tlist(best_path->path.parent);
List *subplans = NIL;
ListCell *subpaths;
--- 679,685 ----
create_append_plan(PlannerInfo *root, AppendPath *best_path)
{
Append *plan;
! List *tlist = build_path_tlist(root, &best_path->path);
List *subplans = NIL;
ListCell *subpaths;
*************** create_merge_append_plan(PlannerInfo *ro
*** 733,739 ****
{
MergeAppend *node = makeNode(MergeAppend);
Plan *plan = &node->plan;
! List *tlist = build_relation_tlist(best_path->path.parent);
List *pathkeys = best_path->path.pathkeys;
List *subplans = NIL;
ListCell *subpaths;
--- 734,740 ----
{
MergeAppend *node = makeNode(MergeAppend);
Plan *plan = &node->plan;
! List *tlist = build_path_tlist(root, &best_path->path);
List *pathkeys = best_path->path.pathkeys;
List *subplans = NIL;
ListCell *subpaths;
*************** create_material_plan(PlannerInfo *root,
*** 862,868 ****
subplan = create_plan_recurse(root, best_path->subpath);
/* We don't want any excess columns in the materialized tuples */
! disuse_physical_tlist(subplan, best_path->subpath);
plan = make_material(subplan);
--- 863,869 ----
subplan = create_plan_recurse(root, best_path->subpath);
/* We don't want any excess columns in the materialized tuples */
! disuse_physical_tlist(root, subplan, best_path->subpath);
plan = make_material(subplan);
*************** create_unique_plan(PlannerInfo *root, Un
*** 911,917 ****
* should be left as-is if we don't need to add any expressions; but if we
* do have to add expressions, then a projection step will be needed at
* runtime anyway, so we may as well remove unneeded items. Therefore
! * newtlist starts from build_relation_tlist() not just a copy of the
* subplan's tlist; and we don't install it into the subplan unless we are
* sorting or stuff has to be added.
*/
--- 912,918 ----
* should be left as-is if we don't need to add any expressions; but if we
* do have to add expressions, then a projection step will be needed at
* runtime anyway, so we may as well remove unneeded items. Therefore
! * newtlist starts from build_path_tlist() not just a copy of the
* subplan's tlist; and we don't install it into the subplan unless we are
* sorting or stuff has to be added.
*/
*************** create_unique_plan(PlannerInfo *root, Un
*** 919,925 ****
uniq_exprs = best_path->uniq_exprs;
/* initialize modified subplan tlist as just the "required" vars */
! newtlist = build_relation_tlist(best_path->path.parent);
nextresno = list_length(newtlist) + 1;
newitems = false;
--- 920,926 ----
uniq_exprs = best_path->uniq_exprs;
/* initialize modified subplan tlist as just the "required" vars */
! newtlist = build_path_tlist(root, &best_path->path);
nextresno = list_length(newtlist) + 1;
newitems = false;
*************** create_unique_plan(PlannerInfo *root, Un
*** 1009,1015 ****
* subplan tlist.
*/
plan = (Plan *) make_agg(root,
! build_relation_tlist(best_path->path.parent),
NIL,
AGG_HASHED,
NULL,
--- 1010,1016 ----
* subplan tlist.
*/
plan = (Plan *) make_agg(root,
! build_path_tlist(root, &best_path->path),
NIL,
AGG_HASHED,
NULL,
*************** create_nestloop_plan(PlannerInfo *root,
*** 2028,2034 ****
Plan *inner_plan)
{
NestLoop *join_plan;
! List *tlist = build_relation_tlist(best_path->path.parent);
List *joinrestrictclauses = best_path->joinrestrictinfo;
List *joinclauses;
List *otherclauses;
--- 2029,2035 ----
Plan *inner_plan)
{
NestLoop *join_plan;
! List *tlist = build_path_tlist(root, &best_path->path);
List *joinrestrictclauses = best_path->joinrestrictinfo;
List *joinclauses;
List *otherclauses;
*************** create_mergejoin_plan(PlannerInfo *root,
*** 2118,2124 ****
Plan *outer_plan,
Plan *inner_plan)
{
! List *tlist = build_relation_tlist(best_path->jpath.path.parent);
List *joinclauses;
List *otherclauses;
List *mergeclauses;
--- 2119,2125 ----
Plan *outer_plan,
Plan *inner_plan)
{
! List *tlist = build_path_tlist(root, &best_path->jpath.path);
List *joinclauses;
List *otherclauses;
List *mergeclauses;
*************** create_mergejoin_plan(PlannerInfo *root,
*** 2186,2192 ****
*/
if (best_path->outersortkeys)
{
! disuse_physical_tlist(outer_plan, best_path->jpath.outerjoinpath);
outer_plan = (Plan *)
make_sort_from_pathkeys(root,
outer_plan,
--- 2187,2193 ----
*/
if (best_path->outersortkeys)
{
! disuse_physical_tlist(root, outer_plan, best_path->jpath.outerjoinpath);
outer_plan = (Plan *)
make_sort_from_pathkeys(root,
outer_plan,
*************** create_mergejoin_plan(PlannerInfo *root,
*** 2199,2205 ****
if (best_path->innersortkeys)
{
! disuse_physical_tlist(inner_plan, best_path->jpath.innerjoinpath);
inner_plan = (Plan *)
make_sort_from_pathkeys(root,
inner_plan,
--- 2200,2206 ----
if (best_path->innersortkeys)
{
! disuse_physical_tlist(root, inner_plan, best_path->jpath.innerjoinpath);
inner_plan = (Plan *)
make_sort_from_pathkeys(root,
inner_plan,
*************** create_hashjoin_plan(PlannerInfo *root,
*** 2413,2419 ****
Plan *outer_plan,
Plan *inner_plan)
{
! List *tlist = build_relation_tlist(best_path->jpath.path.parent);
List *joinclauses;
List *otherclauses;
List *hashclauses;
--- 2414,2420 ----
Plan *outer_plan,
Plan *inner_plan)
{
! List *tlist = build_path_tlist(root, &best_path->jpath.path);
List *joinclauses;
List *otherclauses;
List *hashclauses;
*************** create_hashjoin_plan(PlannerInfo *root,
*** 2470,2480 ****
best_path->jpath.outerjoinpath->parent->relids);
/* We don't want any excess columns in the hashed tuples */
! disuse_physical_tlist(inner_plan, best_path->jpath.innerjoinpath);
/* If we expect batching, suppress excess columns in outer tuples too */
if (best_path->num_batches > 1)
! disuse_physical_tlist(outer_plan, best_path->jpath.outerjoinpath);
/*
* If there is a single join clause and we can identify the outer variable
--- 2471,2481 ----
best_path->jpath.outerjoinpath->parent->relids);
/* We don't want any excess columns in the hashed tuples */
! disuse_physical_tlist(root, inner_plan, best_path->jpath.innerjoinpath);
/* If we expect batching, suppress excess columns in outer tuples too */
if (best_path->num_batches > 1)
! disuse_physical_tlist(root, outer_plan, best_path->jpath.outerjoinpath);
/*
* If there is a single join clause and we can identify the outer variable
*************** replace_nestloop_params_mutator(Node *no
*** 2604,2619 ****
Assert(phv->phlevelsup == 0);
/*
! * If not to be replaced, just return the PlaceHolderVar unmodified.
! * We use bms_overlap as a cheap/quick test to see if the PHV might be
! * evaluated in the outer rels, and then grab its PlaceHolderInfo to
! * tell for sure.
*/
! if (!bms_overlap(phv->phrels, root->curOuterRels))
! return node;
! if (!bms_is_subset(find_placeholder_info(root, phv, false)->ph_eval_at,
! root->curOuterRels))
! return node;
/* Create a Param representing the PlaceHolderVar */
param = assign_nestloop_param_placeholdervar(root, phv);
/* Is this param already listed in root->curOuterParams? */
--- 2605,2641 ----
Assert(phv->phlevelsup == 0);
/*
! * Check whether we need to replace the PHV. We use bms_overlap as a
! * cheap/quick test to see if the PHV might be evaluated in the outer
! * rels, and then grab its PlaceHolderInfo to tell for sure.
*/
! if (!bms_overlap(phv->phrels, root->curOuterRels) ||
! !bms_is_subset(find_placeholder_info(root, phv, false)->ph_eval_at,
! root->curOuterRels))
! {
! /*
! * We can't replace the whole PHV, but we might still need to
! * replace Vars or PHVs within its expression, in case it ends up
! * actually getting evaluated here. (It might get evaluated in
! * this plan node, or some child node; in the latter case we don't
! * really need to process the expression here, but we haven't got
! * enough info to tell if that's the case.) Flat-copy the PHV
! * node and then recurse on its expression.
! *
! * Note that after doing this, we might have different
! * representations of the contents of the same PHV in different
! * parts of the plan tree. This is OK because equal() will just
! * match on phid/phlevelsup, so setrefs.c will still recognize an
! * upper-level reference to a lower-level copy of the same PHV.
! */
! PlaceHolderVar *newphv = makeNode(PlaceHolderVar);
!
! memcpy(newphv, phv, sizeof(PlaceHolderVar));
! newphv->phexpr = (Expr *)
! replace_nestloop_params_mutator((Node *) phv->phexpr,
! root);
! return (Node *) newphv;
! }
/* Create a Param representing the PlaceHolderVar */
param = assign_nestloop_param_placeholdervar(root, phv);
/* Is this param already listed in root->curOuterParams? */
diff --git a/src/backend/optimizer/plan/initsplan.c b/src/backend/optimizer/plan/initsplan.c
index 839ed9dde4049a7bff81d6909776f0ba3e4550a9..0f2c6752c445dece8214fe19ac20d2c4da07edb9 100644
*** a/src/backend/optimizer/plan/initsplan.c
--- b/src/backend/optimizer/plan/initsplan.c
*************** int join_collapse_limit;
*** 37,43 ****
static void extract_lateral_references(PlannerInfo *root, RelOptInfo *brel,
Index rtindex);
! static void add_lateral_info(PlannerInfo *root, Index rhs, Relids lhs);
static List *deconstruct_recurse(PlannerInfo *root, Node *jtnode,
bool below_outer_join,
Relids *qualscope, Relids *inner_join_rels);
--- 37,43 ----
static void extract_lateral_references(PlannerInfo *root, RelOptInfo *brel,
Index rtindex);
! static void add_lateral_info(PlannerInfo *root, Relids lhs, Relids rhs);
static List *deconstruct_recurse(PlannerInfo *root, Node *jtnode,
bool below_outer_join,
Relids *qualscope, Relids *inner_join_rels);
*************** add_vars_to_targetlist(PlannerInfo *root
*** 177,182 ****
--- 177,184 ----
RelOptInfo *rel = find_base_rel(root, var->varno);
int attno = var->varattno;
+ if (bms_is_subset(where_needed, rel->relids))
+ continue;
Assert(attno >= rel->min_attr && attno <= rel->max_attr);
attno -= rel->min_attr;
if (rel->attr_needed[attno] == NULL)
*************** extract_lateral_references(PlannerInfo *
*** 369,382 ****
* create_lateral_join_info
* For each LATERAL subquery, create LateralJoinInfo(s) and add them to
* root->lateral_info_list, and fill in the per-rel lateral_relids sets.
*
* This has to run after deconstruct_jointree, because we need to know the
! * final ph_eval_at values for referenced PlaceHolderVars.
*/
void
create_lateral_join_info(PlannerInfo *root)
{
Index rti;
/* We need do nothing if the query contains no LATERAL RTEs */
if (!root->hasLateralRTEs)
--- 371,387 ----
* create_lateral_join_info
* For each LATERAL subquery, create LateralJoinInfo(s) and add them to
* root->lateral_info_list, and fill in the per-rel lateral_relids sets.
+ * Also generate LateralJoinInfo(s) to represent any lateral references
+ * within PlaceHolderVars.
*
* This has to run after deconstruct_jointree, because we need to know the
! * final ph_eval_at values for PlaceHolderVars.
*/
void
create_lateral_join_info(PlannerInfo *root)
{
Index rti;
+ ListCell *lc;
/* We need do nothing if the query contains no LATERAL RTEs */
if (!root->hasLateralRTEs)
*************** create_lateral_join_info(PlannerInfo *ro
*** 389,395 ****
{
RelOptInfo *brel = root->simple_rel_array[rti];
Relids lateral_relids;
- ListCell *lc;
/* there may be empty slots corresponding to non-baserel RTEs */
if (brel == NULL)
--- 394,399 ----
*************** create_lateral_join_info(PlannerInfo *ro
*** 412,418 ****
{
Var *var = (Var *) node;
! add_lateral_info(root, rti, bms_make_singleton(var->varno));
lateral_relids = bms_add_member(lateral_relids,
var->varno);
}
--- 416,423 ----
{
Var *var = (Var *) node;
! add_lateral_info(root, bms_make_singleton(var->varno),
! brel->relids);
lateral_relids = bms_add_member(lateral_relids,
var->varno);
}
*************** create_lateral_join_info(PlannerInfo *ro
*** 422,428 ****
PlaceHolderInfo *phinfo = find_placeholder_info(root, phv,
false);
! add_lateral_info(root, rti, bms_copy(phinfo->ph_eval_at));
lateral_relids = bms_add_members(lateral_relids,
phinfo->ph_eval_at);
}
--- 427,433 ----
PlaceHolderInfo *phinfo = find_placeholder_info(root, phv,
false);
! add_lateral_info(root, phinfo->ph_eval_at, brel->relids);
lateral_relids = bms_add_members(lateral_relids,
phinfo->ph_eval_at);
}
*************** create_lateral_join_info(PlannerInfo *ro
*** 460,503 ****
}
}
}
}
/*
* add_lateral_info
* Add a LateralJoinInfo to root->lateral_info_list, if needed
*
! * We suppress redundant list entries. The passed lhs set must be freshly
! * made; we free it if not used in a new list entry.
*/
static void
! add_lateral_info(PlannerInfo *root, Index rhs, Relids lhs)
{
LateralJoinInfo *ljinfo;
! ListCell *l;
! Assert(!bms_is_member(rhs, lhs));
/*
! * If an existing list member has the same RHS and an LHS that is a subset
! * of the new one, it's redundant, but we don't trouble to get rid of it.
! * The only case that is really worth worrying about is identical entries,
! * and we handle that well enough with this simple logic.
*/
! foreach(l, root->lateral_info_list)
{
! ljinfo = (LateralJoinInfo *) lfirst(l);
! if (rhs == ljinfo->lateral_rhs &&
bms_is_subset(lhs, ljinfo->lateral_lhs))
- {
- bms_free(lhs);
return;
- }
}
/* Not there, so make a new entry */
ljinfo = makeNode(LateralJoinInfo);
! ljinfo->lateral_rhs = rhs;
! ljinfo->lateral_lhs = lhs;
root->lateral_info_list = lappend(root->lateral_info_list, ljinfo);
}
--- 465,558 ----
}
}
}
+
+ /*
+ * Now check for lateral references within PlaceHolderVars. Unlike the
+ * case for unflattened LATERAL RTEs, the referencing location could be a
+ * join.
+ */
+ foreach(lc, root->placeholder_list)
+ {
+ PlaceHolderInfo *phinfo = (PlaceHolderInfo *) lfirst(lc);
+ Relids eval_at = phinfo->ph_eval_at;
+
+ /* if there are lateral refs in it, add them to lateral_vars */
+ if (phinfo->ph_lateral != NULL)
+ {
+ List *vars = pull_var_clause((Node *) phinfo->ph_var->phexpr,
+ PVC_RECURSE_AGGREGATES,
+ PVC_INCLUDE_PLACEHOLDERS);
+ ListCell *lc2;
+
+ foreach(lc2, vars)
+ {
+ Node *node = (Node *) lfirst(lc2);
+
+ if (IsA(node, Var))
+ {
+ Var *var = (Var *) node;
+
+ if (!bms_is_member(var->varno, eval_at))
+ add_lateral_info(root,
+ bms_make_singleton(var->varno),
+ eval_at);
+ }
+ else if (IsA(node, PlaceHolderVar))
+ {
+ PlaceHolderVar *other_phv = (PlaceHolderVar *) node;
+ PlaceHolderInfo *other_phi;
+
+ other_phi = find_placeholder_info(root, other_phv,
+ false);
+ if (!bms_is_subset(other_phi->ph_eval_at, eval_at))
+ add_lateral_info(root, other_phi->ph_eval_at, eval_at);
+ }
+ else
+ Assert(false);
+ }
+
+ list_free(vars);
+ }
+ }
}
/*
* add_lateral_info
* Add a LateralJoinInfo to root->lateral_info_list, if needed
*
! * We suppress redundant list entries. The passed Relids are copied if saved.
*/
static void
! add_lateral_info(PlannerInfo *root, Relids lhs, Relids rhs)
{
LateralJoinInfo *ljinfo;
! ListCell *lc;
! /* Sanity-check the input */
! Assert(!bms_is_empty(lhs));
! Assert(!bms_is_empty(rhs));
! Assert(!bms_overlap(lhs, rhs));
/*
! * The input is redundant if it has the same RHS and an LHS that is a
! * subset of an existing entry's. If an existing entry has the same RHS
! * and an LHS that is a subset of the new one, it's redundant, but we
! * don't trouble to get rid of it. The only case that is really worth
! * worrying about is identical entries, and we handle that well enough
! * with this simple logic.
*/
! foreach(lc, root->lateral_info_list)
{
! ljinfo = (LateralJoinInfo *) lfirst(lc);
! if (bms_equal(rhs, ljinfo->lateral_rhs) &&
bms_is_subset(lhs, ljinfo->lateral_lhs))
return;
}
/* Not there, so make a new entry */
ljinfo = makeNode(LateralJoinInfo);
! ljinfo->lateral_lhs = bms_copy(lhs);
! ljinfo->lateral_rhs = bms_copy(rhs);
root->lateral_info_list = lappend(root->lateral_info_list, ljinfo);
}
diff --git a/src/backend/optimizer/plan/planmain.c b/src/backend/optimizer/plan/planmain.c
index 42a98945a38d07972ee2e8b5de16b3639196eebd..284929f125e9d8c6eef502d0e112321e65cf5bbf 100644
*** a/src/backend/optimizer/plan/planmain.c
--- b/src/backend/optimizer/plan/planmain.c
*************** query_planner(PlannerInfo *root, List *t
*** 176,187 ****
joinlist = deconstruct_jointree(root);
/*
- * Create the LateralJoinInfo list now that we have finalized
- * PlaceHolderVar eval levels.
- */
- create_lateral_join_info(root);
-
- /*
* Reconsider any postponed outer-join quals now that we have built up
* equivalence classes. (This could result in further additions or
* mergings of classes.)
--- 176,181 ----
*************** query_planner(PlannerInfo *root, List *t
*** 226,231 ****
--- 220,232 ----
add_placeholders_to_base_rels(root);
/*
+ * Create the LateralJoinInfo list now that we have finalized
+ * PlaceHolderVar eval levels and made any necessary additions to the
+ * lateral_vars lists for lateral references within PlaceHolderVars.
+ */
+ create_lateral_join_info(root);
+
+ /*
* We should now have size estimates for every actual table involved in
* the query, and we also know which if any have been deleted from the
* query by join removal; so we can compute total_table_pages.
diff --git a/src/backend/optimizer/prep/prepjointree.c b/src/backend/optimizer/prep/prepjointree.c
index 52842931ec5552c04fdec41e013c36e1897fd4e4..1178b0fc99680d7677efdb8c9a7c0d3521772fe7 100644
*** a/src/backend/optimizer/prep/prepjointree.c
--- b/src/backend/optimizer/prep/prepjointree.c
*************** typedef struct pullup_replace_vars_conte
*** 41,46 ****
--- 41,48 ----
PlannerInfo *root;
List *targetlist; /* tlist of subquery being pulled up */
RangeTblEntry *target_rte; /* RTE of subquery */
+ Relids relids; /* relids within subquery, as numbered after
+ * pullup (set only if target_rte->lateral) */
bool *outer_hasSubLinks; /* -> outer query's hasSubLinks */
int varno; /* varno of subquery */
bool need_phvs; /* do we need PlaceHolderVars? */
*************** pull_up_simple_subquery(PlannerInfo *roo
*** 884,897 ****
/*
* The subquery's targetlist items are now in the appropriate form to
* insert into the top query, but if we are under an outer join then
! * non-nullable items may have to be turned into PlaceHolderVars. If we
! * are dealing with an appendrel member then anything that's not a simple
! * Var has to be turned into a PlaceHolderVar. Set up appropriate context
! * data for pullup_replace_vars.
*/
rvcontext.root = root;
rvcontext.targetlist = subquery->targetList;
rvcontext.target_rte = rte;
rvcontext.outer_hasSubLinks = &parse->hasSubLinks;
rvcontext.varno = varno;
rvcontext.need_phvs = (lowest_nulling_outer_join != NULL ||
--- 886,904 ----
/*
* The subquery's targetlist items are now in the appropriate form to
* insert into the top query, but if we are under an outer join then
! * non-nullable items and lateral references may have to be turned into
! * PlaceHolderVars. If we are dealing with an appendrel member then
! * anything that's not a simple Var has to be turned into a
! * PlaceHolderVar. Set up required context data for pullup_replace_vars.
*/
rvcontext.root = root;
rvcontext.targetlist = subquery->targetList;
rvcontext.target_rte = rte;
+ if (rte->lateral)
+ rvcontext.relids = get_relids_in_jointree((Node *) subquery->jointree,
+ true);
+ else /* won't need relids */
+ rvcontext.relids = NULL;
rvcontext.outer_hasSubLinks = &parse->hasSubLinks;
rvcontext.varno = varno;
rvcontext.need_phvs = (lowest_nulling_outer_join != NULL ||
*************** pullup_replace_vars_callback(Var *var,
*** 1674,1681 ****
if (newnode && IsA(newnode, Var) &&
((Var *) newnode)->varlevelsup == 0)
{
! /* Simple Vars always escape being wrapped */
! wrap = false;
}
else if (newnode && IsA(newnode, PlaceHolderVar) &&
((PlaceHolderVar *) newnode)->phlevelsup == 0)
--- 1681,1698 ----
if (newnode && IsA(newnode, Var) &&
((Var *) newnode)->varlevelsup == 0)
{
! /*
! * Simple Vars always escape being wrapped, unless they are
! * lateral references to something outside the subquery being
! * pulled up. (Even then, we could omit the PlaceHolderVar if
! * the referenced rel is under the same lowest outer join, but
! * it doesn't seem worth the trouble to check that.)
! */
! if (rcon->target_rte->lateral &&
! !bms_is_member(((Var *) newnode)->varno, rcon->relids))
! wrap = true;
! else
! wrap = false;
}
else if (newnode && IsA(newnode, PlaceHolderVar) &&
((PlaceHolderVar *) newnode)->phlevelsup == 0)
*************** pullup_replace_vars_callback(Var *var,
*** 1691,1699 ****
else
{
/*
! * If it contains a Var of current level, and does not contain
! * any non-strict constructs, then it's certainly nullable so
! * we don't need to insert a PlaceHolderVar.
*
* This analysis could be tighter: in particular, a non-strict
* construct hidden within a lower-level PlaceHolderVar is not
--- 1708,1717 ----
else
{
/*
! * If it contains a Var of the subquery being pulled up, and
! * does not contain any non-strict constructs, then it's
! * certainly nullable so we don't need to insert a
! * PlaceHolderVar.
*
* This analysis could be tighter: in particular, a non-strict
* construct hidden within a lower-level PlaceHolderVar is not
*************** pullup_replace_vars_callback(Var *var,
*** 1702,1709 ****
*
* Note: in future maybe we should insert a PlaceHolderVar
* anyway, if the tlist item is expensive to evaluate?
*/
! if (contain_vars_of_level((Node *) newnode, 0) &&
!contain_nonstrict_functions((Node *) newnode))
{
/* No wrap needed */
--- 1720,1733 ----
*
* Note: in future maybe we should insert a PlaceHolderVar
* anyway, if the tlist item is expensive to evaluate?
+ *
+ * For a LATERAL subquery, we have to check the actual var
+ * membership of the node, but if it's non-lateral then any
+ * level-zero var must belong to the subquery.
*/
! if ((rcon->target_rte->lateral ?
! bms_overlap(pull_varnos((Node *) newnode), rcon->relids) :
! contain_vars_of_level((Node *) newnode, 0)) &&
!contain_nonstrict_functions((Node *) newnode))
{
/* No wrap needed */
diff --git a/src/backend/optimizer/util/placeholder.c b/src/backend/optimizer/util/placeholder.c
index c2ff2229e2619355019d4d4f9d03c9b0c5ead0f2..b6f198b712137045a8fb88fa895609983987deca 100644
*** a/src/backend/optimizer/util/placeholder.c
--- b/src/backend/optimizer/util/placeholder.c
*************** find_placeholder_info(PlannerInfo *root,
*** 70,75 ****
--- 70,76 ----
bool create_new_ph)
{
PlaceHolderInfo *phinfo;
+ Relids rels_used;
ListCell *lc;
/* if this ever isn't true, we'd need to be able to look in parent lists */
*************** find_placeholder_info(PlannerInfo *root,
*** 90,96 ****
phinfo->phid = phv->phid;
phinfo->ph_var = copyObject(phv);
! phinfo->ph_eval_at = pull_varnos((Node *) phv);
/* ph_eval_at may change later, see update_placeholder_eval_levels */
phinfo->ph_needed = NULL; /* initially it's unused */
phinfo->ph_may_need = NULL;
--- 91,114 ----
phinfo->phid = phv->phid;
phinfo->ph_var = copyObject(phv);
!
! /*
! * Any referenced rels that are outside the PHV's syntactic scope are
! * LATERAL references, which should be included in ph_lateral but not in
! * ph_eval_at. If no referenced rels are within the syntactic scope,
! * force evaluation at the syntactic location.
! */
! rels_used = pull_varnos((Node *) phv->phexpr);
! phinfo->ph_lateral = bms_difference(rels_used, phv->phrels);
! if (bms_is_empty(phinfo->ph_lateral))
! phinfo->ph_lateral = NULL; /* make it exactly NULL if empty */
! phinfo->ph_eval_at = bms_int_members(rels_used, phv->phrels);
! /* If no contained vars, force evaluation at syntactic location */
! if (bms_is_empty(phinfo->ph_eval_at))
! {
! phinfo->ph_eval_at = bms_copy(phv->phrels);
! Assert(!bms_is_empty(phinfo->ph_eval_at));
! }
/* ph_eval_at may change later, see update_placeholder_eval_levels */
phinfo->ph_needed = NULL; /* initially it's unused */
phinfo->ph_may_need = NULL;
*************** void
*** 250,257 ****
mark_placeholder_maybe_needed(PlannerInfo *root, PlaceHolderInfo *phinfo,
Relids relids)
{
- Relids est_eval_level;
-
/* Mark the PHV as possibly needed at the given syntactic level */
phinfo->ph_may_need = bms_add_members(phinfo->ph_may_need, relids);
--- 268,273 ----
*************** mark_placeholder_maybe_needed(PlannerInf
*** 260,277 ****
* lower-level PHVs. We need to get those into the PlaceHolderInfo list,
* but they aren't going to be needed where the outer PHV is referenced.
* Rather, they'll be needed where the outer PHV is evaluated. We can
! * estimate that conservatively as the syntactic location of the PHV's
! * expression, but not less than the level of any Vars it contains.
! * (Normally the Vars would come from below the syntactic location anyway,
! * but this might not be true if the PHV contains any LATERAL references.)
*/
- est_eval_level = bms_union(phinfo->ph_var->phrels, phinfo->ph_eval_at);
-
- /* Now recurse to take care of any such PHVs */
mark_placeholders_in_expr(root, (Node *) phinfo->ph_var->phexpr,
! est_eval_level);
!
! bms_free(est_eval_level);
}
/*
--- 276,286 ----
* lower-level PHVs. We need to get those into the PlaceHolderInfo list,
* but they aren't going to be needed where the outer PHV is referenced.
* Rather, they'll be needed where the outer PHV is evaluated. We can
! * estimate that (conservatively) as the syntactic location of the PHV's
! * expression. Recurse to take care of any such PHVs.
*/
mark_placeholders_in_expr(root, (Node *) phinfo->ph_var->phexpr,
! phinfo->ph_var->phrels);
}
/*
*************** update_placeholder_eval_levels(PlannerIn
*** 361,366 ****
--- 370,378 ----
}
} while (found_some);
+ /* Can't move the PHV's eval_at level above its syntactic level */
+ Assert(bms_is_subset(eval_at, syn_level));
+
phinfo->ph_eval_at = eval_at;
}
}
*************** update_placeholder_eval_levels(PlannerIn
*** 371,381 ****
*
* This is called after we've finished determining the eval_at levels for
* all placeholders. We need to make sure that all vars and placeholders
! * needed to evaluate each placeholder will be available at the join level
! * where the evaluation will be done. Note that this loop can have
! * side-effects on the ph_needed sets of other PlaceHolderInfos; that's okay
! * because we don't examine ph_needed here, so there are no ordering issues
! * to worry about.
*/
void
fix_placeholder_input_needed_levels(PlannerInfo *root)
--- 383,396 ----
*
* This is called after we've finished determining the eval_at levels for
* all placeholders. We need to make sure that all vars and placeholders
! * needed to evaluate each placeholder will be available at the scan or join
! * level where the evaluation will be done. (It might seem that scan-level
! * evaluations aren't interesting, but that's not so: a LATERAL reference
! * within a placeholder's expression needs to cause the referenced var or
! * placeholder to be marked as needed in the scan where it's evaluated.)
! * Note that this loop can have side-effects on the ph_needed sets of other
! * PlaceHolderInfos; that's okay because we don't examine ph_needed here, so
! * there are no ordering issues to worry about.
*/
void
fix_placeholder_input_needed_levels(PlannerInfo *root)
*************** fix_placeholder_input_needed_levels(Plan
*** 385,411 ****
foreach(lc, root->placeholder_list)
{
PlaceHolderInfo *phinfo = (PlaceHolderInfo *) lfirst(lc);
! Relids eval_at = phinfo->ph_eval_at;
!
! /* No work unless it'll be evaluated above baserel level */
! if (bms_membership(eval_at) == BMS_MULTIPLE)
! {
! List *vars = pull_var_clause((Node *) phinfo->ph_var->phexpr,
! PVC_RECURSE_AGGREGATES,
! PVC_INCLUDE_PLACEHOLDERS);
! add_vars_to_targetlist(root, vars, eval_at, false);
! list_free(vars);
! }
}
}
/*
* add_placeholders_to_base_rels
! * Add any required PlaceHolderVars to base rels' targetlists.
*
* If any placeholder can be computed at a base rel and is needed above it,
! * add it to that rel's targetlist. This might look like it could be merged
* with fix_placeholder_input_needed_levels, but it must be separate because
* join removal happens in between, and can change the ph_eval_at sets. There
* is essentially the same logic in add_placeholders_to_joinrel, but we can't
--- 400,422 ----
foreach(lc, root->placeholder_list)
{
PlaceHolderInfo *phinfo = (PlaceHolderInfo *) lfirst(lc);
! List *vars = pull_var_clause((Node *) phinfo->ph_var->phexpr,
! PVC_RECURSE_AGGREGATES,
! PVC_INCLUDE_PLACEHOLDERS);
! add_vars_to_targetlist(root, vars, phinfo->ph_eval_at, false);
! list_free(vars);
}
}
/*
* add_placeholders_to_base_rels
! * Add any required PlaceHolderVars to base rels' targetlists, and
! * update lateral_vars lists for lateral references contained in them.
*
* If any placeholder can be computed at a base rel and is needed above it,
! * add it to that rel's targetlist, and add any lateral references it requires
! * to the rel's lateral_vars list. This might look like it could be merged
* with fix_placeholder_input_needed_levels, but it must be separate because
* join removal happens in between, and can change the ph_eval_at sets. There
* is essentially the same logic in add_placeholders_to_joinrel, but we can't
*************** add_placeholders_to_base_rels(PlannerInf
*** 421,434 ****
PlaceHolderInfo *phinfo = (PlaceHolderInfo *) lfirst(lc);
Relids eval_at = phinfo->ph_eval_at;
! if (bms_membership(eval_at) == BMS_SINGLETON &&
! bms_nonempty_difference(phinfo->ph_needed, eval_at))
{
int varno = bms_singleton_member(eval_at);
RelOptInfo *rel = find_base_rel(root, varno);
! rel->reltargetlist = lappend(rel->reltargetlist,
! copyObject(phinfo->ph_var));
}
}
}
--- 432,483 ----
PlaceHolderInfo *phinfo = (PlaceHolderInfo *) lfirst(lc);
Relids eval_at = phinfo->ph_eval_at;
! if (bms_membership(eval_at) == BMS_SINGLETON)
{
int varno = bms_singleton_member(eval_at);
RelOptInfo *rel = find_base_rel(root, varno);
! /* add it to reltargetlist if needed above the rel scan level */
! if (bms_nonempty_difference(phinfo->ph_needed, eval_at))
! rel->reltargetlist = lappend(rel->reltargetlist,
! copyObject(phinfo->ph_var));
! /* if there are lateral refs in it, add them to lateral_vars */
! if (phinfo->ph_lateral != NULL)
! {
! List *vars = pull_var_clause((Node *) phinfo->ph_var->phexpr,
! PVC_RECURSE_AGGREGATES,
! PVC_INCLUDE_PLACEHOLDERS);
! ListCell *lc2;
!
! foreach(lc2, vars)
! {
! Node *node = (Node *) lfirst(lc2);
!
! if (IsA(node, Var))
! {
! Var *var = (Var *) node;
!
! if (var->varno != varno)
! rel->lateral_vars = lappend(rel->lateral_vars,
! var);
! }
! else if (IsA(node, PlaceHolderVar))
! {
! PlaceHolderVar *other_phv = (PlaceHolderVar *) node;
! PlaceHolderInfo *other_phi;
!
! other_phi = find_placeholder_info(root, other_phv,
! false);
! if (!bms_is_subset(other_phi->ph_eval_at, eval_at))
! rel->lateral_vars = lappend(rel->lateral_vars,
! other_phv);
! }
! else
! Assert(false);
! }
!
! list_free(vars);
! }
}
}
}
diff --git a/src/backend/optimizer/util/var.c b/src/backend/optimizer/util/var.c
index 7eaf8d27bf08bf5dd1776d203876adb8396c73b3..adde10a85ab05f031e0a792f0c27cebdeabea9d2 100644
*** a/src/backend/optimizer/util/var.c
--- b/src/backend/optimizer/util/var.c
*************** pull_varnos_walker(Node *node, pull_varn
*** 161,168 ****
if (IsA(node, PlaceHolderVar))
{
/*
! * Normally, we can just take the varnos in the contained expression.
! * But if it is variable-free, use the PHV's syntactic relids.
*/
PlaceHolderVar *phv = (PlaceHolderVar *) node;
pull_varnos_context subcontext;
--- 161,173 ----
if (IsA(node, PlaceHolderVar))
{
/*
! * A PlaceHolderVar acts as a variable of its syntactic scope, or
! * lower than that if it references only a subset of the rels in its
! * syntactic scope. It might also contain lateral references, but we
! * should ignore such references when computing the set of varnos in
! * an expression tree. Also, if the PHV contains no variables within
! * its syntactic scope, it will be forced to be evaluated exactly at
! * the syntactic scope, so take that as the relid set.
*/
PlaceHolderVar *phv = (PlaceHolderVar *) node;
pull_varnos_context subcontext;
*************** pull_varnos_walker(Node *node, pull_varn
*** 170,181 ****
subcontext.varnos = NULL;
subcontext.sublevels_up = context->sublevels_up;
(void) pull_varnos_walker((Node *) phv->phexpr, &subcontext);
!
! if (bms_is_empty(subcontext.varnos) &&
! phv->phlevelsup == context->sublevels_up)
! context->varnos = bms_add_members(context->varnos, phv->phrels);
! else
! context->varnos = bms_join(context->varnos, subcontext.varnos);
return false;
}
if (IsA(node, Query))
--- 175,189 ----
subcontext.varnos = NULL;
subcontext.sublevels_up = context->sublevels_up;
(void) pull_varnos_walker((Node *) phv->phexpr, &subcontext);
! if (phv->phlevelsup == context->sublevels_up)
! {
! subcontext.varnos = bms_int_members(subcontext.varnos,
! phv->phrels);
! if (bms_is_empty(subcontext.varnos))
! context->varnos = bms_add_members(context->varnos,
! phv->phrels);
! }
! context->varnos = bms_join(context->varnos, subcontext.varnos);
return false;
}
if (IsA(node, Query))
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index c0a636b9d7c9f6dc4f4367188741af0d07cef3c3..7a868079d87e257cd17a21c83280406dacdd789e 100644
*** a/src/include/nodes/relation.h
--- b/src/include/nodes/relation.h
*************** typedef struct SpecialJoinInfo
*** 1344,1373 ****
/*
* "Lateral join" info.
*
! * Lateral references in subqueries constrain the join order in a way that's
! * somewhat like outer joins, though different in detail. We construct one or
! * more LateralJoinInfos for each RTE with lateral references, and add them to
! * the PlannerInfo node's lateral_info_list.
*
! * lateral_rhs is the relid of a baserel with lateral references, and
! * lateral_lhs is a set of relids of baserels it references, all of which
! * must be present on the LHS to compute a parameter needed by the RHS.
! * Typically, lateral_lhs is a singleton, but it can include multiple rels
! * if the RHS references a PlaceHolderVar with a multi-rel ph_eval_at level.
! * We disallow joining to only part of the LHS in such cases, since that would
! * result in a join tree with no convenient place to compute the PHV.
*
* When an appendrel contains lateral references (eg "LATERAL (SELECT x.col1
* UNION ALL SELECT y.col2)"), the LateralJoinInfos reference the parent
* baserel not the member otherrels, since it is the parent relid that is
* considered for joining purposes.
*/
typedef struct LateralJoinInfo
{
NodeTag type;
! Index lateral_rhs; /* a baserel containing lateral refs */
! Relids lateral_lhs; /* some base relids it references */
} LateralJoinInfo;
/*
--- 1344,1381 ----
/*
* "Lateral join" info.
*
! * Lateral references constrain the join order in a way that's somewhat like
! * outer joins, though different in detail. We construct a LateralJoinInfo
! * for each lateral cross-reference, placing them in the PlannerInfo node's
! * lateral_info_list.
*
! * For unflattened LATERAL RTEs, we generate LateralJoinInfo(s) in which
! * lateral_rhs is the relid of the LATERAL baserel, and lateral_lhs is a set
! * of relids of baserels it references, all of which must be present on the
! * LHS to compute a parameter needed by the RHS. Typically, lateral_lhs is
! * a singleton, but it can include multiple rels if the RHS references a
! * PlaceHolderVar with a multi-rel ph_eval_at level. We disallow joining to
! * only part of the LHS in such cases, since that would result in a join tree
! * with no convenient place to compute the PHV.
*
* When an appendrel contains lateral references (eg "LATERAL (SELECT x.col1
* UNION ALL SELECT y.col2)"), the LateralJoinInfos reference the parent
* baserel not the member otherrels, since it is the parent relid that is
* considered for joining purposes.
+ *
+ * If any LATERAL RTEs were flattened into the parent query, it is possible
+ * that the query now contains PlaceHolderVars containing lateral references,
+ * representing expressions that need to be evaluated at particular spots in
+ * the jointree but contain lateral references to Vars from elsewhere. These
+ * give rise to LateralJoinInfos in which lateral_rhs is the evaluation point
+ * of a PlaceHolderVar and lateral_lhs is the set of lateral rels it needs.
*/
typedef struct LateralJoinInfo
{
NodeTag type;
! Relids lateral_lhs; /* rels needed to compute a lateral value */
! Relids lateral_rhs; /* rel where lateral value is needed */
} LateralJoinInfo;
/*
*************** typedef struct AppendRelInfo
*** 1465,1470 ****
--- 1473,1482 ----
* then allow it to bubble up like a Var until the ph_needed join level.
* ph_needed has the same definition as attr_needed for a regular Var.
*
+ * The PlaceHolderVar's expression might contain LATERAL references to vars
+ * coming from outside its syntactic scope. If so, those rels are *not*
+ * included in ph_eval_at, but they are recorded in ph_lateral.
+ *
* ph_may_need is an initial estimate of ph_needed, formed using the
* syntactic locations of references to the PHV. We need this in order to
* determine whether the PHV reference forces a join ordering constraint:
*************** typedef struct PlaceHolderInfo
*** 1490,1495 ****
--- 1502,1508 ----
Index phid; /* ID for PH (unique within planner run) */
PlaceHolderVar *ph_var; /* copy of PlaceHolderVar tree */
Relids ph_eval_at; /* lowest level we can evaluate value at */
+ Relids ph_lateral; /* relids of contained lateral refs, if any */
Relids ph_needed; /* highest level the value is needed at */
Relids ph_may_need; /* highest level it might be needed at */
int32 ph_width; /* estimated attribute width */
diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out
index 31c2a320a6d1aa47ef355c6901266d7a1aeb518b..afa2a13326408a7b10619408f713c58f7cc748c2 100644
*** a/src/test/regress/expected/join.out
--- b/src/test/regress/expected/join.out
*************** select v.* from
*** 3577,3582 ****
--- 3577,3721 ----
-4567890123456789 |
(20 rows)
+ explain (verbose, costs off)
+ select * from
+ int8_tbl a left join
+ lateral (select *, a.q2 as x from int8_tbl b) ss on a.q2 = ss.q1;
+ QUERY PLAN
+ ------------------------------------------
+ Nested Loop Left Join
+ Output: a.q1, a.q2, b.q1, b.q2, (a.q2)
+ -> Seq Scan on public.int8_tbl a
+ Output: a.q1, a.q2
+ -> Seq Scan on public.int8_tbl b
+ Output: b.q1, b.q2, a.q2
+ Filter: (a.q2 = b.q1)
+ (7 rows)
+
+ select * from
+ int8_tbl a left join
+ lateral (select *, a.q2 as x from int8_tbl b) ss on a.q2 = ss.q1;
+ q1 | q2 | q1 | q2 | x
+ ------------------+-------------------+------------------+-------------------+------------------
+ 123 | 456 | | |
+ 123 | 4567890123456789 | 4567890123456789 | 123 | 4567890123456789
+ 123 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789
+ 123 | 4567890123456789 | 4567890123456789 | -4567890123456789 | 4567890123456789
+ 4567890123456789 | 123 | 123 | 456 | 123
+ 4567890123456789 | 123 | 123 | 4567890123456789 | 123
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 123 | 4567890123456789
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | -4567890123456789 | 4567890123456789
+ 4567890123456789 | -4567890123456789 | | |
+ (10 rows)
+
+ explain (verbose, costs off)
+ select * from
+ int8_tbl a left join
+ lateral (select *, coalesce(a.q2, 42) as x from int8_tbl b) ss on a.q2 = ss.q1;
+ QUERY PLAN
+ ----------------------------------------------------------------
+ Nested Loop Left Join
+ Output: a.q1, a.q2, b.q1, b.q2, (COALESCE(a.q2, 42::bigint))
+ -> Seq Scan on public.int8_tbl a
+ Output: a.q1, a.q2
+ -> Seq Scan on public.int8_tbl b
+ Output: b.q1, b.q2, COALESCE(a.q2, 42::bigint)
+ Filter: (a.q2 = b.q1)
+ (7 rows)
+
+ select * from
+ int8_tbl a left join
+ lateral (select *, coalesce(a.q2, 42) as x from int8_tbl b) ss on a.q2 = ss.q1;
+ q1 | q2 | q1 | q2 | x
+ ------------------+-------------------+------------------+-------------------+------------------
+ 123 | 456 | | |
+ 123 | 4567890123456789 | 4567890123456789 | 123 | 4567890123456789
+ 123 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789
+ 123 | 4567890123456789 | 4567890123456789 | -4567890123456789 | 4567890123456789
+ 4567890123456789 | 123 | 123 | 456 | 123
+ 4567890123456789 | 123 | 123 | 4567890123456789 | 123
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 123 | 4567890123456789
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | -4567890123456789 | 4567890123456789
+ 4567890123456789 | -4567890123456789 | | |
+ (10 rows)
+
+ -- lateral reference in a PlaceHolderVar evaluated at join level
+ explain (verbose, costs off)
+ select * from
+ int8_tbl a left join lateral
+ (select b.q1 as bq1, c.q1 as cq1, least(a.q1,b.q1,c.q1) from
+ int8_tbl b cross join int8_tbl c) ss
+ on a.q2 = ss.bq1;
+ QUERY PLAN
+ -------------------------------------------------------------
+ Nested Loop Left Join
+ Output: a.q1, a.q2, b.q1, c.q1, (LEAST(a.q1, b.q1, c.q1))
+ -> Seq Scan on public.int8_tbl a
+ Output: a.q1, a.q2
+ -> Nested Loop
+ Output: b.q1, c.q1, LEAST(a.q1, b.q1, c.q1)
+ Join Filter: (a.q2 = b.q1)
+ -> Seq Scan on public.int8_tbl b
+ Output: b.q1, b.q2
+ -> Materialize
+ Output: c.q1
+ -> Seq Scan on public.int8_tbl c
+ Output: c.q1
+ (13 rows)
+
+ select * from
+ int8_tbl a left join lateral
+ (select b.q1 as bq1, c.q1 as cq1, least(a.q1,b.q1,c.q1) from
+ int8_tbl b cross join int8_tbl c) ss
+ on a.q2 = ss.bq1;
+ q1 | q2 | bq1 | cq1 | least
+ ------------------+-------------------+------------------+------------------+------------------
+ 123 | 456 | | |
+ 123 | 4567890123456789 | 4567890123456789 | 123 | 123
+ 123 | 4567890123456789 | 4567890123456789 | 123 | 123
+ 123 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 123
+ 123 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 123
+ 123 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 123
+ 123 | 4567890123456789 | 4567890123456789 | 123 | 123
+ 123 | 4567890123456789 | 4567890123456789 | 123 | 123
+ 123 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 123
+ 123 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 123
+ 123 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 123
+ 123 | 4567890123456789 | 4567890123456789 | 123 | 123
+ 123 | 4567890123456789 | 4567890123456789 | 123 | 123
+ 123 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 123
+ 123 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 123
+ 123 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 123
+ 4567890123456789 | 123 | 123 | 123 | 123
+ 4567890123456789 | 123 | 123 | 123 | 123
+ 4567890123456789 | 123 | 123 | 4567890123456789 | 123
+ 4567890123456789 | 123 | 123 | 4567890123456789 | 123
+ 4567890123456789 | 123 | 123 | 4567890123456789 | 123
+ 4567890123456789 | 123 | 123 | 123 | 123
+ 4567890123456789 | 123 | 123 | 123 | 123
+ 4567890123456789 | 123 | 123 | 4567890123456789 | 123
+ 4567890123456789 | 123 | 123 | 4567890123456789 | 123
+ 4567890123456789 | 123 | 123 | 4567890123456789 | 123
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 123 | 123
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 123 | 123
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 123 | 123
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 123 | 123
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 123 | 123
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 123 | 123
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789
+ 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789 | 4567890123456789
+ 4567890123456789 | -4567890123456789 | | |
+ (42 rows)
+
-- case requiring nested PlaceHolderVars
explain (verbose, costs off)
select * from
*************** select * from
*** 3595,3601 ****
Output: c.q1, c.q2, a.q1, a.q2, b.q1, d.q1, (COALESCE(b.q2, 42::bigint)), (COALESCE((COALESCE(b.q2, 42::bigint)), d.q2))
Hash Cond: (d.q1 = c.q2)
-> Nested Loop
! Output: a.q1, a.q2, b.q1, d.q1, (COALESCE(b.q2, 42::bigint)), COALESCE((COALESCE(b.q2, 42::bigint)), d.q2)
-> Hash Left Join
Output: a.q1, a.q2, b.q1, (COALESCE(b.q2, 42::bigint))
Hash Cond: (a.q2 = b.q1)
--- 3734,3740 ----
Output: c.q1, c.q2, a.q1, a.q2, b.q1, d.q1, (COALESCE(b.q2, 42::bigint)), (COALESCE((COALESCE(b.q2, 42::bigint)), d.q2))
Hash Cond: (d.q1 = c.q2)
-> Nested Loop
! Output: a.q1, a.q2, b.q1, d.q1, (COALESCE(b.q2, 42::bigint)), (COALESCE((COALESCE(b.q2, 42::bigint)), d.q2))
-> Hash Left Join
Output: a.q1, a.q2, b.q1, (COALESCE(b.q2, 42::bigint))
Hash Cond: (a.q2 = b.q1)
*************** select * from
*** 3605,3621 ****
Output: b.q1, (COALESCE(b.q2, 42::bigint))
-> Seq Scan on public.int8_tbl b
Output: b.q1, COALESCE(b.q2, 42::bigint)
! -> Materialize
! Output: d.q1, d.q2
! -> Seq Scan on public.int8_tbl d
! Output: d.q1, d.q2
-> Hash
Output: c.q1, c.q2
-> Seq Scan on public.int8_tbl c
Output: c.q1, c.q2
-> Result
Output: (COALESCE((COALESCE(b.q2, 42::bigint)), d.q2))
! (26 rows)
-- test some error cases where LATERAL should have been used but wasn't
select f1,g from int4_tbl a, (select f1 as g) ss;
--- 3744,3758 ----
Output: b.q1, (COALESCE(b.q2, 42::bigint))
-> Seq Scan on public.int8_tbl b
Output: b.q1, COALESCE(b.q2, 42::bigint)
! -> Seq Scan on public.int8_tbl d
! Output: d.q1, COALESCE((COALESCE(b.q2, 42::bigint)), d.q2)
-> Hash
Output: c.q1, c.q2
-> Seq Scan on public.int8_tbl c
Output: c.q1, c.q2
-> Result
Output: (COALESCE((COALESCE(b.q2, 42::bigint)), d.q2))
! (24 rows)
-- test some error cases where LATERAL should have been used but wasn't
select f1,g from int4_tbl a, (select f1 as g) ss;
diff --git a/src/test/regress/sql/join.sql b/src/test/regress/sql/join.sql
index 656766acd3fbb499ef41da99e63bc79fcea0b108..f6009141d2437341d7a9c42d896977acf1f88743 100644
*** a/src/test/regress/sql/join.sql
--- b/src/test/regress/sql/join.sql
*************** select v.* from
*** 995,1000 ****
--- 995,1028 ----
left join int4_tbl z on z.f1 = x.q2,
lateral (select x.q1,y.q1 from dual union all select x.q2,y.q2 from dual) v(vx,vy);
+ explain (verbose, costs off)
+ select * from
+ int8_tbl a left join
+ lateral (select *, a.q2 as x from int8_tbl b) ss on a.q2 = ss.q1;
+ select * from
+ int8_tbl a left join
+ lateral (select *, a.q2 as x from int8_tbl b) ss on a.q2 = ss.q1;
+ explain (verbose, costs off)
+ select * from
+ int8_tbl a left join
+ lateral (select *, coalesce(a.q2, 42) as x from int8_tbl b) ss on a.q2 = ss.q1;
+ select * from
+ int8_tbl a left join
+ lateral (select *, coalesce(a.q2, 42) as x from int8_tbl b) ss on a.q2 = ss.q1;
+
+ -- lateral reference in a PlaceHolderVar evaluated at join level
+ explain (verbose, costs off)
+ select * from
+ int8_tbl a left join lateral
+ (select b.q1 as bq1, c.q1 as cq1, least(a.q1,b.q1,c.q1) from
+ int8_tbl b cross join int8_tbl c) ss
+ on a.q2 = ss.bq1;
+ select * from
+ int8_tbl a left join lateral
+ (select b.q1 as bq1, c.q1 as cq1, least(a.q1,b.q1,c.q1) from
+ int8_tbl b cross join int8_tbl c) ss
+ on a.q2 = ss.bq1;
+
-- case requiring nested PlaceHolderVars
explain (verbose, costs off)
select * from
I wrote:
So attached is a draft patch for this. It's not complete yet because
there are various comments that are now wrong and need to be updated;
but I think the code is functioning correctly.
Hm, spoke too soon :-(. This query causes an assertion failure, with or
without my draft patch:
select c.*,a.*,ss1.q1,ss2.q1,ss3.* from
int8_tbl c left join (
int8_tbl a left join
(select q1, coalesce(q2,f1) as x from int8_tbl b, int4_tbl b2) ss1
on a.q2 = ss1.q1
cross join
lateral (select q1, coalesce(ss1.x,q2) as y from int8_tbl d) ss2
) on c.q2 = ss2.q1,
lateral (select * from int4_tbl i where ss2.y > f1) ss3;
TRAP: FailedAssertion("!(bms_is_subset(phinfo->ph_needed, phinfo->ph_may_need))", File: "initsplan.c", Line: 213)
What's happening is that distribute_qual_to_rels concludes (correctly)
that the "ss2.y > f1" clause must be postponed until after the nest of
left joins, since those could null ss2.y. So the PlaceHolderVar for
ss2.y is marked as being needed at the topmost join level. However,
find_placeholders_in_jointree had only marked the PHV as being "maybe
needed" to scan the "i" relation, since that's what the syntactic
location of the reference implies. Since we depend on the assumption
that ph_needed is always a subset of ph_may_need, there's an assertion
that fires if that stops being true, and that's what's crashing.
After some thought about this, I'm coming to the conclusion that lateral
references destroy the ph_maybe_needed optimization altogether: we
cannot derive an accurate estimate of where a placeholder will end up in
the final qual distribution, short of essentially doing all the work in
deconstruct_jointree over again. I guess in principle we could repeat
deconstruct_jointree until we had stable estimates of the ph_needed
locations, but that would be expensive and probably would induce a lot
of new planner bugs (since the data structure changes performed during
deconstruct_jointree aren't designed to be backed out easily).
The only place where ph_may_need is actually used is in this bit in
make_outerjoininfo():
/*
* Examine PlaceHolderVars. If a PHV is supposed to be evaluated within
* this join's nullable side, and it may get used above this join, then
* ensure that min_righthand contains the full eval_at set of the PHV.
* This ensures that the PHV actually can be evaluated within the RHS.
* Note that this works only because we should already have determined the
* final eval_at level for any PHV syntactically within this join.
*/
foreach(l, root->placeholder_list)
{
PlaceHolderInfo *phinfo = (PlaceHolderInfo *) lfirst(l);
Relids ph_syn_level = phinfo->ph_var->phrels;
/* Ignore placeholder if it didn't syntactically come from RHS */
if (!bms_is_subset(ph_syn_level, right_rels))
continue;
/* We can also ignore it if it's certainly not used above this join */
/* XXX this test is probably overly conservative */
if (bms_is_subset(phinfo->ph_may_need, min_righthand))
continue;
/* Else, prevent join from being formed before we eval the PHV */
min_righthand = bms_add_members(min_righthand, phinfo->ph_eval_at);
}
Looking at it again, it's not really clear that skipping placeholders in
this way results in very much optimization --- sometimes we can avoid
constraining join order, but how often? I tried diking out the check
on ph_may_need from this loop, and saw no changes in the regression test
results (not that that proves a whole lot about optimization of complex
queries). So I'm pretty tempted to just remove ph_may_need, along with
the machinery that computes it.
Another possibility would be to keep the optimization, but disable it in
queries that use LATERAL. I don't much care for that though --- seems
too Rube Goldbergish, and in any case I have a lot less faith in the
whole concept now than I had before I started digging into this issue.
Thoughts?
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 07/03/2013 08:32 PM, Tom Lane wrote:
Another possibility would be to keep the optimization, but disable it in
queries that use LATERAL. I don't much care for that though --- seems
too Rube Goldbergish, and in any case I have a lot less faith in the
whole concept now than I had before I started digging into this issue.Thoughts?
I noticed EXPLAIN in some regression tests. So if they all pass after
removal of this optimization, it might indicate that it was really
insignificant. But alternatively it may just be a lack of focus on this
feature in the test queries. Digging for (non-LATERAL) queries or rather
patterns where the ph_may_need optimization clearly appears to be
important sounds to me like a good SQL exercise, but I'm afraid I won't
have time for it in the next few days.
//Antonin Houska (Tony)
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 07/04/2013 06:11 PM, Antonin Houska wrote:
On 07/03/2013 08:32 PM, Tom Lane wrote:
Another possibility would be to keep the optimization, but disable it in
queries that use LATERAL. I don't much care for that though --- seems
too Rube Goldbergish, and in any case I have a lot less faith in the
whole concept now than I had before I started digging into this issue.Thoughts?
I noticed EXPLAIN in some regression tests. So if they all pass after
removal of this optimization, it might indicate that it was really
insignificant. But alternatively it may just be a lack of focus on
this feature in the test queries. Digging for (non-LATERAL) queries or
rather patterns where the ph_may_need optimization clearly appears to
be important sounds to me like a good SQL exercise, but I'm afraid I
won't have time for it in the next few days.
I constructed a query that triggers the optimization - see attachment
with comments. (Note that the relid sets are derived from my current
knowledge of the logic. I haven't figured out how to check them easily
in gdb session.)
The intention was that the top-level OJ references LHS of the join below
rather than the RHS. That should increase the likelihood that the PHV
becomes the only obstacle for join commuting. And therefore the
ph_may_need optimization should unblock some combinations that would be
impossible otherwise.
However I could not see the condition
if (bms_is_subset(phinfo->ph_may_need, min_righthand))
continue;
met for the top-level join even though the supposed ph_may_need did not
contain tab1. Then it struck me that min_righthand can be the problem.
So I changed the join clause to reference RHS of j1, hoping that it
should make min_righthand bigger. And that really triggered the condition.
EXPLAIN shows the same plan with or without the ph_may_need
optimization, but that might be data problem (my tables are empty).
More important is the fact that I could only avoid addition of the PHV's
eval_at to min_righthand at the cost of adding the whole j1 join (i.e.
more than just eval_at).
Although the idea behind ph_may_need is clever, I can now imagine that
other techniques of the planner can substitute for it. There might be
examples showing the opposite but such are beyond my imagination.
// Antonin Houska (Tony)
I have couple of questions.
On Wed, Jun 26, 2013 at 1:30 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
I've been studying the bug reported at
/messages/by-id/20130617235236.GA1636@jeremyevans.local
that the planner can do the wrong thing with queries likeSELECT * FROM
i LEFT JOIN LATERAL (SELECT * FROM j WHERE i.n = j.n) j ON true;I think the fundamental problem is that, because the "i.n = j.n" clause
appears syntactically in WHERE, the planner is treating it as if it were
an inner-join clause; but really it ought to be considered a clause of
the upper LEFT JOIN. That is, semantically this query ought to be
equivalent toSELECT * FROM
i LEFT JOIN LATERAL (SELECT * FROM j) j ON i.n = j.n;However, because distribute_qual_to_rels doesn't see the clause as being
attached to the outer join, it's not marked with the correct properties
and ends up getting evaluated in the wrong place (as a "filter" clause
not a "join filter" clause). The bug is masked in the test cases we've
used so far because those cases are designed to let the clause get
pushed down into the scan of the inner relation --- but if it doesn't
get pushed down, it's evaluated the wrong way.After some contemplation, I think that the most practical way to fix
this is for deconstruct_recurse and distribute_qual_to_rels to
effectively move such a qual to the place where it logically belongs;
that is, rather than processing it when we look at the lower WHERE
clause, set it aside for a moment and then add it back when looking at
the ON clause of the appropriate outer join. This should be reasonably
easy to do by keeping a list of "postponed lateral clauses" while we're
scanning the join tree.For there to *be* a unique "appropriate outer join", we need to require
that a LATERAL-using qual clause that's under an outer join contain
lateral references only to the outer side of the nearest enclosing outer
join. There's no such restriction in the spec of course, but we can
make it so by refusing to flatten a sub-select if pulling it up would
result in having a clause in the outer query that violates this rule.
There's already some code in prepjointree.c (around line 1300) that
attempts to enforce this, though now that I look at it again I'm not
sure it's covering all the bases. We may need to extend that check.
Why do we need this restriction? Wouldn't a place (specifically join qual
at such a place) in join tree where all the participating relations are
present, serve as a place where the clause can be applied. E.g. in the query
select * from tab1 left join tab2 t2 using (val) left join lateral (select
val from tab2 where val2 = tab1.val * t2.val) t3 using (val);
Can't we apply (as a join qual) the qual val2 = tab1.val * t2.val at a
place where we are computing join between tab1, t2 and t3?
I'm inclined to process all LATERAL-using qual clauses this way, ie
postpone them till we recurse back up to a place where they can
logically be evaluated. That won't make any real difference when no
outer joins are present, but it will eliminate the ugliness that right
now distribute_qual_to_rels is prevented from sanity-checking the scope
of the references in a qual when LATERAL is present. If we do it like
this, we can resurrect full enforcement of that sanity check, and then
throw an error if any "postponed" quals are left over when we're done
recursing.
Parameterized nested loop join would always be able to evaluate a LATERAL
query. Instead of throwing error, why can't we choose that as the default
strategy whenever we fail to flatten subquery?
Can we put the clause with lateral references at its appropriate place
while flattening the subquery? IMO, that will be cleaner and lesser work
than first pulling the clause and then putting it back again? Right, now,
we do not have that capability in pull_up_subqueries() but given its
recursive structure, it might be easier to do it there.
Thoughts, better ideas?
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
--
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Postgres Database Company
Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> writes:
On Wed, Jun 26, 2013 at 1:30 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
For there to *be* a unique "appropriate outer join", we need to require
that a LATERAL-using qual clause that's under an outer join contain
lateral references only to the outer side of the nearest enclosing outer
join. There's no such restriction in the spec of course, but we can
make it so by refusing to flatten a sub-select if pulling it up would
result in having a clause in the outer query that violates this rule.
There's already some code in prepjointree.c (around line 1300) that
attempts to enforce this, though now that I look at it again I'm not
sure it's covering all the bases. We may need to extend that check.
Why do we need this restriction? Wouldn't a place (specifically join qual
at such a place) in join tree where all the participating relations are
present, serve as a place where the clause can be applied.
No. If you hoist a qual that appears below an outer join to above the
outer join, you get wrong results in general: you might eliminate rows
from the outer side of the join, which a qual from within the inner side
should never be able to do.
select * from tab1 left join tab2 t2 using (val) left join lateral (select
val from tab2 where val2 = tab1.val * t2.val) t3 using (val);
Can't we apply (as a join qual) the qual val2 = tab1.val * t2.val at a
place where we are computing join between tab1, t2 and t3?
This particular example doesn't violate the rule I gave above, since
both tab1 and t2 are on the left side of the join to the lateral
subquery, and the qual doesn't have to get hoisted *past* an outer join,
only to the outer join of {tab1,t2} with {t3}.
I'm inclined to process all LATERAL-using qual clauses this way, ie
postpone them till we recurse back up to a place where they can
logically be evaluated. That won't make any real difference when no
outer joins are present, but it will eliminate the ugliness that right
now distribute_qual_to_rels is prevented from sanity-checking the scope
of the references in a qual when LATERAL is present. If we do it like
this, we can resurrect full enforcement of that sanity check, and then
throw an error if any "postponed" quals are left over when we're done
recursing.
Parameterized nested loop join would always be able to evaluate a LATERAL
query. Instead of throwing error, why can't we choose that as the default
strategy whenever we fail to flatten subquery?
I think you misunderstood. That error would only be a sanity check that
we'd accounted for all qual clauses, it's not something a user should
ever see.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Antonin Houska <antonin.houska@gmail.com> writes:
On 07/04/2013 06:11 PM, Antonin Houska wrote:
On 07/03/2013 08:32 PM, Tom Lane wrote:
Another possibility would be to keep the optimization, but disable it in
queries that use LATERAL. I don't much care for that though --- seems
too Rube Goldbergish, and in any case I have a lot less faith in the
whole concept now than I had before I started digging into this issue.
I constructed a query that triggers the optimization - see attachment
with comments.
Thanks for poking at this.
EXPLAIN shows the same plan with or without the ph_may_need
optimization, but that might be data problem (my tables are empty).
Yeah, I didn't have much luck getting a different plan even with data in
the tables. What you'd need for this to be important would be for a join
order that's precluded without the ph_may_need logic to be significantly
better than the join orders that are still allowed. While that's
certainly within the realm of possibility, the difficulty of triggering
the case at all reinforces my feeling that this optimization isn't worth
bothering with. For the moment I'm just going to take it out.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Some time ago, I wrote:
I've been studying the bug reported at
/messages/by-id/20130617235236.GA1636@jeremyevans.local
...
After some contemplation, I think that the most practical way to fix
this is for deconstruct_recurse and distribute_qual_to_rels to
effectively move such a qual to the place where it logically belongs;
that is, rather than processing it when we look at the lower WHERE
clause, set it aside for a moment and then add it back when looking at
the ON clause of the appropriate outer join. This should be reasonably
easy to do by keeping a list of "postponed lateral clauses" while we're
scanning the join tree.
Here's a draft patch for this. The comments need a bit more work
probably, but barring objection I want to push this in before this
afternoon's 9.3rc1 wrap.
regards, tom lane
Attachments:
postpone-lateral-quals.patchtext/x-diff; charset=us-ascii; name=postpone-lateral-quals.patchDownload
diff --git a/src/backend/optimizer/plan/initsplan.c b/src/backend/optimizer/plan/initsplan.c
index 98f601c..e055088 100644
*** a/src/backend/optimizer/plan/initsplan.c
--- b/src/backend/optimizer/plan/initsplan.c
*************** int from_collapse_limit;
*** 36,47 ****
int join_collapse_limit;
static void extract_lateral_references(PlannerInfo *root, RelOptInfo *brel,
Index rtindex);
static void add_lateral_info(PlannerInfo *root, Relids lhs, Relids rhs);
static List *deconstruct_recurse(PlannerInfo *root, Node *jtnode,
bool below_outer_join,
! Relids *qualscope, Relids *inner_join_rels);
static SpecialJoinInfo *make_outerjoininfo(PlannerInfo *root,
Relids left_rels, Relids right_rels,
Relids inner_join_rels,
--- 36,56 ----
int join_collapse_limit;
+ /* Elements of the postponed_qual_list used during deconstruct_recurse */
+ typedef struct PostponedQual
+ {
+ Node *qual; /* a qual clause waiting to be processed */
+ Relids relids; /* the set of baserels it references */
+ } PostponedQual;
+
+
static void extract_lateral_references(PlannerInfo *root, RelOptInfo *brel,
Index rtindex);
static void add_lateral_info(PlannerInfo *root, Relids lhs, Relids rhs);
static List *deconstruct_recurse(PlannerInfo *root, Node *jtnode,
bool below_outer_join,
! Relids *qualscope, Relids *inner_join_rels,
! List **postponed_qual_list);
static SpecialJoinInfo *make_outerjoininfo(PlannerInfo *root,
Relids left_rels, Relids right_rels,
Relids inner_join_rels,
*************** static void distribute_qual_to_rels(Plan
*** 53,59 ****
Relids qualscope,
Relids ojscope,
Relids outerjoin_nonnullable,
! Relids deduced_nullable_relids);
static bool check_outerjoin_delay(PlannerInfo *root, Relids *relids_p,
Relids *nullable_relids_p, bool is_pushed_down);
static bool check_equivalence_delay(PlannerInfo *root,
--- 62,69 ----
Relids qualscope,
Relids ojscope,
Relids outerjoin_nonnullable,
! Relids deduced_nullable_relids,
! List **postponed_qual_list);
static bool check_outerjoin_delay(PlannerInfo *root, Relids *relids_p,
Relids *nullable_relids_p, bool is_pushed_down);
static bool check_equivalence_delay(PlannerInfo *root,
*************** add_lateral_info(PlannerInfo *root, Reli
*** 630,644 ****
List *
deconstruct_jointree(PlannerInfo *root)
{
Relids qualscope;
Relids inner_join_rels;
/* Start recursion at top of jointree */
Assert(root->parse->jointree != NULL &&
IsA(root->parse->jointree, FromExpr));
! return deconstruct_recurse(root, (Node *) root->parse->jointree, false,
! &qualscope, &inner_join_rels);
}
/*
--- 640,662 ----
List *
deconstruct_jointree(PlannerInfo *root)
{
+ List *result;
Relids qualscope;
Relids inner_join_rels;
+ List *postponed_qual_list = NIL;
/* Start recursion at top of jointree */
Assert(root->parse->jointree != NULL &&
IsA(root->parse->jointree, FromExpr));
! result = deconstruct_recurse(root, (Node *) root->parse->jointree, false,
! &qualscope, &inner_join_rels,
! &postponed_qual_list);
!
! /* Shouldn't be any leftover quals */
! Assert(postponed_qual_list == NIL);
!
! return result;
}
/*
*************** deconstruct_jointree(PlannerInfo *root)
*** 656,668 ****
* *inner_join_rels gets the set of base Relids syntactically included in
* inner joins appearing at or below this jointree node (do not modify
* or free this, either)
* Return value is the appropriate joinlist for this jointree node
*
* In addition, entries will be added to root->join_info_list for outer joins.
*/
static List *
deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join,
! Relids *qualscope, Relids *inner_join_rels)
{
List *joinlist;
--- 674,689 ----
* *inner_join_rels gets the set of base Relids syntactically included in
* inner joins appearing at or below this jointree node (do not modify
* or free this, either)
+ * *postponed_qual_list: list of PostponedQual structs, which we can add
+ * quals to if they turn out to belong to a higher join level
* Return value is the appropriate joinlist for this jointree node
*
* In addition, entries will be added to root->join_info_list for outer joins.
*/
static List *
deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join,
! Relids *qualscope, Relids *inner_join_rels,
! List **postponed_qual_list)
{
List *joinlist;
*************** deconstruct_recurse(PlannerInfo *root, N
*** 685,690 ****
--- 706,712 ----
else if (IsA(jtnode, FromExpr))
{
FromExpr *f = (FromExpr *) jtnode;
+ List *child_postponed_quals = NIL;
int remaining;
ListCell *l;
*************** deconstruct_recurse(PlannerInfo *root, N
*** 707,713 ****
sub_joinlist = deconstruct_recurse(root, lfirst(l),
below_outer_join,
&sub_qualscope,
! inner_join_rels);
*qualscope = bms_add_members(*qualscope, sub_qualscope);
sub_members = list_length(sub_joinlist);
remaining--;
--- 729,736 ----
sub_joinlist = deconstruct_recurse(root, lfirst(l),
below_outer_join,
&sub_qualscope,
! inner_join_rels,
! &child_postponed_quals);
*qualscope = bms_add_members(*qualscope, sub_qualscope);
sub_members = list_length(sub_joinlist);
remaining--;
*************** deconstruct_recurse(PlannerInfo *root, N
*** 729,734 ****
--- 752,774 ----
*inner_join_rels = *qualscope;
/*
+ * Try to process any quals postponed by children. If they need
+ * further postponement, add them to my output postponed_qual_list.
+ */
+ foreach(l, child_postponed_quals)
+ {
+ PostponedQual *pq = (PostponedQual *) lfirst(l);
+
+ if (bms_is_subset(pq->relids, *qualscope))
+ distribute_qual_to_rels(root, pq->qual,
+ false, below_outer_join, JOIN_INNER,
+ *qualscope, NULL, NULL, NULL,
+ NULL);
+ else
+ *postponed_qual_list = lappend(*postponed_qual_list, pq);
+ }
+
+ /*
* Now process the top-level quals.
*/
foreach(l, (List *) f->quals)
*************** deconstruct_recurse(PlannerInfo *root, N
*** 737,748 ****
distribute_qual_to_rels(root, qual,
false, below_outer_join, JOIN_INNER,
! *qualscope, NULL, NULL, NULL);
}
}
else if (IsA(jtnode, JoinExpr))
{
JoinExpr *j = (JoinExpr *) jtnode;
Relids leftids,
rightids,
left_inners,
--- 777,790 ----
distribute_qual_to_rels(root, qual,
false, below_outer_join, JOIN_INNER,
! *qualscope, NULL, NULL, NULL,
! postponed_qual_list);
}
}
else if (IsA(jtnode, JoinExpr))
{
JoinExpr *j = (JoinExpr *) jtnode;
+ List *child_postponed_quals = NIL;
Relids leftids,
rightids,
left_inners,
*************** deconstruct_recurse(PlannerInfo *root, N
*** 771,780 ****
case JOIN_INNER:
leftjoinlist = deconstruct_recurse(root, j->larg,
below_outer_join,
! &leftids, &left_inners);
rightjoinlist = deconstruct_recurse(root, j->rarg,
below_outer_join,
! &rightids, &right_inners);
*qualscope = bms_union(leftids, rightids);
*inner_join_rels = *qualscope;
/* Inner join adds no restrictions for quals */
--- 813,824 ----
case JOIN_INNER:
leftjoinlist = deconstruct_recurse(root, j->larg,
below_outer_join,
! &leftids, &left_inners,
! &child_postponed_quals);
rightjoinlist = deconstruct_recurse(root, j->rarg,
below_outer_join,
! &rightids, &right_inners,
! &child_postponed_quals);
*qualscope = bms_union(leftids, rightids);
*inner_join_rels = *qualscope;
/* Inner join adds no restrictions for quals */
*************** deconstruct_recurse(PlannerInfo *root, N
*** 784,793 ****
case JOIN_ANTI:
leftjoinlist = deconstruct_recurse(root, j->larg,
below_outer_join,
! &leftids, &left_inners);
rightjoinlist = deconstruct_recurse(root, j->rarg,
true,
! &rightids, &right_inners);
*qualscope = bms_union(leftids, rightids);
*inner_join_rels = bms_union(left_inners, right_inners);
nonnullable_rels = leftids;
--- 828,839 ----
case JOIN_ANTI:
leftjoinlist = deconstruct_recurse(root, j->larg,
below_outer_join,
! &leftids, &left_inners,
! &child_postponed_quals);
rightjoinlist = deconstruct_recurse(root, j->rarg,
true,
! &rightids, &right_inners,
! &child_postponed_quals);
*qualscope = bms_union(leftids, rightids);
*inner_join_rels = bms_union(left_inners, right_inners);
nonnullable_rels = leftids;
*************** deconstruct_recurse(PlannerInfo *root, N
*** 795,804 ****
case JOIN_SEMI:
leftjoinlist = deconstruct_recurse(root, j->larg,
below_outer_join,
! &leftids, &left_inners);
rightjoinlist = deconstruct_recurse(root, j->rarg,
below_outer_join,
! &rightids, &right_inners);
*qualscope = bms_union(leftids, rightids);
*inner_join_rels = bms_union(left_inners, right_inners);
/* Semi join adds no restrictions for quals */
--- 841,852 ----
case JOIN_SEMI:
leftjoinlist = deconstruct_recurse(root, j->larg,
below_outer_join,
! &leftids, &left_inners,
! &child_postponed_quals);
rightjoinlist = deconstruct_recurse(root, j->rarg,
below_outer_join,
! &rightids, &right_inners,
! &child_postponed_quals);
*qualscope = bms_union(leftids, rightids);
*inner_join_rels = bms_union(left_inners, right_inners);
/* Semi join adds no restrictions for quals */
*************** deconstruct_recurse(PlannerInfo *root, N
*** 807,816 ****
case JOIN_FULL:
leftjoinlist = deconstruct_recurse(root, j->larg,
true,
! &leftids, &left_inners);
rightjoinlist = deconstruct_recurse(root, j->rarg,
true,
! &rightids, &right_inners);
*qualscope = bms_union(leftids, rightids);
*inner_join_rels = bms_union(left_inners, right_inners);
/* each side is both outer and inner */
--- 855,866 ----
case JOIN_FULL:
leftjoinlist = deconstruct_recurse(root, j->larg,
true,
! &leftids, &left_inners,
! &child_postponed_quals);
rightjoinlist = deconstruct_recurse(root, j->rarg,
true,
! &rightids, &right_inners,
! &child_postponed_quals);
*qualscope = bms_union(leftids, rightids);
*inner_join_rels = bms_union(left_inners, right_inners);
/* each side is both outer and inner */
*************** deconstruct_recurse(PlannerInfo *root, N
*** 853,859 ****
ojscope = NULL;
}
! /* Process the qual clauses */
foreach(l, (List *) j->quals)
{
Node *qual = (Node *) lfirst(l);
--- 903,934 ----
ojscope = NULL;
}
! /*
! * Try to process any quals postponed by children. If they need
! * further postponement, add them to my output postponed_qual_list.
! */
! foreach(l, child_postponed_quals)
! {
! PostponedQual *pq = (PostponedQual *) lfirst(l);
!
! if (bms_is_subset(pq->relids, *qualscope))
! distribute_qual_to_rels(root, pq->qual,
! false, below_outer_join, j->jointype,
! *qualscope,
! ojscope, nonnullable_rels, NULL,
! NULL);
! else
! {
! /*
! * We should not be postponing any quals past an outer join.
! * If this Assert fires, pull_up_subqueries() messed up.
! */
! Assert(j->jointype == JOIN_INNER);
! *postponed_qual_list = lappend(*postponed_qual_list, pq);
! }
! }
!
! /* Process the JOIN's qual clauses */
foreach(l, (List *) j->quals)
{
Node *qual = (Node *) lfirst(l);
*************** deconstruct_recurse(PlannerInfo *root, N
*** 861,867 ****
distribute_qual_to_rels(root, qual,
false, below_outer_join, j->jointype,
*qualscope,
! ojscope, nonnullable_rels, NULL);
}
/* Now we can add the SpecialJoinInfo to join_info_list */
--- 936,943 ----
distribute_qual_to_rels(root, qual,
false, below_outer_join, j->jointype,
*qualscope,
! ojscope, nonnullable_rels, NULL,
! postponed_qual_list);
}
/* Now we can add the SpecialJoinInfo to join_info_list */
*************** make_outerjoininfo(PlannerInfo *root,
*** 1154,1160 ****
* the appropriate list for each rel. Alternatively, if the clause uses a
* mergejoinable operator and is not delayed by outer-join rules, enter
* the left- and right-side expressions into the query's list of
! * EquivalenceClasses.
*
* 'clause': the qual clause to be distributed
* 'is_deduced': TRUE if the qual came from implied-equality deduction
--- 1230,1237 ----
* the appropriate list for each rel. Alternatively, if the clause uses a
* mergejoinable operator and is not delayed by outer-join rules, enter
* the left- and right-side expressions into the query's list of
! * EquivalenceClasses. Alternatively, if the clause needs to be treated
! * as belonging to a higher join level, just add it to postponed_qual_list.
*
* 'clause': the qual clause to be distributed
* 'is_deduced': TRUE if the qual came from implied-equality deduction
*************** make_outerjoininfo(PlannerInfo *root,
*** 1170,1175 ****
--- 1247,1254 ----
* equal qualscope)
* 'deduced_nullable_relids': if is_deduced is TRUE, the nullable relids to
* impute to the clause; otherwise NULL
+ * 'postponed_qual_list': list of PostponedQual structs, which we can add
+ * this qual to if it turns out to belong to a higher join level
*
* 'qualscope' identifies what level of JOIN the qual came from syntactically.
* 'ojscope' is needed if we decide to force the qual up to the outer-join
*************** distribute_qual_to_rels(PlannerInfo *roo
*** 1190,1196 ****
Relids qualscope,
Relids ojscope,
Relids outerjoin_nonnullable,
! Relids deduced_nullable_relids)
{
Relids relids;
bool is_pushed_down;
--- 1269,1276 ----
Relids qualscope,
Relids ojscope,
Relids outerjoin_nonnullable,
! Relids deduced_nullable_relids,
! List **postponed_qual_list)
{
Relids relids;
bool is_pushed_down;
*************** distribute_qual_to_rels(PlannerInfo *roo
*** 1207,1226 ****
relids = pull_varnos(clause);
/*
! * Normally relids is a subset of qualscope, and we like to check that
! * here as a crosscheck on the parser and rewriter. That need not be the
! * case when there are LATERAL RTEs, however: the clause could contain
! * references to rels outside its syntactic scope as a consequence of
! * pull-up of such references from a LATERAL subquery below it. So, only
! * check if the query contains no LATERAL RTEs.
! *
! * However, if it's an outer-join clause, we always insist that relids be
! * a subset of ojscope. This is safe because is_simple_subquery()
! * disallows pullup of LATERAL subqueries that could cause the restriction
! * to be violated.
*/
- if (!root->hasLateralRTEs && !bms_is_subset(relids, qualscope))
- elog(ERROR, "JOIN qualification cannot refer to other relations");
if (ojscope && !bms_is_subset(relids, ojscope))
elog(ERROR, "JOIN qualification cannot refer to other relations");
--- 1287,1322 ----
relids = pull_varnos(clause);
/*
! * In ordinary SQL, a WHERE or JOIN/ON clause can't reference any rels
! * that aren't within its syntactic scope; however, if we pulled up a
! * LATERAL subquery then we might find such references in quals that have
! * been pulled up. We need to treat such quals as belonging to the join
! * level that includes every rel they reference. Although we could make
! * pull_up_subqueries() place such quals correctly to begin with, it's
! * easier to handle it here. When we find a clause that contains Vars
! * outside its syntactic scope, we add it to the postponed_clauses list,
! * and process it once we've recursed back up to the appropriate join
! * level.
! */
! if (!bms_is_subset(relids, qualscope))
! {
! PostponedQual *pq = (PostponedQual *) palloc(sizeof(PostponedQual));
!
! Assert(root->hasLateralRTEs); /* shouldn't happen otherwise */
! Assert(jointype == JOIN_INNER); /* mustn't postpone past outer join */
! Assert(!is_deduced); /* shouldn't be deduced, either */
! pq->qual = clause;
! pq->relids = relids;
! *postponed_qual_list = lappend(*postponed_qual_list, pq);
! return;
! }
!
! /*
! * In any case, if it's an outer-join clause, we insist that relids be a
! * subset of ojscope. (It's pull_up_subqueries()'s responsibility to not
! * pull up a LATERAL subquery if that would cause this to fail; the
! * semantics that would result from such a situation are unclear.)
*/
if (ojscope && !bms_is_subset(relids, ojscope))
elog(ERROR, "JOIN qualification cannot refer to other relations");
*************** process_implied_equality(PlannerInfo *ro
*** 1874,1880 ****
*/
distribute_qual_to_rels(root, (Node *) clause,
true, below_outer_join, JOIN_INNER,
! qualscope, NULL, NULL, nullable_relids);
}
/*
--- 1970,1977 ----
*/
distribute_qual_to_rels(root, (Node *) clause,
true, below_outer_join, JOIN_INNER,
! qualscope, NULL, NULL, nullable_relids,
! NULL);
}
/*
diff --git a/src/backend/optimizer/prep/prepjointree.c b/src/backend/optimizer/prep/prepjointree.c
index 875baef..bb6d983 100644
*** a/src/backend/optimizer/prep/prepjointree.c
--- b/src/backend/optimizer/prep/prepjointree.c
*************** static bool is_simple_union_all(Query *s
*** 84,89 ****
--- 84,91 ----
static bool is_simple_union_all_recurse(Node *setOp, Query *setOpQuery,
List *colTypes);
static bool is_safe_append_member(Query *subquery);
+ static bool jointree_contains_lateral_outer_refs(Node *jtnode, bool restricted,
+ Relids safe_upper_varnos);
static void replace_vars_in_jointree(Node *jtnode,
pullup_replace_vars_context *context,
JoinExpr *lowest_nulling_outer_join);
*************** is_simple_subquery(Query *subquery, Rang
*** 1303,1322 ****
return false;
/*
! * If the subquery is LATERAL, and we're below any outer join, and the
! * subquery contains lateral references to rels outside the outer join,
! * don't pull up. Doing so would risk creating outer-join quals that
! * contain references to rels outside the outer join, which is a semantic
! * mess that doesn't seem worth addressing at the moment.
*/
! if (rte->lateral && lowest_outer_join != NULL)
{
! Relids lvarnos = pull_varnos_of_level((Node *) subquery, 1);
! Relids jvarnos = get_relids_in_jointree((Node *) lowest_outer_join,
! true);
! if (!bms_is_subset(lvarnos, jvarnos))
return false;
}
/*
--- 1305,1351 ----
return false;
/*
! * If the subquery is LATERAL, check to see if its WHERE or JOIN/ON quals
! * contain any lateral references to rels outside an upper outer join
! * (including the case where the outer join is within the subquery
! * itself). If so, don't pull up. Doing so would result in a situation
! * where we need to postpone quals from below an outer join to above it,
! * which is probably completely wrong and in any case is a complication
! * that doesn't seem worth addressing at the moment.
*/
! if (rte->lateral)
{
! bool restricted;
! Relids safe_upper_varnos;
! if (lowest_outer_join != NULL)
! {
! restricted = true;
! safe_upper_varnos = get_relids_in_jointree((Node *) lowest_outer_join,
! true);
! }
! else
! {
! restricted = false;
! safe_upper_varnos = NULL; /* doesn't matter */
! }
!
! if (jointree_contains_lateral_outer_refs((Node *) subquery->jointree,
! restricted, safe_upper_varnos))
return false;
+
+ /*
+ * If there's an upper outer join, also disallow any targetlist
+ * references outside it, since these might get pulled into quals
+ * above this subquery.
+ */
+ if (lowest_outer_join != NULL)
+ {
+ Relids lvarnos = pull_varnos_of_level((Node *) subquery->targetList, 1);
+
+ if (!bms_is_subset(lvarnos, safe_upper_varnos))
+ return false;
+ }
}
/*
*************** is_simple_subquery(Query *subquery, Rang
*** 1344,1355 ****
* correctly generate a Result plan for a jointree that's totally empty,
* but we can't cope with an empty FromExpr appearing lower down in a
* jointree: we identify join rels via baserelid sets, so we couldn't
! * distinguish a join containing such a FromExpr from one without it.
! * This would for example break the PlaceHolderVar mechanism, since we'd
! * have no way to identify where to evaluate a PHV coming out of the
! * subquery. Not worth working hard on this, just to collapse
! * SubqueryScan/Result into Result; especially since the SubqueryScan can
! * often be optimized away by setrefs.c anyway.
*/
if (subquery->jointree->fromlist == NIL)
return false;
--- 1373,1384 ----
* correctly generate a Result plan for a jointree that's totally empty,
* but we can't cope with an empty FromExpr appearing lower down in a
* jointree: we identify join rels via baserelid sets, so we couldn't
! * distinguish a join containing such a FromExpr from one without it. This
! * would for example break the PlaceHolderVar mechanism, since we'd have
! * no way to identify where to evaluate a PHV coming out of the subquery.
! * Not worth working hard on this, just to collapse SubqueryScan/Result
! * into Result; especially since the SubqueryScan can often be optimized
! * away by setrefs.c anyway.
*/
if (subquery->jointree->fromlist == NIL)
return false;
*************** is_safe_append_member(Query *subquery)
*** 1467,1472 ****
--- 1496,1575 ----
}
/*
+ * jointree_contains_lateral_outer_refs
+ * Check for disallowed lateral references in a jointree's quals
+ *
+ * If restricted is false, all level-1 Vars are allowed (but we still must
+ * search the jointree, since it might contain outer joins below which there
+ * will be restrictions). If restricted is true, return TRUE when any qual
+ * in the jointree contains level-1 Vars coming from outside the rels listed
+ * in safe_upper_varnos.
+ */
+ static bool
+ jointree_contains_lateral_outer_refs(Node *jtnode, bool restricted,
+ Relids safe_upper_varnos)
+ {
+ if (jtnode == NULL)
+ return false;
+ if (IsA(jtnode, RangeTblRef))
+ return false;
+ else if (IsA(jtnode, FromExpr))
+ {
+ FromExpr *f = (FromExpr *) jtnode;
+ ListCell *l;
+
+ /* First, recurse to check child joins */
+ foreach(l, f->fromlist)
+ {
+ if (jointree_contains_lateral_outer_refs(lfirst(l),
+ restricted,
+ safe_upper_varnos))
+ return true;
+ }
+
+ /* Then check the top-level quals */
+ if (restricted &&
+ !bms_is_subset(pull_varnos_of_level(f->quals, 1),
+ safe_upper_varnos))
+ return true;
+ }
+ else if (IsA(jtnode, JoinExpr))
+ {
+ JoinExpr *j = (JoinExpr *) jtnode;
+
+ /*
+ * If this is an outer join, we mustn't allow any upper lateral
+ * references in or below it.
+ */
+ if (j->jointype != JOIN_INNER)
+ {
+ restricted = true;
+ safe_upper_varnos = NULL;
+ }
+
+ /* Check the child joins */
+ if (jointree_contains_lateral_outer_refs(j->larg,
+ restricted,
+ safe_upper_varnos))
+ return true;
+ if (jointree_contains_lateral_outer_refs(j->rarg,
+ restricted,
+ safe_upper_varnos))
+ return true;
+
+ /* Check the JOIN's qual clauses */
+ if (restricted &&
+ !bms_is_subset(pull_varnos_of_level(j->quals, 1),
+ safe_upper_varnos))
+ return true;
+ }
+ else
+ elog(ERROR, "unrecognized node type: %d",
+ (int) nodeTag(jtnode));
+ return false;
+ }
+
+ /*
* Helper routine for pull_up_subqueries: do pullup_replace_vars on every
* expression in the jointree, without changing the jointree structure itself.
* Ugly, but there's no other way...
diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out
index fc3e168..98aacd3 100644
*** a/src/test/regress/expected/join.out
--- b/src/test/regress/expected/join.out
*************** explain (costs off)
*** 3161,3167 ****
Nested Loop Left Join
-> Seq Scan on int4_tbl x
-> Index Scan using tenk1_unique1 on tenk1
! Index Cond: (unique1 = x.f1)
(4 rows)
-- check scoping of lateral versus parent references
--- 3161,3167 ----
Nested Loop Left Join
-> Seq Scan on int4_tbl x
-> Index Scan using tenk1_unique1 on tenk1
! Index Cond: (x.f1 = unique1)
(4 rows)
-- check scoping of lateral versus parent references
*************** select * from int4_tbl i left join
*** 3648,3659 ****
lateral (select * from int2_tbl j where i.f1 = j.f1) k on true;
QUERY PLAN
-------------------------------------------
! Nested Loop Left Join
Output: i.f1, j.f1
! Filter: (i.f1 = j.f1)
-> Seq Scan on public.int4_tbl i
Output: i.f1
! -> Materialize
Output: j.f1
-> Seq Scan on public.int2_tbl j
Output: j.f1
--- 3648,3659 ----
lateral (select * from int2_tbl j where i.f1 = j.f1) k on true;
QUERY PLAN
-------------------------------------------
! Hash Left Join
Output: i.f1, j.f1
! Hash Cond: (i.f1 = j.f1)
-> Seq Scan on public.int4_tbl i
Output: i.f1
! -> Hash
Output: j.f1
-> Seq Scan on public.int2_tbl j
Output: j.f1
*************** select * from int4_tbl i left join
*** 3661,3670 ****
select * from int4_tbl i left join
lateral (select * from int2_tbl j where i.f1 = j.f1) k on true;
! f1 | f1
! ----+----
! 0 | 0
! (1 row)
explain (verbose, costs off)
select * from int4_tbl i left join
--- 3661,3674 ----
select * from int4_tbl i left join
lateral (select * from int2_tbl j where i.f1 = j.f1) k on true;
! f1 | f1
! -------------+----
! 0 | 0
! 123456 |
! -123456 |
! 2147483647 |
! -2147483647 |
! (5 rows)
explain (verbose, costs off)
select * from int4_tbl i left join
*************** select * from int4_tbl i left join
*** 3691,3696 ****
--- 3695,3723 ----
-2147483647 |
(5 rows)
+ explain (verbose, costs off)
+ select * from int4_tbl a,
+ lateral (
+ select * from int4_tbl b left join int8_tbl c on (b.f1 = q1 and a.f1 = q2)
+ ) ss;
+ QUERY PLAN
+ -------------------------------------------------
+ Nested Loop
+ Output: a.f1, b.f1, c.q1, c.q2
+ -> Seq Scan on public.int4_tbl a
+ Output: a.f1
+ -> Hash Left Join
+ Output: b.f1, c.q1, c.q2
+ Hash Cond: (b.f1 = c.q1)
+ -> Seq Scan on public.int4_tbl b
+ Output: b.f1
+ -> Hash
+ Output: c.q1, c.q2
+ -> Seq Scan on public.int8_tbl c
+ Output: c.q1, c.q2
+ Filter: (a.f1 = c.q2)
+ (14 rows)
+
-- lateral reference in a PlaceHolderVar evaluated at join level
explain (verbose, costs off)
select * from
diff --git a/src/test/regress/sql/join.sql b/src/test/regress/sql/join.sql
index 36853dd..c0ed8b0 100644
*** a/src/test/regress/sql/join.sql
--- b/src/test/regress/sql/join.sql
*************** select * from int4_tbl i left join
*** 1022,1027 ****
--- 1022,1032 ----
lateral (select coalesce(i) from int2_tbl j where i.f1 = j.f1) k on true;
select * from int4_tbl i left join
lateral (select coalesce(i) from int2_tbl j where i.f1 = j.f1) k on true;
+ explain (verbose, costs off)
+ select * from int4_tbl a,
+ lateral (
+ select * from int4_tbl b left join int8_tbl c on (b.f1 = q1 and a.f1 = q2)
+ ) ss;
-- lateral reference in a PlaceHolderVar evaluated at join level
explain (verbose, costs off)