ERROR: too late to create a new PlaceHolderInfo
While fixing the assertion failure with LATERAL UNION ALL reported by
Fujii-san [1]/messages/by-id/CAHGQGwGZBZmb7Yq+UTT0Lp5wvztQZtMT94ehcMHQs9Dk6jaaXQ@mail.gmail.com, I ran into this:
create table t (a int);
select *
from t t1 left join (t t2 join t t3 on true) j on true,
lateral ((select j offset 0) union all (select null offset 0)) ss;
ERROR: too late to create a new PlaceHolderInfo
This goes back to v16, when add_nullingrels_if_needed() was first
introduced.
When a LATERAL UNION ALL subquery is pulled up as an appendrel,
find_lateral_references looks only at the parent RTE's subquery,
assuming the children are copies of parts of it. But the parent's
subquery and the children get their join alias Vars expanded
separately, later in subquery_planner. If the expansion needs a PHV
(here, the whole-row Var of j, which is nulled by the left join), each
flatten_join_alias_vars call makes its own PHV with a new phid. So
the children end up asking for PHVs that have no PlaceHolderInfo.
A straightforward fix is to expand join alias Vars in the LATERAL
subquery in pull_up_simple_union_all, before its rtable gets copied,
so that all the copies share the same PHVs. Doing this that early
should be OK, since a LATERAL item can only reference FROM items to
its left, and pull_up_subqueries has already processed those, so their
join alias lists are final. See attached.
This feels more like a band-aid though. I think the more principled
fix would be a PHV cache in add_nullingrels_if_needed, so that we
don't generate identical PHVs with different IDs, like what rv_cache
does in pullup_replace_vars_callback. But since
flatten_join_alias_vars is called many times per query level, the
cache would have to live in PlannerInfo, which seems too invasive to
back-patch.
Any thoughts?
[1]: /messages/by-id/CAHGQGwGZBZmb7Yq+UTT0Lp5wvztQZtMT94ehcMHQs9Dk6jaaXQ@mail.gmail.com
- Richard
On Thu, Sep 17, 2026 at 10:14 AM Richard Guo <guofenglinux@gmail.com> wrote:
A straightforward fix is to expand join alias Vars in the LATERAL
subquery in pull_up_simple_union_all, before its rtable gets copied,
so that all the copies share the same PHVs.
Ugh, this doesn't work if the UNION ALL is nested inside another
LATERAL subquery:
select *
from t t1 left join (t t2 join t t3 on true) j on true,
lateral (select * from ((select j offset 0) union all (select null
offset 0)) s) ss;
ERROR: too late to create a new PlaceHolderInfo
Here the UNION ALL gets pulled up while ss is being pulled up, using
ss's own subroot. At that point its RTE isn't LATERAL, and j belongs
to the outer query anyway, so v1 doesn't expand anything. We could
patch pull_up_simple_subquery the same way, but that makes the fix
more like a band-aid.
This feels more like a band-aid though. I think the more principled
fix would be a PHV cache in add_nullingrels_if_needed, so that we
don't generate identical PHVs with different IDs, like what rv_cache
does in pullup_replace_vars_callback. But since
flatten_join_alias_vars is called many times per query level, the
cache would have to live in PlannerInfo, which seems too invasive to
back-patch.
I still think the PHV cache is the better approach. Attached v2 does
that: add_nullingrels_if_needed now keeps the PHVs it makes in a new
PlannerInfo field, and reuses one when the same expression is expanded
again. This fixes both queries, no matter where the copies come from,
and it also avoids generating duplicate PHVs for the same alias
elsewhere in the query.
The problem with it is that it breaks ABI and can't be back-patched.
Given that there have been no field reports since v16, maybe we can
fix it on master only?
- Richard
Richard Guo <guofenglinux@gmail.com> writes:
This feels more like a band-aid though. I think the more principled
fix would be a PHV cache in add_nullingrels_if_needed, so that we
don't generate identical PHVs with different IDs, like what rv_cache
does in pullup_replace_vars_callback. But since
flatten_join_alias_vars is called many times per query level, the
cache would have to live in PlannerInfo, which seems too invasive to
back-patch.
I agree with that sounding more principled, but I wonder if we should
think bigger than just tweaking add_nullingrels_if_needed: if we're
desirous of de-duplicating PHVs, why not do that across the board,
for every place that makes PHVs? So we'd mechanize this in
make_placeholder_expr's assignment of phid rather than somewhere else.
The problem with it is that it breaks ABI and can't be back-patched.
Given that there have been no field reports since v16, maybe we can
fix it on master only?
Maybe. In the past, when we needed a new struct field, we've sometimes
decided that adding it at the end of the struct in the back branches
would be adequately ABI-compatible. I think we might be able to get
away with that here too, although I'm a bit worried whether any
extensions might be doing makeNode(PlannerInfo).
regards, tom lane
I wrote:
I agree with that sounding more principled, but I wonder if we should
think bigger than just tweaking add_nullingrels_if_needed: if we're
desirous of de-duplicating PHVs, why not do that across the board,
for every place that makes PHVs? So we'd mechanize this in
make_placeholder_expr's assignment of phid rather than somewhere else.
Actually, we can't be too gung-ho about that: we should not merge
PHVs if their expressions are volatile. It's not quite clear to me
whether that's a problem for the join-alias-Vars case.
regards, tom lane
On Fri, Sep 18, 2026 at 12:41 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
I wrote:
I agree with that sounding more principled, but I wonder if we should
think bigger than just tweaking add_nullingrels_if_needed: if we're
desirous of de-duplicating PHVs, why not do that across the board,
for every place that makes PHVs? So we'd mechanize this in
make_placeholder_expr's assignment of phid rather than somewhere else.
Actually, we can't be too gung-ho about that: we should not merge
PHVs if their expressions are volatile. It's not quite clear to me
whether that's a problem for the join-alias-Vars case.
Thanks for looking at this.
I don't think volatility is a problem for the join alias case. A join
alias expansion starts out as Vars and COALESCEs of the join's inputs.
Those are handled without a PHV (is_standard_join_alias_expression).
We need a PHV only for a whole-row reference, which expands to a
RowExpr of those entries, or when an input Var has since been replaced
by some other expression. That happens through subquery or VALUES
pullup, which refuse volatile expressions.
As for doing this in make_placeholder_expr, I'm not sure it would buy
us much. pullup_replace_vars_callback already shares PHVs per
subquery output column through rv_cache. It doesn't have the problem
we have here, where the same reference exists in two copies of a
subquery that are expanded separately. Also, PHVs made during pullup
have their phrels rewritten afterwards, so cache entries for them
would go stale right away, and the lookups would mostly be wasted. So
I'm inclined to keep this in add_nullingrels_if_needed.
Regarding back-patching, I'm still hesitant. The new field is only
read in add_nullingrels_if_needed, so adding it at the end of the
struct would probably work. But if some extension does do
makeNode(PlannerInfo) and pass that to flatten_join_alias_vars, we'd
be reading a garbage pointer there. This patch might also lead to
plan changes, since expansions of the same join alias that used to get
separate PHVs now share one. But maybe we can get this patch into
v19.
- Richard
I found that v2 still has a loose end at the pull_up_simple_subquery
boundary. When a subquery is pulled up, its targetlist, and its
jointree quals if it is LATERAL, are flattened using the subroot,
while the rest of it is flattened later using the parent root. Since
each root has its own join_alias_phvs, an alias Var that appears in
both places still gets two different PHVs:
select *
from (select v.k
from t t1 left join (t t2 join t t3 on true) j on true
left join lateral (select j as k from t t4) v on true
where v.k is not null) ss;
ERROR: too late to create a new PlaceHolderInfo
Here the PHV that pullup makes for v.k ends up in both the targetlist
and the WHERE clause of ss. The j inside it is expanded by ss's
subroot in one copy and by the parent root in the other.
To fix, attached v3 makes pull_up_simple_subquery handle the subroot's
join_alias_phvs the same way as its append_rel_list: adjust it with
OffsetVarNodes and IncrementVarSublevelsUp, then append it to the
parent's list, so that the parent's flattening reuses the subroot's
PHVs.
I'd like to push this to master and v19, and leave v18 and older
alone due to ABI concerns.
I plan to push it soon, after going over the code a bit more, so that
I have at least some time to react to any bugs found by the automated
LLM-based tools that now scan new commits.
- Richard
On Thu, Nov 5, 2026 at 11:25 AM Richard Guo <guofenglinux@gmail.com> wrote:
I'd like to push this to master and v19, and leave v18 and older
alone due to ABI concerns.
Hmm, while fuzzing v3 I found that this bug can also cause wrong
results, not just errors.
create table a (k int);
create table b (k int);
insert into a values (1);
select ss.x, count(*)
from a left join (b cross join (select 1 as c) s) j on a.k = j.k
cross join lateral (select j as x) ss
where ss.x::text is not null
group by rollup (ss.x);
x | count
------+-------
NULL | 1
NULL | 1
(2 rows)
b is empty, so a's row is null-extended, ss.x is NULL, and the answer
should be a single (NULL, 0) row. We get the above wrong results on
every branch back to v16. Because of the grouping sets, pullup wraps
ss.x in a PHV, and the j inside it is flattened separately in the
targetlist and in the WHERE clause. The WHERE clause's inner PHV then
matches nothing in the join's input, so setrefs.c computes its
expression above the left join, yielding ROW(NULL, 1) instead of NULL.
This wrong-result is fixed by the patch.
Given the wrong results, I think we need to reconsider back-patching
to v16. Is fixing this in the back branches worth the risk of an ABI
break?
- Richard
On Thu, Nov 5, 2026 at 5:29 PM Richard Guo <guofenglinux@gmail.com> wrote:
Given the wrong results, I think we need to reconsider back-patching
to v16. Is fixing this in the back branches worth the risk of an ABI
break?
I searched some open-source extensions and found several that allocate
their own PlannerInfo, such as TimescaleDB, HypoPG and pg_ivm, and
Apache AGE even passes one to flatten_join_alias_vars. This close to
the minor releases, I didn't want to risk an ABI break, so I've pushed
it to master and v19 only.
(v16-v18 still give these wrong results silently, but I don't know
what to do about them for now.)
- Richard