Free indexed_tlist memory explicitly within set_plan_refs()
While trying to fix a largely unrelated bug, I noticed that the new
build_tlist_index() call for the "excluded" targetlist (used by ON
CONFLICT DO UPDATE queries) does not have its memory subsequently
freed by the caller. Since every other call to build_tlist_index()
does this, and comments above build_tlist_index() encourage it, I
think the new caller should do the same.
Attached patch adds such a pfree() call.
--
Peter Geoghegan
Attachments:
add-pfree-indexed_tlist.patchtext/x-patch; charset=US-ASCII; name=add-pfree-indexed_tlist.patchDownload
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index 90e13e4..8afe6a3 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -776,6 +776,8 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset)
linitial_int(splan->resultRelations),
rtoffset);
+ pfree(itlist);
+
splan->exclRelTlist =
fix_scan_list(root, splan->exclRelTlist, rtoffset);
}
On Mon, May 25, 2015 at 10:17 AM, Peter Geoghegan <pg@heroku.com> wrote:
While trying to fix a largely unrelated bug, I noticed that the new
build_tlist_index() call for the "excluded" targetlist (used by ON
CONFLICT DO UPDATE queries) does not have its memory subsequently
freed by the caller. Since every other call to build_tlist_index()
does this, and comments above build_tlist_index() encourage it, I
think the new caller should do the same.Attached patch adds such a pfree() call.
Yep. This looks correct to me.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Sun, May 24, 2015 at 6:17 PM, Peter Geoghegan <pg@heroku.com> wrote:
Attached patch adds such a pfree() call.
Attached, revised version incorporates this small fix, while adding an
additional big fix, and a number of small style tweaks.
This is mainly concerned with fixing the bug I was trying to fix when
I spotted the minor pfree() issue:
postgres=# insert into upsert (key, val) values('Foo', 'Bar') on
conflict (key) do update set val = excluded.val where excluded.* is
not null;
ERROR: XX000: variable not found in subplan target lists
LOCATION: fix_join_expr_mutator, setrefs.c:2003
postgres=# insert into upsert (key, val) values(('Foo', 'Bar') on
conflict (key) do update set val = excluded.val where excluded.ctid is
not null;
ERROR: XX000: variable not found in subplan target lists
LOCATION: fix_join_expr_mutator, setrefs.c:2003
The first query shown should clearly finish processing by the
optimizer without raising this error message (execution should work
correctly too, of course). The second query shown should fail with a
user visible error message about referencing the excluded
pseudo-relation's ctid column not making sense.
The basic problem is that there wasn't much thought put into how the
excluded pseudo-relation's "reltargetlist" is generated -- it
currently comes from a call to expandRelAttrs() during parse analysis,
which, on its own, doesn't allow whole row Vars to work.
One approach to fixing this is to follow the example of RETURNING
lists with references to more than one relation:
preprocess_targetlist() handles this by calling pull_var_clause() and
making new TargetEntry entries for Vars found to not be referencing
the target (and not already in the targetlist for some other reason).
Another approach, preferred by Andres, is to have query_planner() do
more. I understand that the idea there is to make excluded.* closer to
a regular table, in that it can be expected to have a baserel, and
within the executor we have something closer to a bona-fide scan
reltargetlist, that we can expect to have all Vars appear in. This
should be enough to make the reltargetlist have the appropriate Vars
more or less in the regular course of planning, including excluded.*
whole row Vars. To make this work we could call
add_vars_to_targetlist(), and call add_base_rels_to_query() and then
build_base_rel_tlists() within query_planner() (while moving a few
other things around).
However, the ordering dependencies within query_planner() seemed quite
complicated to me, and I didn't want to modify such an important
routine just to fix this bug. RETURNING seemed like a perfectly good
precedent to follow, so that's what I did. Now, it might have been
that I misunderstood Andres when we discussed this problem on
Jabber/IM, but ISTM that the second idea doesn't have much advantage
over the first (I'm sure that Andres will be able to explain what he'd
like to do better here -- it was a quick conversation). I did
prototype the second approach, and the code footprint of what I have
here (the first approach) seems lower than it would have to be with
the second. Besides, I didn't see a convenient choke point to reject
system column references with the second approach. Attached patch
fixes the bug using the first approach. Tests were added demonstrating
that the cases above are fixed.
A second attached patch fixes another, largely independent bug. I
noticed array assignment with ON CONFLICT DO UPDATE was broken. See
commit message for full details.
Thoughts?
--
Peter Geoghegan
Attachments:
0002-Fix-bug-with-assigned-to-expressions-with-indirectio.patchtext/x-patch; charset=US-ASCII; name=0002-Fix-bug-with-assigned-to-expressions-with-indirectio.patchDownload
From 72076f565b142debe39eb1e5a6cac27100b76fdb Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <peter.geoghegan86@gmail.com>
Date: Thu, 28 May 2015 00:18:19 -0700
Subject: [PATCH 2/2] Fix bug with assigned-to expressions with indirection
Handling of assigned-to expressions with indirection (as used with
UPDATE targetlists, where, for example, assigned-to expressions allow
array elements to be directly assigned to) could previously become
confused. The problem was that ParseState was consulted to determine if
an INSERT-appropriate or UPDATE-appropriate behavior should be used, and
so for INSERT statements with ON CONFLICT DO UPDATE, an
INSERT-targetlist-applicable codepath could incorrectly be taken.
To fix, allow ParseState to reflect that an individual statement can be
both p_is_insert and p_is_update at the same time.
---
src/backend/parser/analyze.c | 3 +++
src/backend/parser/parse_target.c | 3 ++-
src/include/parser/parse_node.h | 2 +-
src/test/regress/expected/arrays.out | 21 +++++++++++++++++++++
src/test/regress/sql/arrays.sql | 13 +++++++++++++
5 files changed, 40 insertions(+), 2 deletions(-)
diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c
index fc463fa..be474dc 100644
--- a/src/backend/parser/analyze.c
+++ b/src/backend/parser/analyze.c
@@ -891,6 +891,9 @@ transformOnConflictClause(ParseState *pstate,
/* Process DO UPDATE */
if (onConflictClause->action == ONCONFLICT_UPDATE)
{
+ /* p_is_update must be set here, after INSERT targetlist processing */
+ pstate->p_is_update = true;
+
exclRte = addRangeTableEntryForRelation(pstate,
pstate->p_target_relation,
makeAlias("excluded", NIL),
diff --git a/src/backend/parser/parse_target.c b/src/backend/parser/parse_target.c
index 1b3fcd6..ebde013 100644
--- a/src/backend/parser/parse_target.c
+++ b/src/backend/parser/parse_target.c
@@ -469,13 +469,14 @@ transformAssignedExpr(ParseState *pstate,
{
Node *colVar;
- if (pstate->p_is_insert)
+ if (!pstate->p_is_update)
{
/*
* The command is INSERT INTO table (col.something) ... so there
* is not really a source value to work with. Insert a NULL
* constant as the source value.
*/
+ Assert(pstate->p_is_insert);
colVar = (Node *) makeNullConst(attrtype, attrtypmod,
attrcollation);
}
diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h
index 3103b71..e0f5641 100644
--- a/src/include/parser/parse_node.h
+++ b/src/include/parser/parse_node.h
@@ -152,7 +152,7 @@ struct ParseState
bool p_hasSubLinks;
bool p_hasModifyingCTE;
bool p_is_insert;
- bool p_is_update;
+ bool p_is_update; /* not mutually exclusive with p_is_insert */
bool p_locked_from_parent;
Relation p_target_relation;
RangeTblEntry *p_target_rangetblentry;
diff --git a/src/test/regress/expected/arrays.out b/src/test/regress/expected/arrays.out
index 5f1532f..73fb5a2 100644
--- a/src/test/regress/expected/arrays.out
+++ b/src/test/regress/expected/arrays.out
@@ -1116,6 +1116,27 @@ select * from arr_tbl where f1 >= '{1,2,3}' and f1 < '{1,5,3}';
{1,2,10}
(2 rows)
+-- test ON CONFLICT DO UPDATE with arrays
+create temp table arr_pk_tbl (pk int4 primary key, f1 int[]);
+insert into arr_pk_tbl values (1, '{1,2,3}');
+insert into arr_pk_tbl values (1, '{3,4,5}') on conflict (pk)
+ do update set f1[1] = excluded.f1[1], f1[3] = excluded.f1[3]
+ returning pk, f1;
+ pk | f1
+----+---------
+ 1 | {3,2,5}
+(1 row)
+
+insert into arr_pk_tbl(pk, f1[1:2]) values (1, '{6,7,8}') on conflict (pk)
+ do update set f1[1] = excluded.f1[1],
+ f1[2] = excluded.f1[2],
+ f1[3] = excluded.f1[3]
+ returning pk, f1;
+ pk | f1
+----+------------
+ 1 | {6,7,NULL}
+(1 row)
+
-- note: if above selects don't produce the expected tuple order,
-- then you didn't get an indexscan plan, and something is busted.
reset enable_seqscan;
diff --git a/src/test/regress/sql/arrays.sql b/src/test/regress/sql/arrays.sql
index 562134b..b1dd651 100644
--- a/src/test/regress/sql/arrays.sql
+++ b/src/test/regress/sql/arrays.sql
@@ -306,6 +306,19 @@ set enable_seqscan to off;
set enable_bitmapscan to off;
select * from arr_tbl where f1 > '{1,2,3}' and f1 <= '{1,5,3}';
select * from arr_tbl where f1 >= '{1,2,3}' and f1 < '{1,5,3}';
+
+-- test ON CONFLICT DO UPDATE with arrays
+create temp table arr_pk_tbl (pk int4 primary key, f1 int[]);
+insert into arr_pk_tbl values (1, '{1,2,3}');
+insert into arr_pk_tbl values (1, '{3,4,5}') on conflict (pk)
+ do update set f1[1] = excluded.f1[1], f1[3] = excluded.f1[3]
+ returning pk, f1;
+insert into arr_pk_tbl(pk, f1[1:2]) values (1, '{6,7,8}') on conflict (pk)
+ do update set f1[1] = excluded.f1[1],
+ f1[2] = excluded.f1[2],
+ f1[3] = excluded.f1[3]
+ returning pk, f1;
+
-- note: if above selects don't produce the expected tuple order,
-- then you didn't get an indexscan plan, and something is busted.
reset enable_seqscan;
--
1.9.1
0001-Fix-bug-with-whole-row-Vars-in-excluded-targetlist.patchtext/x-patch; charset=US-ASCII; name=0001-Fix-bug-with-whole-row-Vars-in-excluded-targetlist.patchDownload
From 39c85acd7b9c11d6b7c6612459bb4fb8f36ef45b Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <peter.geoghegan86@gmail.com>
Date: Mon, 25 May 2015 01:08:30 -0700
Subject: [PATCH 1/2] Fix bug with whole row Vars in excluded targetlist
Follow the approach taken with RETURNING clauses containing references
to multiple relations; pull up Vars referencing non-target relations
(limited here to the ON CONFLICT DO UPDATE excluded pseudo-relation)
that appear in ON CONFLICT DO UPDATE, and add the Vars to the excluded
relation targetlist as resjunk.
Have setrefs.c infrastructure avoid changing the varattno of whole row
Var resjunk attributes from ON CONFLICT related fix_join_expr() calls.
Failing to do so would now cause the executor to misidentify the Var as
a scalar Var rather than a whole row Var (it specifically recognizes
"varattno == InvalidAttrNumber" as representing the whole row case).
setrefs.c is also changed to call build_tlist_index_other_vars() rather
than build_tlist_index(); this isn't important for correctness, but it
seems preferable to make the code more consistent with the well
established set_returning_clause_references() case.
In passing, make a few minor stylistic tweaks, and reject Vars
referencing the excluded.* pseudo relation in an invalid manner. These
system column Vars are never useful or meaningful, since, of course,
excluded is not a real table.
---
src/backend/executor/execMain.c | 2 +
src/backend/executor/nodeModifyTable.c | 9 +++-
src/backend/optimizer/plan/planner.c | 14 +++--
src/backend/optimizer/plan/setrefs.c | 38 +++++++++++---
src/backend/optimizer/prep/preptlist.c | 74 +++++++++++++++++++++++++--
src/include/optimizer/prep.h | 4 +-
src/test/regress/expected/insert_conflict.out | 39 ++++++++++++++
src/test/regress/sql/insert_conflict.sql | 22 ++++++++
8 files changed, 179 insertions(+), 23 deletions(-)
diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c
index a1561ce..f2c0c64 100644
--- a/src/backend/executor/execMain.c
+++ b/src/backend/executor/execMain.c
@@ -1246,6 +1246,8 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo,
resultRelInfo->ri_ConstraintExprs = NULL;
resultRelInfo->ri_junkFilter = NULL;
resultRelInfo->ri_projectReturning = NULL;
+ resultRelInfo->ri_onConflictSetProj = NULL;
+ resultRelInfo->ri_onConflictSetWhere = NIL;
}
/*
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index 874ca6a..30a092b 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -1505,13 +1505,18 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags)
mtstate->resultRelInfo = estate->es_result_relations + node->resultRelIndex;
mtstate->mt_arowmarks = (List **) palloc0(sizeof(List *) * nplans);
mtstate->mt_nplans = nplans;
- mtstate->mt_onconflict = node->onConflictAction;
- mtstate->mt_arbiterindexes = node->arbiterIndexes;
/* set up epqstate with dummy subplan data for the moment */
EvalPlanQualInit(&mtstate->mt_epqstate, estate, NULL, NIL, node->epqParam);
mtstate->fireBSTriggers = true;
+ /* initialize ON CONFLICT data */
+ mtstate->mt_onconflict = node->onConflictAction;
+ mtstate->mt_arbiterindexes = node->arbiterIndexes;
+ mtstate->mt_existing = NULL;
+ mtstate->mt_excludedtlist = NIL;
+ mtstate->mt_conflproj = NULL;
+
/*
* call ExecInitNode on each of the plans to be executed and save the
* results into the array "mt_plans". This is also a convenient place to
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 8afde2b..4ce6ee85 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -488,7 +488,7 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
EXPRKIND_TARGET);
parse->onConflict->onConflictWhere =
- preprocess_expression(root, (Node *) parse->onConflict->onConflictWhere,
+ preprocess_expression(root, parse->onConflict->onConflictWhere,
EXPRKIND_QUAL);
}
@@ -1273,7 +1273,6 @@ grouping_planner(PlannerInfo *root, double tuple_fraction)
bool use_hashed_grouping = false;
WindowFuncLists *wflists = NULL;
List *activeWindows = NIL;
- OnConflictExpr *onconfl;
int maxref = 0;
int *tleref_to_colnum_map;
List *rollup_lists = NIL;
@@ -1364,12 +1363,11 @@ grouping_planner(PlannerInfo *root, double tuple_fraction)
/* Preprocess targetlist */
tlist = preprocess_targetlist(root, tlist);
- onconfl = parse->onConflict;
- if (onconfl)
- onconfl->onConflictSet =
- preprocess_onconflict_targetlist(onconfl->onConflictSet,
- parse->resultRelation,
- parse->rtable);
+ /* Preprocess targetlist for ON CONFLICT DO UPDATE */
+ if (parse->onConflict)
+ preprocess_onconflict_targetlist(parse->onConflict,
+ parse->resultRelation,
+ parse->rtable);
/*
* Expand any rangetable entries that have security barrier quals.
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index a7f65dd..fa683ab 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -33,6 +33,7 @@ typedef struct
Index varno; /* RT index of Var */
AttrNumber varattno; /* attr number of Var */
AttrNumber resno; /* TLE position of Var */
+ bool resjunk; /* TLE is resjunk? */
} tlist_vinfo;
typedef struct
@@ -41,6 +42,7 @@ typedef struct
int num_vars; /* number of plain Var tlist entries */
bool has_ph_vars; /* are there PlaceHolderVar entries? */
bool has_non_vars; /* are there other entries? */
+ bool change_resjunk; /* change varattno of resjunk entries? */
tlist_vinfo vars[FLEXIBLE_ARRAY_MEMBER]; /* has num_vars entries */
} indexed_tlist;
@@ -106,6 +108,8 @@ static void set_join_references(PlannerInfo *root, Join *join, int rtoffset);
static void set_upper_references(PlannerInfo *root, Plan *plan, int rtoffset);
static void set_dummy_tlist_references(Plan *plan, int rtoffset);
static indexed_tlist *build_tlist_index(List *tlist);
+static indexed_tlist *build_tlist_index_other_vars(List *tlist,
+ Index ignore_rel);
static Var *search_indexed_tlist_for_var(Var *var,
indexed_tlist *itlist,
Index newvarno,
@@ -762,7 +766,15 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset)
{
indexed_tlist *itlist;
- itlist = build_tlist_index(splan->exclRelTlist);
+ itlist = build_tlist_index_other_vars(splan->exclRelTlist,
+ linitial_int(splan->resultRelations));
+
+ /*
+ * Don't allow fix_join_expr to change the varattno of
+ * resjunk TLE's Vars. This can occur due to whole row Var
+ * excluded.* references, but for no other reason.
+ */
+ itlist->change_resjunk = false;
splan->onConflictSet =
fix_join_expr(root, splan->onConflictSet,
@@ -775,6 +787,7 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset)
NULL, itlist,
linitial_int(splan->resultRelations),
rtoffset);
+ pfree(itlist);
splan->exclRelTlist =
fix_scan_list(root, splan->exclRelTlist, rtoffset);
@@ -1722,6 +1735,7 @@ build_tlist_index(List *tlist)
itlist->tlist = tlist;
itlist->has_ph_vars = false;
itlist->has_non_vars = false;
+ itlist->change_resjunk = true;
/* Find the Vars and fill in the index array */
vinfo = itlist->vars;
@@ -1736,6 +1750,7 @@ build_tlist_index(List *tlist)
vinfo->varno = var->varno;
vinfo->varattno = var->varattno;
vinfo->resno = tle->resno;
+ vinfo->resjunk = tle->resjunk;
vinfo++;
}
else if (tle->expr && IsA(tle->expr, PlaceHolderVar))
@@ -1772,6 +1787,7 @@ build_tlist_index_other_vars(List *tlist, Index ignore_rel)
itlist->tlist = tlist;
itlist->has_ph_vars = false;
itlist->has_non_vars = false;
+ itlist->change_resjunk = true;
/* Find the desired Vars and fill in the index array */
vinfo = itlist->vars;
@@ -1788,6 +1804,7 @@ build_tlist_index_other_vars(List *tlist, Index ignore_rel)
vinfo->varno = var->varno;
vinfo->varattno = var->varattno;
vinfo->resno = tle->resno;
+ vinfo->resjunk = tle->resjunk;
vinfo++;
}
}
@@ -1827,7 +1844,11 @@ search_indexed_tlist_for_var(Var *var, indexed_tlist *itlist,
Var *newvar = copyVar(var);
newvar->varno = newvarno;
- newvar->varattno = vinfo->resno;
+ if (itlist->change_resjunk || !vinfo->resjunk)
+ newvar->varattno = vinfo->resno;
+ else
+ Assert(newvar->varattno == InvalidAttrNumber);
+
if (newvar->varnoold > 0)
newvar->varnoold += rtoffset;
return newvar;
@@ -1917,10 +1938,13 @@ search_indexed_tlist_for_sortgroupref(Node *node,
*
* This is used in two different scenarios: a normal join clause, where all
* the Vars in the clause *must* be replaced by OUTER_VAR or INNER_VAR
- * references; and a RETURNING clause, which may contain both Vars of the
- * target relation and Vars of other relations. In the latter case we want
- * to replace the other-relation Vars by OUTER_VAR references, while leaving
- * target Vars alone.
+ * references; and a RETURNING/ON CONFLICT DO UPDATE SET/ON CONFLICT DO UPDATE
+ * WHERE clause, which may contain both Vars of the target relation and Vars of
+ * other relations. In the latter case we want to replace the other-relation
+ * Vars by OUTER_VAR references (or INNER_VAR references for ON CONFLICT),
+ * while leaving target Vars alone. Also, for ON CONFLICT, caller has us avoid
+ * incrementing resjunk TLE Vars' varattno -- this is for the benefit of
+ * excluded.* whole row Vars.
*
* For a normal join, acceptable_rel should be zero so that any failure to
* match a Var will be reported as an error. For the RETURNING case, pass
@@ -1978,7 +2002,7 @@ fix_join_expr_mutator(Node *node, fix_join_expr_context *context)
return (Node *) newvar;
}
- /* Then in the outer */
+ /* Then in the inner */
if (context->inner_itlist)
{
newvar = search_indexed_tlist_for_var(var,
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 6b0c689..e6a5933 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -38,6 +38,8 @@
static List *expand_targetlist(List *tlist, int command_type,
Index result_relation, List *range_table);
+static void pull_var_targetlist_clause(OnConflictExpr *onconfl, List* clause,
+ int result_relation);
/*
@@ -185,12 +187,28 @@ preprocess_targetlist(PlannerInfo *root, List *tlist)
* preprocess_onconflict_targetlist
* Process ON CONFLICT SET targetlist.
*
- * Returns the new targetlist.
+ * Modifies passed ON CONFLICT expression in-place.
*/
-List *
-preprocess_onconflict_targetlist(List *tlist, int result_relation, List *range_table)
+void
+preprocess_onconflict_targetlist(OnConflictExpr *onconfl, int result_relation,
+ List *range_table)
{
- return expand_targetlist(tlist, CMD_UPDATE, result_relation, range_table);
+ List *tlist;
+
+ tlist = expand_targetlist(onconfl->onConflictSet, CMD_UPDATE,
+ result_relation, range_table);
+
+ /*
+ * Perform similar pull up process to that perfomred by
+ * preprocess_targetlist() for multi-relation RETURNING lists. This is
+ * only necessary for the benefit of excluded.* whole row Vars, and to
+ * enforce that excluded.* system column references are disallowed.
+ */
+ pull_var_targetlist_clause(onconfl, tlist, result_relation);
+ pull_var_targetlist_clause(onconfl, (List *) onconfl->onConflictWhere,
+ result_relation);
+
+ onconfl->onConflictSet = tlist;
}
@@ -376,6 +394,54 @@ expand_targetlist(List *tlist, int command_type,
return new_tlist;
}
+/*
+ * pull_var_targetlist_clause
+ * Given an ON CONFLICT clause as generated by the parser and a result
+ * relation, add resjunk entries for any Vars used that are not
+ * associated with the target and are not excluded.* non-resjunk
+ * TLE Vars.
+ */
+static void
+pull_var_targetlist_clause(OnConflictExpr *onconfl, List* clause,
+ int result_relation)
+{
+ List *vars;
+ ListCell *l;
+
+ vars = pull_var_clause((Node *) clause,
+ PVC_RECURSE_AGGREGATES,
+ PVC_INCLUDE_PLACEHOLDERS);
+ foreach(l, vars)
+ {
+ Var *var = (Var *) lfirst(l);
+ TargetEntry *tle;
+
+ if (IsA(var, Var) && var->varno == result_relation)
+ continue; /* don't need it */
+
+ if (IsA(var, Var) && var->varno == onconfl->exclRelIndex)
+ {
+ if (var->varattno < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("system columns from excluded table cannot be referenced")));
+ else if (var->varattno > 0) /* don't need it */
+ continue;
+ }
+
+ if (tlist_member((Node *) var, onconfl->exclRelTlist))
+ continue; /* already got it */
+
+ tle = makeTargetEntry((Expr *) var,
+ list_length(clause) + 1,
+ NULL,
+ true);
+
+ onconfl->exclRelTlist = lappend(onconfl->exclRelTlist, tle);
+ }
+
+ list_free(vars);
+}
/*
* Locate PlanRowMark for given RT index, or return NULL if none
diff --git a/src/include/optimizer/prep.h b/src/include/optimizer/prep.h
index 7b8c0a9..8d61c7c 100644
--- a/src/include/optimizer/prep.h
+++ b/src/include/optimizer/prep.h
@@ -45,8 +45,8 @@ extern void expand_security_quals(PlannerInfo *root, List *tlist);
*/
extern List *preprocess_targetlist(PlannerInfo *root, List *tlist);
-extern List *preprocess_onconflict_targetlist(List *tlist,
- int result_relation, List *range_table);
+extern void preprocess_onconflict_targetlist(OnConflictExpr *onconfl,
+ int result_relation, List *range_table);
extern PlanRowMark *get_plan_rowmark(List *rowmarks, Index rtindex);
diff --git a/src/test/regress/expected/insert_conflict.out b/src/test/regress/expected/insert_conflict.out
index eca9690..442d9fc 100644
--- a/src/test/regress/expected/insert_conflict.out
+++ b/src/test/regress/expected/insert_conflict.out
@@ -363,6 +363,45 @@ ERROR: there is no unique or exclusion constraint matching the ON CONFLICT spec
insert into insertconflicttest values (23, 'Blackberry') on conflict (fruit) where fruit like '%berry' do update set fruit = excluded.fruit;
ERROR: there is no unique or exclusion constraint matching the ON CONFLICT specification
drop index partial_key_index;
+--
+-- Test resjunk in excluded.* pseudo-relation
+--
+create unique index plain on insertconflicttest(key);
+-- Succeeds, updates existing row:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* != excluded.* returning *;
+ key | fruit
+-----+-----------
+ 23 | Jackfruit
+(1 row)
+
+-- No update this time, though:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* != excluded.* returning *;
+ key | fruit
+-----+-------
+(0 rows)
+
+-- Predicate changed to require match rather than non-match, so updates once more:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* = excluded.* returning *;
+ key | fruit
+-----+-----------
+ 23 | Jackfruit
+(1 row)
+
+-- Assign:
+insert into insertconflicttest as i values (23, 'Avocado') on conflict (key) do update set fruit = excluded.*::text
+ returning *;
+ key | fruit
+-----+--------------
+ 23 | (23,Avocado)
+(1 row)
+
+-- Forbidden, fails:
+insert into insertconflicttest values (23, 'Blackberry') on conflict (key) do update set fruit = excluded.ctid;
+ERROR: system columns from excluded table cannot be referenced
+drop index plain;
-- Cleanup
drop table insertconflicttest;
-- ******************************************************************
diff --git a/src/test/regress/sql/insert_conflict.sql b/src/test/regress/sql/insert_conflict.sql
index a0bdd7f..a2b6aba 100644
--- a/src/test/regress/sql/insert_conflict.sql
+++ b/src/test/regress/sql/insert_conflict.sql
@@ -212,6 +212,28 @@ insert into insertconflicttest values (23, 'Blackberry') on conflict (fruit) whe
drop index partial_key_index;
+--
+-- Test resjunk in excluded.* pseudo-relation
+--
+create unique index plain on insertconflicttest(key);
+
+-- Succeeds, updates existing row:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* != excluded.* returning *;
+-- No update this time, though:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* != excluded.* returning *;
+-- Predicate changed to require match rather than non-match, so updates once more:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* = excluded.* returning *;
+-- Assign:
+insert into insertconflicttest as i values (23, 'Avocado') on conflict (key) do update set fruit = excluded.*::text
+ returning *;
+-- Forbidden, fails:
+insert into insertconflicttest values (23, 'Blackberry') on conflict (key) do update set fruit = excluded.ctid;
+
+drop index plain;
+
-- Cleanup
drop table insertconflicttest;
--
1.9.1
On Thu, May 28, 2015 at 2:37 PM, Peter Geoghegan <pg@heroku.com> wrote:
A second attached patch fixes another, largely independent bug. I
noticed array assignment with ON CONFLICT DO UPDATE was broken. See
commit message for full details.
Finally, here is a third patch, fixing the final bug that I discussed
with you privately. There are now fixes for all bugs that I'm
currently aware of.
This concerns a thinko in unique index inference. See the commit
message for full details.
--
Peter Geoghegan
Attachments:
0003-Fix-bug-in-unique-index-inference.patchtext/x-patch; charset=US-ASCII; name=0003-Fix-bug-in-unique-index-inference.patchDownload
From 904f46f4d9e7758c588254e2c2bb3ef3ef93f3c9 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <peter.geoghegan86@gmail.com>
Date: Thu, 28 May 2015 15:45:48 -0700
Subject: [PATCH 3/3] Fix bug in unique index inference
ON CONFLICT unique index inference had a thinko that could affect cases
where the user-supplied inference clause required that an attribute
match a particular (user named) collation and/or opclass.
Firstly, infer_collation_opclass_match() matched on opclass and/or
collation. Secondly, the attribute had to be in the list of attributes
or expressions known to be in the definition of the index under
consideration. The second step wasn't correct though, because having
some match doesn't necessarily mean that the second step found the same
index attribute as the (collation/opclass wise) match from the first
step.
To fix, make infer_collation_opclass_match() more discriminating in its
second step.
---
src/backend/optimizer/util/plancat.c | 45 +++++++++++++++++----------
src/test/regress/expected/insert_conflict.out | 18 +++++++++++
src/test/regress/sql/insert_conflict.sql | 12 +++++++
3 files changed, 59 insertions(+), 16 deletions(-)
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index b04dc2e..1b81f4c 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -52,7 +52,7 @@ get_relation_info_hook_type get_relation_info_hook = NULL;
static bool infer_collation_opclass_match(InferenceElem *elem, Relation idxRel,
- Bitmapset *inferAttrs, List *idxExprs);
+ List *idxExprs);
static int32 get_rel_data_width(Relation rel, int32 *attr_widths);
static List *get_relation_constraints(PlannerInfo *root,
Oid relationObjectId, RelOptInfo *rel,
@@ -615,8 +615,7 @@ infer_arbiter_indexes(PlannerInfo *root)
* this for both expressions and ordinary (non-expression)
* attributes appearing as inference elements.
*/
- if (!infer_collation_opclass_match(elem, idxRel, inferAttrs,
- idxExprs))
+ if (!infer_collation_opclass_match(elem, idxRel, idxExprs))
goto next;
/*
@@ -681,11 +680,10 @@ next:
* infer_collation_opclass_match - ensure infer element opclass/collation match
*
* Given unique index inference element from inference specification, if
- * collation was specified, or if opclass (represented here as opfamily +
- * opcintype) was specified, verify that there is at least one matching
- * indexed attribute (occasionally, there may be more). Skip this in the
- * common case where inference specification does not include collation or
- * opclass (instead matching everything, regardless of cataloged
+ * collation was specified, or if opclass was specified, verify that there is
+ * at least one matching indexed attribute (occasionally, there may be more).
+ * Skip this in the common case where inference specification does not include
+ * collation or opclass (instead matching everything, regardless of cataloged
* collation/opclass of indexed attribute).
*
* At least historically, Postgres has not offered collations or opclasses
@@ -707,11 +705,12 @@ next:
*/
static bool
infer_collation_opclass_match(InferenceElem *elem, Relation idxRel,
- Bitmapset *inferAttrs, List *idxExprs)
+ List *idxExprs)
{
AttrNumber natt;
- Oid inferopfamily = InvalidOid; /* OID of att opfamily */
- Oid inferopcinputtype = InvalidOid; /* OID of att opfamily */
+ Oid inferopfamily = InvalidOid; /* OID of opclass opfamily */
+ Oid inferopcinputtype = InvalidOid; /* OID of opclass input type */
+ int nplain = 0; /* # plain attrs observed */
/*
* If inference specification element lacks collation/opclass, then no
@@ -734,6 +733,10 @@ infer_collation_opclass_match(InferenceElem *elem, Relation idxRel,
Oid opfamily = idxRel->rd_opfamily[natt - 1];
Oid opcinputtype = idxRel->rd_opcintype[natt - 1];
Oid collation = idxRel->rd_indcollation[natt - 1];
+ int attno = idxRel->rd_index->indkey.values[natt - 1];
+
+ if (attno != 0)
+ nplain++;
if (elem->inferopclass != InvalidOid &&
(inferopfamily != opfamily || inferopcinputtype != opcinputtype))
@@ -749,12 +752,22 @@ infer_collation_opclass_match(InferenceElem *elem, Relation idxRel,
continue;
}
- if ((IsA(elem->expr, Var) &&
- bms_is_member(((Var *) elem->expr)->varattno, inferAttrs)) ||
- list_member(idxExprs, elem->expr))
+ /* If one matching index att found, good enough -- return true */
+ if (IsA(elem->expr, Var))
{
- /* Found one match - good enough */
- return true;
+ if (((Var *) elem->expr)->varattno == attno)
+ return true;
+ }
+ else
+ {
+ Node *nattExpr = list_nth(idxExprs, (natt - 1) - nplain);
+
+ /*
+ * Note that unlike routines like match_index_to_operand(), we're
+ * unconcerned about RelabelType. An exact match is required.
+ */
+ if (equal(elem->expr, nattExpr))
+ return true;
}
}
diff --git a/src/test/regress/expected/insert_conflict.out b/src/test/regress/expected/insert_conflict.out
index 442d9fc..8aa0918 100644
--- a/src/test/regress/expected/insert_conflict.out
+++ b/src/test/regress/expected/insert_conflict.out
@@ -141,6 +141,24 @@ drop index collation_index_key;
drop index both_index_key;
drop index both_index_expr_key;
--
+-- Make sure that cross matching of attribute opclass/collation does not occur
+--
+create unique index cross_match on insertconflicttest(lower(fruit) collate "C", upper(fruit) text_pattern_ops);
+-- fails:
+explain (costs off) insert into insertconflicttest values(0, 'Crowberry') on conflict (lower(fruit) text_pattern_ops, upper(fruit) collate "C") do nothing;
+ERROR: there is no unique or exclusion constraint matching the ON CONFLICT specification
+-- works:
+explain (costs off) insert into insertconflicttest values(0, 'Crowberry') on conflict (lower(fruit) collate "C", upper(fruit) text_pattern_ops) do nothing;
+ QUERY PLAN
+-----------------------------------------
+ Insert on insertconflicttest
+ Conflict Resolution: NOTHING
+ Conflict Arbiter Indexes: cross_match
+ -> Result
+(4 rows)
+
+drop index cross_match;
+--
-- Single key tests
--
create unique index key_index on insertconflicttest(key);
diff --git a/src/test/regress/sql/insert_conflict.sql b/src/test/regress/sql/insert_conflict.sql
index a2b6aba..72440af 100644
--- a/src/test/regress/sql/insert_conflict.sql
+++ b/src/test/regress/sql/insert_conflict.sql
@@ -58,6 +58,18 @@ drop index both_index_key;
drop index both_index_expr_key;
--
+-- Make sure that cross matching of attribute opclass/collation does not occur
+--
+create unique index cross_match on insertconflicttest(lower(fruit) collate "C", upper(fruit) text_pattern_ops);
+
+-- fails:
+explain (costs off) insert into insertconflicttest values(0, 'Crowberry') on conflict (lower(fruit) text_pattern_ops, upper(fruit) collate "C") do nothing;
+-- works:
+explain (costs off) insert into insertconflicttest values(0, 'Crowberry') on conflict (lower(fruit) collate "C", upper(fruit) text_pattern_ops) do nothing;
+
+drop index cross_match;
+
+--
-- Single key tests
--
create unique index key_index on insertconflicttest(key);
--
1.9.1
On Thu, May 28, 2015 at 6:31 PM, Peter Geoghegan <pg@heroku.com> wrote:
This concerns a thinko in unique index inference. See the commit
message for full details.
It seems I missed a required defensive measure here. Attached patch
adds it, too.
--
Peter Geoghegan
Attachments:
0004-Additional-defensive-measure.patchtext/x-patch; charset=US-ASCII; name=0004-Additional-defensive-measure.patchDownload
From c5aee669bbdf58f38f1063934a1d93286506de7b Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <peter.geoghegan86@gmail.com>
Date: Fri, 29 May 2015 02:53:41 -0700
Subject: [PATCH 4/4] Additional defensive measure
---
src/backend/optimizer/util/plancat.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 1b81f4c..4f38972 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -758,7 +758,7 @@ infer_collation_opclass_match(InferenceElem *elem, Relation idxRel,
if (((Var *) elem->expr)->varattno == attno)
return true;
}
- else
+ else if (attno == 0)
{
Node *nattExpr = list_nth(idxExprs, (natt - 1) - nplain);
--
1.9.1
On Thu, May 28, 2015 at 2:37 PM, Peter Geoghegan <pg@heroku.com> wrote:
Attached, revised version incorporates this small fix, while adding an
additional big fix, and a number of small style tweaks.This is mainly concerned with fixing the bug I was trying to fix when
I spotted the minor pfree() issue:postgres=# insert into upsert (key, val) values('Foo', 'Bar') on
conflict (key) do update set val = excluded.val where excluded.* is
not null;
ERROR: XX000: variable not found in subplan target lists
LOCATION: fix_join_expr_mutator, setrefs.c:2003
My fix for this issue
(0001-Fix-bug-with-whole-row-Vars-in-excluded-targetlist.patch) still
missed something. There needs to be additional handling in
ruleutils.c:
postgres=# explain insert into upsert as u
values (1, 'fooz') on conflict (key)
do update set val = excluded.val where excluded.* is not null;
ERROR: XX000: bogus varattno for INNER_VAR var: 0
LOCATION: get_variable, ruleutils.c:5904
I'll look for a fix for this additional issue tomorrow.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Sat, May 30, 2015 at 1:07 AM, Peter Geoghegan <pg@heroku.com> wrote:
My fix for this issue
(0001-Fix-bug-with-whole-row-Vars-in-excluded-targetlist.patch) still
missed something. There needs to be additional handling in
ruleutils.c:
Debugging this allowed me to come up with a significantly simplified
approach. Attached is a new version of the original fix. Details are
in commit message -- there is no actual need to have
search_indexed_tlist_for_var() care about Vars being resjunk in a
special way, which is a good thing. There is also no need for further
ruleutils.c specialization, as I implied before.
Some deparsing tests are now included on top of what was already in
the first version.
--
Peter Geoghegan
Attachments:
0001-Fix-bug-with-whole-row-Vars-in-excluded-targetlist.patchtext/x-patch; charset=US-ASCII; name=0001-Fix-bug-with-whole-row-Vars-in-excluded-targetlist.patchDownload
From 9d2f2b51ed3640a6a89313b7be3365168746ee00 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <peter.geoghegan86@gmail.com>
Date: Mon, 25 May 2015 01:08:30 -0700
Subject: [PATCH 1/6] Fix bug with whole row Vars in excluded targetlist
Follow the approach taken with RETURNING clauses containing references
to multiple relations; pull up Vars referencing non-target relations
(limited here to the ON CONFLICT DO UPDATE excluded pseudo-relation)
that appear in ON CONFLICT DO UPDATE, and add the Vars to the excluded
relation targetlist as resjunk.
setrefs.c is also changed to call build_tlist_index_other_vars() rather
than build_tlist_index(); this isn't important for correctness, but it
seems preferable to make the code more consistent with the well
established set_returning_clause_references() case.
In passing, make a few minor stylistic tweaks, and reject Vars
referencing the excluded.* pseudo relation in an invalid manner. These
system column Vars are never useful or meaningful, since, of course,
excluded is not a real table.
---
src/backend/executor/execMain.c | 2 +
src/backend/executor/nodeModifyTable.c | 9 +++-
src/backend/optimizer/plan/planner.c | 14 +++--
src/backend/optimizer/plan/setrefs.c | 18 ++++---
src/backend/optimizer/prep/preptlist.c | 74 +++++++++++++++++++++++++--
src/include/optimizer/prep.h | 4 +-
src/test/regress/expected/insert_conflict.out | 50 ++++++++++++++++++
src/test/regress/expected/rules.out | 18 +++----
src/test/regress/sql/insert_conflict.sql | 24 +++++++++
src/test/regress/sql/rules.sql | 2 +-
10 files changed, 183 insertions(+), 32 deletions(-)
diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c
index a1561ce..f2c0c64 100644
--- a/src/backend/executor/execMain.c
+++ b/src/backend/executor/execMain.c
@@ -1246,6 +1246,8 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo,
resultRelInfo->ri_ConstraintExprs = NULL;
resultRelInfo->ri_junkFilter = NULL;
resultRelInfo->ri_projectReturning = NULL;
+ resultRelInfo->ri_onConflictSetProj = NULL;
+ resultRelInfo->ri_onConflictSetWhere = NIL;
}
/*
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index 874ca6a..30a092b 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -1505,13 +1505,18 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags)
mtstate->resultRelInfo = estate->es_result_relations + node->resultRelIndex;
mtstate->mt_arowmarks = (List **) palloc0(sizeof(List *) * nplans);
mtstate->mt_nplans = nplans;
- mtstate->mt_onconflict = node->onConflictAction;
- mtstate->mt_arbiterindexes = node->arbiterIndexes;
/* set up epqstate with dummy subplan data for the moment */
EvalPlanQualInit(&mtstate->mt_epqstate, estate, NULL, NIL, node->epqParam);
mtstate->fireBSTriggers = true;
+ /* initialize ON CONFLICT data */
+ mtstate->mt_onconflict = node->onConflictAction;
+ mtstate->mt_arbiterindexes = node->arbiterIndexes;
+ mtstate->mt_existing = NULL;
+ mtstate->mt_excludedtlist = NIL;
+ mtstate->mt_conflproj = NULL;
+
/*
* call ExecInitNode on each of the plans to be executed and save the
* results into the array "mt_plans". This is also a convenient place to
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 8afde2b..4ce6ee85 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -488,7 +488,7 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
EXPRKIND_TARGET);
parse->onConflict->onConflictWhere =
- preprocess_expression(root, (Node *) parse->onConflict->onConflictWhere,
+ preprocess_expression(root, parse->onConflict->onConflictWhere,
EXPRKIND_QUAL);
}
@@ -1273,7 +1273,6 @@ grouping_planner(PlannerInfo *root, double tuple_fraction)
bool use_hashed_grouping = false;
WindowFuncLists *wflists = NULL;
List *activeWindows = NIL;
- OnConflictExpr *onconfl;
int maxref = 0;
int *tleref_to_colnum_map;
List *rollup_lists = NIL;
@@ -1364,12 +1363,11 @@ grouping_planner(PlannerInfo *root, double tuple_fraction)
/* Preprocess targetlist */
tlist = preprocess_targetlist(root, tlist);
- onconfl = parse->onConflict;
- if (onconfl)
- onconfl->onConflictSet =
- preprocess_onconflict_targetlist(onconfl->onConflictSet,
- parse->resultRelation,
- parse->rtable);
+ /* Preprocess targetlist for ON CONFLICT DO UPDATE */
+ if (parse->onConflict)
+ preprocess_onconflict_targetlist(parse->onConflict,
+ parse->resultRelation,
+ parse->rtable);
/*
* Expand any rangetable entries that have security barrier quals.
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index a7f65dd..24d50bf 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -106,6 +106,8 @@ static void set_join_references(PlannerInfo *root, Join *join, int rtoffset);
static void set_upper_references(PlannerInfo *root, Plan *plan, int rtoffset);
static void set_dummy_tlist_references(Plan *plan, int rtoffset);
static indexed_tlist *build_tlist_index(List *tlist);
+static indexed_tlist *build_tlist_index_other_vars(List *tlist,
+ Index ignore_rel);
static Var *search_indexed_tlist_for_var(Var *var,
indexed_tlist *itlist,
Index newvarno,
@@ -762,7 +764,9 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset)
{
indexed_tlist *itlist;
- itlist = build_tlist_index(splan->exclRelTlist);
+ itlist = build_tlist_index_other_vars(splan->exclRelTlist,
+ linitial_int(splan->resultRelations));
+
splan->onConflictSet =
fix_join_expr(root, splan->onConflictSet,
@@ -775,6 +779,7 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset)
NULL, itlist,
linitial_int(splan->resultRelations),
rtoffset);
+ pfree(itlist);
splan->exclRelTlist =
fix_scan_list(root, splan->exclRelTlist, rtoffset);
@@ -1917,10 +1922,11 @@ search_indexed_tlist_for_sortgroupref(Node *node,
*
* This is used in two different scenarios: a normal join clause, where all
* the Vars in the clause *must* be replaced by OUTER_VAR or INNER_VAR
- * references; and a RETURNING clause, which may contain both Vars of the
- * target relation and Vars of other relations. In the latter case we want
- * to replace the other-relation Vars by OUTER_VAR references, while leaving
- * target Vars alone.
+ * references; and a RETURNING/ON CONFLICT DO UPDATE SET/ON CONFLICT DO UPDATE
+ * WHERE clause, which may contain both Vars of the target relation and Vars of
+ * other relations. In the latter case we want to replace the other-relation
+ * Vars by OUTER_VAR references (or INNER_VAR references for ON CONFLICT),
+ * while leaving target Vars alone.
*
* For a normal join, acceptable_rel should be zero so that any failure to
* match a Var will be reported as an error. For the RETURNING case, pass
@@ -1978,7 +1984,7 @@ fix_join_expr_mutator(Node *node, fix_join_expr_context *context)
return (Node *) newvar;
}
- /* Then in the outer */
+ /* Then in the inner */
if (context->inner_itlist)
{
newvar = search_indexed_tlist_for_var(var,
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 6b0c689..6b7adb7 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -38,6 +38,8 @@
static List *expand_targetlist(List *tlist, int command_type,
Index result_relation, List *range_table);
+static void pull_var_targetlist_clause(OnConflictExpr *onconfl, List* clause,
+ int result_relation);
/*
@@ -185,12 +187,28 @@ preprocess_targetlist(PlannerInfo *root, List *tlist)
* preprocess_onconflict_targetlist
* Process ON CONFLICT SET targetlist.
*
- * Returns the new targetlist.
+ * Modifies passed ON CONFLICT expression in-place.
*/
-List *
-preprocess_onconflict_targetlist(List *tlist, int result_relation, List *range_table)
+void
+preprocess_onconflict_targetlist(OnConflictExpr *onconfl, int result_relation,
+ List *range_table)
{
- return expand_targetlist(tlist, CMD_UPDATE, result_relation, range_table);
+ List *tlist;
+
+ tlist = expand_targetlist(onconfl->onConflictSet, CMD_UPDATE,
+ result_relation, range_table);
+
+ /*
+ * Perform similar pull up process to that performed by
+ * preprocess_targetlist() for multi-relation RETURNING lists. This is
+ * only necessary for the benefit of excluded.* whole row Vars, and to
+ * enforce that excluded.* system column references are disallowed.
+ */
+ pull_var_targetlist_clause(onconfl, tlist, result_relation);
+ pull_var_targetlist_clause(onconfl, (List *) onconfl->onConflictWhere,
+ result_relation);
+
+ onconfl->onConflictSet = tlist;
}
@@ -376,6 +394,54 @@ expand_targetlist(List *tlist, int command_type,
return new_tlist;
}
+/*
+ * pull_var_targetlist_clause
+ * Given an ON CONFLICT clause as generated by the parser and a result
+ * relation, add resjunk entries for any Vars used that are not
+ * associated with the target and are not excluded.* non-resjunk
+ * TLE Vars.
+ */
+static void
+pull_var_targetlist_clause(OnConflictExpr *onconfl, List* clause,
+ int result_relation)
+{
+ List *vars;
+ ListCell *l;
+
+ vars = pull_var_clause((Node *) clause,
+ PVC_RECURSE_AGGREGATES,
+ PVC_INCLUDE_PLACEHOLDERS);
+ foreach(l, vars)
+ {
+ Var *var = (Var *) lfirst(l);
+ TargetEntry *tle;
+
+ if (IsA(var, Var) && var->varno == result_relation)
+ continue; /* don't need it */
+
+ if (IsA(var, Var) && var->varno == onconfl->exclRelIndex)
+ {
+ if (var->varattno < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
+ errmsg("system columns from excluded table cannot be referenced")));
+ else if (var->varattno > 0) /* don't need it */
+ continue;
+ }
+
+ if (tlist_member((Node *) var, onconfl->exclRelTlist))
+ continue; /* already got it */
+
+ if (var->varattno != InvalidAttrNumber)
+ elog(ERROR, "cannot pull up non-wholerow Var in excluded targetlist");
+
+ tle = makeTargetEntry((Expr *) var, var->varattno, NULL, true);
+
+ onconfl->exclRelTlist = lappend(onconfl->exclRelTlist, tle);
+ }
+
+ list_free(vars);
+}
/*
* Locate PlanRowMark for given RT index, or return NULL if none
diff --git a/src/include/optimizer/prep.h b/src/include/optimizer/prep.h
index 7b8c0a9..8d61c7c 100644
--- a/src/include/optimizer/prep.h
+++ b/src/include/optimizer/prep.h
@@ -45,8 +45,8 @@ extern void expand_security_quals(PlannerInfo *root, List *tlist);
*/
extern List *preprocess_targetlist(PlannerInfo *root, List *tlist);
-extern List *preprocess_onconflict_targetlist(List *tlist,
- int result_relation, List *range_table);
+extern void preprocess_onconflict_targetlist(OnConflictExpr *onconfl,
+ int result_relation, List *range_table);
extern PlanRowMark *get_plan_rowmark(List *rowmarks, Index rtindex);
diff --git a/src/test/regress/expected/insert_conflict.out b/src/test/regress/expected/insert_conflict.out
index eca9690..0d8b26b 100644
--- a/src/test/regress/expected/insert_conflict.out
+++ b/src/test/regress/expected/insert_conflict.out
@@ -363,6 +363,56 @@ ERROR: there is no unique or exclusion constraint matching the ON CONFLICT spec
insert into insertconflicttest values (23, 'Blackberry') on conflict (fruit) where fruit like '%berry' do update set fruit = excluded.fruit;
ERROR: there is no unique or exclusion constraint matching the ON CONFLICT specification
drop index partial_key_index;
+--
+-- Test resjunk in excluded.* pseudo-relation
+--
+create unique index plain on insertconflicttest(key);
+-- Succeeds, updates existing row:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* != excluded.* returning *;
+ key | fruit
+-----+-----------
+ 23 | Jackfruit
+(1 row)
+
+-- No update this time, though:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* != excluded.* returning *;
+ key | fruit
+-----+-------
+(0 rows)
+
+-- Predicate changed to require match rather than non-match, so updates once more:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* = excluded.* returning *;
+ key | fruit
+-----+-----------
+ 23 | Jackfruit
+(1 row)
+
+-- Assign:
+insert into insertconflicttest as i values (23, 'Avocado') on conflict (key) do update set fruit = excluded.*::text
+ returning *;
+ key | fruit
+-----+--------------
+ 23 | (23,Avocado)
+(1 row)
+
+-- deparse whole row var in WHERE clause:
+explain (costs off) insert into insertconflicttest as i values (23, 'Avocado') on conflict (key) do update set fruit = excluded.fruit where excluded.* is null;
+ QUERY PLAN
+-----------------------------------------
+ Insert on insertconflicttest i
+ Conflict Resolution: UPDATE
+ Conflict Arbiter Indexes: plain
+ Conflict Filter: (excluded.* IS NULL)
+ -> Result
+(5 rows)
+
+-- Forbidden, fails:
+insert into insertconflicttest values (23, 'Blackberry') on conflict (key) do update set fruit = excluded.ctid;
+ERROR: system columns from excluded table cannot be referenced
+drop index plain;
-- Cleanup
drop table insertconflicttest;
-- ******************************************************************
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 60c1f40..23dda7d 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2893,7 +2893,7 @@ CREATE RULE hat_upsert AS ON INSERT TO hats
ON CONFLICT (hat_name)
DO UPDATE
SET hat_name = hat_data.hat_name, hat_color = excluded.hat_color
- WHERE excluded.hat_color <> 'forbidden'
+ WHERE excluded.hat_color <> 'forbidden' AND hat_data.* != excluded.*
RETURNING *;
SELECT definition FROM pg_rules WHERE tablename = 'hats' ORDER BY rulename;
definition
@@ -2901,7 +2901,7 @@ SELECT definition FROM pg_rules WHERE tablename = 'hats' ORDER BY rulename;
CREATE RULE hat_upsert AS +
ON INSERT TO hats DO INSTEAD INSERT INTO hat_data (hat_name, hat_color) +
VALUES (new.hat_name, new.hat_color) ON CONFLICT(hat_name) DO UPDATE SET hat_name = hat_data.hat_name, hat_color = excluded.hat_color+
- WHERE (excluded.hat_color <> 'forbidden'::bpchar) +
+ WHERE ((excluded.hat_color <> 'forbidden'::bpchar) AND (hat_data.* <> excluded.*)) +
RETURNING hat_data.hat_name, +
hat_data.hat_color;
(1 row)
@@ -2949,19 +2949,19 @@ SELECT tablename, rulename, definition FROM pg_rules
hats | hat_upsert | CREATE RULE hat_upsert AS +
| | ON INSERT TO hats DO INSTEAD INSERT INTO hat_data (hat_name, hat_color) +
| | VALUES (new.hat_name, new.hat_color) ON CONFLICT(hat_name) DO UPDATE SET hat_name = hat_data.hat_name, hat_color = excluded.hat_color+
- | | WHERE (excluded.hat_color <> 'forbidden'::bpchar) +
+ | | WHERE ((excluded.hat_color <> 'forbidden'::bpchar) AND (hat_data.* <> excluded.*)) +
| | RETURNING hat_data.hat_name, +
| | hat_data.hat_color;
(1 row)
-- ensure explain works for on insert conflict rules
explain (costs off) INSERT INTO hats VALUES ('h8', 'forbidden') RETURNING *;
- QUERY PLAN
-----------------------------------------------------------------
+ QUERY PLAN
+-------------------------------------------------------------------------------------------------
Insert on hat_data
Conflict Resolution: UPDATE
Conflict Arbiter Indexes: hat_data_unique_idx
- Conflict Filter: (excluded.hat_color <> 'forbidden'::bpchar)
+ Conflict Filter: ((excluded.hat_color <> 'forbidden'::bpchar) AND (hat_data.* <> excluded.*))
-> Result
(5 rows)
@@ -2988,12 +2988,12 @@ EXPLAIN (costs off) WITH data(hat_name, hat_color) AS (
INSERT INTO hats
SELECT * FROM data
RETURNING *;
- QUERY PLAN
-----------------------------------------------------------------
+ QUERY PLAN
+-------------------------------------------------------------------------------------------------
Insert on hat_data
Conflict Resolution: UPDATE
Conflict Arbiter Indexes: hat_data_unique_idx
- Conflict Filter: (excluded.hat_color <> 'forbidden'::bpchar)
+ Conflict Filter: ((excluded.hat_color <> 'forbidden'::bpchar) AND (hat_data.* <> excluded.*))
CTE data
-> Values Scan on "*VALUES*"
-> CTE Scan on data
diff --git a/src/test/regress/sql/insert_conflict.sql b/src/test/regress/sql/insert_conflict.sql
index a0bdd7f..42a9b54 100644
--- a/src/test/regress/sql/insert_conflict.sql
+++ b/src/test/regress/sql/insert_conflict.sql
@@ -212,6 +212,30 @@ insert into insertconflicttest values (23, 'Blackberry') on conflict (fruit) whe
drop index partial_key_index;
+--
+-- Test resjunk in excluded.* pseudo-relation
+--
+create unique index plain on insertconflicttest(key);
+
+-- Succeeds, updates existing row:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* != excluded.* returning *;
+-- No update this time, though:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* != excluded.* returning *;
+-- Predicate changed to require match rather than non-match, so updates once more:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* = excluded.* returning *;
+-- Assign:
+insert into insertconflicttest as i values (23, 'Avocado') on conflict (key) do update set fruit = excluded.*::text
+ returning *;
+-- deparse whole row var in WHERE clause:
+explain (costs off) insert into insertconflicttest as i values (23, 'Avocado') on conflict (key) do update set fruit = excluded.fruit where excluded.* is null;
+-- Forbidden, fails:
+insert into insertconflicttest values (23, 'Blackberry') on conflict (key) do update set fruit = excluded.ctid;
+
+drop index plain;
+
-- Cleanup
drop table insertconflicttest;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
index 561e2fd..4299a5b 100644
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -1105,7 +1105,7 @@ CREATE RULE hat_upsert AS ON INSERT TO hats
ON CONFLICT (hat_name)
DO UPDATE
SET hat_name = hat_data.hat_name, hat_color = excluded.hat_color
- WHERE excluded.hat_color <> 'forbidden'
+ WHERE excluded.hat_color <> 'forbidden' AND hat_data.* != excluded.*
RETURNING *;
SELECT definition FROM pg_rules WHERE tablename = 'hats' ORDER BY rulename;
--
1.9.1
On Sat, May 30, 2015 at 3:12 PM, Peter Geoghegan <pg@heroku.com> wrote:
Debugging this allowed me to come up with a significantly simplified
approach. Attached is a new version of the original fix. Details are
in commit message -- there is no actual need to have
search_indexed_tlist_for_var() care about Vars being resjunk in a
special way, which is a good thing.
It feels wrong to not have the additional, paranoid IsVar() check
within pull_var_targetlist_clause() check added in most recent
revision, even though it should not be necessary. Attached delta patch
adds this check.
I need to stop working on weekends...
--
Peter Geoghegan
Attachments:
additional-paranoid-check.patchtext/x-patch; charset=US-ASCII; name=additional-paranoid-check.patchDownload
From 8542ff2623d7f2142ecb8c21c572b47d67500231 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <peter.geoghegan86@gmail.com>
Date: Sat, 30 May 2015 15:25:55 -0700
Subject: [PATCH 2/7] Add additional, paranoid check
---
src/backend/optimizer/prep/preptlist.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c
index 6b7adb7..8c418d2 100644
--- a/src/backend/optimizer/prep/preptlist.c
+++ b/src/backend/optimizer/prep/preptlist.c
@@ -432,7 +432,7 @@ pull_var_targetlist_clause(OnConflictExpr *onconfl, List* clause,
if (tlist_member((Node *) var, onconfl->exclRelTlist))
continue; /* already got it */
- if (var->varattno != InvalidAttrNumber)
+ if (!IsA(var, Var) || var->varattno != InvalidAttrNumber)
elog(ERROR, "cannot pull up non-wholerow Var in excluded targetlist");
tle = makeTargetEntry((Expr *) var, var->varattno, NULL, true);
--
1.9.1
On 2015-05-28 14:37:43 -0700, Peter Geoghegan wrote:
To fix, allow ParseState to reflect that an individual statement can be
both p_is_insert and p_is_update at the same time.
/* Process DO UPDATE */ if (onConflictClause->action == ONCONFLICT_UPDATE) { + /* p_is_update must be set here, after INSERT targetlist processing */ + pstate->p_is_update = true; +
It's not particularly pretty that you document in the commit message
that both is_insert and is_update can be set at the same time, and then
it has constraints like the above.
But that's more crummy API's fault than yours.
I'm right now not really coming up with a better idea how to fix
this. So I guess I'll apply something close to this tomorrow.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Sun, Jul 12, 2015 at 1:45 PM, Andres Freund <andres@anarazel.de> wrote:
But that's more crummy API's fault than yours.
As you probably noticed, the only reason the p_is_update and
p_is_insert fields exist is for transformAssignedExpr() -- in fact, in
the master branch, nothing checks the value of p_is_update (although I
suppose a hook into a third-party module could see that module test
ParseState.p_is_update, so that isn't quite true).
I'm right now not really coming up with a better idea how to fix
this. So I guess I'll apply something close to this tomorrow.
Sounds good.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2015-07-12 22:45:18 +0200, Andres Freund wrote:
I'm right now not really coming up with a better idea how to fix
this. So I guess I'll apply something close to this tomorrow.
Took a bit longer than that :(
I've pushed a version of your patch. I just opted to remove p_is_update
instead of allowing both to be set at the same time. To me that seemed
simpler.
Thanks for the fix!
Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2015-05-28 18:31:56 -0700, Peter Geoghegan wrote:
Subject: [PATCH 3/3] Fix bug in unique index inference
ON CONFLICT unique index inference had a thinko that could affect cases
where the user-supplied inference clause required that an attribute
match a particular (user named) collation and/or opclass.Firstly, infer_collation_opclass_match() matched on opclass and/or
collation. Secondly, the attribute had to be in the list of attributes
or expressions known to be in the definition of the index under
consideration. The second step wasn't correct though, because having
some match doesn't necessarily mean that the second step found the same
index attribute as the (collation/opclass wise) match from the first
step.
Yes, makes sense.
+ else + { + Node *nattExpr = list_nth(idxExprs, (natt - 1) - nplain); + + /* + * Note that unlike routines like match_index_to_operand(), we're + * unconcerned about RelabelType. An exact match is required. + */ + if (equal(elem->expr, nattExpr)) + return true;
Why is that?
Regads,
Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, Jul 24, 2015 at 2:58 AM, Andres Freund <andres@anarazel.de> wrote:
I've pushed a version of your patch. I just opted to remove p_is_update
instead of allowing both to be set at the same time. To me that seemed
simpler.
I would be hesitant to remove a struct field from a struct that
appears as a hook argument. Someone's extension (that uses parser
hooks) might have been relying on that. Perhaps not a big deal.
Thanks
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On July 24, 2015 7:57:43 PM GMT+02:00, Peter Geoghegan <pg@heroku.com> wrote:
On Fri, Jul 24, 2015 at 2:58 AM, Andres Freund <andres@anarazel.de>
wrote:I've pushed a version of your patch. I just opted to remove
p_is_update
instead of allowing both to be set at the same time. To me that
seemed
simpler.
I would be hesitant to remove a struct field from a struct that
appears as a hook argument. Someone's extension (that uses parser
hooks) might have been relying on that. Perhaps not a big deal.
They'd also be affected by the change in meaning...
---
Please excuse brevity and formatting - I am writing this on my mobile phone.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, Jul 24, 2015 at 3:08 AM, Andres Freund <andres@anarazel.de> wrote:
+ else + { + Node *nattExpr = list_nth(idxExprs, (natt - 1) - nplain); + + /* + * Note that unlike routines like match_index_to_operand(), we're + * unconcerned about RelabelType. An exact match is required. + */ + if (equal(elem->expr, nattExpr)) + return true;Why is that?
No very strong reason. RelabelType exists to represent a dummy
coercion between two binary-compatible types. I think that a unique
index inference specification (which is novel in some ways) does not
need to do anything special for this case.
Each inference specification attribute that is an expression should
match some attribute in some index's cataloged definition. The
inference specification looks very much like the CREATE UNIQUE INDEX
that created the unique index that is inferred (usually, they'll be
identical). No need to make it any more complicated than that.
In fact, I don't think it's possible to construct a case where it
could even be argued that it matters. I'm not very caffeinated at the
moment, so I'm not sure of that.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Hi,
To recap for other readers: There's a problem with ON CONFLICT when the
SET or ON CONFLICT ... WHERE clause references excluded.* (i.e. as a
whole row var). The problem is that setrefs.c in
fix_join_expr_mutator() currently won't find a matching entry in the
indexed tlist and thus error out with
elog(ERROR, "variable not found in subplan target lists");
The reason is that the targetlist we build the index list on just
contains the attributes in excluded.*.
Peter's patch upthread fixes this by pulling expressions from
onConflictSet/Where into the targetlist. I disliked this - much less
than initially - a bit because that seems a bit crufty given that we're
not actually getting data from a child node. This is different to
RETURNING where the targetlist massaging is actually important to get
the data up the tree.
An actually trivial, although not all that pretty, fix is to simply
accept wholerow references in fix_join_expr_mutator(), even if not in
the targetlist. As far as I can see the problem right now really can
only be hit for whole row references.
A variant of the second approach is to have a fix_onconflict_expr()
mutator that has such special handler.
Any opinions on either approach?
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Sat, Sep 19, 2015 at 5:11 PM, Andres Freund <andres@anarazel.de> wrote:
Peter's patch upthread fixes this by pulling expressions from
onConflictSet/Where into the targetlist. I disliked this - much less
than initially - a bit because that seems a bit crufty given that we're
not actually getting data from a child node. This is different to
RETURNING where the targetlist massaging is actually important to get
the data up the tree.
Maybe the massaging is somewhat justified by the fact that it's just
as good a place as any to reject system columns, and that needs to
happen somewhere. I know that you suggested that this be done during
parse analysis, but not sure how attached you are to that. Might also
be a good choke point for detecting when unexpected vars/expressions
appear in the targetlist due to unforeseen circumstances/bugs. I
actually cover a couple of "can't happen" cases at the same time,
IIRC.
Continuing to follow RETURNING may have some value, even if the
analogy is a bit more forced here.
An actually trivial, although not all that pretty, fix is to simply
accept wholerow references in fix_join_expr_mutator(), even if not in
the targetlist. As far as I can see the problem right now really can
only be hit for whole row references.
I am concerned about the risk of adding bugs to unrelated code paths
that this could create. I must admit that this concern is mostly
driven by paranoia, and not a seasoned appreciation of problems that
could arise from ordinary post-processing of join expressions.
A variant of the second approach is to have a fix_onconflict_expr()
mutator that has such special handler.
I suppose you could add a fix_join_expr_context field that had
fix_join_expr_mutator() avoid the special handler for post-processing
of join expressions. That might be a bit ugly too, but would involve
less code duplication.
Any opinions on either approach?
I think that I favor my original solution, although only by a tiny
margin. I will avoid offering either a -1 or a +1 to any proposal
here, although they all sound basically reasonable to me. A more
complete targetlist representation would have been something that I'd
probably vote against, since it seems complex and invasive, but that
doesn't matter now. In short, I defer to others here.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2015-09-19 18:40:14 -0700, Peter Geoghegan wrote:
An actually trivial, although not all that pretty, fix is to simply
accept wholerow references in fix_join_expr_mutator(), even if not in
the targetlist. As far as I can see the problem right now really can
only be hit for whole row references.I am concerned about the risk of adding bugs to unrelated code paths
that this could create.
How? This is a must-not-reach code path currently?
Stuff I want to fix by tomorrow:
* Whole row var references to exclude
* wrong offsets for columns after dropped ones
* INSTEAD DO UPDATE for tables with oids
Do you know of anything else?
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, Sep 24, 2015 at 8:25 AM, Andres Freund <andres@anarazel.de> wrote:
Stuff I want to fix by tomorrow:
* Whole row var references to exclude
* wrong offsets for columns after dropped ones
* INSTEAD DO UPDATE for tables with oidsDo you know of anything else?
You said something in Dallas about the test case developed by Amit
Langote touching on a different bug to the regression test I came up
with. If that is the case, then you didn't list that one separately.
Otherwise, no.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2015-09-24 17:25:21 +0200, Andres Freund wrote:
Stuff I want to fix by tomorrow:
* Whole row var references to exclude
* wrong offsets for columns after dropped ones
* INSTEAD DO UPDATE for tables with oidsDo you know of anything else?
So, took a bit longer than "tomorrow. I fought for a long while with a
mysterious issue, which turned out to be separate bug: The excluded
relation was affected by row level security policies, which doesn't make
sense.
My proposal in this WIP patch is to make it a bit clearer that
'EXCLUDED' isn't a real relation. I played around with adding a
different rtekind, but that's too heavy a hammer. What I instead did was
to set relkind to composite - which seems to signal pretty well that
we're not dealing with a real relation. That immediately fixes the RLS
issue as fireRIRrules has the following check:
if (rte->rtekind != RTE_RELATION ||
rte->relkind != RELKIND_RELATION)
continue;
It also makes it relatively straightforward to fix the system column
issue by adding an additional relkind check to scanRTEForColumn's system
column handling.
WRT to the wholerow issue: There's currently two reasons we need a
targetlist entry for excluded wholerow vars: 1) setrefs.c errors out
without - that can relativley easily be worked around 2) ruleutils.c
expects an entry in the child tlist. That could also be worked around,
but it's a bit more verbose. I'm inclined to not go the pullup route
but instead simply unconditionally add a wholerow var to the excluded
tlist.
Peter, what do you think?
Andres
Attachments:
0001-wip-upsert.patchtext/x-patch; charset=us-asciiDownload
>From f11fc12993beabf4d280b979c062682b6612c989 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@anarazel.de>
Date: Tue, 29 Sep 2015 17:08:36 +0200
Subject: [PATCH] wip-upsert
---
src/backend/parser/analyze.c | 76 +++++++++++++++----
src/backend/parser/parse_relation.c | 7 +-
src/test/regress/expected/insert_conflict.out | 101 ++++++++++++++++++++++++++
src/test/regress/expected/rules.out | 18 ++---
src/test/regress/sql/insert_conflict.sql | 56 ++++++++++++++
src/test/regress/sql/rules.sql | 2 +-
6 files changed, 234 insertions(+), 26 deletions(-)
diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c
index a0dfbf9..528825a 100644
--- a/src/backend/parser/analyze.c
+++ b/src/backend/parser/analyze.c
@@ -891,33 +891,81 @@ transformOnConflictClause(ParseState *pstate,
/* Process DO UPDATE */
if (onConflictClause->action == ONCONFLICT_UPDATE)
{
+ Relation targetrel = pstate->p_target_relation;
+ Var *var;
+ TargetEntry *te;
+ int attno;
+
/*
* All INSERT expressions have been parsed, get ready for potentially
* existing SET statements that need to be processed like an UPDATE.
*/
pstate->p_is_insert = false;
+ /*
+ * Add range table entry for the EXCLUDED pseudo relation; relkind is
+ * set to composite to signal that we're not dealing with an actual
+ * relation.
+ */
exclRte = addRangeTableEntryForRelation(pstate,
- pstate->p_target_relation,
+ targetrel,
makeAlias("excluded", NIL),
false, false);
+ exclRte->relkind = RELKIND_COMPOSITE_TYPE;
exclRelIndex = list_length(pstate->p_rtable);
/*
- * Build a targetlist for the EXCLUDED pseudo relation. Out of
- * simplicity we do that here, because expandRelAttrs() happens to
- * nearly do the right thing; specifically it also works with views.
- * It'd be more proper to instead scan some pseudo scan node, but it
- * doesn't seem worth the amount of code required.
- *
- * The only caveat of this hack is that the permissions expandRelAttrs
- * adds have to be reset. markVarForSelectPriv() will add the exact
- * required permissions back.
+ * Build a targetlist for the EXCLUDED pseudo relation. Have to be
+ * careful to use resnos that correspond to attnos of the underlying
+ * relation.
+ */
+ Assert(pstate->p_next_resno == 1);
+ for (attno = 0; attno < targetrel->rd_rel->relnatts; attno++)
+ {
+ Form_pg_attribute attr = targetrel->rd_att->attrs[attno];
+ char *name;
+
+ if (attr->attisdropped)
+ {
+ /*
+ * can't use atttypid here, but it doesn't really matter
+ * what type the Const claims to be.
+ */
+ var = (Var *) makeNullConst(INT4OID, -1, InvalidOid);
+ name = "";
+ }
+ else
+ {
+ var = makeVar(exclRelIndex, attno + 1,
+ attr->atttypid, attr->atttypmod,
+ attr->attcollation,
+ 0);
+ var->location = -1;
+
+ name = NameStr(attr->attname);
+ }
+
+ Assert(pstate->p_next_resno == attno + 1);
+ te = makeTargetEntry((Expr *) var,
+ pstate->p_next_resno++,
+ name,
+ false);
+
+ /* don't require select access yet */
+ exclRelTlist = lappend(exclRelTlist, te);
+ }
+
+ /*
+ * Additionally add a whole row tlist entry for EXCLUDED. That's
+ * really only needed for ruleutils' benefit, which expects to find
+ * corresponding entries in child tlists. Alternatively we could do
+ * this only when required, but that doesn't seem worth the trouble.
*/
- exclRelTlist = expandRelAttrs(pstate, exclRte,
- exclRelIndex, 0, -1);
- exclRte->requiredPerms = 0;
- exclRte->selectedCols = NULL;
+ var = makeVar(exclRelIndex, InvalidAttrNumber,
+ RelationGetRelid(targetrel),
+ -1, InvalidOid, 0);
+ te = makeTargetEntry((Expr *) var, 0, NULL, true);
+ exclRelTlist = lappend(exclRelTlist, te);
/*
* Add EXCLUDED and the target RTE to the namespace, so that they can
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 0b2dacf..270a78c 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -686,9 +686,12 @@ scanRTEForColumn(ParseState *pstate, RangeTblEntry *rte, char *colname,
return result;
/*
- * If the RTE represents a real table, consider system column names.
+ * If the RTE represents a real relation, consider system column
+ * names. Composites are only used for pseudo-relations like ON CONFLICT's
+ * excluded.
*/
- if (rte->rtekind == RTE_RELATION)
+ if (rte->rtekind == RTE_RELATION &&
+ rte->relkind != RELKIND_COMPOSITE_TYPE)
{
/* quick check to see if name could be a system column */
attnum = specialAttNum(colname);
diff --git a/src/test/regress/expected/insert_conflict.out b/src/test/regress/expected/insert_conflict.out
index 1399f3c..111029d 100644
--- a/src/test/regress/expected/insert_conflict.out
+++ b/src/test/regress/expected/insert_conflict.out
@@ -380,6 +380,58 @@ ERROR: there is no unique or exclusion constraint matching the ON CONFLICT spec
insert into insertconflicttest values (23, 'Blackberry') on conflict (fruit) where fruit like '%berry' do update set fruit = excluded.fruit;
ERROR: there is no unique or exclusion constraint matching the ON CONFLICT specification
drop index partial_key_index;
+--
+-- Test resjunk in excluded.* pseudo-relation
+--
+create unique index plain on insertconflicttest(key);
+-- Succeeds, updates existing row:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* != excluded.* returning *;
+ key | fruit
+-----+-----------
+ 23 | Jackfruit
+(1 row)
+
+-- No update this time, though:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* != excluded.* returning *;
+ key | fruit
+-----+-------
+(0 rows)
+
+-- Predicate changed to require match rather than non-match, so updates once more:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* = excluded.* returning *;
+ key | fruit
+-----+-----------
+ 23 | Jackfruit
+(1 row)
+
+-- Assign:
+insert into insertconflicttest as i values (23, 'Avocado') on conflict (key) do update set fruit = excluded.*::text
+ returning *;
+ key | fruit
+-----+--------------
+ 23 | (23,Avocado)
+(1 row)
+
+-- deparse whole row var in WHERE clause:
+explain (costs off) insert into insertconflicttest as i values (23, 'Avocado') on conflict (key) do update set fruit = excluded.fruit where excluded.* is null;
+ QUERY PLAN
+-----------------------------------------
+ Insert on insertconflicttest i
+ Conflict Resolution: UPDATE
+ Conflict Arbiter Indexes: plain
+ Conflict Filter: (excluded.* IS NULL)
+ -> Result
+(5 rows)
+
+-- Forbidden, fails:
+insert into insertconflicttest values (23, 'Blackberry') on conflict (key) do update set fruit = excluded.ctid;
+ERROR: column excluded.ctid does not exist
+LINE 1: ...ckberry') on conflict (key) do update set fruit = excluded.c...
+ ^
+drop index plain;
-- Cleanup
drop table insertconflicttest;
-- ******************************************************************
@@ -566,3 +618,52 @@ insert into testoids values(3, '2') on conflict (key) do update set data = exclu
(1 row)
DROP TABLE testoids;
+-- check that references to columns after dropped columns are handled correctly
+create table dropcol(key int primary key, drop1 int, keep1 text, drop2 numeric, keep2 float);
+insert into dropcol(key, drop1, keep1, drop2, keep2) values(1, 1, '1', '1', 1);
+-- set using excluded
+insert into dropcol(key, drop1, keep1, drop2, keep2) values(1, 2, '2', '2', 2) on conflict(key)
+ do update set drop1 = excluded.drop1, keep1 = excluded.keep1, drop2 = excluded.drop2, keep2 = excluded.keep2
+ where excluded.drop1 is not null and excluded.keep1 is not null and excluded.drop2 is not null and excluded.keep2 is not null
+ and dropcol.drop1 is not null and dropcol.keep1 is not null and dropcol.drop2 is not null and dropcol.keep2 is not null
+ returning *;
+ key | drop1 | keep1 | drop2 | keep2
+-----+-------+-------+-------+-------
+ 1 | 2 | 2 | 2 | 2
+(1 row)
+
+;
+-- set using existing table
+insert into dropcol(key, drop1, keep1, drop2, keep2) values(1, 3, '3', '3', 3) on conflict(key)
+ do update set drop1 = dropcol.drop1, keep1 = dropcol.keep1, drop2 = dropcol.drop2, keep2 = dropcol.keep2
+ returning *;
+ key | drop1 | keep1 | drop2 | keep2
+-----+-------+-------+-------+-------
+ 1 | 2 | 2 | 2 | 2
+(1 row)
+
+;
+alter table dropcol drop column drop1, drop column drop2;
+-- set using excluded
+insert into dropcol(key, keep1, keep2) values(1, '4', 4) on conflict(key)
+ do update set keep1 = excluded.keep1, keep2 = excluded.keep2
+ where excluded.keep1 is not null and excluded.keep2 is not null
+ and dropcol.keep1 is not null and dropcol.keep2 is not null
+ returning *;
+ key | keep1 | keep2
+-----+-------+-------
+ 1 | 4 | 4
+(1 row)
+
+;
+-- set using existing table
+insert into dropcol(key, keep1, keep2) values(1, '5', 5) on conflict(key)
+ do update set keep1 = dropcol.keep1, keep2 = dropcol.keep2
+ returning *;
+ key | keep1 | keep2
+-----+-------+-------
+ 1 | 4 | 4
+(1 row)
+
+;
+DROP TABLE dropcol;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 44c6740..80374e4 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2900,7 +2900,7 @@ CREATE RULE hat_upsert AS ON INSERT TO hats
ON CONFLICT (hat_name)
DO UPDATE
SET hat_name = hat_data.hat_name, hat_color = excluded.hat_color
- WHERE excluded.hat_color <> 'forbidden'
+ WHERE excluded.hat_color <> 'forbidden' AND hat_data.* != excluded.*
RETURNING *;
SELECT definition FROM pg_rules WHERE tablename = 'hats' ORDER BY rulename;
definition
@@ -2908,7 +2908,7 @@ SELECT definition FROM pg_rules WHERE tablename = 'hats' ORDER BY rulename;
CREATE RULE hat_upsert AS +
ON INSERT TO hats DO INSTEAD INSERT INTO hat_data (hat_name, hat_color) +
VALUES (new.hat_name, new.hat_color) ON CONFLICT(hat_name) DO UPDATE SET hat_name = hat_data.hat_name, hat_color = excluded.hat_color+
- WHERE (excluded.hat_color <> 'forbidden'::bpchar) +
+ WHERE ((excluded.hat_color <> 'forbidden'::bpchar) AND (hat_data.* <> excluded.*)) +
RETURNING hat_data.hat_name, +
hat_data.hat_color;
(1 row)
@@ -2956,19 +2956,19 @@ SELECT tablename, rulename, definition FROM pg_rules
hats | hat_upsert | CREATE RULE hat_upsert AS +
| | ON INSERT TO hats DO INSTEAD INSERT INTO hat_data (hat_name, hat_color) +
| | VALUES (new.hat_name, new.hat_color) ON CONFLICT(hat_name) DO UPDATE SET hat_name = hat_data.hat_name, hat_color = excluded.hat_color+
- | | WHERE (excluded.hat_color <> 'forbidden'::bpchar) +
+ | | WHERE ((excluded.hat_color <> 'forbidden'::bpchar) AND (hat_data.* <> excluded.*)) +
| | RETURNING hat_data.hat_name, +
| | hat_data.hat_color;
(1 row)
-- ensure explain works for on insert conflict rules
explain (costs off) INSERT INTO hats VALUES ('h8', 'forbidden') RETURNING *;
- QUERY PLAN
-----------------------------------------------------------------
+ QUERY PLAN
+-------------------------------------------------------------------------------------------------
Insert on hat_data
Conflict Resolution: UPDATE
Conflict Arbiter Indexes: hat_data_unique_idx
- Conflict Filter: (excluded.hat_color <> 'forbidden'::bpchar)
+ Conflict Filter: ((excluded.hat_color <> 'forbidden'::bpchar) AND (hat_data.* <> excluded.*))
-> Result
(5 rows)
@@ -2995,12 +2995,12 @@ EXPLAIN (costs off) WITH data(hat_name, hat_color) AS (
INSERT INTO hats
SELECT * FROM data
RETURNING *;
- QUERY PLAN
-----------------------------------------------------------------
+ QUERY PLAN
+-------------------------------------------------------------------------------------------------
Insert on hat_data
Conflict Resolution: UPDATE
Conflict Arbiter Indexes: hat_data_unique_idx
- Conflict Filter: (excluded.hat_color <> 'forbidden'::bpchar)
+ Conflict Filter: ((excluded.hat_color <> 'forbidden'::bpchar) AND (hat_data.* <> excluded.*))
CTE data
-> Values Scan on "*VALUES*"
-> CTE Scan on data
diff --git a/src/test/regress/sql/insert_conflict.sql b/src/test/regress/sql/insert_conflict.sql
index efa902e..5653715 100644
--- a/src/test/regress/sql/insert_conflict.sql
+++ b/src/test/regress/sql/insert_conflict.sql
@@ -223,6 +223,30 @@ insert into insertconflicttest values (23, 'Blackberry') on conflict (fruit) whe
drop index partial_key_index;
+--
+-- Test resjunk in excluded.* pseudo-relation
+--
+create unique index plain on insertconflicttest(key);
+
+-- Succeeds, updates existing row:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* != excluded.* returning *;
+-- No update this time, though:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* != excluded.* returning *;
+-- Predicate changed to require match rather than non-match, so updates once more:
+insert into insertconflicttest as i values (23, 'Jackfruit') on conflict (key) do update set fruit = excluded.fruit
+ where i.* = excluded.* returning *;
+-- Assign:
+insert into insertconflicttest as i values (23, 'Avocado') on conflict (key) do update set fruit = excluded.*::text
+ returning *;
+-- deparse whole row var in WHERE clause:
+explain (costs off) insert into insertconflicttest as i values (23, 'Avocado') on conflict (key) do update set fruit = excluded.fruit where excluded.* is null;
+-- Forbidden, fails:
+insert into insertconflicttest values (23, 'Blackberry') on conflict (key) do update set fruit = excluded.ctid;
+
+drop index plain;
+
-- Cleanup
drop table insertconflicttest;
@@ -317,3 +341,35 @@ insert into testoids values(3, '1') on conflict (key) do update set data = exclu
insert into testoids values(3, '2') on conflict (key) do update set data = excluded.data RETURNING *;
DROP TABLE testoids;
+
+
+-- check that references to columns after dropped columns are handled correctly
+create table dropcol(key int primary key, drop1 int, keep1 text, drop2 numeric, keep2 float);
+insert into dropcol(key, drop1, keep1, drop2, keep2) values(1, 1, '1', '1', 1);
+-- set using excluded
+insert into dropcol(key, drop1, keep1, drop2, keep2) values(1, 2, '2', '2', 2) on conflict(key)
+ do update set drop1 = excluded.drop1, keep1 = excluded.keep1, drop2 = excluded.drop2, keep2 = excluded.keep2
+ where excluded.drop1 is not null and excluded.keep1 is not null and excluded.drop2 is not null and excluded.keep2 is not null
+ and dropcol.drop1 is not null and dropcol.keep1 is not null and dropcol.drop2 is not null and dropcol.keep2 is not null
+ returning *;
+;
+-- set using existing table
+insert into dropcol(key, drop1, keep1, drop2, keep2) values(1, 3, '3', '3', 3) on conflict(key)
+ do update set drop1 = dropcol.drop1, keep1 = dropcol.keep1, drop2 = dropcol.drop2, keep2 = dropcol.keep2
+ returning *;
+;
+alter table dropcol drop column drop1, drop column drop2;
+-- set using excluded
+insert into dropcol(key, keep1, keep2) values(1, '4', 4) on conflict(key)
+ do update set keep1 = excluded.keep1, keep2 = excluded.keep2
+ where excluded.keep1 is not null and excluded.keep2 is not null
+ and dropcol.keep1 is not null and dropcol.keep2 is not null
+ returning *;
+;
+-- set using existing table
+insert into dropcol(key, keep1, keep2) values(1, '5', 5) on conflict(key)
+ do update set keep1 = dropcol.keep1, keep2 = dropcol.keep2
+ returning *;
+;
+
+DROP TABLE dropcol;
diff --git a/src/test/regress/sql/rules.sql b/src/test/regress/sql/rules.sql
index 561e2fd..4299a5b 100644
--- a/src/test/regress/sql/rules.sql
+++ b/src/test/regress/sql/rules.sql
@@ -1105,7 +1105,7 @@ CREATE RULE hat_upsert AS ON INSERT TO hats
ON CONFLICT (hat_name)
DO UPDATE
SET hat_name = hat_data.hat_name, hat_color = excluded.hat_color
- WHERE excluded.hat_color <> 'forbidden'
+ WHERE excluded.hat_color <> 'forbidden' AND hat_data.* != excluded.*
RETURNING *;
SELECT definition FROM pg_rules WHERE tablename = 'hats' ORDER BY rulename;
--
2.6.0.rc3
On Tue, Sep 29, 2015 at 8:24 AM, Andres Freund <andres@anarazel.de> wrote:
So, took a bit longer than "tomorrow. I fought for a long while with a
mysterious issue, which turned out to be separate bug: The excluded
relation was affected by row level security policies, which doesn't make
sense.
Why? You certainly thought that it made sense for conventional column
permissions due to potential problems with before row insert triggers.
Why would RLS be different? Some of my concerns with RLS were that it
is different to the existing permissions model in a way that seems a
bit arbitrary. I don't think that they were changed to do anything
special with SELECT ... FOR UPDATE, which has always required some
amount of conventional UPDATE privilege.
I specifically remember discussing this with you off list (on IM,
roughly a couple of weeks prior to initial commit). I recommended that
we err towards a more restrictive behavior in the absence of any
strong principle pushing us one way or the other. You seemed to agree.
My proposal in this WIP patch is to make it a bit clearer that
'EXCLUDED' isn't a real relation. I played around with adding a
different rtekind, but that's too heavy a hammer. What I instead did was
to set relkind to composite - which seems to signal pretty well that
we're not dealing with a real relation. That immediately fixes the RLS
issue as fireRIRrules has the following check:
if (rte->rtekind != RTE_RELATION ||
rte->relkind != RELKIND_RELATION)
continue;
Well, not sure that that's a good thing. Let's discuss.
It also makes it relatively straightforward to fix the system column
issue by adding an additional relkind check to scanRTEForColumn's system
column handling.
That seems fine.
WRT to the wholerow issue: There's currently two reasons we need a
targetlist entry for excluded wholerow vars: 1) setrefs.c errors out
without - that can relativley easily be worked around 2) ruleutils.c
expects an entry in the child tlist. That could also be worked around,
but it's a bit more verbose. I'm inclined to not go the pullup route
but instead simply unconditionally add a wholerow var to the excluded
tlist.
I suppose that we have a tight enough grip on the targetlist that it's
hard to imagine anything else being introduced there spuriously. I had
thought that the pull-up did allow useful additional
defense/sanitization, but that may not be an excellent argument. The
only remaining argument is that my approach is closer to RETURNING,
but that doesn't seem like an excellent argument.
Basically, I think that this is fine.
However, there were a number of small stylistic tweaks made in passing
within my original patch -- minor things around consistency. Please
either restore these, or commit them separately.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On September 29, 2015 8:52:14 PM GMT+02:00, Peter Geoghegan <pg@heroku.com> wrote:
On Tue, Sep 29, 2015 at 8:24 AM, Andres Freund <andres@anarazel.de>
wrote:So, took a bit longer than "tomorrow. I fought for a long while with
a
mysterious issue, which turned out to be separate bug: The excluded
relation was affected by row level security policies, which doesn'tmake
sense.
Why? You certainly thought that it made sense for conventional column
permissions due to potential problems with before row insert triggers.
Why would RLS be different?
What would it mean? And why would it make sense to apply rls to a values list?
nodeModify already has the necessary rls invocations, no?
Andres
---
Please excuse brevity and formatting - I am writing this on my mobile phone.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
* Peter Geoghegan (pg@heroku.com) wrote:
On Tue, Sep 29, 2015 at 8:24 AM, Andres Freund <andres@anarazel.de> wrote:
So, took a bit longer than "tomorrow. I fought for a long while with a
mysterious issue, which turned out to be separate bug: The excluded
relation was affected by row level security policies, which doesn't make
sense.Why? You certainly thought that it made sense for conventional column
permissions due to potential problems with before row insert triggers.
Why would RLS be different? Some of my concerns with RLS were that it
is different to the existing permissions model in a way that seems a
bit arbitrary. I don't think that they were changed to do anything
special with SELECT ... FOR UPDATE, which has always required some
amount of conventional UPDATE privilege.
I'm just about to push a patch to address exactly the SELECT .. FOR
UPDATE case, actually, and to try and make sure that RLS is more in line
with the existing permissions model.. I admit that I've not been
entirely following this thread though, so I'm not quite sure how that's
relevant to this discussion.
From Andres' reply, it looks like this is about the EXCLUDED pseudo
relation which comes from the INSERT'd values themselves, in which case,
I tend to agree with his assessment that it doesn't make sense for those
to be subject to RLS policies, given that it's all user-provided data,
as long as the USING check is done on the row found to be conflicting
and the CHECK constraints are dealt with correctly for any row added,
which I believe is what we had agreed was the correct way to handle this
case in prior discussions.
Thanks!
Stephen
On 2015-09-29 15:49:28 -0400, Stephen Frost wrote:
From Andres' reply, it looks like this is about the EXCLUDED pseudo
relation which comes from the INSERT'd values themselves
Right.
in which case, I tend to agree with his assessment that it doesn't
make sense for those to be subject to RLS policies, given that it's
all user-provided data, as long as the USING check is done on the row
found to be conflicting and the CHECK constraints are dealt with
correctly for any row added, which I believe is what we had agreed was
the correct way to handle this case in prior discussions.
Yes, that what I think as well. At this point we'll already have
executed insert rls stuff on the EXCLUDED tuple:
/*
* Check any RLS INSERT WITH CHECK policies
*
* ExecWithCheckOptions() will skip any WCOs which are not of the kind
* we are looking for at this point.
*/
if (resultRelInfo->ri_WithCheckOptions != NIL)
ExecWithCheckOptions(WCO_RLS_INSERT_CHECK,
resultRelInfo, slot, estate);
and before executing the actual projection we also checked the existing
tuple:
ExecWithCheckOptions(WCO_RLS_CONFLICT_CHECK, resultRelInfo,
mtstate->mt_existing,
mtstate->ps.state);
after the update triggers have, if applicable run, we run the the normal
checks there as well because it's just ExecUpdate()
if (resultRelInfo->ri_WithCheckOptions != NIL)
ExecWithCheckOptions(WCO_RLS_UPDATE_CHECK,
resultRelInfo, slot, estate);
so I do indeed think that there's no point in layering RLS above
EXCLUDED.
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2015-09-29 11:52:14 -0700, Peter Geoghegan wrote:
On Tue, Sep 29, 2015 at 8:24 AM, Andres Freund <andres@anarazel.de> wrote:
So, took a bit longer than "tomorrow. I fought for a long while with a
mysterious issue, which turned out to be separate bug: The excluded
relation was affected by row level security policies, which doesn't make
sense.Why? You certainly thought that it made sense for conventional column
permissions due to potential problems with before row insert triggers.
I don't see how those compare:
I specifically remember discussing this with you off list (on IM,
roughly a couple of weeks prior to initial commit). I recommended that
we err towards a more restrictive behavior in the absence of any
strong principle pushing us one way or the other. You seemed to agree.
I don't think this really is comparable. Comparing this with a plain
INSERT or UPDATE this would be akin to running RLS on the RETURNING
tuple - which we currently don't.
I think this is was just a bug.
I suppose that we have a tight enough grip on the targetlist that it's
hard to imagine anything else being introduced there spuriously. I had
thought that the pull-up did allow useful additional
defense/sanitization, but that may not be an excellent argument. The
only remaining argument is that my approach is closer to RETURNING,
but that doesn't seem like an excellent argument.
I indeed don't think this is comparable to RETURNING - the pullup there
is into an actual querytree above unrelated relations.
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, Oct 1, 2015 at 3:42 AM, Andres Freund <andres@anarazel.de> wrote:
Yes, that what I think as well. At this point we'll already have
executed insert rls stuff on the EXCLUDED tuple:
/*
* Check any RLS INSERT WITH CHECK policies
*
* ExecWithCheckOptions() will skip any WCOs which are not of the kind
* we are looking for at this point.
*/
if (resultRelInfo->ri_WithCheckOptions != NIL)
ExecWithCheckOptions(WCO_RLS_INSERT_CHECK,
resultRelInfo, slot, estate);
and before executing the actual projection we also checked the existing
tuple:
ExecWithCheckOptions(WCO_RLS_CONFLICT_CHECK, resultRelInfo,
mtstate->mt_existing,
mtstate->ps.state);after the update triggers have, if applicable run, we run the the normal
checks there as well because it's just ExecUpdate()
if (resultRelInfo->ri_WithCheckOptions != NIL)
ExecWithCheckOptions(WCO_RLS_UPDATE_CHECK,
resultRelInfo, slot, estate);so I do indeed think that there's no point in layering RLS above
EXCLUDED.
I see your point, I think. It might be a problem if we weren't already
making the statement error out, but we are.
However, we're checking the excluded tuple (the might-be-inserted,
might-be-excluded tuple that reflects before row insert trigger
effects) with WCO_RLS_INSERT_CHECK, not WCO_RLS_UPDATE_CHECK. The
WCO_RLS_UPDATE_CHECK applies to the tuple to be appended to the
relation (the tuple that an UPDATE makes supersede some existing
tuple, a new row version).
We all seem to be in agreement that excluded.* ought to be subject to
column-level privilege enforcement, mostly due to possible leaks with
before row insert triggers (these could be SoD; a malicious UPSERT
could be written a certain way). None of the checks in the code above
are the exact RLS equivalent of the principle we have for column
privileges, AFAICT, because update-applicable policies (everything but
insert-applicable policies, actually) are not checked against the
excluded tuple. Shouldn't select-applicable policies also be applied
to the excluded tuples, just as with UPDATE ... FROM "join from"
tables, which excluded is kinda similar to?
I'm not trying to be pedantic; I just don't grok the underlying
principles here. Couldn't a malicious WHERE clause leak the excluded.*
tuple contents (and cause the UPDATE to not proceed) before the
WCO_RLS_CONFLICT_CHECK call site was reached, while also preventing it
from ever actually being reached (with a malicious function that
returns false after stashing excluded.* elsewhere)? You can put
volatile functions in UPDATE WHERE clauses, even if it is generally a
bad idea.
Perhaps I'm simply not following you here, though. I think that this
is one challenge with having per-command policies with a system that
checks permissions dynamically (not during parse analysis). Note that
I'm not defending the status quo of the master branch -- I'm just a
little uneasy about what the ideal, least surprising behavior is here.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2015-10-01 16:13:12 -0700, Peter Geoghegan wrote:
However, we're checking the excluded tuple (the might-be-inserted,
might-be-excluded tuple that reflects before row insert trigger
effects) with WCO_RLS_INSERT_CHECK, not WCO_RLS_UPDATE_CHECK. The
WCO_RLS_UPDATE_CHECK applies to the tuple to be appended to the
relation (the tuple that an UPDATE makes supersede some existing
tuple, a new row version).
You can already see the effects of an INSERT modified by before triggers
via RETURNING. No?
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, Oct 1, 2015 at 3:53 AM, Andres Freund <andres@anarazel.de> wrote:
I specifically remember discussing this with you off list (on IM,
roughly a couple of weeks prior to initial commit). I recommended that
we err towards a more restrictive behavior in the absence of any
strong principle pushing us one way or the other. You seemed to agree.I don't think this really is comparable. Comparing this with a plain
INSERT or UPDATE this would be akin to running RLS on the RETURNING
tuple - which we currently don't.I think this is was just a bug.
Maybe that's the problem here; I still thought that we were planning
on changing RLS in this regard, but it actually seems we changed
course, looking at the 9.5 open items list.
I would say that that's a clear divergence between RLS and column
privileges. That might be fine, but it doesn't match my prior
understanding of RLS (or, more accurately, how it was likely to change
pre-release).
If that's the design that we want for RLS across the board, then I'm
happy to defer to that decision.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, Oct 1, 2015 at 4:17 PM, Andres Freund <andres@anarazel.de> wrote:
On 2015-10-01 16:13:12 -0700, Peter Geoghegan wrote:
However, we're checking the excluded tuple (the might-be-inserted,
might-be-excluded tuple that reflects before row insert trigger
effects) with WCO_RLS_INSERT_CHECK, not WCO_RLS_UPDATE_CHECK. The
WCO_RLS_UPDATE_CHECK applies to the tuple to be appended to the
relation (the tuple that an UPDATE makes supersede some existing
tuple, a new row version).You can already see the effects of an INSERT modified by before triggers
via RETURNING. No?
I'm not saying that I agree with the decision to not do anything
special with RLS + RETURNING in general. I'm also not going to say
that I disagree with it. As I said, I missed that decision until just
now. I agree that it's obviously true that what you propose is
consistent with what is now considered ideal behavior for RLS (that's
what I get from the wiki page comments on RLS + RETURNING).
FWIW, I think that this technically wasn't a bug, because it was based
on a deliberate design decision that I thought (not without
justification) was consistent with what we wanted for RLS across the
board. But, again, happy to go along with what you say in light of
this new information.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2015-10-01 16:26:07 -0700, Peter Geoghegan wrote:
FWIW, I think that this technically wasn't a bug
Meh. In which scenario would do a policy applied to EXCLUDED actually
anything reasonable?
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, Oct 1, 2015 at 4:26 PM, Peter Geoghegan <pg@heroku.com> wrote:
You can already see the effects of an INSERT modified by before triggers
via RETURNING. No?I'm not saying that I agree with the decision to not do anything
special with RLS + RETURNING in general. I'm also not going to say
that I disagree with it. As I said, I missed that decision until just
now. I agree that it's obviously true that what you propose is
consistent with what is now considered ideal behavior for RLS (that's
what I get from the wiki page comments on RLS + RETURNING).
I see now that commit 4f3b2a8883 changed things for UPDATE and DELETE
statements, but not INSERT statements. I guess my unease is because
that isn't entirely consistent with INSERT + RETURNING and the GRANT
system. Logically, the only possible justification for our long
standing INSERT and RETURNING behavior with GRANT (the fact that it
requires SELECT privilege for rows returned, just like UPDATE and
DELETE) is that before row insert triggers could do something secret
(e.g. they could be security definer). It doesn't seem to be too much
of a stretch to suppose the same should apply with RLS.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, Oct 1, 2015 at 4:49 PM, Andres Freund <andres@anarazel.de> wrote:
On 2015-10-01 16:26:07 -0700, Peter Geoghegan wrote:
FWIW, I think that this technically wasn't a bug
Meh. In which scenario would do a policy applied to EXCLUDED actually
anything reasonable?
I agree that it's very unlikely to matter. Consistency is something
that is generally valued, though.
I'm not going to object if you want to continue with committing
something that changes excluded + RLS. I was just explaining my view
of the matter.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2015-10-01 16:55:23 -0700, Peter Geoghegan wrote:
On Thu, Oct 1, 2015 at 4:49 PM, Andres Freund <andres@anarazel.de> wrote:
On 2015-10-01 16:26:07 -0700, Peter Geoghegan wrote:
FWIW, I think that this technically wasn't a bug
Meh. In which scenario would do a policy applied to EXCLUDED actually
anything reasonable?I agree that it's very unlikely to matter. Consistency is something
that is generally valued, though.
I don't think you get my gist.
I'm can't see how the current code can do anything sensible at all. What
do you think is going to be the effect of an excluded row that doesn't
meet security quals? Even if it worked in the sense that the correct
data were accessed and every - which I doubt is completely the case as
things stands given there's no actual scan node and stuff - you'd still
have EXCLUDED.* being used in the projection for the new version of the
tuple.
As far as I can see the only correct thing you could do in that
situation is error out.
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, Oct 1, 2015 at 5:12 PM, Andres Freund <andres@anarazel.de> wrote:
I'm can't see how the current code can do anything sensible at all. What
do you think is going to be the effect of an excluded row that doesn't
meet security quals? Even if it worked in the sense that the correct
data were accessed and every - which I doubt is completely the case as
things stands given there's no actual scan node and stuff - you'd still
have EXCLUDED.* being used in the projection for the new version of the
tuple.As far as I can see the only correct thing you could do in that
situation is error out.
I agree. I wasn't defending the current code (although that might have
been made unclear by the "technically wasn't a bug" remark).
Note that I'm not telling you what I think needs to happen. I'm just
explaining my understanding of what has happened.
--
Peter Geoghegan
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
My proposal in this WIP patch is to make it a bit clearer that
'EXCLUDED' isn't a real relation. I played around with adding a
different rtekind, but that's too heavy a hammer. What I instead did was
to set relkind to composite - which seems to signal pretty well that
we're not dealing with a real relation. That immediately fixes the RLS
issue as fireRIRrules has the following check:
if (rte->rtekind != RTE_RELATION ||
rte->relkind != RELKIND_RELATION)
continue;
It also makes it relatively straightforward to fix the system column
issue by adding an additional relkind check to scanRTEForColumn's system
column handling.
That works, but also precludes referencing 'oid' in a WITH OIDs table
via EXCLUDED.oid - to me that looks correct since a to-be-inserted row
can't yet have an oid assigned. Differing opinions?
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Peter, all,
* Peter Geoghegan (pg@heroku.com) wrote:
I see now that commit 4f3b2a8883 changed things for UPDATE and DELETE
statements, but not INSERT statements. I guess my unease is because
that isn't entirely consistent with INSERT + RETURNING and the GRANT
system. Logically, the only possible justification for our long
standing INSERT and RETURNING behavior with GRANT (the fact that it
requires SELECT privilege for rows returned, just like UPDATE and
DELETE) is that before row insert triggers could do something secret
(e.g. they could be security definer). It doesn't seem to be too much
of a stretch to suppose the same should apply with RLS.
I had intended to address with policies what is addressed through
permissions with 7d8db3e, but the coverage for INSERT+RETURNING was only
done when ON CONFLICT was in use.
I've fixed that by applying the SELECT policies as WCOs for both the
INSERT and UPDATE RETURNING cases. This matches the permissions system,
where we require SELECT rights on the table for an INSERT RETURNING
query.
Thanks!
Stephen
On 2015-10-05 08:01:00 -0400, Stephen Frost wrote:
Peter, all,
I had intended to address with policies what is addressed through
permissions with 7d8db3e, but the coverage for INSERT+RETURNING was only
done when ON CONFLICT was in use.I've fixed that by applying the SELECT policies as WCOs for both the
INSERT and UPDATE RETURNING cases. This matches the permissions system,
where we require SELECT rights on the table for an INSERT RETURNING
query.
This really needs tests verifying the behaviour...
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andres,
On Monday, October 5, 2015, Andres Freund <andres@anarazel.de> wrote:
On 2015-10-05 08:01:00 -0400, Stephen Frost wrote:
Peter, all,
I had intended to address with policies what is addressed through
permissions with 7d8db3e, but the coverage for INSERT+RETURNING was only
done when ON CONFLICT was in use.I've fixed that by applying the SELECT policies as WCOs for both the
INSERT and UPDATE RETURNING cases. This matches the permissions system,
where we require SELECT rights on the table for an INSERT RETURNING
query.This really needs tests verifying the behaviour...
Good point, will add.
Thanks!
Stephen
Stephen Frost <sfrost@snowman.net> writes:
I had intended to address with policies what is addressed through
permissions with 7d8db3e, but the coverage for INSERT+RETURNING was only
done when ON CONFLICT was in use.
I've fixed that by applying the SELECT policies as WCOs for both the
INSERT and UPDATE RETURNING cases. This matches the permissions system,
where we require SELECT rights on the table for an INSERT RETURNING
query.
What of DELETE RETURNING?
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
Stephen Frost <sfrost@snowman.net> writes:
I had intended to address with policies what is addressed through
permissions with 7d8db3e, but the coverage for INSERT+RETURNING was only
done when ON CONFLICT was in use.I've fixed that by applying the SELECT policies as WCOs for both the
INSERT and UPDATE RETURNING cases. This matches the permissions system,
where we require SELECT rights on the table for an INSERT RETURNING
query.What of DELETE RETURNING?
That was handled in 7d8db3e.
Per previous discussion, UPDATE and DELETE RETURNING apply SELECT
policies as security quals, meaning only the records visible through the
SELECT policy are eligible for consideration. INSERT+RETURNING has only
WithCheckOptions, no security quals, which is what makes it different
from the other cases. The INSERT+ON CONFLICT+RETURNING case had been
covered already and I had mistakenly thought it was also covering
INSERT+RETURNING. In fixing that, I realized that Peter makes a good
point that UPDATE+RETURNING should also have SELECT policies applied as
WithCheckOptions.
I'm about to push updated regression tests, as suggested by Andres.
Thanks!
Stephen
* Andres Freund (andres@anarazel.de) wrote:
On 2015-10-05 08:01:00 -0400, Stephen Frost wrote:
Peter, all,
I had intended to address with policies what is addressed through
permissions with 7d8db3e, but the coverage for INSERT+RETURNING was only
done when ON CONFLICT was in use.I've fixed that by applying the SELECT policies as WCOs for both the
INSERT and UPDATE RETURNING cases. This matches the permissions system,
where we require SELECT rights on the table for an INSERT RETURNING
query.This really needs tests verifying the behaviour...
Added.
Thanks!
Stephen