Improve perfomance for index search ANY(ARRAY[]) condition with single item

Started by Dima Pavlovabout 9 years ago3 messageshackers
Beta feature

Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.

appliesbuild failedCI history

Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:

git clone --branch t37142_1 https://github.com/hackorum-dev/postgres.git

In a checkout you already have, add the fork once:

git remote add hackorum https://github.com/hackorum-dev/postgres.git

then, for this patchset and every later one:

git fetch hackorum t37142_1 && git checkout t37142_1

Patchset v1 (message #1) is on t37142_1

Jump to latest
#1Dima Pavlov
imyfess@gmail.com

Hello,

The problems I tried to solve here:
1. Improve perfomance for index search ANY(ARRAY[...]) condition with
single item
2. I saw tons of users code like: if len(array) == 1: sql +=
'{}'.format(array[0]) else: sql += 'ANY(ARRAY[{}])'.format(array)
So there will be less lines of code and it will be clearer.
3. Confusing moment that "IN" works well with single item, and
"ANY(ARRAY[])" doesn't without any real reason.

The problem was discussed on stackoverflow:
https://stackoverflow.com/questions/45061966/index-usage-with-single-item-anyarray

That's my first patch so I will be grateful for constructive criticism.
---------------------------------------------------------------------------

CREATE TABLE public.t (id serial, a integer, b integer);

INSERT INTO t(a, b)
SELECT round(random()*1000), round(random()*1000)
FROM generate_series(1, 1000000);

CREATE INDEX "i_1" ON public.t USING btree (a, b);
CREATE INDEX "i_2" ON public.t USING btree (b);

---------------------------------------------------------------------------

If "a = 50" in the first query, everything is ok, appropriate index "i_1"
is used:

SELECT * FROM t WHERE a = 50 ORDER BY b LIMIT 1

"Limit (cost=0.42..4.03 rows=1 width=12) (actual time=0.085..0.085 rows=1
loops=1)"
" Buffers: shared hit=1 read=3"
" -> Index Scan using i_1 on t (cost=0.42..4683.12 rows=1300 width=12)
(actual time=0.084..0.084 rows=1 loops=1)"
" Index Cond: (a = 50)"
" Buffers: shared hit=1 read=3"
"Planning time: 0.637 ms"
"Execution time: 0.114 ms"

---------------------------------------------------------------------------

With "a IN (50)" result is the same:

SELECT * FROM t WHERE a IN (50) ORDER BY b LIMIT 1

"Limit (cost=0.42..4.03 rows=1 width=12) (actual time=0.058..0.058 rows=1
loops=1)"
" Buffers: shared hit=4"
" -> Index Scan using i_1 on t (cost=0.42..4683.12 rows=1300 width=12)
(actual time=0.056..0.056 rows=1 loops=1)"
" Index Cond: (a = 50)"
" Buffers: shared hit=4"
"Planning time: 0.287 ms"
"Execution time: 0.105 ms"

---------------------------------------------------------------------------

The problem is when I try to use "a = ANY(ARRAY[50])". Wrong index "i_2" is
used instead of "i_1" and execution time becomes x25 longer:

SELECT * FROM t WHERE a = ANY(ARRAY[50]) ORDER BY b LIMIT 1

"Limit (cost=0.42..38.00 rows=1 width=12) (actual time=2.591..2.591 rows=1
loops=1)"
" Buffers: shared hit=491 read=4"
" -> Index Scan using i_2 on t (cost=0.42..48853.65 rows=1300 width=12)
(actual time=2.588..2.588 rows=1 loops=1)"
" Filter: (a = ANY ('{50}'::integer[]))"
" Rows Removed by Filter: 520"
" Buffers: shared hit=491 read=4"
"Planning time: 0.251 ms"
"Execution time: 2.627 ms"

Attachments:

t37142_1
improve-single-item-array.patchapplication/octet-stream; name=improve-single-item-array.patchDownload+102-0
#2Tom Lane
tgl@sss.pgh.pa.us
In reply to: Dima Pavlov (#1)
Re: Improve perfomance for index search ANY(ARRAY[]) condition with single item

Dima Pavlov <imyfess@gmail.com> writes:

The problem was discussed on stackoverflow:
https://stackoverflow.com/questions/45061966/index-usage-with-single-item-anyarray

Usually, we're not very happy with submissions that reference external
pages for justification; we like to have all the relevant material in our
mail archives.

That's my first patch so I will be grateful for constructive criticism.

I think it would be better to do this in the planner, see specifically
eval_const_expressions. That would allow the transformation to succeed
in more cases, like where the array is coming from a parameter rather
than an ARRAY[] construct. That is, you should be able to do this not
only for ARRAY[x] but for any single-element constant array.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#3Tom Lane
tgl@sss.pgh.pa.us
In reply to: Tom Lane (#2)
Re: Improve perfomance for index search ANY(ARRAY[]) condition with single item

I wrote:

Dima Pavlov <imyfess@gmail.com> writes:

That's my first patch so I will be grateful for constructive criticism.

I think it would be better to do this in the planner, see specifically
eval_const_expressions.

BTW, currently eval_const_expressions doesn't know anything about
ScalarArrayOpExpr, but there's a patch pending to improve that:

https://commitfest.postgresql.org/14/1136/

You should probably build your revised patch as a follow-on to that
one, else we're going to have merge conflicts.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers