Query planning on partitioned table causes postgres 13.4 to consume all memory

Started by Duncan Sandsover 4 years ago3 messagesbugs
Jump to latest
#1Duncan Sands
duncan.sands@deepbluecap.com

Postgresql version: 13.4
O/S version: Ubuntu 21.04

How to reproduce:
(1) Create "data" schema and its tables by executing the commands from the
attached file d.sql. Each of the two created tables has less than 20 rows.
(2) Execute or EXPLAIN this query

DELETE FROM data.files

WHERE path=ANY(

SELECT path

FROM (

SELECT meta, path,

first_value(path)

OVER (PARTITION BY meta ORDER BY priority) AS first

FROM data.files

) f,

data.metadata m

WHERE f.path <> f.first

AND f.meta = m.id

AND m.syd = 667

);

(3) Observe that the EXPLAIN/query never completes, and memory usage goes up
and up.

For example:

duncan=> \i /tmp/d.sql

...
duncan=> DELETE FROM data.files

WHERE path=ANY(

SELECT path

FROM (

SELECT meta, path,

first_value(path)

OVER (PARTITION BY meta ORDER BY priority) AS first

FROM data.files

) f,

data.metadata m

WHERE f.path <> f.first

AND f.meta = m.id

AND m.syd = 667

);

... wailing and gnashing of teeth ...

Attachments:

d.sqlapplication/sql; name=d.sqlDownload
#2Tom Lane
tgl@sss.pgh.pa.us
In reply to: Duncan Sands (#1)
Re: Query planning on partitioned table causes postgres 13.4 to consume all memory

Duncan Sands <duncan.sands@deepbluecap.com> writes:

[ planning DELETE on a thousand-partition table takes forever ]

FWIW, this situation has been very much improved for v14 [1]https://git.postgresql.org/gitweb/?p=postgresql.git&amp;a=commitdiff&amp;h=86dc90056.
In older branches, the best advice I can give you is "don't use
so many partitions". Especially not with hash partitioning,
where the query WHERE clause generally won't translate to any
useful pruning of the partitions.

(Personally, I think that hash partitioning is an evil that
we shouldn't have implemented at all. Or at least there
should be stronger warnings about it in the manual than there
are now.)

regards, tom lane

[1]: https://git.postgresql.org/gitweb/?p=postgresql.git&amp;a=commitdiff&amp;h=86dc90056

#3Duncan Sands
duncan.sands@deepbluecap.com
In reply to: Tom Lane (#2)
Re: Query planning on partitioned table causes postgres 13.4 to consume all memory

Hi Tom,

On 19/09/2021 18:03, Tom Lane wrote:

Duncan Sands <duncan.sands@deepbluecap.com> writes:

[ planning DELETE on a thousand-partition table takes forever ]

FWIW, this situation has been very much improved for v14 [1].

thanks, part (2) of that commit indeed looks like it should solve it.

Best wishes, Duncan.

Show quoted text

In older branches, the best advice I can give you is "don't use
so many partitions". Especially not with hash partitioning,
where the query WHERE clause generally won't translate to any
useful pruning of the partitions.

(Personally, I think that hash partitioning is an evil that
we shouldn't have implemented at all. Or at least there
should be stronger warnings about it in the manual than there
are now.)

regards, tom lane

[1] https://git.postgresql.org/gitweb/?p=postgresql.git&amp;a=commitdiff&amp;h=86dc90056