Query planning on partitioned table causes postgres 13.4 to consume all memory
Postgresql version: 13.4
O/S version: Ubuntu 21.04
How to reproduce:
(1) Create "data" schema and its tables by executing the commands from the
attached file d.sql. Each of the two created tables has less than 20 rows.
(2) Execute or EXPLAIN this query
DELETE FROM data.files
WHERE path=ANY(
SELECT path
FROM (
SELECT meta, path,
first_value(path)
OVER (PARTITION BY meta ORDER BY priority) AS first
FROM data.files
) f,
data.metadata m
WHERE f.path <> f.first
AND f.meta = m.id
AND m.syd = 667
);
(3) Observe that the EXPLAIN/query never completes, and memory usage goes up
and up.
For example:
duncan=> \i /tmp/d.sql
...
duncan=> DELETE FROM data.files
WHERE path=ANY(
SELECT path
FROM (
SELECT meta, path,
first_value(path)
OVER (PARTITION BY meta ORDER BY priority) AS first
FROM data.files
) f,
data.metadata m
WHERE f.path <> f.first
AND f.meta = m.id
AND m.syd = 667
);
... wailing and gnashing of teeth ...
Attachments:
Duncan Sands <duncan.sands@deepbluecap.com> writes:
[ planning DELETE on a thousand-partition table takes forever ]
FWIW, this situation has been very much improved for v14 [1]https://git.postgresql.org/gitweb/?p=postgresql.git&a=commitdiff&h=86dc90056.
In older branches, the best advice I can give you is "don't use
so many partitions". Especially not with hash partitioning,
where the query WHERE clause generally won't translate to any
useful pruning of the partitions.
(Personally, I think that hash partitioning is an evil that
we shouldn't have implemented at all. Or at least there
should be stronger warnings about it in the manual than there
are now.)
regards, tom lane
[1]: https://git.postgresql.org/gitweb/?p=postgresql.git&a=commitdiff&h=86dc90056
Hi Tom,
On 19/09/2021 18:03, Tom Lane wrote:
Duncan Sands <duncan.sands@deepbluecap.com> writes:
[ planning DELETE on a thousand-partition table takes forever ]
FWIW, this situation has been very much improved for v14 [1].
thanks, part (2) of that commit indeed looks like it should solve it.
Best wishes, Duncan.
Show quoted text
In older branches, the best advice I can give you is "don't use
so many partitions". Especially not with hash partitioning,
where the query WHERE clause generally won't translate to any
useful pruning of the partitions.(Personally, I think that hash partitioning is an evil that
we shouldn't have implemented at all. Or at least there
should be stronger warnings about it in the manual than there
are now.)regards, tom lane
[1] https://git.postgresql.org/gitweb/?p=postgresql.git&a=commitdiff&h=86dc90056