BUG #18851: Queries with xxx NOT IN (SELECT xxx FROM table) fail to run (or run very slowly) on v17 (v14 ok)

The following bug has been logged on the website:

Bug reference: 18851
Logged by: Joan Sánchez Sabé
Email address: joan@sanchezsabe.com
PostgreSQL version: 17.4
Operating system: Windows 11 24H2 (26100.3194)
Description:

I have experienced a misbehaviour (IMHO) in PostgreSQL 17 that didn't occur
in PostgreSQL 14.

Minimum reproducible scenario to show the error:

1) Table with a primary key (integer) and 500_000 records on it.

CREATE TABLE t1 AS
(
SELECT
id
FROM
generate_series(1, 1000000) AS s(id)
ORDER BY
random()
LIMIT
500000
) ;
ALTER TABLE t1
ADD PRIMARY KEY (id);

ANALYZE t1 ;

2) Second table, with a smaller (200_000) number of values of the same type,
some of them overlapping the first table, without a primary key (nor any
index).

CREATE TABLE t2 AS
(
SELECT
id
FROM
generate_series(1, 1000000) AS s(id)
ORDER BY
random()
LIMIT
200000
) ;

3) The following query, which just looks for values in t2 not in t1, takes
about 200 ms on my computer using "PostgreSQL 14.17, compiled by Visual C++
build 1942, 64-bit"; running on Windows 11 Pro 24H2 (26100.3194).

SELECT
t2.id
FROM
t2
WHERE
t2.id NOT IN (SELECT id FROM t1) ;

Output: a random list of around 100_000 numbers.

The query plan is as follows:

Seq Scan on public.t2 (cost=8463.00..11848.00 rows=100000 width=4)
(actual time=122.093..186.668 rows=99566 loops=1)
Output: t2.id
Filter: (NOT (hashed SubPlan 1))
Rows Removed by Filter: 100434
SubPlan 1
-> Seq Scan on public.t1 (cost=0.00..7213.00 rows=500000 width=4)
(actual time=0.017..20.573 rows=500000 loops=1)
Output: t1.id
Planning Time: 0.173 ms
Execution Time: 192.002 ms

You can check a (smallish) version on https://dbfiddle.uk/aHTC8YQd

--------
The same query, executed on "PostgreSQL 17.4 on x86_64-windows, compiled by
msvc-19.42.34436, 64-bit)" takes "forever". I actually gave up after 10
minutes.

The query plan was:

Seq Scan on public.t2 (cost=0.00..1294403396.00 rows=100000 width=4)
Output: t2.id
Filter: (NOT (ANY (t2.id = (SubPlan 1).col1)))
SubPlan 1
-> Materialize (cost=0.00..11694.00 rows=500000 width=4)
Output: t1.id
-> Seq Scan on public.t1 (cost=0.00..7240.00 rows=500000
width=4)
Output: t1.id

The same query works on smaller tables (sizes 100_000 and 40_000,
respectively; i.e.: 5 x smaller). In that case, the execution plan is the
same as with PostgreSQL 14 (with values divided by aprox. 5, as expected)

Seq Scan on public.t2 (cost=1698.00..2502.00 rows=24480 width=4)
(actual time=20.813..28.825 rows=20016 loops=1)
Output: t2.id
Filter: (NOT (ANY (t2.id = (hashed SubPlan 1).col1)))
Rows Removed by Filter: 19984
SubPlan 1
-> Seq Scan on public.t1 (cost=0.00..1448.00 rows=100000 width=4)
(actual time=0.013..4.844 rows=100000 loops=1)
Output: t1.id
Planning Time: 0.205 ms
Execution Time: 30.565 ms

So, it seems that the problem appears when the execution planner adds the
"Materialize" node.

If necessary, I can add configuration files for both versions of
PostgreSQL.

Many thanks in advance