scary patch contest

On 25 Aug 2026, at 18:48, Robert Haas <robertmhaas@gmail.com> wrote:

#2 has been cut back in scope enough that I don't think we would have accepted
the feature originally if it had been subject to those limitations, which IMHO
is a reasonable argument for a revert but also not conclusive.

I don't think that is an argument for reverting, I think it's an argument for
blocking progress by making perfect the enemy of good.

I'm pretty scared about all of #1-#3 having a long tail of bugs that we haven't
found yet, in pretty critical areas.

I'll prepare a revert.

--
Daniel Gustafsson

On Tue, Aug 25, 2026 at 08:13:10PM +0200, Daniel Gustafsson wrote:

On 25 Aug 2026, at 18:48, Robert Haas <robertmhaas@gmail.com> wrote:

#2 has been cut back in scope enough that I don't think we would have accepted
the feature originally if it had been subject to those limitations, which IMHO
is a reasonable argument for a revert but also not conclusive.

I don't think that is an argument for reverting, I think it's an argument for
blocking progress by making perfect the enemy of good.

I'm pretty scared about all of #1-#3 having a long tail of bugs that we haven't
found yet, in pretty critical areas.

I'll prepare a revert.

Uh, I am confused. We are now considering reverting these?

1. RI fast-path FK checks / batching (2da86c1ef9b, b7b27eb41a5,
2. REPACK / REPACK CONCURRENTLY (ac58465e061, 28d534e2ae0, Antonin
3. Online data checksums (f19c0eccae9, Daniel Gustafsson and Magnus

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

Do not let urgent matters crowd out time for investment in the future.

On Tue, Aug 25, 2026 at 2:13 PM Daniel Gustafsson <daniel@yesql.se> wrote:

On 25 Aug 2026, at 18:48, Robert Haas <robertmhaas@gmail.com> wrote:

I'm pretty scared about all of #1-#3 having a long tail of bugs that we haven't
found yet, in pretty critical areas.

I'll prepare a revert.

I don't feel like there is consensus that we need to revert this yet.
One thing that I'm wondering is if the ease with which LLMs allow
people to pressure test features means we are finding more bugs sooner
than we have in the past. With online checksums, as far as I
understand, the design has consensus and the implementation has
required follow-on commits but not more than other similar sized
features that weren't reverted. I understand that the concern is 1)
data corruption is especially bad and 2) if there were bugs so far,
who knows what more could be lurking. But I tend to wonder if 2 is
changing because of how easy it is to look for bugs now.

- Melanie

On Tue, Aug 25, 2026 at 02:50:37PM -0400, Melanie Plageman wrote:

I don't feel like there is consensus that we need to revert this yet.
One thing that I'm wondering is if the ease with which LLMs allow
people to pressure test features means we are finding more bugs sooner
than we have in the past. With online checksums, as far as I
understand, the design has consensus and the implementation has
required follow-on commits but not more than other similar sized
features that weren't reverted. I understand that the concern is 1)
data corruption is especially bad and 2) if there were bugs so far,
who knows what more could be lurking. But I tend to wonder if 2 is
changing because of how easy it is to look for bugs now.

If we look back at previously released features, does anyone feel these
PG 19 features received an unusualy high number of cleanup patches? And
if so, are there more because of AI, as Melanie is asking?

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

Do not let urgent matters crowd out time for investment in the future.

On Tue, Aug 25, 2026 at 2:13 PM Daniel Gustafsson <daniel@yesql.se> wrote:

#2 has been cut back in scope enough that I don't think we would have accepted
the feature originally if it had been subject to those limitations, which IMHO
is a reasonable argument for a revert but also not conclusive.

I don't think that is an argument for reverting, I think it's an argument for
blocking progress by making perfect the enemy of good.

I'm not sure I understand what you're saying here. My guess is that
you're arguing that we should have been willing to take REPACK
CONCURRENTLY even if we'd known from the beginning that the
one-at-a-time-per-cluster restriction would have been necessary. I
would have argued against that, but perhaps I would have been wrong to
do so, and it's certainly fair if you feel otherwise.

I'm pretty scared about all of #1-#3 having a long tail of bugs that we haven't
found yet, in pretty critical areas.

I'll prepare a revert.

My sincere intention was to start a conversation, not jump straight to
a conclusion. If you were already half-convinced that you should be
thinking about reverting and this pushed you over the line, I suppose
that's fair enough. Otherwise, I think some discussion would be a good
idea.

--
Robert Haas
EDB: http://www.enterprisedb.com

Robert Haas <robertmhaas@gmail.com> wrote:

2. REPACK / REPACK CONCURRENTLY (ac58465e061, 28d534e2ae0, Antonin
Houska c/b Álvaro Herrera): 28 fixes including data loss (missingvals
discarded, also affecting VACUUM FULL/CLUSTER), broken toasted-tuple
and generated-column handling, and two security-flavored ACL fixes.

One of the two "two security-flavored ACL fixes" is probably e13851080ce - I
think this was a pre-existing problem, rather than REPACK (CONCURRENTLY)
regression.

Not sure what the other fix was.

--
Antonin Houska
Web: https://www.cybertec-postgresql.com

On 2026-08-25 Tu 2:54 PM, Robert Haas wrote:

On Tue, Aug 25, 2026 at 2:13 PM Daniel Gustafsson <daniel@yesql.se> wrote:

#2 has been cut back in scope enough that I don't think we would have accepted
the feature originally if it had been subject to those limitations, which IMHO
is a reasonable argument for a revert but also not conclusive.

I don't think that is an argument for reverting, I think it's an argument for
blocking progress by making perfect the enemy of good.

I'm not sure I understand what you're saying here. My guess is that
you're arguing that we should have been willing to take REPACK
CONCURRENTLY even if we'd known from the beginning that the
one-at-a-time-per-cluster restriction would have been necessary. I
would have argued against that, but perhaps I would have been wrong to
do so, and it's certainly fair if you feel otherwise.

Given the unsatisfactory state of all the alternatives, I think it is
still worth having, modulo any remaining bugs.

I'm pretty scared about all of #1-#3 having a long tail of bugs that we haven't
found yet, in pretty critical areas.

I'll prepare a revert.

My sincere intention was to start a conversation, not jump straight to
a conclusion. If you were already half-convinced that you should be
thinking about reverting and this pushed you over the line, I suppose
that's fair enough. Otherwise, I think some discussion would be a good
idea.

Agreed, let's not be hasty, even though time is now very short. I do
remember one case about 10 years ago that people told me they found
scary, but in the event it hasn't given rise to the huge number of bugs
that were predicted. I do think we need to be smarter about using LLMs
methodically to detect and test for bugs more comprehensively, not as an
alternative to human efforts but as a useful supplement.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

ср, 26 авг. 2026 г. в 00:29, Andrew Dunstan <andrew@dunslane.net>:

I do think we need to be smarter about using LLMs
methodically to detect and test for bugs more comprehensively, not as an
alternative to human efforts but as a useful supplement.

As someone who does fuzzing, finds bugs in the code, and spends free time
going through bug mailing lists and trying to fix issues - let me share my
perspective.

Melanie made a good point - LLMs are making it easier for people to find
bugs, and that's definitely true. But at the same time, they also help us
respond to those findings more quickly.

That said, only those who do nothing are mistake-free.

Maybe we should be writing more tests and thinking more broadly during the
design phase. That alone won't solve everything, but it could help catch
things earlier.

From where I stand, fixing a bug is only half the battle. We need the
change to be merged into master. Reviews are generally easy to come by -
there's almost always someone who can do them. However, making sure that a
committer or the subsystem maintainer notices something in the bug mailing
list requires effort and a bit of ingenuity.

--
Regards,
Rachitskiy Andrey

On Tue, Aug 25, 2026 at 02:53:54PM -0400, Bruce Momjian wrote:

On Tue, Aug 25, 2026 at 02:50:37PM -0400, Melanie Plageman wrote:

I don't feel like there is consensus that we need to revert this yet.
One thing that I'm wondering is if the ease with which LLMs allow
people to pressure test features means we are finding more bugs sooner
than we have in the past. With online checksums, as far as I
understand, the design has consensus and the implementation has
required follow-on commits but not more than other similar sized
features that weren't reverted. I understand that the concern is 1)
data corruption is especially bad and 2) if there were bugs so far,
who knows what more could be lurking. But I tend to wonder if 2 is
changing because of how easy it is to look for bugs now.

If we look back at previously released features, does anyone feel these
PG 19 features received an unusualy high number of cleanup patches? And
if so, are there more because of AI, as Melanie is asking?

I do get the feeling that we're finding more with LLMs. Beyond that, the
RMT can use LLMs to easily find threads that ought to be added as open
items but weren't for whatever reason (e.g., the barrage of e-mails I just
sent out).

--
nathan

On Tue, Aug 25, 2026 at 2:50 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:

I don't feel like there is consensus that we need to revert this yet.

+1.

One thing that I'm wondering is if the ease with which LLMs allow
people to pressure test features means we are finding more bugs sooner
than we have in the past. With online checksums, as far as I
understand, the design has consensus and the implementation has
required follow-on commits but not more than other similar sized
features that weren't reverted.

I'm not sure about that last part, because this came out of an
LLM-driven analysis of features that required a lot of post-commit
fixing. It's not unique in that, but it has required a lot of fixes.

I understand that the concern is 1)
data corruption is especially bad and 2) if there were bugs so far,
who knows what more could be lurking. But I tend to wonder if 2 is
changing because of how easy it is to look for bugs now.

I don't know. I don't especially care about low-probability bugs that
are unlikely to be hit in practice, but it's not clear to me the stuff
thus far fixed should be placed in that category. In any event, if
we've fixed most of the stuff real users are likely to hit, that's
good enough. If we haven't, that's potentially a problem. Even then,
if the remaining bugs are things that have few architectural
implications and can likely be cleaned up in a minor release without
causing much trouble, that may still be fine, the exception being if
there are really a lot of them. But if we put out a .0 release and
then discover, for example, an online-checksum bug that can't be fixed
without bumping catversion, that would be very awkward. I'm not saying
that particular thing is likely, but just that once .0 goes out we
have fewer options to resolve remaining problems and everything gets
more painful because there are real production clusters out there to
worry about. I'm not trying to prejudge the outcome here; I don't
actually know what the right thing to do is. But I think it's worth
talking about, and hearing from Daniel on how he's feeling about the
feature at this point.

--
Robert Haas
EDB: http://www.enterprisedb.com

Hi,

On 8/25/26 4:26 PM, Robert Haas wrote:

I'm not sure about that last part, because this came out of an
LLM-driven analysis of features that required a lot of post-commit
fixing. It's not unique in that, but it has required a lot of fixes.

My suggestion would be use multiple LLMs and see if they agree as using
orangu /auto_review give very different results based on the LLM loaded.

You didn't state which LLM you used with Claude. A lot of people still
liked Opus 4.8 more than Opus 5, and few like Fable 5.

Best regards,
Jesper

Robert Haas <robertmhaas@gmail.com> writes:

I asked Claude to evaluate which v19 patches were the scariest based
on the number and type of bugs fixed post-freeze. Results below, with
a few particularly cutting remarks from the LLM edited out.

...

5. SQL/PGQ property graphs (2f094e7ac69, Peter Eisentraut and Ashutosh
Bapat): 17 fixes, but lower severity — wrong collation, unresolved
literals, broken LATERAL references, deparse and pg_dump ACL bugs,
plus a series of after-the-fact prohibitions.

...

FWIW, I am quite afraid of #5, mainly because we are *still*
discussing fixes that will require parsetree and/or catalog changes
and thus catversion bumps. (There are two such threads on the
open-items list, and possibly more that I've not noticed.) At this
point I'd be willing to bet dinner that if we ship it in v19 there
will be post-release bug discoveries that are unfixable until v20.

regards, tom lane

On Tue, Aug 25, 2026 at 6:33 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

FWIW, I am quite afraid of #5, mainly because we are *still*
discussing fixes that will require parsetree and/or catalog changes
and thus catversion bumps. (There are two such threads on the
open-items list, and possibly more that I've not noticed.) At this
point I'd be willing to bet dinner that if we ship it in v19 there
will be post-release bug discoveries that are unfixable until v20.

Without opining on that feature too specifically, I don't think
features that may need catversion bumps belong in v19 at this point. I
guess if some feature needs a trivial catversion bump and we're pretty
confident nothing more will be needed afterward then it might be OK to
let it slide, but really, the time for working out what the catalogs
should look like was sometime well in advance of feature freeze, not
four months after it. My gut reaction is that anything that still
needs a catversion bump at this point should be reverted at least out
of v19, *especially* if it's not clear what that catversion bump needs
to look like.

--
Robert Haas
EDB: http://www.enterprisedb.com

On Wed, Aug 26, 2026 at 3:54 AM Bruce Momjian <bruce@momjian.us> wrote:

If we look back at previously released features, does anyone feel these
PG 19 features received an unusualy high number of cleanup patches? And
if so, are there more because of AI, as Melanie is asking?

Yeah, I feel the same way.

I expected that AI assistance would make new features more stable by
feature freeze, but it seems that that hasn't turned out to be the
case.

- Richard

On Wed, Aug 26, 2026 at 12:24 AM Bruce Momjian <bruce@momjian.us> wrote:

On Tue, Aug 25, 2026 at 02:50:37PM -0400, Melanie Plageman wrote:

I don't feel like there is consensus that we need to revert this yet.
One thing that I'm wondering is if the ease with which LLMs allow
people to pressure test features means we are finding more bugs sooner
than we have in the past. With online checksums, as far as I
understand, the design has consensus and the implementation has
required follow-on commits but not more than other similar sized
features that weren't reverted. I understand that the concern is 1)
data corruption is especially bad and 2) if there were bugs so far,
who knows what more could be lurking. But I tend to wonder if 2 is
changing because of how easy it is to look for bugs now.

If we look back at previously released features, does anyone feel these
PG 19 features received an unusualy high number of cleanup patches? And
if so, are there more because of AI, as Melanie is asking?

I think so. Now, with AI it is relatively easier to understand the
code and find the problems. So more people are able to find and
provide the solution to problems. IMO, this is good for long term
stability of the project even though in the short-term, we see more
cleanup patches post-commit. I am hoping that slowly we will also use
AI more before committing and reduce such cleanup work after-words.

--
With Regards,
Amit Kapila.

On 26 Aug 2026, at 02:58, Richard Guo <guofenglinux@gmail.com> wrote:

I expected that AI assistance would make new features more stable by
feature freeze, but it seems that that hasn't turned out to be the
case.

Keep in mind that most (if not all) large features in 19 were written, reviewed
and tested, before AI tools were either available or even remotely as good as
they are now. I think the volume and type of postcommit fixes for 19 is a
clear indication of this. Feature freeze for 20 will likely be a different
story again.

--
Daniel Gustafsson

On Tue, Aug 25, 2026 at 10:18 PM Robert Haas <robertmhaas@gmail.com> wrote:

I asked Claude to evaluate which v19 patches were the scariest based
on the number and type of bugs fixed post-freeze. Results below, with
a few particularly cutting remarks from the LLM edited out. I also
omitted the list of runners-up that it generated, none of which seemed
to me to be of enough concern to be worth discussion. I do think these
are worth discussing, though: are any of them likely to still be in
bad enough shape that we should consider reverting them before v19
ships? Past performance is no guarantee of future results, but it's
not a contraindication either.

1. RI fast-path FK checks / batching (2da86c1ef9b, b7b27eb41a5,
Junwang Zhao and Amit Langote): ~16 fixes including an out-of-bounds
write on re-entry, a crash under nested SPI, a wrong-user-id index
scan, and five distinct classes of incorrect FK enforcement
(cross-type, domains, column order, nullable keys, non-btree indexes).
The transactional design was retrofitted post-freeze
(subtransaction/firing-cycle tracking, deferred-trigger integration,
xact.c cleanup).

2. REPACK / REPACK CONCURRENTLY (ac58465e061, 28d534e2ae0, Antonin
Houska c/b Álvaro Herrera): 28 fixes including data loss (missingvals
discarded, also affecting VACUUM FULL/CLUSTER), broken toasted-tuple
and generated-column handling, and two security-flavored ACL fixes.
Its prerequisite (database-specific logical replication snapshots,
0d3dba38c777) was reverted as fundamentally flawed, leaving v19's
REPACK CONCURRENTLY restricted to one process cluster-wide.

3. Online data checksums (f19c0eccae9, Daniel Gustafsson and Magnus
Hagander): ~25 substantive fixes, mostly state-machine holes — invalid
transitions in checkpoints and promotion, initial state missing from
the control file, basebackup false checksum failures, wrong WAL for
unlogged relations. A corruption-detection feature producing false
positives is exactly the wrong failure mode.

4. UPDATE/DELETE FOR PORTION OF (8e72d914c52, Paul Jungwirth c/b Peter
Eisentraut): 17 fixes, three of them security — leftover-row inserts
bypassed INSERT RLS policies, WITH CHECK OPTION went unenforced on
DELETE, and UPDATE permission on the range column wasn't required. The
interaction matrix (views, generated columns, inheritance, rules,
WHERE CURRENT OF, FDW partitions) had to be forbidden or fixed piece
by piece after freeze.

5. SQL/PGQ property graphs (2f094e7ac69, Peter Eisentraut and Ashutosh
Bapat): 17 fixes, but lower severity — wrong collation, unresolved
literals, broken LATERAL references, deparse and pg_dump ACL bugs,
plus a series of after-the-fact prohibitions.

6. postgres_fdw statistics import (28972b6fc3d, Corey Huinker and
Etsuro Fujita): committed on freeze day, then needed 7 fixes including
a syntax error, a quoted-column deparsing bug, an SPI removal, and a
user-facing option rename in August. High defect density for ~one file
of contrib code.

I don't currently have a firm position on what we should do here. I
think it's pretty clear that none of these were as robust at commit
time as we would like, but that doesn't mean that they're still
broken. I personally think #4 and #6 are relatively lower risk on a
going-forward basis, because they're only going to come up in fairly
niche situations. That somewhat applies to #5 as well, but it's a
bigger feature so I'm more concerned.

I agree with your judgement about #4 and #6. For others, I don't know
based on the current number and severity of issues how we should
decide whether to revert or not because it is quite possible that this
time we found more bugs early with the help of AI. If we are concerned
about their quality, one possibility is to do a post-review of some of
those.

#2 has been cut back in scope

enough that I don't think we would have accepted the feature
originally if it had been subject to those limitations,

Yeah, this is a valid point. But in favor of this feature, we had
patches for the current limitations but they were not mature enough to
be committed or released. I hope that we will work on removing these
limitations in future releases. As per my knowledge, we had a
consensus on the design for the current committed work and the same
has been reviewed as well. Sure, we had found more bugs than is
normally expected but that could be both due to AI and the feature is
committed near to feature-freeze. I vote to keep #2.

--
With Regards,
Amit Kapila.

On Tue, Aug 25, 2026 at 9:29 PM Andrew Dunstan <andrew@dunslane.net> wrote:

On 2026-08-25 Tu 2:54 PM, Robert Haas wrote:

On Tue, Aug 25, 2026 at 2:13 PM Daniel Gustafsson <daniel@yesql.se> wrote:

#2 has been cut back in scope enough that I don't think we would have accepted
the feature originally if it had been subject to those limitations, which IMHO
is a reasonable argument for a revert but also not conclusive.

I don't think that is an argument for reverting, I think it's an argument for
blocking progress by making perfect the enemy of good.

I'm not sure I understand what you're saying here. My guess is that
you're arguing that we should have been willing to take REPACK
CONCURRENTLY even if we'd known from the beginning that the
one-at-a-time-per-cluster restriction would have been necessary. I
would have argued against that, but perhaps I would have been wrong to
do so, and it's certainly fair if you feel otherwise.

Given the unsatisfactory state of all the alternatives, I think it is
still worth having, modulo any remaining bugs.

Very well said, +1.

I also think that those are complex features which are much more complex and
that's why nobody probably tried to solve those problems for years (decades),
which caused avalanche of 3rd party solutions in the first place, which
caused even more hairy issues.

I'm pretty scared about all of #1-#3 having a long tail of bugs that we haven't
found yet, in pretty critical areas.

I'll prepare a revert.

My sincere intention was to start a conversation, not jump straight to
a conclusion. If you were already half-convinced that you should be
thinking about reverting and this pushed you over the line, I suppose
that's fair enough. Otherwise, I think some discussion would be a good
idea.

Agreed, let's not be hasty, even though time is now very short. I do
remember one case about 10 years ago that people told me they found
scary, but in the event it hasn't given rise to the huge number of bugs
that were predicted. I do think we need to be smarter about using LLMs
methodically to detect and test for bugs more comprehensively, not as an
alternative to human efforts but as a useful supplement.

General idea: I never fully understood why we couldn't also mark some features
as bleeding edge / experimental / kind of developer preview if there's some
doubt (some things like 'debug_io_direct' and recent
'debug_logical_replication_streaming' come to mind and already doing this;
one could have GUCs to unlock some experimental stuff for the brave ones.
E.g. in the case debug_io_direct it's pretty clear that that Direct I/O was
never finished, yet it's there, which is awesome. It's just the naming
('debug_' prefix) that doesn't really fit. techpreview_%s IMHO would be
way better to unlock certain stuff.

-J.