Preferred way to submit a very large batch of bugs?
Hi all,
I know the mailing lists have been hit with a flood of automated bug
reports lately and
the last thing I want to do is create more headaches for the team.
Over the past few
months, I've been building a suite of tools for finding bugs. That
work has surfaced
over 1000 previously unreported bugs ranging from crashes, to
corruption, to MVCC
violations. I've included three of the bugs so you can get a sense of
some of the findings.
What's the least disruptive way for me to submit these issues? I've prepared a
searchable catalog that makes it easy to browse through the bugs based on the
category and severity of the issue, but I wanted to reach out first
before sharing anything.
Cheers,
Michael
** Hot standby query gets wrong results from index built by plain
CREATE INDEX **
With hot_standby_feedback = off (the default), a REPEATABLE READ
transaction on a hot standby can silently observe two different row
counts for the same query within a single snapshot after a plain CREATE
INDEX runs on the primary. Specifically, if rows visible to the standby
snapshot are deleted and committed on the primary and a new index is
then built on the table (without VACUUM or heap pruning, so all heap
tuples remain on the page), the primary treats the deleted tuples as
HEAPTUPLE_DEAD and omits them from the new index. Once the index build
is replayed on the standby, a sequential scan in the open REPEATABLE
READ transaction still returns all rows visible to its snapshot, whereas
an index scan or index-only scan using the new index omits the deleted
rows. No recovery conflict is raised even with
max_standby_streaming_delay at its default of 30s, breaking the
REPEATABLE READ guarantee that a transaction sees a consistent snapshot
unless canceled by a recovery conflict.
Reproduction (primary with a streaming hot standby, default
hot_standby_feedback = off and max_standby_streaming_delay = 30s or -1):
-- Primary:
CREATE TABLE t (id int, k int) WITH (autovacuum_enabled = off, fillfactor = 50);
INSERT INTO t SELECT g, g FROM generate_series(1, 1000) g;
-- Standby (after replay):
BEGIN ISOLATION LEVEL REPEATABLE READ;
SET enable_indexscan = off;
SET enable_bitmapscan = off;
SET enable_indexonlyscan = off;
SELECT count(*) FROM t WHERE k > 0;
-- actual: 1000
-- Primary:
DELETE FROM t WHERE id <= 500;
CREATE INDEX t_k ON t(k);
-- Standby (after replay of CREATE INDEX, in the same transaction):
SELECT count(*) FROM t WHERE k > 0;
-- actual: 1000
SET enable_seqscan = off;
SET enable_bitmapscan = off;
SET enable_indexscan = on;
SET enable_indexonlyscan = on;
SELECT count(*) FROM t WHERE k > 0;
-- actual: 500
-- expected: 1000 (or transaction canceled by recovery conflict)
COMMIT;
** SQL-language function plan cache freezes parse-time literal coercions **
In PostgreSQL 18 and later, non-inlined SQL-language functions cache
their parse-analyzed Query tree across calls for the lifetime of the
session. Because unknown-literal input conversion happens during parse
analysis, parse-time literal coercions inside a SQL function body are
frozen on the first call until a relation or function invalidation or a
search_path change occurs. Consequently, INSERT ... VALUES ('now', ...)
inside a SQL function writes the first call's transaction timestamp in
all subsequent transactions, and literals whose parsing depends on GUC
settings such as DateStyle or TimeZone retain the meaning from the
first call after those GUCs change. In PostgreSQL 17 and earlier, a
non-inlined SQL function body is re-parsed in every outer query, so
'now' evaluates to the current transaction's start time and GUC-
dependent literals follow current settings, matching direct query
execution. While the PL/pgSQL documentation (plpgsql.sgml, "Plan
Caching") warns about this plan-caching behavior for 'now', SQL-language
functions never had this behavior prior to 18 and neither the PostgreSQL
18 documentation nor the release notes state that SQL functions are
affected.
Reproduction:
CREATE TABLE one(x int);
INSERT INTO one VALUES (1);
CREATE FUNCTION fd() RETURNS date LANGUAGE sql AS $$SELECT
'01/02/2024'::date FROM one$$;
CREATE FUNCTION ftz() RETURNS text LANGUAGE sql AS $$SELECT
('2024-01-01 00:00'::timestamptz)::timestamp::text FROM one$$;
CREATE TABLE log(ts timestamptz, msg text);
CREATE FUNCTION log_it(m text) RETURNS void LANGUAGE sql AS $$ INSERT
INTO log VALUES ('now', m) $$;
SET TimeZone = 'UTC';
SET DateStyle = 'ISO, MDY';
SELECT fd(); -- 2024-01-02
SELECT ftz(); -- 2024-01-01 00:00:00
SELECT log_it('first');
SET DateStyle = 'ISO, DMY';
SET TimeZone = 'Asia/Tokyo';
SELECT fd(); -- actual (18): 2024-01-02 (stale
MDY parse)
-- expected (17): 2024-02-01
SELECT ftz(); -- actual (18): 2024-01-01
09:00:00 (stale UTC parse)
-- expected (17): 2024-01-01 00:00:00
SELECT '01/02/2024'::date AS fresh; -- 2024-02-01
SELECT pg_sleep(1.1);
SELECT log_it('second');
SELECT count(DISTINCT ts) AS distinct_ts FROM log;
-- actual (18): 1 (second row got the first call's 'now')
-- expected (17): 2
** ON UPDATE CASCADE into narrower-typmod FK leaves orphans or re-points rows **
When an ON UPDATE CASCADE foreign key references a primary key column
with a wider typmod (for example, varchar(n) or char(n) referencing
text, numeric(p,s) referencing numeric, or timestamp(p) referencing
timestamp), the cascade query updates the referencing column through
assignment coercion to the FK column's typmod. This coercion can alter
the cascaded value without raising an error: varchar(n) and char(n)
drop trailing blanks per the SQL length-coercion rule, and numeric(p,s)
and timestamp(p) round. As a result, the FK row either keeps its old
stored value so the FK re-check is skipped and an orphan row is
committed, or receives a rounded value that matches a different PK row
and silently re-points the reference. This violates the CREATE TABLE
documentation for ON UPDATE CASCADE ("update the values of the
referencing column(s) to the new values of the referenced column(s)")
and breaks the invariant that a validated, enforced foreign key holds
after every statement (a brute-force check and re-adding the constraint
with ALTER TABLE ADD FOREIGN KEY both fail with SQLSTATE 23503).
Reproduction:
-- A: varchar(n) FK referencing text PK: cascade value truncated
-- (trailing blanks), FK re-check skipped -> committed orphan
CREATE TABLE pv (k text PRIMARY KEY);
INSERT INTO pv VALUES ('abc');
CREATE TABLE fv (r varchar(3) REFERENCES pv ON UPDATE CASCADE);
INSERT INTO fv VALUES ('abc');
UPDATE pv SET k = 'abc ';
SELECT r, length(r) FROM fv;
-- r | length
-- -----+--------
-- abc | 3
-- (1 row)
SELECT count(*) AS orphans FROM fv WHERE NOT EXISTS (SELECT 1 FROM pv
WHERE pv.k = fv.r);
-- orphans
-- ---------
-- 1
-- (1 row)
ALTER TABLE fv ADD CONSTRAINT recheck FOREIGN KEY (r) REFERENCES pv;
-- ERROR: insert or update on table "fv" violates foreign key
constraint "recheck"
-- DETAIL: Key (r)=(abc) is not present in table "pv".
-- B: char(n) FK referencing text PK
CREATE TABLE pt (k text PRIMARY KEY);
INSERT INTO pt VALUES ('ab');
CREATE TABLE ft (r char(4) REFERENCES pt ON UPDATE CASCADE);
INSERT INTO ft VALUES ('ab');
UPDATE pt SET k = 'ab ';
ALTER TABLE ft ADD CONSTRAINT recheck FOREIGN KEY (r) REFERENCES pt;
-- ERROR: insert or update on table "ft" violates foreign key
constraint "recheck"
-- DETAIL: Key (r)=(ab ) is not present in table "pt".
-- C: numeric(5,1) FK referencing numeric PK: cascade rounds the new
-- key, FK row silently re-points to another PK row
CREATE TABLE pn (k numeric PRIMARY KEY);
INSERT INTO pn VALUES (1.2), (1.3);
CREATE TABLE fn (r numeric(5,1) REFERENCES pn ON UPDATE CASCADE);
INSERT INTO fn VALUES (1.2);
UPDATE pn SET k = 1.25 WHERE k = 1.2;
SELECT * FROM fn;
-- r
-- -----
-- 1.3
-- (1 row)
UPDATE pn SET k = 9 WHERE k = 1.25;
SELECT * FROM fn;
-- r
-- -----
-- 1.3
-- (1 row)
-- D: timestamp(0) FK referencing timestamp PK
CREATE TABLE pts (k timestamp PRIMARY KEY);
INSERT INTO pts VALUES ('2020-01-01 00:00:00'), ('2020-01-01 00:00:01');
CREATE TABLE fts (r timestamp(0) REFERENCES pts ON UPDATE CASCADE);
INSERT INTO fts VALUES ('2020-01-01 00:00:00');
UPDATE pts SET k = '2020-01-01 00:00:00.7' WHERE k = '2020-01-01 00:00:00';
SELECT * FROM fts;
-- r
-- ---------------------
-- 2020-01-01 00:00:01
-- (1 row)
Expected behavior:
After any statement, a validated, enforced foreign key constraint must
hold (0 orphans, and re-adding the same constraint with ALTER TABLE ADD
FOREIGN KEY must succeed), and cascading an update must not silently
re-point a referencing row to a different PK row. Either the cascade
must fail when the coerced FK value differs from the new PK key (or
does not equal it under the FK equality operator), or the FK check must
run on cascaded rows.
On 2026-Nov-02, Michael Malis wrote:
Hi all,
I know the mailing lists have been hit with a flood of automated bug
reports lately and the last thing I want to do is create more
headaches for the team. Over the past few months, I've been building
a suite of tools for finding bugs. That work has surfaced over 1000
previously unreported bugs ranging from crashes, to corruption, to
MVCC violations.
Hello Michael,
As a first triage step, it's worth separating out the bugs that are
potential security problems. Crashes per se aren't security bugs, but
buffer overwrites or memory disclosure while the server is on its way
down are likely security bugs. Those are better reported directly to
security@postgresql.org with no further public trail. (But if a minor
release is close to be published, say in the following two weeks, it's
not a bad idea to wait for that and see if it's going to be fixed there
first; a bug reported too close to the deadline is unlikely to be dealt
with in that release. Reporting immediately after the minors are out
maximizes time to prepare fixes without zero-daying the project.)
For non-security bugs, the preferred way is to get them submitted via
the bug submission form [1]https://www.postgresql.org/account/submitbug/. Flooding the list with hundreds of bugs is
unlikely to be welcome though, so I suppose you'll need to categorize/
prioritize so that we can get the important things out of the way
without drowning in trivial issues.
[1]: https://www.postgresql.org/account/submitbug/
I've included three of the bugs so you can get a sense of some of the
findings.
Fun :-( (Please do not consider these as having been reported. We're
unlikely to follow through based on these samples in your broader
questions.)
Thanks,
--
Álvaro Herrera
On Tue, 03 Nov 2026, Michael Malis <malis@pgrust.com> wrote:
I know the mailing lists have been hit with a flood of automated bug
reports lately and
the last thing I want to do is create more headaches for the team.
Over the past few
months, I've been building a suite of tools for finding bugs. That
work has surfaced
over 1000 previously unreported bugs ranging from crashes, to
corruption, to MVCC
violations. I've included three of the bugs so you can get a sense of
some of the findings.
I did the same thing, and I am facing basically the same issue.
As Alvaro already mentioned, my first concern was security issues - separating the bulk of it is relatively easy with LLMs, but even that requires careful manual review as even when you tell them the criteria used by the security team, they still misclassify issues. In my case, around ~5% of all findings were security-suspect issues initially, but after reading them manually I only considered half of them as such based on the postgres guidelines, so the rest of them could be declassified.
Unfortunately the other direction is also true: I have several examples where an LLM identified an issue as not-security, but after looking into a bit more and writing a patch for it, I realized that it is security related.
And that second part is the main reason why so far I only published my open-issue list with a limited set of people instead of somewhere public and posting a link to it to hackers to parallelize the work, and then we are slowly sending patches about them to the list one-by-one.
Hi,
On 2026-11-02 18:06:23 -0800, Michael Malis wrote:
I know the mailing lists have been hit with a flood of automated bug reports
lately and the last thing I want to do is create more headaches for the
team. Over the past few months, I've been building a suite of tools for
finding bugs. That work has surfaced over 1000 previously unreported bugs
ranging from crashes, to corruption, to MVCC violations. I've included three
of the bugs so you can get a sense of some of the findings.
What's the least disruptive way for me to submit these issues? I've prepared a
searchable catalog that makes it easy to browse through the bugs based on the
category and severity of the issue, but I wanted to reach out first
before sharing anything.
A few things that I think would be important:
- Make sure there aren't many duplicates. The same root cause can have many
different consequences. Of course you can't be expected to perfecly pinpoint
everything, but some effort is important.
- Minimize the heck out of reproducers. I think we have all grown extremely
tired of overcomplicated LLM generated repros. Ensure the simplest
implementation of a repro is chosen.
regression test > isolation test > tap test >>> random python/C/... script
- Make sure they're practically relevant. Make the server crash by doing
insane shit as superuser isn't interesting. Coming up with absurdly
convoluted query patterns that produce arguably kind of maybe inconsistent
results isn't likely actionable at the moment.
- Try to separate out security issues.
- Sending a thousand separate reports at once is likely just to overwhelm
everyone. Sending all of them at once is unlikely to be manageable. The
least bad idea I have is to group them into related issues and submit those
together.
- If some of the tooling for discovering these kind of issues could be morphed
into something that others can be used, or even better that can be made into
something the server detects, it could be helpful.
Going through the examples:
** Hot standby query gets wrong results from index built by plain
CREATE INDEX **
This clearly is an issue we need to fix.
Reproduction (primary with a streaming hot standby, default
hot_standby_feedback = off and max_standby_streaming_delay = 30s or -1):
This is an example where including a minimal tap test might be helpful
(assuming this is done with LLM help), or even better an extension of one of
the existing tap tests. Thats easier to deal with than something requiring a
sequence of manual steps between multiple nodes.
** SQL-language function plan cache freezes parse-time literal coercions **
In PostgreSQL 18 and later, non-inlined SQL-language functions cache
their parse-analyzed Query tree across calls for the lifetime of the
session. Because unknown-literal input conversion happens during parse
analysis, parse-time literal coercions inside a SQL function body are
frozen on the first call until a relation or function invalidation or a
search_path change occurs. Consequently, INSERT ... VALUES ('now', ...)
inside a SQL function writes the first call's transaction timestamp in
all subsequent transactions, and literals whose parsing depends on GUC
settings such as DateStyle or TimeZone retain the meaning from the
first call after those GUCs change. In PostgreSQL 17 and earlier, a
non-inlined SQL function body is re-parsed in every outer query, so
'now' evaluates to the current transaction's start time and GUC-
dependent literals follow current settings, matching direct query
execution. While the PL/pgSQL documentation (plpgsql.sgml, "Plan
Caching") warns about this plan-caching behavior for 'now', SQL-language
functions never had this behavior prior to 18 and neither the PostgreSQL
18 documentation nor the release notes state that SQL functions are
affected.
Personally this seems like one that is at most worth reporting as a
documentation improvement. And given the flood we are facing, I'm not sure I'd
start with sending relatively minor stuff like this. As you say, this is
already present in a bunch of places.
** ON UPDATE CASCADE into narrower-typmod FK leaves orphans or re-points rows **
When an ON UPDATE CASCADE foreign key references a primary key column
with a wider typmod (for example, varchar(n) or char(n) referencing
text, numeric(p,s) referencing numeric, or timestamp(p) referencing
timestamp), the cascade query updates the referencing column through
assignment coercion to the FK column's typmod. This coercion can alter
the cascaded value without raising an error: varchar(n) and char(n)
drop trailing blanks per the SQL length-coercion rule, and numeric(p,s)
and timestamp(p) round. As a result, the FK row either keeps its old
stored value so the FK re-check is skipped and an orphan row is
committed, or receives a rounded value that matches a different PK row
and silently re-points the reference. This violates the CREATE TABLE
documentation for ON UPDATE CASCADE ("update the values of the
referencing column(s) to the new values of the referenced column(s)")
and breaks the invariant that a validated, enforced foreign key holds
after every statement (a brute-force check and re-adding the constraint
with ALTER TABLE ADD FOREIGN KEY both fail with SQLSTATE 23503).
To me this falls somewhere between "Don't do that, maybe it'd not hurt to warn
against" and "It'd be nice if we errored out" categories.
I'd again not start with this kind of thing, we have much bigger fish to fry,
and I don't think this is of huge practical relevance. But I could imagine
others disagreeing.
Greetings,
Andres Freund
Hi Michael,
I have been sending AI-found bugs this year. Here is what worked.
Before I send one, I have Claude search the archives and the commit
log for it. There are some very good postgres mcp server such as
https://pg.ddx.io/ is good for that use case I also have skills for this
and am happy to share them.
I reproduce every bug by hand, which catches false positives. Then I
ask what kind of user/workload would hit it. A superuser-only case on a
platform you have never heard of is probably not worth sending.
A SELECT * from table that returns wrong results is a big deal.
I review every fix myself. I asked Claude to push it to a private GitHub
repo,
so I get a diff view and can ask about any line and review it.
One thing I found is that the LLM often fixes the symptom, not the root
cause.
I am sure I have made some noise too. I am happy to follow whatever
AI policy the community ends up with.
Thanks,
Shihao
Ok. What I'll do is submit the 5 most serious bugs and 5 most serious
security issues I've found. Assuming those are well received, I'll submit
more.
Cheers,
Michael
On Wed, Nov 04, 2026 at 11:17 AM, shihao zhong <zhong950419@gmail.com>
wrote:
Show quoted text
Hi Michael,
I have been sending AI-found bugs this year. Here is what worked.
Before I send one, I have Claude search the archives and the commit
log for it. There are some very good postgres mcp server such as
https://pg.ddx.io/ is good for that use case I also have skills for this
and am happy to share them.I reproduce every bug by hand, which catches false positives. Then I
ask what kind of user/workload would hit it. A superuser-only case on a
platform you have never heard of is probably not worth sending.
A SELECT * from table that returns wrong results is a big deal.I review every fix myself. I asked Claude to push it to a private GitHub
repo,
so I get a diff view and can ask about any line and review it.One thing I found is that the LLM often fixes the symptom, not the root
cause.I am sure I have made some noise too. I am happy to follow whatever
AI policy the community ends up with.Thanks,
Shihao
Hi,
On Thu, Nov 5, 2026 at 3:18 AM shihao zhong <zhong950419@gmail.com> wrote:
Hi Michael,
I have been sending AI-found bugs this year. Here is what worked.
Before I send one, I have Claude search the archives and the commit
log for it. There are some very good postgres mcp server such as
https://pg.ddx.io/ is good for that use case I also have skills for this
and am happy to share them.
I reproduce every bug by hand, which catches false positives. Then I
ask what kind of user/workload would hit it. A superuser-only case on a
platform you have never heard of is probably not worth sending.
A SELECT * from table that returns wrong results is a big deal.I review every fix myself. I asked Claude to push it to a private GitHub repo,
so I get a diff view and can ask about any line and review it.One thing I found is that the LLM often fixes the symptom, not the root cause.
It seems somewhat helpful to instruct it to identify the root cause
before suggesting any fixing strategies.
I am sure I have made some noise too. I am happy to follow whatever
AI policy the community ends up with.
The current mailing list and commitfest seems unable to scale up well
with the growing numbers of bug reports. We don't have an issue ticket
system for tracking those bugs and there're no strict owners for them
as well except for the reported bugs of the newly-minted features.
Submitting a huge amount of bug reports at once seems like it would
blow up the whole thing. Therefore, besides the suggestions given by
other hackers, I am wondering whether it's helpful to build a
temporary site or use existing tools to track those bugs after
filtering, auditing and grouping them. This seems able to meet future
needs to some extent as well. That said, not sure about the degree of
that demand since there's a likely ceiling to the current growth of
bug reports -- the existing stock of them would be exhausted one day
after massive hunting and the bar of getting into the core has raised
because new features are already under heavy-scrutiny by LLMs, which
are quite good at catching edge cases. Another complementary tool that
seems helpful is something like the sashiko review system from the
linux community [1]https://github.com/sashiko-dev/sashiko.
[1]: https://github.com/sashiko-dev/sashiko
--
Regards,
Xuneng Zhou
HighGo Software Co., Ltd.
On Thu, 05 Nov 2026, Xuneng Zhou <xunengzhou@gmail.com> wrote:
I am wondering whether it's helpful to build a
temporary site or use existing tools to track those bugs after
filtering, auditing and grouping them.
That's the direction we are going with hackorum.dev. It already makes reading/searching/using the lists much easier, and we are working on many features that will move it even more towards a proper bug tracker / review tool based on the mailing list. You can also suggest ideas or PRs :) (we are already working on LLM-assisted automatic grouping, for example, soon available)
Another complementary tool that
seems helpful is something like the sashiko review system from the
linux community
Thanks, that's a useful link I didn't know about. I already built something similar during the year (for now privately) for reviewing patches, which is based on the claude/codex CLI, so it works with subscriptions instead of API based billing. The long term goal with that is also to automatically run it for every patch submission (and commits, as sometimes there are changes between the last patch and the committed code), and to make it public, but I want to make it really good before I publish it.
That said, not sure about the degree of
that demand since there's a likely ceiling to the current growth of
bug reports
I would be still more worried about the part that without a human actually looking at it, we are just risking disclosing/finding security issues more easier. I know LLMs are already capable of finding many, but we shouldn't make it even more easier by just dumping lots of unreviewed bugreports on the internet.
On Thu, Nov 5, 2026 at 4:03 PM Zsolt Parragi <zsolt.parragi@percona.com> wrote:
On Thu, 05 Nov 2026, Xuneng Zhou <xunengzhou@gmail.com> wrote:
I am wondering whether it's helpful to build a
temporary site or use existing tools to track those bugs after
filtering, auditing and grouping them.That's the direction we are going with hackorum.dev. It already makes
reading/searching/using the lists much easier, and we are working on
many features that will move it even more towards a proper bug tracker
/ review tool based on the mailing list. You can also suggest ideas or
PRs :) (we are already working on LLM-assisted automatic grouping, for
example, soon available)
Thanks for informing me about this. It looks useful. I'll take a closer look.
Another complementary tool that
seems helpful is something like the sashiko review system from the
linux communityThanks, that's a useful link I didn't know about. I already built
something similar during the year (for now privately) for reviewing
patches, which is based on the claude/codex CLI, so it works with
subscriptions instead of API based billing. The long term goal with
that is also to automatically run it for every patch submission (and
commits, as sometimes there are changes between the last patch and the
committed code), and to make it public, but I want to make it really
good before I publish it.
There're lots of discussions and controversies on the use of this tool
in the linux community. Nevertheless, It seems to be a
growingly-useful tool thanks to the advancement of both the model and
harness.. I was thinking of building something similar for a while,
but was not able to do it because of the limited bandwidth. You can
share it to the mailing list when it is mature. But I also think
there're advantages of sharing a basically-working POC project, it
would receive helpful potential feedback and contributions at the
early stage.
That said, not sure about the degree of
that demand since there's a likely ceiling to the current growth of
bug reportsI would be still more worried about the part that without a human
actually looking at it, we are just risking disclosing/finding
security issues more easier. I know LLMs are already capable of
finding many, but we shouldn't make it even more easier by just
dumping lots of unreviewed bugreports on the internet.
Yeah, it needs to separate security issues from others.
--
Regards,
Xuneng Zhou
HighGo Software Co., Ltd.