Untrusted relations and WAL
Hi hackers,
I know that everybody who can do anything about vulnerabilities is busy
with the workload LLMs generate for pgsql-security these days. Still,
the list does not forbid ideas for fun, and this one has been bothering
me. There is no patch attached, and there will not be one: I understand
the problem is beyond what I can solve alone, and current AI will not
lift me to the required level either. I would be glad to hear thoughts.
Our threat model is clear and well served by the security team: "a
security vulnerability in PostgreSQL is an issue that allows a user to
gain access to privileges or data that they do not have permission to
use, or allows a user to execute arbitrary code through a PostgreSQL
process" [0]https://www.postgresql.org/support/security/. It covers a lot: the wire protocol, SQL constructs,
database objects, ACLs, type input functions. And yet a typical exploit
is just an SQL query. Frankly, a bit boring. Let's add attack surface in
the form of new bytes!
Today, letting a friend run your $PGDATA on their machine is handing them
a Trojan horse, no weaker than an .exe or an ELF. No surprise: we have
functions, which are small programs, i.e. code, i.e. executable code,
and running somebody else's code is what security people call RCE.
Cleaning the catalog is a tractable problem. A sufficiently patched
pg_upgrade can drop every give_me_superuser() SECURITY DEFINER, recreate
extensions from local packages, throw away remaining C functions, and so
on, until the catalog is fresh. I have prototyped this far. Making
relations and WAL safe is where my imagination ends.
Why would anyone give PGDATA to a friend who does not trust its bytes?
* A managed PostgreSQL service could onboard an on-premise database
without downtime: make the cluster a physical replica, then switch
the load. Much faster than pg_dump/pg_restore, and much simpler than
setting up logical replication for an arbitrary schema.
Just a small GUC dance around shared_buffers\max_connections needed.
* Compute/storage separation systems like Neon or Aurora. Storage
processes WAL from many tenants, so attacking it with a zero-day
freshly published by our security team (there is no shortage) is
worth somebody's time. I believe, and this is worth checking, that
they confine the WAL parser in a container and hand out page images
that nothing interprets further: pages go to compute as they are.
As soon as we want to do anything interesting with pages in a place
shared by tenants (put them into Iceberg?), the page contents can
attack us. In the managed case a per-database container is also not
a perfect answer, but I do not want to describe control plane attacks
on the list.
It would be wonderful to harden every redo function: no combination of
bytes and WAL records shall... do what, exactly? What would a security
model even look like here?
This year brotli had CVE-2026-111497 [1]https://nvd.nist.gov/vuln/detail/cve-2026-111497: decompressing a crafted
stream corrupts decoder state and can lead to code execution. I
reported exactly the same property of pglz to pgsql-security in 2019.
Back then I could not imagine a real, widely deployed general-purpose
codec having this kind of bug. The answer was that our model does not
let a user hand compressed bytes to pglz, which is true. It was fixed
later anyway, and that is good. But we have many such places: the
binary layout of jsonb can attack the jsonb code, clever
bytes can attack arrays, the TOAST machinery, tsvector, range types -
all of them are parsers of bytes an attacker chose, with no history of
security hardening because the model says they never see untrusted
input.
So the questions:
1. Is it true that we cannot, even in theory, state a security model in
which WAL and relations are untrusted? Or is there a clever way to
draw a comprehensible boundary there?
2. Alternatively, could a large hardening project over everything we
read from disk make sense? Although it seems we already have plenty
of real reports on the surfaces we did promise to defend.
Thank you for reading this far. Any ideas are welcome.
Best regards, Andrey Borodin.
[0]: https://www.postgresql.org/support/security/
[1]: https://nvd.nist.gov/vuln/detail/cve-2026-111497
Hi,
On 2026-11-08 23:53:37 +0500, Andrey Borodin wrote:
It would be wonderful to harden every redo function: no combination of
bytes and WAL records shall... do what, exactly? What would a security
model even look like here?
I think this is just about impossible. Obviously the catalogs need to be
modifiable by WAL. Once you can update the catalog in arbitrary ways, you're
able to make the database do just about arbitrary things, including obviously
arbitrary code execution.
Introducing a security boundary for physiological replication / crash recovery
makes IMO no sense whatsoever. It's not mathematically impossible, if you make
postgres not have any extensibility and have all crucial configuration
(including authn & authz) outside the database, but there wouldn't be much of
postgres left.
So the questions:
1. Is it true that we cannot, even in theory, state a security model in
which WAL and relations are untrusted? Or is there a clever way to
draw a comprehensible boundary there?
Correct.
2. Alternatively, could a large hardening project over everything we
read from disk make sense?
Sure, we can add some more or a a lot more hardening of what we read. I think
that's entirely unrelated to make it safe to replay arbitrarily corrupt WAL.
Although it seems we already have plenty of real reports on the surfaces we
did promise to defend.
TBH, even just the idea of expanding the scope of what we consider vulns feels
like somewhat an insult at the moment. If you want to make everyone in
postgres maintenance quit from exhaustion, that's the way.
Greetings,
Andres Freund
On 08/11/2026 8:53 PM, Andrey Borodin wrote:
Hi hackers,
I know that everybody who can do anything about vulnerabilities is busy
with the workload LLMs generate for pgsql-security these days. Still,
the list does not forbid ideas for fun, and this one has been bothering
me. There is no patch attached, and there will not be one: I understand
the problem is beyond what I can solve alone, and current AI will not
lift me to the required level either. I would be glad to hear thoughts.Our threat model is clear and well served by the security team: "a
security vulnerability in PostgreSQL is an issue that allows a user to
gain access to privileges or data that they do not have permission to
use, or allows a user to execute arbitrary code through a PostgreSQL
process" [0]. It covers a lot: the wire protocol, SQL constructs,
database objects, ACLs, type input functions. And yet a typical exploit
is just an SQL query. Frankly, a bit boring. Let's add attack surface in
the form of new bytes!Today, letting a friend run your $PGDATA on their machine is handing them
a Trojan horse, no weaker than an .exe or an ELF. No surprise: we have
functions, which are small programs, i.e. code, i.e. executable code,
and running somebody else's code is what security people call RCE.Cleaning the catalog is a tractable problem. A sufficiently patched
pg_upgrade can drop every give_me_superuser() SECURITY DEFINER, recreate
extensions from local packages, throw away remaining C functions, and so
on, until the catalog is fresh. I have prototyped this far. Making
relations and WAL safe is where my imagination ends.Why would anyone give PGDATA to a friend who does not trust its bytes?
* A managed PostgreSQL service could onboard an on-premise database
without downtime: make the cluster a physical replica, then switch
the load. Much faster than pg_dump/pg_restore, and much simpler than
setting up logical replication for an arbitrary schema.
Just a small GUC dance around shared_buffers\max_connections needed.* Compute/storage separation systems like Neon or Aurora. Storage
processes WAL from many tenants, so attacking it with a zero-day
freshly published by our security team (there is no shortage) is
worth somebody's time. I believe, and this is worth checking, that
they confine the WAL parser in a container and hand out page images
that nothing interprets further: pages go to compute as they are.
As soon as we want to do anything interesting with pages in a place
shared by tenants (put them into Iceberg?), the page contents can
attack us. In the managed case a per-database container is also not
a perfect answer, but I do not want to describe control plane attacks
on the list.
In Neon we from the very beginning do not consider compute as trusted
source of data.
This is why applying WAL at page server is performed by separate process
with restricted set of system calls.
So even if some sequence of WAL records allows a hacker to get full
control on this process, it still can not do anything harmful.
We thought about rewriting all redo handler in Rust which can make such
cracks much more difficult.
But we refused this idea because it will greatly complicate integration
with upstream changes.
Actually we didn't encountered any attempts to crack page server by
generating invalid WAL records.
But even legal WAL records can affect page server. For example "CRATE
DATABASE ... STRATEGY FILE_COPY" can generate single arbitrary large WAL
record.
Which can just cause OOM at PS. Also arbitrary large WAL records can be
generated with wal_level=logical. But we filter such data and do not
send them to page server.