Stop asserting that Unicode normalization consumes its whole input

Started by Tristan Partin2 days ago1 messageshackers
Beta feature

Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.

appliessuccessCI history

You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:

docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t253847
psql -h localhost -U postgres

Built from patchset v1 (message #1), September 20, 2026 at 05:11 PM.

Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:

git clone --branch t253847_1 https://github.com/hackorum-dev/postgres.git

In a checkout you already have, add the fork once:

git remote add hackorum https://github.com/hackorum-dev/postgres.git

then, for this patchset and every later one:

git fetch hackorum t253847_1 && git checkout t253847_1

Patchset v1 (message #1) is on t253847_1

Jump to latest
#1Tristan Partin
tristan@partin.io

normalize() and IS NORMALIZED determine how many code points their input
holds with pg_mbstrlen_with_len(). They then decode exactly that many
code points, and assert afterward that this consumed the entire datum.
That invariant does not hold because pg_mbstrlen_with_len() stops at the
first of "limit" or a NUL.

Postgres doesn't allow text types to contain NUL, so reaching this
requires work, but src/test/regress/sql/encoding.sql already constructs
such values to pin down how the string functions behave on them. It just
never passed them to normalize() or IS NORMALIZED, which is why this
went unnoticed.

--
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)

Attachments:

t253847_1
v1-0001-Stop-asserting-that-Unicode-normalization-consume.patchtext/x-patch; charset=utf-8; name=v1-0001-Stop-asserting-that-Unicode-normalization-consume.patchDownload+16-3