Add ASCII fast path to Unicode normalization functions

Started by Andrew Dunstan2 months ago16 messageshackers
Jump to latest
#1Andrew Dunstan
andrew@dunslane.net

Hi,

A linkedin post comparing CedarDB's new Unicode normalization support
to PostgreSQL's caught my eye [1]https://lnkd.in/p/eKUqSj73: same results, but a claimed 30x
speedup on "SELECT count(*) FROM hits WHERE url IS NORMALIZED" over
ClickBench's hits table. Most of that turned out to be down to CedarDB
using all available threads by default versus our
max_parallel_workers_per_gather of 2. But even at the matched thread
count they reported a 6x edge, attributed to two things: an ASCII fast
path (most URLs are already normalized ASCII, so you can skip decoding
entirely), and vectorized byte scanning for the ASCII check itself.

I went and looked, and unicode_is_normalized(), unicode_assigned(), and
normalize() all decode every string to an array of char32_t codepoints,
one utf8_to_unicode()/pg_utf_mblen() call at a time, before doing any
real work -- including on input that's already pure ASCII. The attached
patch adds a fast path: scan the raw bytes for anything with the high
bit set, using the SIMD-vectorized is_valid_ascii() we already have
(currently only used inside pg_utf8_verifystr()). If nothing is found,
the string is trivially normalized (ASCII code points have no
canonical or compatibility decomposition, and a combining class of
zero) and every code point in it is assigned, so all three functions
can return immediately.

I deliberately didn't copy CedarDB's trick of comparing byte length to
codepoint count -- getting the codepoint count means calling
pg_mbstrlen_with_len(), exactly the scalar work this patch avoids.
Scanning raw bytes with is_valid_ascii() instead reuses SIMD
infrastructure we already have, and is cheaper to begin with: a single
reduction versus a population count.

Benchmarked with data sized to fit comfortably under shared_buffers rather
than triggering the seqscan ring-buffer bypass, which otherwise swamps the
comparison at larger table sizes: ~10x on pure ASCII, ~4x on an 85/15
ASCII/non-ASCII mix, and no measurable regression on non-ASCII input
that still needs the full decode-and-quickcheck path.

Regression tests cover the ASCII-hit case for all three functions, plus
a boundary sweep that plants a non-NFC sequence at varying offsets
around ASCII padding, to catch any off-by-one in the SIMD-chunk/scalar-
remainder split.

cheers

andrew

[1]: https://lnkd.in/p/eKUqSj73

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Attachments:

t253785_1
0001-Add-ASCII-fast-path-to-Unicode-normalization-functio.patchtext/x-patch; charset=UTF-8; name=0001-Add-ASCII-fast-path-to-Unicode-normalization-functio.patchDownload+139-1
#2David E. Wheeler
david@justatheory.com
In reply to: Andrew Dunstan (#1)
Re: Add ASCII fast path to Unicode normalization functions

On Sep 14, 2026, at 10:50, Andrew Dunstan <andrew@dunslane.net> wrote:

Benchmarked with data sized to fit comfortably under shared_buffers rather
than triggering the seqscan ring-buffer bypass, which otherwise swamps the
comparison at larger table sizes: ~10x on pure ASCII, ~4x on an 85/15
ASCII/non-ASCII mix, and no measurable regression on non-ASCII input
that still needs the full decode-and-quickcheck path.

Well this looks super simple and like a very nice speedup! Thanks for digging into that!

D

#3Chao Li
li.evan.chao@gmail.com
In reply to: Andrew Dunstan (#1)
Re: Add ASCII fast path to Unicode normalization functions

On Sep 14, 2026, at 22:50, Andrew Dunstan <andrew@dunslane.net> wrote:

Hi,

A linkedin post comparing CedarDB's new Unicode normalization support
to PostgreSQL's caught my eye [1]: same results, but a claimed 30x
speedup on "SELECT count(*) FROM hits WHERE url IS NORMALIZED" over
ClickBench's hits table. Most of that turned out to be down to CedarDB
using all available threads by default versus our
max_parallel_workers_per_gather of 2. But even at the matched thread
count they reported a 6x edge, attributed to two things: an ASCII fast
path (most URLs are already normalized ASCII, so you can skip decoding
entirely), and vectorized byte scanning for the ASCII check itself.

I went and looked, and unicode_is_normalized(), unicode_assigned(), and
normalize() all decode every string to an array of char32_t codepoints,
one utf8_to_unicode()/pg_utf_mblen() call at a time, before doing any
real work -- including on input that's already pure ASCII. The attached
patch adds a fast path: scan the raw bytes for anything with the high
bit set, using the SIMD-vectorized is_valid_ascii() we already have
(currently only used inside pg_utf8_verifystr()). If nothing is found,
the string is trivially normalized (ASCII code points have no
canonical or compatibility decomposition, and a combining class of
zero) and every code point in it is assigned, so all three functions
can return immediately.

I deliberately didn't copy CedarDB's trick of comparing byte length to
codepoint count -- getting the codepoint count means calling
pg_mbstrlen_with_len(), exactly the scalar work this patch avoids.
Scanning raw bytes with is_valid_ascii() instead reuses SIMD
infrastructure we already have, and is cheaper to begin with: a single
reduction versus a population count.

Benchmarked with data sized to fit comfortably under shared_buffers rather
than triggering the seqscan ring-buffer bypass, which otherwise swamps the
comparison at larger table sizes: ~10x on pure ASCII, ~4x on an 85/15
ASCII/non-ASCII mix, and no measurable regression on non-ASCII input
that still needs the full decode-and-quickcheck path.

Regression tests cover the ASCII-hit case for all three functions, plus
a boundary sweep that plants a non-NFC sequence at varying offsets
around ASCII padding, to catch any off-by-one in the SIMD-chunk/scalar-
remainder split.

cheers

andrew

[1] https://lnkd.in/p/eKUqSj73

--
Andrew Dunstan
EDB: https://www.enterprisedb.com
<0001-Add-ASCII-fast-path-to-Unicode-normalization-functio.patch>

The patch looks good to me.

I also did some benchmark testing on my MacBook Air M4. I used clean builds with -O2 and without -g.

# unicode_is_normalized()

* all ascii: master 572ms; patch 68ms; Huge improvement
* mixed: master 529ms; patch 64ms; Big improvement
* non-ascii: master 399ms; patch 397ms; Roughly unchanged
* late-non-ascii: master 1069ms; patch 1074ms; Roughly unchanged; This is the worse case, most of chars are ascii, and only unicode appear in the end

# unicode_normalize_func()

* all ascii: master 1659ms; patch 68ms; Huge improvement
* mixed: master 1909ms; patch 233ms; Big improvement
* non-ascii: master 1432ms; patch 1452ms; Roughly unchanged
* late-non-ascii: master 3572ms; patch 3578ms; Roughly unchanged

# unicode_assigned()

* all ascii: master 258ms; patch 60ms; Big improvement
* mixed: master 237ms; patch 61ms; Big improvement
* non-ascii: master 221ms; patch 219ms; Roughly unchanged
* late-non-ascii: master 531ms; patch 533ms; Roughly unchanged

The late-non-ascii case is intended to be a worst case for the added ascii scan: most of the string is ascii, with the first non-ascii character appearing near the end.

For pure ascii and mixed ascii/non-ascii input, all three functions show substantial improvements. For pure non-ascii input, including the late-non-ascii case, performance is roughly unchanged.

So this looks like a worthwhile performance optimization to me.

The attached is my test script.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

Attachments:

test_ascii_perf.sqlapplication/octet-stream; name=test_ascii_perf.sql; x-unix-mode=0644Download
#4Andrew Dunstan
andrew@dunslane.net
In reply to: Chao Li (#3)
Re: Add ASCII fast path to Unicode normalization functions

On 2026-09-15 Tu 12:22 AM, Chao Li wrote:

On Sep 14, 2026, at 22:50, Andrew Dunstan <andrew@dunslane.net> wrote:

Hi,

A linkedin post comparing CedarDB's new Unicode normalization support
to PostgreSQL's caught my eye [1]: same results, but a claimed 30x
speedup on "SELECT count(*) FROM hits WHERE url IS NORMALIZED" over
ClickBench's hits table. Most of that turned out to be down to CedarDB
using all available threads by default versus our
max_parallel_workers_per_gather of 2. But even at the matched thread
count they reported a 6x edge, attributed to two things: an ASCII fast
path (most URLs are already normalized ASCII, so you can skip decoding
entirely), and vectorized byte scanning for the ASCII check itself.

I went and looked, and unicode_is_normalized(), unicode_assigned(), and
normalize() all decode every string to an array of char32_t codepoints,
one utf8_to_unicode()/pg_utf_mblen() call at a time, before doing any
real work -- including on input that's already pure ASCII. The attached
patch adds a fast path: scan the raw bytes for anything with the high
bit set, using the SIMD-vectorized is_valid_ascii() we already have
(currently only used inside pg_utf8_verifystr()). If nothing is found,
the string is trivially normalized (ASCII code points have no
canonical or compatibility decomposition, and a combining class of
zero) and every code point in it is assigned, so all three functions
can return immediately.

I deliberately didn't copy CedarDB's trick of comparing byte length to
codepoint count -- getting the codepoint count means calling
pg_mbstrlen_with_len(), exactly the scalar work this patch avoids.
Scanning raw bytes with is_valid_ascii() instead reuses SIMD
infrastructure we already have, and is cheaper to begin with: a single
reduction versus a population count.

Benchmarked with data sized to fit comfortably under shared_buffers rather
than triggering the seqscan ring-buffer bypass, which otherwise swamps the
comparison at larger table sizes: ~10x on pure ASCII, ~4x on an 85/15
ASCII/non-ASCII mix, and no measurable regression on non-ASCII input
that still needs the full decode-and-quickcheck path.

Regression tests cover the ASCII-hit case for all three functions, plus
a boundary sweep that plants a non-NFC sequence at varying offsets
around ASCII padding, to catch any off-by-one in the SIMD-chunk/scalar-
remainder split.

[1] https://lnkd.in/p/eKUqSj73

The patch looks good to me.

I also did some benchmark testing on my MacBook Air M4. I used clean builds with -O2 and without -g.

# unicode_is_normalized()

* all ascii: master 572ms; patch 68ms; Huge improvement
* mixed: master 529ms; patch 64ms; Big improvement
* non-ascii: master 399ms; patch 397ms; Roughly unchanged
* late-non-ascii: master 1069ms; patch 1074ms; Roughly unchanged; This is the worse case, most of chars are ascii, and only unicode appear in the end

# unicode_normalize_func()

* all ascii: master 1659ms; patch 68ms; Huge improvement
* mixed: master 1909ms; patch 233ms; Big improvement
* non-ascii: master 1432ms; patch 1452ms; Roughly unchanged
* late-non-ascii: master 3572ms; patch 3578ms; Roughly unchanged

# unicode_assigned()

* all ascii: master 258ms; patch 60ms; Big improvement
* mixed: master 237ms; patch 61ms; Big improvement
* non-ascii: master 221ms; patch 219ms; Roughly unchanged
* late-non-ascii: master 531ms; patch 533ms; Roughly unchanged

The late-non-ascii case is intended to be a worst case for the added ascii scan: most of the string is ascii, with the first non-ascii character appearing near the end.

For pure ascii and mixed ascii/non-ascii input, all three functions show substantial improvements. For pure non-ascii input, including the late-non-ascii case, performance is roughly unchanged.

So this looks like a worthwhile performance optimization to me.

The attached is my test script.

Great, thanks for the review and tests.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

#5Nazir Bilal Yavuz
byavuz81@gmail.com
In reply to: Andrew Dunstan (#4)
Re: Add ASCII fast path to Unicode normalization functions

Hi,

Thank you for working on this!

On Tue, 15 Sept 2026 at 14:31, Andrew Dunstan <andrew@dunslane.net> wrote:

On 2026-09-15 Tu 12:22 AM, Chao Li wrote:

The attached is my test script.

Great, thanks for the review and tests.

What do you think about continuing from where text_ascii_check() is
left? I wrote a patch for this and benchmarked with Chao's script.

Timings are v1 vs v2, not master vs v2.

# unicode_is_normalized()

* all ascii: 89.115ms | 87.803ms
* mixed: 249.239ms | 126.011ms -> improvement
* non-ascii: 1243.496ms | 1244.052ms
* late-non-ascii: 3632.410ms | 370.966ms -> improvement

# unicode_normalize_func()

* all ascii: 84.458ms | 84.636ms
* mixed: 578.684ms | 226.093ms -> improvement
* non-ascii: 3651.573ms | 3644.026ms
* late-non-ascii: 10513.861ms | 942.470ms -> improvement

# unicode_assigned()

* all ascii: 61.147ms | 60.658ms
* mixed: 124.193ms | 86.186ms -> improvement
* non-ascii: 507.130ms | 510.247ms
* late-non-ascii: 1169.617ms | 166.924ms -> improvement

Do you think these results worth the additional complexity?

--
Regards,
Nazir Bilal Yavuz
Microsoft

Attachments:

t253785_5
v2-0001-Add-ASCII-fast-path-to-Unicode-normalization-func.patchtext/x-patch; charset=US-ASCII; name=v2-0001-Add-ASCII-fast-path-to-Unicode-normalization-func.patchDownload+139-1
nocfbot-v2-0002-Return-index-at-text_ascii_check.patchtext/x-patch; charset=US-ASCII; name=nocfbot-v2-0002-Return-index-at-text_ascii_check.patchDownload+53-23
#6Andrew Dunstan
andrew@dunslane.net
In reply to: Nazir Bilal Yavuz (#5)
Re: Add ASCII fast path to Unicode normalization functions

On 2026-09-15 Tu 8:51 AM, Nazir Bilal Yavuz wrote:

Hi,

Thank you for working on this!

On Tue, 15 Sept 2026 at 14:31, Andrew Dunstan <andrew@dunslane.net> wrote:

On 2026-09-15 Tu 12:22 AM, Chao Li wrote:

The attached is my test script.

Great, thanks for the review and tests.

What do you think about continuing from where text_ascii_check() is
left? I wrote a patch for this and benchmarked with Chao's script.

Timings are v1 vs v2, not master vs v2.

# unicode_is_normalized()

* all ascii: 89.115ms | 87.803ms
* mixed: 249.239ms | 126.011ms -> improvement
* non-ascii: 1243.496ms | 1244.052ms
* late-non-ascii: 3632.410ms | 370.966ms -> improvement

# unicode_normalize_func()

* all ascii: 84.458ms | 84.636ms
* mixed: 578.684ms | 226.093ms -> improvement
* non-ascii: 3651.573ms | 3644.026ms
* late-non-ascii: 10513.861ms | 942.470ms -> improvement

# unicode_assigned()

* all ascii: 61.147ms | 60.658ms
* mixed: 124.193ms | 86.186ms -> improvement
* non-ascii: 507.130ms | 510.247ms
* late-non-ascii: 1169.617ms | 166.924ms -> improvement

Do you think these results worth the additional complexity?

Yes, I think it is - it's not that much extra. This is neat, in effect
avoiding duplicating the work.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

#7Tristan Partin
tristan@neon.tech
In reply to: Andrew Dunstan (#1)
Re: Add ASCII fast path to Unicode normalization functions

On Mon Sep 14, 2026 at 9:55 AM CDT, Andrew Dunstan wrote:

Hi,

A linkedin post comparing CedarDB's new Unicode normalization support
to PostgreSQL's caught my eye [1]: same results, but a claimed 30x
speedup on "SELECT count(*) FROM hits WHERE url IS NORMALIZED" over
ClickBench's hits table. Most of that turned out to be down to CedarDB
using all available threads by default versus our
max_parallel_workers_per_gather of 2. But even at the matched thread
count they reported a 6x edge, attributed to two things: an ASCII fast
path (most URLs are already normalized ASCII, so you can skip decoding
entirely), and vectorized byte scanning for the ASCII check itself.

I went and looked, and unicode_is_normalized(), unicode_assigned(), and
normalize() all decode every string to an array of char32_t codepoints,
one utf8_to_unicode()/pg_utf_mblen() call at a time, before doing any
real work -- including on input that's already pure ASCII. The attached
patch adds a fast path: scan the raw bytes for anything with the high
bit set, using the SIMD-vectorized is_valid_ascii() we already have
(currently only used inside pg_utf8_verifystr()). If nothing is found,
the string is trivially normalized (ASCII code points have no
canonical or compatibility decomposition, and a combining class of
zero) and every code point in it is assigned, so all three functions
can return immediately.

I deliberately didn't copy CedarDB's trick of comparing byte length to
codepoint count -- getting the codepoint count means calling
pg_mbstrlen_with_len(), exactly the scalar work this patch avoids.
Scanning raw bytes with is_valid_ascii() instead reuses SIMD
infrastructure we already have, and is cheaper to begin with: a single
reduction versus a population count.

Benchmarked with data sized to fit comfortably under shared_buffers rather
than triggering the seqscan ring-buffer bypass, which otherwise swamps the
comparison at larger table sizes: ~10x on pure ASCII, ~4x on an 85/15
ASCII/non-ASCII mix, and no measurable regression on non-ASCII input
that still needs the full decode-and-quickcheck path.

Regression tests cover the ASCII-hit case for all three functions, plus
a boundary sweep that plants a non-NFC sequence at varying offsets
around ASCII padding, to catch any off-by-one in the SIMD-chunk/scalar-
remainder split.

[1] https://lnkd.in/p/eKUqSj73

Hey Andrew,

I saw the same LinkedIn post too, and I also started working on
a similar patch that I was benchmarking last week. So I'll provide some
review and some results that I saw.

FWIW, here is the function that I added to ascii.h:

/*
* Wrapper around is_valid_ascii() such that a string of any length can be
* passed in. If you know your string to have a length of a multiple of
* sizeof(Vector8), stick with is_valid_ascii(). It will avoid a few
* instructions.
*/
static inline bool
is_all_valid_ascii(const unsigned char *s, int len)
{
int chunk_len;

Assert(len >= 0);

if (len <= 0)
return true;

chunk_len = len - (len % sizeof(Vector8));

if (chunk_len > 0 && !is_valid_ascii(s, chunk_len))
return false;

for (int i = chunk_len; i < len; i++)
{
if (s[i] == '\0' || IS_HIGHBIT_SET(s[i]))
return false;
}

return true;
}

As you can see, it is basically the same as yours except this function
operates on a string of unsigned chars versus you're operating on a text
object. Not sure one is specifically better than the other, except
is_all_valid_ascii() might be more reusable than text_is_ascii(). That
can always be changed later though.

I think some of the comment for text_is_ascii() is a little verbose. For
instance, the paragraph about unicode normalization is not really
relevant to the text_is_ascii() function. It would probably fit better
where we actually do unicode normalization.

I don't think text_is_ascii() is a great function name since we
explicitly reject NUL characters. text_is_valid_ascii() is much more
indicative of what you are actually checking, and it reuses the same
wording as the is_valid_ascii() function. I would also maybe converge on
either "pure ASCII" or "valid ASCII". Not really sure if they are the
same, but I see both mentioned in the codebase.

Here are my benchmark findings, which concur with your analysis:

The benchmarks below use the ClickBench hits table (100M rows), sampled
into four tables. Times are medians (ms) of 5 timed runs after 2 warm-up
runs on a release build (-Dbuildtype=release -Dcassert=false), with
max_parallel_workers_per_gather = 0 and all data cached in memory. The
baseline is unpatched master as of a12600b762c.

Three tables come from the URL column:

- bench_ascii: 20M ASCII-only URLs.
- bench_nonascii: 14,963,181 URLs containing at least one non-ASCII byte.
These are still mostly ASCII; a URL with one Cyrillic path segment
lands here.
- bench_mixed: 20M URLs, unfiltered, so ~15% contain at least one
non-ASCII byte, which is the natural ratio of the dataset.

The URL column has no rows that are predominantly non-ASCII rather than
merely containing some, so the fourth table comes from the Title column:

- bench_cyrillic: 5M Russian titles. This is just about the worst case
for this patch. This patch optimistically looks for ASCII characters.
Russian lacks ASCII characters, except for spaces and punctuation, as
far as I know.

unicode_is_normalized() - SELECT count(*) FROM <table> WHERE <text> IS NORMALIZED;
table baseline (ms) patched (ms) speedup
bench_ascii 19329.5 1812.2 10.67x
bench_nonascii 45370.8 42319.4 1.07x
bench_mixed 26918.0 10218.9 2.63x
bench_cyrillic 4797.6 4978.1 0.96x
bench_cyrillic* 4827.9 4861.2 0.99x

unicode_assigned() - SELECT count(*) FROM <table> WHERE unicode_assigned(<text>);
table baseline (ms) patched (ms) speedup
bench_ascii 8889.9 1586.1 5.60x
bench_nonascii 24269.1 21474.7 1.13x
bench_mixed 13021.6 5620.4 2.32x
bench_cyrillic 6015.9 6210.7 0.97x
bench_cyrillic* 6071.0 6134.4 0.99x

unicode_normalize_func() - SELECT count(normalize(<text>)) FROM <table>;
table baseline (ms) patched (ms) speedup
bench_ascii 59292.9 1714.4 34.59x
bench_nonascii 147948.4 145031.3 1.02x
bench_mixed 86104.6 31805.2 2.71x
bench_cyrillic 14175.4 14553.5 0.97x
bench_cyrillic* 14467.5 14326.5 1.01x

bench_cyrillic* is basically the same as bench_cyrillic, except that
master and the patched tree were compiled with -falign-functions=64
-falign-loops=64. I found that with the changes the hot loops could
straddle a 64-bit instruction fetch boundary.

In essence, I completely agree with your findings. Note that I did not
bump shared_buffers like you did, which was likely a very smart call.

--
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)

#8Andrew Dunstan
andrew@dunslane.net
In reply to: Tristan Partin (#7)
Re: Add ASCII fast path to Unicode normalization functions

On 2026-09-17 Th 6:46 PM, Tristan Partin wrote:

On Mon Sep 14, 2026 at 9:55 AM CDT, Andrew Dunstan wrote:

Hi,

A linkedin post comparing CedarDB's new Unicode normalization support
to PostgreSQL's caught my eye [1]: same results, but a claimed 30x
speedup on "SELECT count(*) FROM hits WHERE url IS NORMALIZED" over
ClickBench's hits table. Most of that turned out to be down to CedarDB
using all available threads by default versus our
max_parallel_workers_per_gather of 2. But even at the matched thread
count they reported a 6x edge, attributed to two things: an ASCII fast
path (most URLs are already normalized ASCII, so you can skip decoding
entirely), and vectorized byte scanning for the ASCII check itself.

I went and looked, and unicode_is_normalized(), unicode_assigned(), and
normalize() all decode every string to an array of char32_t codepoints,
one utf8_to_unicode()/pg_utf_mblen() call at a time, before doing any
real work -- including on input that's already pure ASCII. The attached
patch adds a fast path: scan the raw bytes for anything with the high
bit set, using the SIMD-vectorized is_valid_ascii() we already have
(currently only used inside pg_utf8_verifystr()). If nothing is found,
the string is trivially normalized (ASCII code points have no
canonical or compatibility decomposition, and a combining class of
zero) and every code point in it is assigned, so all three functions
can return immediately.

I deliberately didn't copy CedarDB's trick of comparing byte length to
codepoint count -- getting the codepoint count means calling
pg_mbstrlen_with_len(), exactly the scalar work this patch avoids.
Scanning raw bytes with is_valid_ascii() instead reuses SIMD
infrastructure we already have, and is cheaper to begin with: a single
reduction versus a population count.

Benchmarked with data sized to fit comfortably under shared_buffers rather
than triggering the seqscan ring-buffer bypass, which otherwise swamps the
comparison at larger table sizes: ~10x on pure ASCII, ~4x on an 85/15
ASCII/non-ASCII mix, and no measurable regression on non-ASCII input
that still needs the full decode-and-quickcheck path.

Regression tests cover the ASCII-hit case for all three functions, plus
a boundary sweep that plants a non-NFC sequence at varying offsets
around ASCII padding, to catch any off-by-one in the SIMD-chunk/scalar-
remainder split.

[1] https://lnkd.in/p/eKUqSj73

Hey Andrew,

I saw the same LinkedIn post too, and I also started working on
a similar patch that I was benchmarking last week. So I'll provide some
review and some results that I saw.

FWIW, here is the function that I added to ascii.h:

/*
* Wrapper around is_valid_ascii() such that a string of any length can be
* passed in. If you know your string to have a length of a multiple of
* sizeof(Vector8), stick with is_valid_ascii(). It will avoid a few
* instructions.
*/
static inline bool
is_all_valid_ascii(const unsigned char *s, int len)
{
int chunk_len;

Assert(len >= 0);

if (len <= 0)
return true;

chunk_len = len - (len % sizeof(Vector8));

if (chunk_len > 0 && !is_valid_ascii(s, chunk_len))
return false;

for (int i = chunk_len; i < len; i++)
{
if (s[i] == '\0' || IS_HIGHBIT_SET(s[i]))
return false;
}

return true;
}

As you can see, it is basically the same as yours except this function
operates on a string of unsigned chars versus you're operating on a text
object. Not sure one is specifically better than the other, except
is_all_valid_ascii() might be more reusable than text_is_ascii(). That
can always be changed later though.

I don't have a strong opinion about it. It would just mean that you
would migrate the VARDATA_ANY etc. stuff to the call sites.

I think some of the comment for text_is_ascii() is a little verbose. For
instance, the paragraph about unicode normalization is not really
relevant to the text_is_ascii() function. It would probably fit better
where we actually do unicode normalization.

Fair point

I don't think text_is_ascii() is a great function name since we
explicitly reject NUL characters. text_is_valid_ascii() is much more
indicative of what you are actually checking, and it reuses the same
wording as the is_valid_ascii() function. I would also maybe converge on
either "pure ASCII" or "valid ASCII". Not really sure if they are the
same, but I see both mentioned in the codebase.

I'm not fixed on the name, but we might want something different if we
go with Bilal's useful optimization from upthread.

Here are my benchmark findings, which concur with your analysis:

The benchmarks below use the ClickBench hits table (100M rows), sampled
into four tables. Times are medians (ms) of 5 timed runs after 2 warm-up
runs on a release build (-Dbuildtype=release -Dcassert=false), with
max_parallel_workers_per_gather = 0 and all data cached in memory. The
baseline is unpatched master as of a12600b762c.

Three tables come from the URL column:

- bench_ascii: 20M ASCII-only URLs.
- bench_nonascii: 14,963,181 URLs containing at least one non-ASCII byte.
These are still mostly ASCII; a URL with one Cyrillic path segment
lands here.
- bench_mixed: 20M URLs, unfiltered, so ~15% contain at least one
non-ASCII byte, which is the natural ratio of the dataset.

The URL column has no rows that are predominantly non-ASCII rather than
merely containing some, so the fourth table comes from the Title column:

- bench_cyrillic: 5M Russian titles. This is just about the worst case
for this patch. This patch optimistically looks for ASCII characters.
Russian lacks ASCII characters, except for spaces and punctuation, as
far as I know.

unicode_is_normalized() - SELECT count(*) FROM <table> WHERE <text> IS NORMALIZED;
table baseline (ms) patched (ms) speedup
bench_ascii 19329.5 1812.2 10.67x
bench_nonascii 45370.8 42319.4 1.07x
bench_mixed 26918.0 10218.9 2.63x
bench_cyrillic 4797.6 4978.1 0.96x
bench_cyrillic* 4827.9 4861.2 0.99x

unicode_assigned() - SELECT count(*) FROM <table> WHERE unicode_assigned(<text>);
table baseline (ms) patched (ms) speedup
bench_ascii 8889.9 1586.1 5.60x
bench_nonascii 24269.1 21474.7 1.13x
bench_mixed 13021.6 5620.4 2.32x
bench_cyrillic 6015.9 6210.7 0.97x
bench_cyrillic* 6071.0 6134.4 0.99x

unicode_normalize_func() - SELECT count(normalize(<text>)) FROM <table>;
table baseline (ms) patched (ms) speedup
bench_ascii 59292.9 1714.4 34.59x
bench_nonascii 147948.4 145031.3 1.02x
bench_mixed 86104.6 31805.2 2.71x
bench_cyrillic 14175.4 14553.5 0.97x
bench_cyrillic* 14467.5 14326.5 1.01x

bench_cyrillic* is basically the same as bench_cyrillic, except that
master and the patched tree were compiled with -falign-functions=64
-falign-loops=64. I found that with the changes the hot loops could
straddle a 64-bit instruction fetch boundary.

In essence, I completely agree with your findings. Note that I did not
bump shared_buffers like you did, which was likely a very smart call.

When time permits I'll send a new version, incorporating your review and
Bilal's suggestion.

Thanks for the review!

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

#9Tristan Partin
tristan@neon.tech
In reply to: Nazir Bilal Yavuz (#5)
Re: Add ASCII fast path to Unicode normalization functions

On Tue Sep 15, 2026 at 8:01 AM CDT, Nazir Bilal Yavuz wrote:

Hi,

Thank you for working on this!

On Tue, 15 Sept 2026 at 14:31, Andrew Dunstan <andrew@dunslane.net> wrote:

On 2026-09-15 Tu 12:22 AM, Chao Li wrote:

The attached is my test script.

Great, thanks for the review and tests.

What do you think about continuing from where text_ascii_check() is
left? I wrote a patch for this and benchmarked with Chao's script.

Timings are v1 vs v2, not master vs v2.

# unicode_is_normalized()

* all ascii: 89.115ms | 87.803ms
* mixed: 249.239ms | 126.011ms -> improvement
* non-ascii: 1243.496ms | 1244.052ms
* late-non-ascii: 3632.410ms | 370.966ms -> improvement

# unicode_normalize_func()

* all ascii: 84.458ms | 84.636ms
* mixed: 578.684ms | 226.093ms -> improvement
* non-ascii: 3651.573ms | 3644.026ms
* late-non-ascii: 10513.861ms | 942.470ms -> improvement

# unicode_assigned()

* all ascii: 61.147ms | 60.658ms
* mixed: 124.193ms | 86.186ms -> improvement
* non-ascii: 507.130ms | 510.247ms
* late-non-ascii: 1169.617ms | 166.924ms -> improvement

Do you think these results worth the additional complexity?

This is a nice change. Great idea.

--
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)

#10Andrew Dunstan
andrew@dunslane.net
In reply to: Andrew Dunstan (#8)
Re: Add ASCII fast path to Unicode normalization functions

On 2026-09-18 Fr 10:11 AM, Andrew Dunstan wrote:

When time permits I'll send a new version, incorporating your review
and Bilal's suggestion.

Here's v3, rebased on current master. It folds in Bilal's change to pick
up where the ASCII scan stopped rather than decoding the whole string, and
deals with Tristan's review comments:

- The helper is now valid_ascii_prefix_len(), which works on raw bytes
  rather than a text value, and returns the length of the valid ASCII
  prefix it found, or the full length if the whole string is valid ASCII.
  That gets rid of the -1 return in v2, and settles on "valid ASCII"
  throughout, as Tristan suggested.

- The paragraph about Unicode normalization has moved out of the helper's
  comment and into the callers, where it actually matters.

- As in Bilal's patch, normalize() and IS NORMALIZED restart with the last
  ASCII character before the non-ASCII part, since a following combining
  mark can compose with it. ASCII characters have no decomposition and
  are all starters, so nothing earlier can be affected, and the quick
  check gives the same answer starting there as from the beginning of the
  string.

- There's a new regression test sweeping ASCII prefixes of 0 to 40 bytes
  for normalize() (NFC and NFD) and unicode_assigned(), including a
  combining mark right after the ASCII. It only lists lengths that give
  the wrong answer, so the expected output is short. I checked that it
  does fail if the last ASCII character isn't included in the part that
  gets normalized.

I'll add it to the next CF.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Attachments:

v3-0001-Add-ASCII-fast-path-to-Unicode-normalization-func.patchtext/x-patch; charset=UTF-8; name=v3-0001-Add-ASCII-fast-path-to-Unicode-normalization-func.patchDownload+206-11
#11Ayush Tiwari
ayushtiwari.slg01@gmail.com
In reply to: Andrew Dunstan (#10)
Re: Add ASCII fast path to Unicode normalization functions

Hi,

On Sat, 3 Oct 2026 at 19:57, Andrew Dunstan <andrew@dunslane.net> wrote:

On 2026-09-18 Fr 10:11 AM, Andrew Dunstan wrote:

When time permits I'll send a new version, incorporating your review
and Bilal's suggestion.

Here's v3, rebased on current master. It folds in Bilal's change to pick
up where the ASCII scan stopped rather than decoding the whole string, and
deals with Tristan's review comments:

- The helper is now valid_ascii_prefix_len(), which works on raw bytes
rather than a text value, and returns the length of the valid ASCII
prefix it found, or the full length if the whole string is valid ASCII.
That gets rid of the -1 return in v2, and settles on "valid ASCII"
throughout, as Tristan suggested.

- The paragraph about Unicode normalization has moved out of the helper's
comment and into the callers, where it actually matters.

- As in Bilal's patch, normalize() and IS NORMALIZED restart with the last
ASCII character before the non-ASCII part, since a following combining
mark can compose with it. ASCII characters have no decomposition and
are all starters, so nothing earlier can be affected, and the quick
check gives the same answer starting there as from the beginning of the
string.

- There's a new regression test sweeping ASCII prefixes of 0 to 40 bytes
for normalize() (NFC and NFD) and unicode_assigned(), including a
combining mark right after the ASCII. It only lists lengths that give
the wrong answer, so the expected output is short. I checked that it
does fail if the last ASCII character isn't included in the part that
gets normalized.

I'll add it to the next CF.

I see CI tests failing for this patch. I haven't gone through the patch
yet, but from the logs it looks related to missing trailing spaces in
the newly added expected-output headings. The pg_upgrade failure seems
to be in its regression setup, not the upgrade itself.

Regards,
Ayush

#12Andrew Dunstan
andrew@dunslane.net
In reply to: Ayush Tiwari (#11)
Re: Add ASCII fast path to Unicode normalization functions

On 2026-10-03 Sa 3:41 PM, Ayush Tiwari wrote:

Hi,

On Sat, 3 Oct 2026 at 19:57, Andrew Dunstan <andrew@dunslane.net> wrote:

On 2026-09-18 Fr 10:11 AM, Andrew Dunstan wrote:

When time permits I'll send a new version, incorporating your review
and Bilal's suggestion.

Here's v3, rebased on current master. It folds in Bilal's change to pick
up where the ASCII scan stopped rather than decoding the whole string, and
deals with Tristan's review comments:

- The helper is now valid_ascii_prefix_len(), which works on raw bytes
rather than a text value, and returns the length of the valid ASCII
prefix it found, or the full length if the whole string is valid ASCII.
That gets rid of the -1 return in v2, and settles on "valid ASCII"
throughout, as Tristan suggested.

- The paragraph about Unicode normalization has moved out of the helper's
comment and into the callers, where it actually matters.

- As in Bilal's patch, normalize() and IS NORMALIZED restart with the last
ASCII character before the non-ASCII part, since a following combining
mark can compose with it. ASCII characters have no decomposition and
are all starters, so nothing earlier can be affected, and the quick
check gives the same answer starting there as from the beginning of the
string.

- There's a new regression test sweeping ASCII prefixes of 0 to 40 bytes
for normalize() (NFC and NFD) and unicode_assigned(), including a
combining mark right after the ASCII. It only lists lengths that give
the wrong answer, so the expected output is short. I checked that it
does fail if the last ASCII character isn't included in the part that
gets normalized.

I'll add it to the next CF.

I see CI tests failing for this patch. I haven't gone through the patch
yet, but from the logs it looks related to missing trailing spaces in
the newly added expected-output headings. The pg_upgrade failure seems
to be in its regression setup, not the upgrade itself.

Sorry about that. Here's a v4 that should have the whitespace issue fixed.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Attachments:

t253785_12
v4-0001-Add-ASCII-fast-path-to-Unicode-normalization-func.patchtext/x-patch; charset=UTF-8; name=v4-0001-Add-ASCII-fast-path-to-Unicode-normalization-func.patchDownload+206-11
#13Rui Zhao
zhaorui126@gmail.com
In reply to: Andrew Dunstan (#12)
Re: Add ASCII fast path to Unicode normalization functions

Hi Andrew,

Thanks for working on this. I tried two further optimizations on top
of v4. They can be applied independently:

1. unicode_assigned() still calls pg_mbstrlen_with_len() on the
remaining input, then walks it again to check each code point. 0001
uses the byte length to walk to the end directly. This also avoids
scanning the whole suffix before returning false for an unassigned
code point near the beginning.

2. unicode_normalize_func() and unicode_is_normalized() allocate a C
string for the normalization form on every call, even when the input
is all ASCII. 0002 compares the form directly in the text value,
preserving case-insensitive matching and rejecting extra bytes.
It only builds a C string when reporting an invalid form.

0003 replaces the constant-folded VALUES test with stored short and
compressed text values.

Core regression and the Unicode normalization checks passed with these
patches.

Regards,
Rui

Attachments:

nocfbot-0001-Avoid-counting-code-points-before-checking-Unicode-a.patchapplication/octet-stream; name=nocfbot-0001-Avoid-counting-code-points-before-checking-Unicode-a.patchDownload+4-5
nocfbot-0002-Avoid-allocating-a-C-string-for-the-normalization-fo.patchapplication/octet-stream; name=nocfbot-0002-Avoid-allocating-a-C-string-for-the-normalization-fo.patchDownload+25-11
nocfbot-0003-Exercise-ASCII-normalization-with-stored-text-values.patchapplication/octet-stream; name=nocfbot-0003-Exercise-ASCII-normalization-with-stored-text-values.patchDownload+14-5
#14Andrew Dunstan
andrew@dunslane.net
In reply to: Rui Zhao (#13)
Re: Add ASCII fast path to Unicode normalization functions

On 2026-10-04 Su 12:42 PM, Rui Zhao wrote:

Hi Andrew,

Thanks for working on this. I tried two further optimizations on top
of v4. They can be applied independently:

1. unicode_assigned() still calls pg_mbstrlen_with_len() on the
remaining input, then walks it again to check each code point. 0001
uses the byte length to walk to the end directly. This also avoids
scanning the whole suffix before returning false for an unassigned
code point near the beginning.

2. unicode_normalize_func() and unicode_is_normalized() allocate a C
string for the normalization form on every call, even when the input
is all ASCII. 0002 compares the form directly in the text value,
preserving case-insensitive matching and rejecting extra bytes.
It only builds a C string when reporting an invalid form.

0003 replaces the constant-folded VALUES test with stored short and
compressed text values.

Core regression and the Unicode normalization checks passed with these
patches.

OK, that all looks good. The attached combines all this.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Attachments:

t253785_14
v5-0001-Add-ASCII-fast-path-to-Unicode-normalization-func.patchtext/x-patch; charset=UTF-8; name=v5-0001-Add-ASCII-fast-path-to-Unicode-normalization-func.patchDownload+252-24
#15Tristan Partin
tristan@neon.tech
In reply to: Andrew Dunstan (#14)
Re: Add ASCII fast path to Unicode normalization functions

On Tue Oct 6, 2026 at 9:22 PM UTC, Andrew Dunstan wrote:

On 2026-10-04 Su 12:42 PM, Rui Zhao wrote:

Hi Andrew,

Thanks for working on this. I tried two further optimizations on top
of v4. They can be applied independently:

1. unicode_assigned() still calls pg_mbstrlen_with_len() on the
remaining input, then walks it again to check each code point. 0001
uses the byte length to walk to the end directly. This also avoids
scanning the whole suffix before returning false for an unassigned
code point near the beginning.

2. unicode_normalize_func() and unicode_is_normalized() allocate a C
string for the normalization form on every call, even when the input
is all ASCII. 0002 compares the form directly in the text value,
preserving case-insensitive matching and rejecting extra bytes.
It only builds a C string when reporting an invalid form.

0003 replaces the constant-folded VALUES test with stored short and
compressed text values.

Core regression and the Unicode normalization checks passed with these
patches.

OK, that all looks good. The attached combines all this.

Thanks for the updates Andrew. They looks pretty good to me. A few
nitpicks:

+/*
+ * Return the length of an initial portion of s[0..len) that is valid ASCII,
+ * that is, contains no zero bytes and no bytes with the high bit set, or
+ * len if all of it is.
+ *
+ * is_valid_ascii() requires a length that is a multiple of sizeof(Vector8),
+ * so check one chunk at a time with it and then the remainder byte by byte.
+ * A failure inside a chunk is reported at the start of that chunk, so the
+ * result can be up to sizeof(Vector8) - 1 bytes short of the full valid
+ * ASCII prefix.
+ */

I would relegate that second paragraph to a comment within the function
itself. I am not sure it is valuable to callers, only to those reading
the actual implementation. It also may be a bit verbose, but that isjust
an opinion.

+   /*
+    * Check the remaining code points without first counting them, stopping
+    * at a NUL as pg_mbstrlen_with_len() would.
+    */
+   p = (unsigned char *) VARDATA_ANY(input) + start;
+   end = (unsigned char *) VARDATA_ANY(input) + len;

I am not sure that it makes sense to mention pg_mbstrlen_with_len()
here. I would be more explicit as to why we are stopping at a NUL.

--
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)

#16Andrew Dunstan
andrew@dunslane.net
In reply to: Tristan Partin (#15)
Re: Add ASCII fast path to Unicode normalization functions

On 2026-10-06 Tu 5:56 PM, Tristan Partin wrote:

On Tue Oct 6, 2026 at 9:22 PM UTC, Andrew Dunstan wrote:

On 2026-10-04 Su 12:42 PM, Rui Zhao wrote:

Hi Andrew,

Thanks for working on this. I tried two further optimizations on top
of v4. They can be applied independently:

1. unicode_assigned() still calls pg_mbstrlen_with_len() on the
remaining input, then walks it again to check each code point. 0001
uses the byte length to walk to the end directly. This also avoids
scanning the whole suffix before returning false for an unassigned
code point near the beginning.

2. unicode_normalize_func() and unicode_is_normalized() allocate a C
string for the normalization form on every call, even when the input
is all ASCII. 0002 compares the form directly in the text value,
preserving case-insensitive matching and rejecting extra bytes.
It only builds a C string when reporting an invalid form.

0003 replaces the constant-folded VALUES test with stored short and
compressed text values.

Core regression and the Unicode normalization checks passed with these
patches.

OK, that all looks good. The attached combines all this.

Thanks for the updates Andrew. They looks pretty good to me. A few
nitpicks:

+/*
+ * Return the length of an initial portion of s[0..len) that is valid ASCII,
+ * that is, contains no zero bytes and no bytes with the high bit set, or
+ * len if all of it is.
+ *
+ * is_valid_ascii() requires a length that is a multiple of sizeof(Vector8),
+ * so check one chunk at a time with it and then the remainder byte by byte.
+ * A failure inside a chunk is reported at the start of that chunk, so the
+ * result can be up to sizeof(Vector8) - 1 bytes short of the full valid
+ * ASCII prefix.
+ */

I would relegate that second paragraph to a comment within the function
itself. I am not sure it is valuable to callers, only to those reading
the actual implementation. It also may be a bit verbose, but that isjust
an opinion.

+   /*
+    * Check the remaining code points without first counting them, stopping
+    * at a NUL as pg_mbstrlen_with_len() would.
+    */
+   p = (unsigned char *) VARDATA_ANY(input) + start;
+   end = (unsigned char *) VARDATA_ANY(input) + len;

I am not sure that it makes sense to mention pg_mbstrlen_with_len()
here. I would be more explicit as to why we are stopping at a NUL.

Thanks. I'm intending to commit this during the November CF, will fix then.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com