[PATCH] Fix SASLprep validation after NFKC normalization
Hi,
I noticed that pg_saslprep() in src/common/saslprep.c performs the prohibited code point and bidirectional checks on input_chars rather than output_chars, even though NFKC normalization has already been performed.
Since NFKC normalization can change both the code point values and the number of code points, this can lead to incorrect results. The existing comments in pg_saslprep() explicitly describe checking the output string after normalization, suggesting that the current behavior is unintentional.
I found two examples that demonstrate the problem:
1. U+0061 followed by U+0340 normalizes to U+00E0, which is valid. The current implementation returns SASLPREP_PROHIBITED instead of SASLPREP_SUCCESS.
2. U+FB1D normalizes to U+05D9 followed by U+05B4. The normalized string violates the bidirectional rules because it does not end with a RandALCat character. The current implementation incorrectly returns SASLPREP_SUCCESS.
While investigating this issue, I also verified all the code point tables used by pg_saslprep() against RFC 3454 and found no discrepancies.
After building and installing the test_saslprep module from src/test/modules/test_saslprep, the issue can be reproduced in a running Postgres instance using:
CREATE EXTENSION IF NOT EXISTS test_saslprep;
WITH cases (name, src, expected_status, expected_res) AS (
VALUES
('step3', decode('61cd80', 'hex'), 'SUCCESS', decode('c3a0', 'hex')),
('step4', decode('efac9d', 'hex'), 'PROHIBITED', NULL::bytea)
)
SELECT name, status, encode(res, 'hex') AS result,
status = expected_status AND res IS NOT DISTINCT FROM expected_res AS passed
FROM cases
CROSS JOIN LATERAL test_saslprep(src);
On the unmodified master branch, both cases return false in the "passed" column. With the patch applied, both return true.
The attached patch changes the prohibited code point and bidirectional checks to operate on the normalized output, rather than the pre-normalization input. It also rejects an empty result after normalization.
The patch includes regression tests for both cases. These tests fail on the unmodified master branch and pass with the patch applied.
Testing was performed with a clean build configured with --enable-tap-tests, --enable-cassert, and --enable-debug. All applicable check-world tests passed with PG_TEST_EXTRA=saslprep, including the additional SASLprep TAP tests and the authentication tests.
The change is intended to correct the existing SASLprep implementation according to RFC 3454 and RFC 4013.
Since Postgres falls back to the original password when SASLprep fails, this change may affect the input to SCRAM key derivation for some passwords. This could cause compatibility issues with existing SCRAM verifiers or between different Postgres versions.
If retaining the current behavior is preferred for compatibility reasons, I think it would be useful to document this explicitly in the code, since the current implementation appears to contradict the processing order specified by RFC 3454.
I would appreciate your review and thoughts on the proposed change, particularly regarding the potential compatibility implications.
Regards,
Robert I Nilsson
Hi,
Attached is v2 of the patch, along with a Python script I wrote to test PostgreSQL's SASLprep implementation against
Python's Unicode 3.2 implementation.
The updated patch addresses two issues:
1. Prohibited output characters and bidirectional properties must be checked after NFKC normalization, using the
normalized string rather than the original input.
2. Code points unassigned in Unicode 3.2 must be rejected before normalization. PostgreSQL's normalizer uses newer
Unicode data, which can normalize some previously unassigned code points into characters that were already assigned
in Unicode 3.2. Checking only after normalization would allow these characters to bypass the unassigned code point
check.
I've also added 13 regression test cases covering both issues. Nine of these fail with the original implementation,
while all 13 pass with the patch.
Additional testing
To verify the changes more thoroughly, I wrote a Python script that uses unicodedata.ucd_3_2_0 and stringprep as a
reference implementation of SASLprep.
The script tests all Unicode scalar values (excluding U+0000 and surrogates), along with selected combinations intended
to exercise normalization and bidirectional rules.
It checks 1,112,210 test cases in total.
The results were:
- Original PostgreSQL implementation: 104 mismatches
- Initial patch: 704 mismatches
- Updated patch (v2): 5 mismatches
The increase with the initial patch revealed the problem with checking unassigned code points only after normalization,
which is now addressed in v2.
The five remaining mismatches are all normalization differences involving the code points U+2F868, U+2F874, U+2F91F,
U+2F95F, and U+2F9BF. These correspond to the five mappings corrected by Unicode Corrigendum #4:
https://www.unicode.org/versions/corrigendum4.html
In all five cases, both PostgreSQL and the reference implementation accept the input, but produce different normalized output.
Test script
I've attached saslprep_diff.py as a standalone testing tool. It requires Python 3 and psycopg, and runs against a
PostgreSQL instance with the test_saslprep extension installed.
I'm not sure how best to integrate this kind of exhaustive testing into PostgreSQL's existing test infrastructure,
especially since the script currently depends on Python and psycopg.
For now, I've included the representative test cases in the regular SQL regression tests and attached the Python script
separately so the more extensive testing can be reproduced.
I'd be happy to adapt the script or its generated test cases if there is a preferred way to include this kind of
coverage in the existing test suite.
I understand that the focus is currently on getting v19 ready, so there is no urgency for a review. I just wanted to make the updated patch and test results available.
Thanks!
Hi again,
Attached is v3 of the patch (now with a new title).
This version addresses the five remaining mismatches reported in my previous email. These were caused by Unicode Corrigendum #4, which changed the normalization mappings of five characters in Unicode 4.0. Since SASLprep (RFC 4013) requires Unicode 3.2, the patch now maps these characters to their original Unicode 3.2 values before normalization, as described in Unicode Standard Annex #15, section 11.3.
I've also made a few additional changes:
- Moved the Unicode 3.2 unassigned code point check into the existing mapping loop, avoiding an additional pass over the input.
- Added regression tests for the five affected characters, bringing the total to 18 test cases.
- Made the regression test ordering independent of the database collation.
I reran the comparison against Python's Unicode 3.2 reference implementation using the same test script as before.
The results are now:
- Original PostgreSQL implementation: 104 mismatches
- Initial patch: 704 mismatches
- v2: 5 mismatches
- v3: 0 mismatches
Checked 1,112,210 cases; mismatches: 0
The updated patch includes the five additional regression tests. The standalone Python test script attached to my previous email remains unchanged.
I would appreciate any review or feedback.
Thanks!
Regards,
Robert I Nilsson