model-writtenrun 8853 - may be wrong, links go to the source messages. Commit-by-commit LLM analysis is in the Commits dialog.
- John Naylor began by removing scanner backup-prevention rules, finding unchanged information-schema parsing performance and a smaller transition table that allowed 16-bit entries #1.
- Tom Lane wanted more headroom below the 16-bit limit, while Andres Freund suggested repetitive workloads; John's resulting pgbench-style benchmark showed a roughly 3% slowdown with backtracking, steering the work toward preserving backup-free scanning #2 #3 #4.
- John introduced one shared state for detecting string continuation, then corrected its fallback rules to eliminate backing-up states and preserve benchmark performance #5 #6.
- A preliminary whitespace simplification was dropped after Tom warned about per-character scanner overhead, and the saved scanner state was moved from a static variable into yyextra to preserve reentrancy #7 #8 #9 #10.
- John extended state sharing to Unicode strings and identifiers and separated UESCAPE from its character literal; Tom then consolidated the series, tightened failure patterns, and verified regression tests with old and new Flex versions #10 #11.
- John propagated the changes to psql and ECPG and added a separate consolidation of ECPG's C-comment states, reporting that check-world passed #12.
- Tom found that malformed escape-character literals could leave psql believing it was inside a string, prompting John to move Unicode de-escaping into grammar productions and let the scanners return separate tokens #13 #14.
- Chapman Flack raised the prospect of supporting Unicode escapes beyond UTF8 databases, while Tom questioned how moving processing into the grammar alone would provide the necessary context #15 #16.
- Tom's grammar review led John to inline the Uescape production, revise precedence, share helper functions, and add surrogate-pair tests, but benchmarks still showed regressions #17 #18.
- Thomas Munro reported a Windows test failure, and Alvaro Herrera identified the tests' dependence on UTF8 server encoding and proposed alternate expectations or removal; Tom deferred that decision while the implementation remained unfinished #19 #20 #21.
- After several months, Tom returned with a rewrite based on the earlier grammar draft: parser.c's existing lookahead filter now converted Unicode token sequences, avoiding invasive grammar changes and restoring ordinary-query performance while accepting slower explicit UESCAPE processing #22.
- John supplied a small error-position fix on top of Tom's rewrite, and Tom recommended that ECPG preserve Unicode spelling rather than assume the eventual runtime encoding #23 #24.
- John completed psql and ECPG integration, unified conversion branches, simplified ECPG string-token reconstruction, and introduced Flex state-stack functions; subsequent benchmarks were encouraging except for explicit UESCAPE clauses #25 #26.
- Tom objected to state-stack overhead in the backend and requested a clearer USCONST token name, so John restored explicit saved states across all three scanners, updated tests, and separated the ECPG comment cleanup again #27 #28.
- Tom committed both final patches together as 7f380c59f8, with light cleanup, reducing the backend transition array from 37,045 to 20,495 entries and saving over 200kB #28 #29.
- Tom left an experimental in-place de-escaping optimization uncommitted because it offered little benefit and could complicate future encoding support; the thread ended with the submitted work landed, John thanking him, and non-UTF8 Unicode handling identified as follow-up work #29 #30.