model-writtenrun 8966 - may be wrong, links go to the source messages. Commit-by-commit LLM analysis is in the Commits dialog.
May - Dec 2019 · 16 entries
- Andrey Borodin opened the thread with work undertaken with Vladimir to accelerate existing pglz compression and decompression, and Michael Paquier encouraged treating the two independently while keeping complexity proportional to the gains #1 #2 #5.
- Andrey benchmarked WAL, database pages and Shakespeare text across Intel, ARM and Power9 systems, then posted two decompression alternatives using non-overlapping memcpy calls, one retaining byte copying for short matches; Gasper Zejn independently confirmed gains on AMD Ryzen #4 #6 #7.
- Andrey subsequently posted Vladimir's separate compressor rewrite using functions, compact history entries and faster comparisons, reporting roughly a 1.43-fold speedup #10.
- Konstantin Knizhnik validated the memcpy approach and reproduced its gains, but his substantially faster LZ4 results redirected discussion toward adding another compression algorithm #11.
- Tomas Vondra initially proposed choosing compression at initdb, while Andres Freund explained that different TOAST values could use different algorithms and new writes could migrate naturally; Andres also opposed making user-trained dictionaries a prerequisite for obtaining a better default compressor #12 #13 #14 #16 #17.
- Petr Jelinek supplied an independent LZ4 patch with a compression_algorithm GUC, system-library linkage, a format discriminator and algorithm byte, mixed-algorithm TOAST support, WAL compression integration and TAP tests #21.
- Tomas found that the LZ4 patch failed to build without LZ4 and broke compressed-WAL replay, tracing the latter to RestoreBlockImage still calling pglz_decompress directly; Petr identified the misplaced header-size definition responsible for the build failure #23 #25 #26.
- LZ4 review expanded into session-level configuration, separate WAL settings and metadata placement: Andres wanted compression identifiers available in TOAST references and WAL records, while Petr defended the embedded byte and later raised big-endian compatibility concerns; these issues produced no revised attachment in this thread #27 #28 #32 #40.
- Returning to decompression, Peter Eisentraut tested the unrestricted memcpy version and requested explanations of offset doubling and output-to-output copying; Andrey added an illustrative example and clearer comments #41 #42.
- Oleg Bartunov suggested the Silesia corpus, whose shorter matches reversed Andrey's earlier benchmark preference, prompting a revision that retained byte copying for matches of eight bytes or fewer #39 #43 #44.
- Because the benchmark conclusions conflicted, Tomas requested current copies of both alternatives and Andrey supplied them together for independent testing #47 #48.
- Tels proposed batching literal bytes as another optimization, but Andrey left that out because earlier experiments showed noisy, dataset-dependent gains #49 #50.
- After a wait for independent verification, Tomas benchmarked real SQL substring operations on two machines and found both alternatives beneficial, generally favoring unrestricted memcpy; Peter preferred that simpler implementation, and Andrey accepted the result #52 #56 #57 #61.
- Tomas and Michael requested comment corrections and a clearer explanation of why memmove cannot implement pglz's overlapping back-references, which Andrey supplied in the final patch #58 #59 #63.
- Tomas committed the final decompression patch as c60e520f6e, using Álvaro Herrera's pgindent-resistant comment formatting and removing an unrelated whitespace change while preserving the executable logic #64 #66.
- The thread ends with decompression committed, no landing shown for Vladimir's compressor rewrite or Petr's LZ4 proposal, and Andrey asking whether LZ4 work should target PostgreSQL 13 or 14 and a future CommitFest #10 #21 #66 #69.