model-writtenrun 7075 - may be wrong, links go to the source messages. Commit-by-commit LLM analysis is in the Commits dialog.
Feb - Mar 2024 · 10 entries
- Melanie Plageman began with #1, proposing table-AM ownership of skip-fetch, scan initialization after bitmap creation, and iterator control that would let BitmapHeapScan use the streaming-read API.
- Mark Dilger tested the initial series and assessed adapting a proprietary table AM in #2 and #4, while Andres Freund's review in #5 exposed snapshot, counter, interrupt-check, and iterator-lifetime problems addressed in #6 and #7.
- An off-list discussion with Andres led Melanie to count bitmap pages even when they contain no visible tuples, producing the separate EXPLAIN fix and simpler lossy-page reporting in #8.
- Tomas Vondra's benchmarks in #10 and #17 exposed performance regressions, while discussion in #23, #32, and #41 clarified that concurrency zero disables prefetching and concurrency one can still overlap reading with executor work.
- Heikki Linnakangas discovered that parallel bitmap scans redundantly serialized snapshots, proposed #37, and added assertions after Dilip Kumar and Robert Haas examined the assumptions; the revised #43 landed unchanged as 84c18acaf6.
- Andres diagnosed a race between the shared main and prefetch iterators in #54 and #58, and Tomas's syscall tracing in #61 confirmed that many supposed prefetches happened after the actual reads.
- Heikki committed the EXPLAIN fix from #50 as 0960ae1967, simplifying counter placement and backpatching it; Melanie resent the remaining series in #64.
- Heikki's suggestions in #66 led to the larger redesign in #71, with common serial/parallel iterator interfaces, prefetching moved into heap AM, and a merged tuple-oriented callback path.
- Tomas found uninitialized heap-scan fields in #72, fixed by #73, while #78 separated independently useful refactoring from streaming-dependent work and identified scan-descriptor bloat as an unresolved design problem.
- Further testing exposed an inverted iterator-position check and incorrect stream destruction, fixed in #95; after the Read Stream API itself landed, #96 removed the bundled dependency and reset streams on rescan.
Apr - Jun 2024 · 9 entries
- Tomas concluded in #97 that streaming adoption needed more performance testing than PostgreSQL 17 allowed, so Melanie's #98 concentrated on independently useful refactoring and specialized bitmap scan descriptors, with the large tail split further in #99.
- Tomas's review archive in #100 prompted an explicit need-tuples Boolean interface, and Tom Lane confirmed in #104 that using the current page's recheck flag for future-page prefetching had been an oversight.
- The initialization and can-skip timing changes landed from #105 and #106 as 1577081e96 and fe1431e39c, with only a message edit to the former.
- The skip-fetch relocation from #105, prefetch-recheck correction from #106, and BlockNumber parameter cleanup from #106 landed as 04e72ed617, 1fdb0ce9b1, and 92641d8d65, with explanatory edits to the first two.
- After those landings, #108 retained three additional preparatory patches, but #110 deferred them because intermediate table-AM changes and descriptor growth needed more thought; the main streaming feature remained unfinished.
- Melanie questioned the newly added empty-tuple assertions in #108, and Tomas and Richard Guo supplied failing query shapes in #114 and #116, showing that a scan can legitimately stop before emitting every pending tuple.
- The assertion repair evolved from #120 to #141, using a temporary table for reliable visibility-map coverage, resetting pending tuples on rescan, and removing unnecessary test settings; the final attachment landed unchanged as a3e6c6f929.
- Melanie revived the complete project in #144, refining iterator ownership in #145 and incorporating Tomas's review patches from #146 into #147.
- Discussion of specialized descriptors and searchable member names continued through #150, #153, and #154; Melanie retained the established prefixes and resumed the work after a long pause in #155.
Aug - Oct 2024 · 5 entries
- The prototype in #156 replaced separate top-level descriptors with a scan-state union and an inherited bitmap heap descriptor, while Thomas Munro's portability warning in #157 required named rather than anonymous unions.
- Melanie integrated that lower-impact design in #159 and incorporated Tomas's naming, cast, and assertion feedback in #161.
- The counter relocation and asynchronous callback control flow from #161 landed as 7bd7aa4d30 and de380a62b5, with message improvements and removal of a redundant include.
- The common iterator abstraction and its BitmapHeapScan use from #161 subsequently landed as 7f9d4187e7 and 1a0da347a7, with named-union accesses and comment corrections.
- The inherited bitmap-specific scan descriptor from #161 landed as f7a8fc10cc, preserving intervening upstream field-type changes and correcting the commit explanation.
Dec 2024 - Apr 2025 · 16 entries
- Richard reported a double-cleanup crash and supplied an inline guard in #163, extending it to endscan in #164; Melanie added regression coverage and committed the repair as 754c610e13.
- With those preparations committed, #167 reduced the remaining project to five patches and abandoned intermediate prefetch relocation in favor of deleting bespoke prefetching when read streams were introduced.
- Nazir Bilal Yavuz reviewed the setup-helper extraction, and #168 landed as a5358c14b2 with a small conditional-compilation guard; #170 carried the remaining four patches.
- Tomas's broader storage benchmarks in #172 revealed problematic default-concurrency cases, and Melanie's analysis in #175 showed that the old parallel prefetch code could effectively exceed its configured concurrency.
- Andres identified premature sequential-stream detection in #192, and the diagnostic variant in #198, tested in #203 and #204, showed that disabling this detection removed many regressions.
- Thomas proposed keeping compact references to immutable bitmap storage in #193, and Melanie implemented deferred offset extraction in #206, reducing per-buffer memory and unnecessary work under the shared iterator lock.
- Thomas's review in #212 led to explicit lossiness and successive extraction-API refinements, culminating in the array-plus-capacity interface in #214.
- The lossy indicator and delayed extraction from #214 landed as b8778c4cd8 and bfe56cdf9a, with an initialization moved between components for the latter; #215 left only streaming integration pending.
- Cloud and NVMe measurements in #216 and #217 supported raising effective_io_concurrency to 16, and the component in #218 landed as ff79b5b2ab after Andres's documentation review.
- The temporary RANDOM-flag workaround in #218 was replaced by Thomas's read-stream improvements developed in thread #51076 and forwarded in #223, the latest matching attachment; his advice component landed as 7ea8cd1566 with revised advice gating and sequential-region completion logic.
- Once the significant cyclic-access regression was resolved, the three integration components of #223 landed as 944e81bf99, 2b73a8cd33, and c3953226a0, retaining their posted code and making only explanatory message changes.
- Jakub Wartak questioned maintenance concurrency remaining below effective concurrency, Andres recommended equal defaults in #228, and #229 landed with an expanded rationale as cc6be07ebd.
- Alexander Lakhin reported and bisected intermittent wrong results in #231, and Andres's deterministic reproduction and analysis in #233 and #234 identified stale recheck state while emitting empty tuples.
- Melanie separated that immediate repair from the proposed removal of the skip-fetch optimization in #235, and the reset plus regression test in #236 landed unchanged as aea916fe55.
- Thomas's new experiment in #238 batched parallel bitmap block assignments to preserve I/O combining, but he abandoned its ramp-down companion and deferred the work beyond PostgreSQL 18 because asynchronous streams complicated fair work distribution.
- At the final message, the main streaming feature and its fixes had landed, while parallel batching, contention, and fairness remained open research; Robert Haas proposed hiding work stealing inside per-participant ReadStreams in #248.