model-writtenrun 8712 - may be wrong, links go to the source messages. Commit-by-commit LLM analysis is in the Commits dialog.
Jul - Aug 2019 · 4 entries
- Jeff Davis proposed keeping existing aggregate groups in memory while partitioning raw tuples for new groups onto disk, and Tomas Vondra supported an incremental implementation with real memory accounting despite unresolved growing transition states such as ARRAY_AGG #1 #2.
- The initial implementation supported grouping sets and JIT, and the next revision added saved hash values, correctness fixes, and regression tests while retaining the same partitioned-spill algorithm #3 #7.
- Adam Lee suggested spilling partial aggregate states to handle skew, but Jeff and Tomas deferred that more complex strategy until the simpler implementation provided a baseline #9 #10 #11.
- Taylor Vesely and Melanie Plageman began reviewing the implementation, and Taylor rebased it after the simplehash prerequisite landed separately while raising questions about skew and the fixed partition count #12.
Nov 2019 - Mar 2020 · 27 entries
- After the autumn gap, Jeff added runtime partition sizing; Tomas's detailed review and Melanie's partition-accounting analysis then led to clearer limits and a substantial execution rework using spilling-specific expressions and direct MinimalTuple replay #14 #15 #16 #17.
- Tomas's question about a removed comment exposed stale execGrouping.c documentation, which Jeff separated from the feature and committed with a nearby typo fix and additional comment cleanup #21 30d47723fd.
- Adam and Melanie supplied and tested a delta that omitted unnecessary spilled columns; Jeff integrated it using a separate virtual slot and attribute arrays, also preserving transitionSpace for hash-table sizing #24 #29.
- Tomas's benchmarks showed that uncosted spilling could replace a faster parallel sorted plan with a slower hash aggregate, prompting Jeff to add disk costs and consolidate hash-entry size estimates #27 #31.
- Heikki Linnakangas found that the costing revision lost its GUC definitions and that projection discovery missed aggregates nested inside expressions; he recommended discovering aggregated and unaggregated columns together #33.
- Adam prototyped logical-tape spilling with shared disk-space reuse, and Jeff adopted it in a substantial rewrite that added lazy read buffers and extensible tape sets, improved rescans, and temporarily removed projection #32 #34.
- Logical tapes exposed severe freelist-sorting overhead; Peter Geoghegan suggested a sorting pathology, and Jeff's investigation led to replacing the repeatedly sorted freelist with a min heap #35 #36 #37.
- Peter's review identified integer-overflow and freelist-allocation risks, which Jeff addressed while splitting the tape changes into three patches; the min-heap member then landed unchanged #41 #44 c02fdc9223.
- An experiment with separately allocated tape arrays showed a reported sorting slowdown, so Jeff retained the narrower tape-extension API and deferred a broader logical-tape redesign #46 #47.
- Jeff extracted three aggregation prerequisites, and the reusable tuple-hash API landed first with a small internal rearrangement of hash/match-state initialization #48 4eaea3db15.
- The combined patch improved bucket selection and spill costing, and Jeff's regression matrix found no clear regression in the tested in-memory, sorting, or JIT cases #51 #52.
- Lazy logical-tape read buffers were extracted from the combined patch and committed with an additional LogicalTapeTell initialization case #52 7fdd919ae7.
- Jeff isolated the nodeAgg.c construction and lookup refactoring, while Melanie separately revived column discovery with a joint walker; the latter exposed compiler warnings and uncertainty about which attachments formed the active series #53 #54 #56.
- Jeff resent the consolidated remainder, and the standalone nodeAgg.c refactoring landed with comment-only corrections; Melanie's projection prerequisite remained outside the main implementation #57 5b618e1f48.
- Tomas's adversarial tests found zero-bucket and grouping-set crashes, and Adam helped investigate; Jeff imposed a bucket floor, cleared invalidated pointers, repaired multiple-table spilling, and strengthened its regression coverage #59 #61 #62 #67.
- To exercise multiple spilling grouping sets without broadly changing existing mixed-aggregate planning, Jeff introduced enable_groupingsets_hash_disk and removed the earlier spill and overflow controls; Tomas confirmed that the crashes were fixed #67 #68.
- Andres Freund proposed an existing expression-evaluation optimization from thread #41402, then landed it so this feature could use simpler aggregate transition machinery; that commit originated in the other thread #63 #69 2742c45080.
- Jeff adapted the main patch to the evaluator changes, cached expression variants, reused hash tables, and added planned partitions and tape buffers to instrumentation #70.
- The transitionSpace prerequisite, originally isolated earlier and still present in the updated main patch, landed separately with Size fields replacing int32 fields #48 #70 c11cb17dc5.
- The optional per-group NULL-check expression step was likewise extracted from the updated main patch and committed with generalized comments and a standalone caller adjustment #70 c954d49046.
- The March remainder dropped landed prerequisites and restored enable_hashagg_disk; Justin Pryzby's review then caught a buffer-accounting assignment error and prompted documentation cleanup and a separately measured hash metadata context #71 #72 #73.
- Disk-based hash aggregation landed from the final attachment with local changes that increased the partition cap, delayed recursive spill initialization, and avoided advancing aggregates for spilled tuples #73 1f39bce021.
- Jeff announced the landing while explicitly leaving projection, partial-state spilling for growing aggregates, improved cardinality and entry-size estimates, better I/O overlap, and a cleaner tape API for future work #74.
- Justin subsequently identified another accumulation mistake and integer-width concerns; Jeff fixed totalGroups accumulation and EXPLAIN reporting and also tuned hash-entry and bucket estimates without posting a separate patch #76 64fe602279.
- Pengzhou Tang questioned whether the grouping-set test forced hashing, then withdrew the concern after recognizing the additional grouping-set planner setting #77 #78.
- Richard Guo reported negative spill costs from fractional batch estimates, Tomas proposed rounding the batch count, and Jeff committed that correction with further guards and initialization fixes #79 #80 7351bfeda3.
- The thread ends with the main feature and both follow-up fixes committed, Jeff explaining the final costing safeguards, and the previously identified projection and broader optimization work deferred #81 #74.