model-writtenrun 8795 - may be wrong, links go to the source messages. Commit-by-commit LLM analysis is in the Commits dialog.
Sep - Oct 2019 · 3 entries
- Amit Khandekar opened the thread with a reproducer that exhausted transient descriptors while decoding hundreds of spilling subtransactions, initially proposing to close files between restore calls and remember their positions #1.
- Alvaro Herrera raised the performance cost of repeated reopening, Robert Haas proposed reusing the existing VFD layer, and Tom Lane argued that logical decoding should track offsets itself without changing that API #2 #5 #10.
- Amit Khandekar implemented VFD-based spill-file reading with local offset tracking, then added a regression test and small cleanups #12 #16.
Nov 2019 - Feb 2020 · 16 entries
- Amit Kapila's review exposed missing cleanup after errors and a separate Windows failure where reading through FileRead at EOF returned an error instead of zero #19 #21.
- Juan José Santamaría Flecha explained the Windows EOF behavior, Thomas Munro posted the corresponding pg_pread fix inline, and Amit Kapila tested it before Thomas committed it as 6969deeb8d #24 #25 #28 #31.
- Further investigation established that an initialization error left iterator state unavailable to PG_CATCH, keeping VFDs open and causing repeated unlink failures on Windows; Amit Khandekar changed initialization to return state through an output parameter so cleanup could run #43 #49 #50.
- Amit Kapila's review prompted safer iterator-state publication and a cheaper test: the expensive contrib regression case moved into a recovery TAP test that restarted the server with a small descriptor limit and set logical_decoding_work_mem low #52 #56 #57 #61.
- Amit Kapila tested the revised case on Windows, polished comments and formatting, and moved offset advancement after error checks, producing the final development-branch submission #62.
- Amit Khandekar supplied a simpler PostgreSQL 11 backport because older VFDs already tracked offsets, then prepared versions through PostgreSQL 9.4; the proposed standalone PostgreSQL 9.6 test was dropped to avoid introducing a new test infrastructure dependency there #66 #68 #69 #70.
- Amit Kapila committed the VFD fix and TAP test as d207038053, matching #62 apart from whitespace and rebase context, and announced the push in #71.
- The new test subsequently crashed on PostgreSQL 10 AIX buildfarm members; Noah Misch traced this to the default 256 MB address-space allowance, while discovering that failed TAP runs could delete useful temporary artifacts #72 #79.
- Tom Lane demonstrated excessive memory consumption on PostgreSQL 10 and explained that forcing max_files_per_process so low was not portable, particularly where named semaphores consume descriptors; he proposed direct debug control of max_safe_fds as a possible replacement testing mechanism #84 #86.
- Amit Kapila posted a test-only revert and committed it as ed7bb5c311, preserving the VFD fix while removing the problematic test from HEAD and affected stable branches and recording the memory issue for investigation #91 #98.
- Tomas Vondra suggested investigating the Generation allocator introduced in PostgreSQL 11, but Tom Lane resisted backporting that infrastructure and questioned whether fragmentation explained this particular workload of equal-sized tuples #94 #95 #96 #97.
- Noah Misch separately fixed temporary-directory retention after nonzero TAP exits and removed the redundant PostgresNode __DIE__ handler; Daniel Gustafsson read the patch, and it landed unchanged apart from rebase context as bf989aaf35 #100 #101.
- Kuntal Ghosh isolated MaxHeapTupleSize rounding as a source of excessive memory use, and Amit Kapila suggested limiting that rounding once enough tuple buffers existed to fill the cache; Kuntal implemented this proposal #102 #103 #104.
- Andres Freund explained how per-subtransaction change limits amplify memory consumption during restore and supplied an alternative patch reducing slab-cached tuple size to 1024 bytes, while Kuntal supplied a separate diagnostic patch reproducing large allocations with Generation contexts #108 #111.
- Kuntal's memory estimates favored the allocation-count approach, and Amit Kapila tested it successfully, but Andres warned that variable-sized allocations had previously caused severe allocator fragmentation and requested overlapping-transaction testing #112 #113 #114 #115.
- At the final message, the VFD and Windows fixes and TAP artifact-retention fix had landed, the spill regression test remained reverted, and the older-branch memory proposals remained unresolved as Amit Kapila acknowledged that leaving those branches unchanged might be preferable to risking performance regressions #116.