model-writtenrun 7347 - may be wrong, links go to the source messages. Commit-by-commit LLM analysis is in the Commits dialog.
May 2021 · 3 entries
- Robert Haas opened the thread by describing archivers overwhelmed by repeated full scans of large archive_status directories, proposing next-filename prediction as an immediate fix and questioning whether status files should eventually be replaced by shared-memory signaling #1.
- Andres Freund proposed gathering multiple filenames per scan and warned that existing archive commands sometimes archive batches and modify status files themselves, establishing an important compatibility constraint #2.
- Stephen Frost explained that timeline history files must reach the archive promptly to reduce conflicting timeline assignments after promotions, and Robert proposed notifying the archiver when that priority must interrupt its fast path #8 #9.
Jul - Nov 2021 · 18 entries
- Dipesh Pandit submitted next-segment prediction, but Stephen identified its missing history-file priority handling; Dipesh's next version added timeline-switch notification and forced rescans #22 #24 #27.
- Dilip Kumar and Jeevan Ladhe reviewed notification interfaces, state ownership, and comments, leading to smaller revisions of the prediction patch #28 #31 #30 #35.
- Nathan Bossart suggested replacing per-segment status files with boundary markers, but Robert and Nathan agreed to defer that broader redesign and concentrate on the directory-scan problem #32 #33 #34.
- Robert objected to dedicating SIGINT to timeline changes and to prediction state being manipulated across multiple functions; after clarifying that the archiver already had shared-memory access, Dipesh introduced shared timeline tracking and consolidated state #36 #38 #39.
- Nathan demonstrated that status files created out of order could remain unarchived indefinitely under continuous next-segment prediction, prompting a generalized force-scan flag and further discussion of atomics, locking, and memory ordering #41 #46 #49 #53 #54.
- The force-scan implementation progressed through boolean flags, notification relocation, and a memory barrier while Nathan emphasized clearing scan requests before scanning and covering every special-file creation path #56 #57 #59 #60.
- Nathan introduced a competing implementation that gathered 64 filenames per scan with a binary heap; his 200,000-file test took roughly eight minutes with batching, five and a half minutes with prediction, and nearly three hours with the existing implementation #62 #63.
- Robert identified that cached filenames needed to be rechecked because an archive command might already have marked them done, while Nathan continued investigating backup-history priority and out-of-order notifications in the prediction branch #64 #65 #67.
- Dipesh added backup-history handling and centralized out-of-order detection using shared archiver state, while Nathan continued to favor the resilience of repeated bounded scans #70 #74 #73.
- Nathan supplied revised versions of both alternatives, adding cached-status handling to batching and proposing spinlocks, stronger error handling, and a priority comparator for prediction; Dipesh then refined Nathan's prediction implementation #76 #77.
- Kyotaro Horiguchi examined the batching limit and preferred prediction, while Robert and Nathan agreed that either optimization still needed timely notification for history files and that either would improve the status quo #80 #82 #83 #86.
- Further prediction reviews exposed skipped-file risks after archival failures and races in comparing new notifications with archiver progress; Dipesh added failure rescans and changed detection to compare successive notifications #88 #89 #90 #92.
- Nathan and Kyotaro identified additional concurrency and early-return problems, and Dipesh combined notification-state checking and updating under one lock while adding the missing rescan requests #93 #95 #96.
- Nathan substantially simplified prediction by replacing explicit out-of-order detection and checkpoint-triggered scans with periodic scans every three minutes, then corrected the timer to start after a scan completed #97 #101.
- Robert judged both alternatives committable but selected batching because it suited possible future multi-file archive commands and matched several participants' preferred design; David Steele explicitly approved the batching proposal and highlighted the value of testing history-file priority #103 #104.
- Nathan posted the final batching revision with expanded comments, retaining the filename array because replacing it with a reversed heap would add unnecessary complexity #105.
- After missed emails delayed closure, Robert committed the final batching attachment unchanged as beb4e9ba16, leaving the prediction alternative and wholesale removal of status files uncommitted #111.
- Alvaro landed 4abf615cc8, combining this thread's final batching patch #105 with larger filename buffers, allocated archiver state, and a heap reset from thread #45300; he abandoned PostgreSQL 13 after an attempted adaptation caused archiver hangs in testing #117.
Nov - Dec 2024 · 3 entries
- Three years later, Alvaro Herrera reopened the thread after production outages on older releases, proposing a PostgreSQL 14 backport of the batching commit together with its subsequent buffer and memory-management fixes #114.
- Robert Haas and Michael Paquier supported the mature implementation's backport, while agreeing that PostgreSQL 13's older archiver architecture made that branch substantially riskier #115 #116.
- The thread ended with batching landed and backported to PostgreSQL 14, PostgreSQL 13 left out, and Nathan noting that a further overhead reduction existed but might be unsuitable for backpatching because it could increase re-archiving after crashes #117 #118.