Patch CI statistics

Pipeline reads CI wall-clock time. Corpus reads today's patchsets laid out over PostgreSQL's history.

Runs
10,781
terminal, in range
Success rate
84.6%
9,126 of 10,781
Median build
2m 52s
successful and failed alike
Median test
5m 18s
successful and failed alike
Median queue wait
0m 00s
queued to started
Infra errors
0.1%
7 runs
GitHub re-attempts
0
attempt > 1 on one run id

Runs by result

terminal runs only; timeouts fold into their failure band

Show data
WhenResultRuns
10 Aug 00:00build failed327
10 Aug 00:00cancelled6
10 Aug 00:00infra error7
10 Aug 00:00success6,466
10 Aug 00:00tests failed843
17 Aug 00:00build failed134
17 Aug 00:00success2,660
17 Aug 00:00tests failed338

Success rate

hollow points sit below 5 terminal runs in the bucket

Show data
WhenSuccess rateTerminal runs
10 Aug 00:0084.5%7,649
17 Aug 00:0084.9%3,132

Build and test duration by era

median seconds; successful runs only

Show data
WhenEraBuild medianTest medianBuild p25-p75
10 Aug 00:00trixie2m 53s5m 21s2m 47s - 2m 55s
17 Aug 00:00trixie2m 52s5m 19s2m 46s - 2m 55s

ccache hit rate

median per bucket; the image's warm cache was baked against one reference commit, so a patch on a different base misses most of it

Show data
WhenHit rateRuns
10 Aug 00:002.5%6,466
17 Aug 00:001.5%2,660

Build time against ccache hit rate

one point per successful run; table shows the first 50

Show data
Hit rateBuild
1.5%2m 57s
1.5%2m 58s
1.2%2m 52s
1.3%2m 08s
1.5%2m 57s
1.5%2m 58s
1.5%2m 55s
1.5%2m 55s
1.5%2m 45s
1.2%2m 48s
1.5%2m 50s
1.5%2m 52s
1.2%2m 52s
1.2%2m 51s
1.5%1m 58s
1.5%2m 47s
1.5%2m 45s
1.5%2m 58s
1.5%2m 57s
2.2%2m 49s
1.5%2m 48s
1.5%2m 50s
1.5%2m 49s
1.5%2m 11s
1.5%2m 53s
1.5%2m 50s
1.5%2m 49s
2.2%1m 53s
2.2%3m 01s
2.2%2m 44s
1.5%2m 52s
1.5%2m 55s
2.2%1m 56s
2.2%2m 46s
1.5%3m 00s
1.5%2m 56s
9.2%2m 38s
1.5%2m 58s
9.4%1m 46s
2.2%2m 30s
1.5%2m 55s
9.3%2m 38s
9.2%1m 47s
1.5%2m 11s
23.5%0m 54s
1.5%2m 55s
1.2%2m 29s
1.5%2m 53s
1.5%3m 01s
1.5%2m 50s

Queue latency

median seconds

Show data
WhenPush to queuedQueued to started
10 Aug 00:00-5767m 28s0m 00s
17 Aug 00:000m 01s0m 00s

Average concurrency

time-weighted average of runs in flight per bucket, against the 30-slot budget

Show data
WhenAvg concurrency
10 Aug 00:0013.2
17 Aug 00:003.9

Push lag

time from the patch being posted to CI queueing it

Show data
WhenLagRuns
10 Aug 00:00same day124
10 Aug 00:00days379
10 Aug 00:00weeks782
10 Aug 00:00months2,898
10 Aug 00:00older3,466
17 Aug 00:00same day59
17 Aug 00:00days132
17 Aug 00:00weeks334
17 Aug 00:00months1,221
17 Aug 00:00older1,386

Push kind

first push, a re-push on a newer master, or a retry of the same head

Show data
WhenKindRuns
10 Aug 00:00first push126
10 Aug 00:00rebase re-push7,519
10 Aug 00:00retry4
17 Aug 00:00first push122
17 Aug 00:00rebase re-push3,010

Suites failed per failing run

one suite is usually a patch bug; dozens is usually one crash cascading

Show data
Suites failedRuns
1394
255
327
4348
574
618
746
817
96
107
1112
1211
134
217
311
337
551
561
586
597
627
767
837
1023
1074
1111
1226
1231
1296
1313
1321
1331
1401
1553
1561
1572
1582
2003

Top failing suites

total failures, and in brackets the number of distinct patchsets - many failures on few patchsets means one broken patch retried; roughly one per patchset means either a flaky suite or a suite that reruns others and inherits their failures

recovery/027_stream_regress
561 (90 branches)
test_plan_advice/001_replan_regress
520 (80 branches)
pg_upgrade/002_pg_upgrade
510 (81 branches)
regress/collate.icu.utf8
69 (15 branches)
regress/subselect
67 (13 branches)
regress/partition_join
61 (14 branches)
postgres_fdw/postgres_fdw
60 (9 branches)
regress/alter_table
59 (11 branches)
regress/copy2
59 (11 branches)
regress/join
58 (12 branches)
regress/partition_split
58 (10 branches)
isolation/horizons
57 (9 branches)
regress/generated_virtual
54 (10 branches)
regress/stats
53 (11 branches)
regress/plpgsql
51 (9 branches)
regress/predicate
51 (9 branches)
regress/with
51 (9 branches)
isolation/eval-plan-qual
50 (8 branches)
regress/aggregates
49 (10 branches)
regress/opr_sanity
46 (8 branches)

Infra health

Everything here should be zero. Non-zero is worth a look, not a panic.

ConditionCount
infra errors7
in flight past 48h0
branches with a failed push2
payloads we could not parse0
runs with no payload6
built but pushed no image38
Retried heads
4
same head_sha pushed again, new run id
Verdict changed
100.0%
4 of 4 - meaningless until this denominator grows
discussionsallactive only
Tracked patchsets
5,463
current, not superseded, all patchsets
Median base age
897 days
behind master, all patchsets
On recent master
34.5%
base 30 days or newer, all patchsets
Has a verdict
89.4%
terminal CI status, all patchsets
Has a runnable image
86.1%
counted by digest, all patchsets
recent
1,885
base age tier, all patchsets
stale
394
base age tier, all patchsets
ancient
3,123
base age tier, all patchsets
unknown
61
base age tier, all patchsets

Master re-check recency

when each patchset was last tried against master; overdue is past 26h with master moved since. Unlike the base age tiles above, this follows the discussions control.

current
52
behind
1,670
overdue
292
never
3,449

Base age distribution

all current patchsets against those in active discussions

Show data
Base ageAllActive discussions
<1d1,575318
1-7d6555
7-30d24599
30-90d6216
90d-1y33210
1-2y3202
2-5y8780
>5y1,9251
unknown6110

Outcome by base commit year

current patchsets, by the year of the master commit they apply to

Show data
Base yearOutcomePatchsets
unknownnever applied61
2012build failed1
2014never applied1
2015build failed3
2015never applied3
2016success22
2016tests failed2
2016never applied2
2017success463
2017tests failed58
2017build failed14
2017never applied63
2018success281
2018tests failed38
2018build failed7
2018never applied60
2019success244
2019tests failed23
2019build failed10
2019never applied74
2020success232
2020tests failed34
2020build failed4
2020never applied52
2021success241
2021tests failed35
2021build failed9
2021never applied35
2022success242
2022cancelled1
2022tests failed36
2022build failed11
2022never applied36
2023success220
2023tests failed26
2023build failed9
2023never applied32
2024success205
2024tests failed36
2024build failed8
2024infra error1
2024never applied23
2025success260
2025tests failed43
2025build failed13
2025infra error1
2025no verdict7
2025never applied36
2026success1,659
2026cancelled1
2026tests failed206
2026build failed76
2026infra error2
2026no verdict36
2026never applied165

Success rate by base commit year

of patchsets carrying a verdict

Show data
Base yearSuccess rateWith a verdict
20120.0%1
20150.0%3
201691.7%24
201786.5%535
201886.2%326
201988.1%277
202085.9%270
202184.6%285
202283.4%290
202386.3%255
202482.0%250
202582.0%317
202686.1%2,052

Apply failures by base commit year

patchsets that would not apply even to their own detected base - a patch-extraction / base-detection signal, not a staleness measure; the rate is in the table

Show data
Base yearPatchsetsFailed to applyFailure rate
unknown6161100.0%
2012100.0%
201411100.0%
20156350.0%
20162627.7%
20175986310.5%
20183866015.5%
20193517421.1%
20203225216.1%
20213203510.9%
20223263611.0%
20232873211.1%
2024273238.4%
20253603610.0%
20262,1451657.7%

Build and test time over PostgreSQL history

median seconds of successful runs, by the base commit's year

Show data
Base yearBuild medianTest median
20161m 41s3m 18s
20171m 46s4m 37s
20181m 50s2m 38s
20191m 55s2m 35s
20202m 00s2m 52s
20212m 11s3m 09s
20222m 14s3m 58s
20232m 51s3m 58s
20242m 43s3m 39s
20252m 45s4m 26s
20262m 52s5m 20s

Test suites over PostgreSQL history

median executed suites of successful runs, by the base commit's year

Show data
Base yearSuites median
2016412
2017451
2018505
2019537
2020566
2021621
2022654
2023777
2024833
2025900
20261019

Per PostgreSQL major

A major whose era image is still a disabled stub is marked - those patchsets sit in won't-retry until the image exists.

PGEraPatchsetsRunsSuccessBuild medianTest medianSuitesImages
9stretch131361.5%1m 34s2m 41s3989
10stretch40440587.9%1m 45s5m 20s443392
11stretch34734887.1%1m 48s2m 35s476339
12buster27527586.9%1m 51s2m 35s513267
13buster27327384.6%1m 57s2m 37s548267
14bullseye29029286.3%2m 08s3m 06s590284
15bullseye27928186.5%2m 11s3m 18s625268
16bookworm26926882.8%2m 46s4m 12s669256
17bookworm25025084.4%2m 54s3m 27s793244
18trixie28028179.0%2m 41s3m 50s858267
19trixie37335186.3%2m 49s4m 42s937339
20trixie1,90630,90283.6%2m 52s5m 18s10181,769

Files the corpus conflicts on

from apply-stage failures; in brackets the number of distinct patchsets

src/bin/pgbench/pgbench.c
11 (11 branches)
doc/src/sgml/ref/pgbench.sgml
10 (10 branches)
src/bin/pgbench/t/001_pgbench_with_server.pl
6 (6 branches)
src/backend/access/transam/xlog.c
5 (5 branches)
src/backend/parser/gram.y
5 (5 branches)
src/bin/pg_dump/pg_dump.c
5 (5 branches)
doc/src/sgml/config.sgml
4 (4 branches)
src/backend/access/common/reloptions.c
4 (4 branches)
src/backend/commands/tablecmds.c
4 (4 branches)
src/backend/postmaster/autovacuum.c
4 (4 branches)
src/include/utils/float.h
4 (4 branches)
templates/pages/about/policies/coc/ja.html
4 (4 branches)
configure.in
3 (3 branches)
doc/src/sgml/perform.sgml
3 (3 branches)
doc/src/sgml/ref/psql-ref.sgml
3 (3 branches)
doc/src/sgml/ref/select.sgml
3 (3 branches)
doc/src/sgml/release-19.sgml
3 (3 branches)
indent.c
3 (3 branches)
run_build.pl
3 (3 branches)
src/backend/commands/indexcmds.c
3 (3 branches)

Verdict coverage in active discussions

patchsets in threads with a message in the last 30 days

Show data
EraWith a verdictNo verdict
bullseye10
trixie46633
unknown011

Patch size against outcome

files touched per patchset, from the recorded submission paths

Show data
Files touchedOutcomePatchsets
unknownsuccess4,069
unknowncancelled2
unknowntests failed537
unknownbuild failed165
unknowninfra error4
unknownno verdict43
unknownnever applied643

Patchset versions per topic

every version, superseded included; split by whether the topic was committed

Show data
VersionsTopicTopics
1not committed4,979
1committed171
2not committed158
2committed36
3not committed45
3committed8
4not committed14
4committed5
5committed2
5not committed8
6committed1
6not committed5
7not committed4
8not committed2
10committed1
11not committed1

Why the orchestrator won't retry

The dashboard shows the top three of this same breakdown.

ReasonPatchsets
base too old3,032
committed118
patchset touches .github/11