Print latency percentiles in pgbench
Hi hackers!
I did a lot of benchmarks in last time, and more are incoming.
While benchmarking with pgbench I keep hitting the same annoyance: the final
report says only "latency average" and "latency stddev",
which tells nothing about the distribution. p95/p99 is usually what you
actually want when evaluating tail latency.
Today the answer is to run with -l and post-process
the log, but logging itself perturbs subsequent transactions (fsync'ed writes
on the hot path), and requires a second step.
What if we add new option, smth like
--latency-percentiles[=p1,p2,...], which extends the final reports with:
latency min = 4.114 ms
latency max = 15.213 ms
latency p50 = 6.097 ms
latency p95 = 12.211 ms
latency p99 = 13.883 ms
Main design points, kept deliberately simple for a first version:
- percentiles are estimated from a fixed-size log-scale histogram
(32 sub-buckets per octave, 4 KB, zero allocations, a few ops per
transaction; no overhead at all when the option is not given). A more
clever estimator (t-digest and friends) can replace it later behind
the same interface, but seemed like overkill to start with;
- the histogram is a separate member of StatsData next to the latency
SimpleStats, so existing stats structs/log files/per-command table and
the default output stay untouched;
- semantics match existing latency stats: successful, non-skipped
transactions only;
Accuracy vs a sorted -l log of the same run: p50 0.535 vs 0.533 ms,
p99 1.152 vs 1.158 ms - within the sub-bucket resolution.
POC-patch attached, any feedback would be appreciated!
--
Best regards,
Roman Khapov
Hi Roman,
POC-patch attached, any feedback would be appreciated!
Thanks for working on this. I've started with the documentation part
of the patch, on master at 30faf13889b.
The doc changes re-indent some existing text without changing it:
- the <varlistentry> and <term> lines of --log-prefix,
- the "when neither --progress, --rate, nor --latency-limit" lines,
- the whole paragraph on retries ("The main report contains the
number of failed transactions...").
The new --latency-percentiles varlistentry is also indented one level
deeper than the entries around it, so --log-prefix now has its
<varlistentry> and <term> at a different depth from its <listitem>.
Because of this, the doc diff is +57/-12, while git diff -w shows
+47/-2. Could you keep the existing indentation and indent the new
entry like its neighbours, so that only the real changes remain?
Regards,
Don Hoang
Hi Roman,
I've continued reviewing the patch, on master at 30faf13889b.
1. --latency-percentiles=nan passes the range check and then fails
an assertion:
$ pgbench --latency-percentiles=nan -t 10 bench
pgbench: pgbench.c:1517: latencyHistogramPercentile: Assertion
`p > 0.0 && p <= 100.0' failed.
Every comparison with NaN is false, so "percentile <= 0.0 ||
percentile > 100.0" does not reject it. Writing the check as
!(percentile > 0.0 && percentile <= 100.0) would.
2. The patch adds no tests. It would be good to cover the parse
errors (0, 101, abc, ",50", 11 values, nan) in
002_pgbench_no_server.pl and check the new output lines in
001_pgbench_with_server.pl.
3. Because the argument is optional, the list has to be attached
with "=". With a space, the list is taken as the database name:
$ pgbench --latency-percentiles 50,99 -t 10 bench
pgbench: error: too many command-line arguments (first is "bench")
$ pgbench --latency-percentiles 50,99 -t 10
pgbench: error: connection to server ... failed: FATAL: database
"50,99" does not exist
Neither message points at the option, so I think the docs should
say that the value must be given as --latency-percentiles=LIST.
4. The docs say the reported percentile "lies within the histogram
bucket that contains the requested rank", but not how wide a
bucket is. It would help to state the resolution (about 3% of the
value) and the cases where it does not hold:
- A percentile can fall outside [min, max] because of the
interpolation inside a bucket. With a script containing only
"\sleep 2 ms" (500 transactions) I got:
latency min = 2.023 ms
latency p0.1 = 2.016 ms
Clamping the result to the min/max that SimpleStats already
tracks would avoid this.
- Below 32 us an octave is a single bucket, so nothing is
interpolated: for 1000 latencies between 20 and 29 us, p1, p50
and p99 all come out as 16 us.
Regards,
Don Hoang
Hello,
+1 to the NaN rejection, tests, and documenting the '=' form.
One small addition: duplicates like --latency-percentiles=50,50
are accepted; probably fine, but worth confirming that's intentional.
Best,
Phil Alger
--
EDB
On Wednesday, November 4th, 2026 at 08:51, Hữu Đôn Hoàng <hoanghuudon02hp@gmail.com> wrote:
Show quoted text
Hi Roman,
I've continued reviewing the patch, on master at 30faf13889b.
1. --latency-percentiles=nan passes the range check and then fails
an assertion:$ pgbench --latency-percentiles=nan -t 10 bench
pgbench: pgbench.c:1517: latencyHistogramPercentile: Assertion
`p > 0.0 && p <= 100.0' failed.Every comparison with NaN is false, so "percentile <= 0.0 ||
percentile > 100.0" does not reject it. Writing the check as
!(percentile > 0.0 && percentile <= 100.0) would.2. The patch adds no tests. It would be good to cover the parse
errors (0, 101, abc, ",50", 11 values, nan) in
002_pgbench_no_server.pl and check the new output lines in
001_pgbench_with_server.pl.3. Because the argument is optional, the list has to be attached
with "=". With a space, the list is taken as the database name:$ pgbench --latency-percentiles 50,99 -t 10 bench
pgbench: error: too many command-line arguments (first is "bench")$ pgbench --latency-percentiles 50,99 -t 10
pgbench: error: connection to server ... failed: FATAL: database
"50,99" does not existNeither message points at the option, so I think the docs should
say that the value must be given as --latency-percentiles=LIST.4. The docs say the reported percentile "lies within the histogram
bucket that contains the requested rank", but not how wide a
bucket is. It would help to state the resolution (about 3% of the
value) and the cases where it does not hold:- A percentile can fall outside [min, max] because of the
interpolation inside a bucket. With a script containing only
"\sleep 2 ms" (500 transactions) I got:latency min = 2.023 ms
latency p0.1 = 2.016 msClamping the result to the min/max that SimpleStats already
tracks would avoid this.- Below 32 us an octave is a single bucket, so nothing is
interpolated: for 1000 latencies between 20 and 29 us, p1, p50
and p99 all come out as 16 us.Regards,
Don Hoang