PSA: New intel MDS vulnerability mitigations cause measurable slowdown

On Wed, May 15, 2019 at 10:31 AM Andres Freund <andres@anarazel.de> wrote:

*Without* disabling SMT, for readonly pgbench, I'm seeing regressions
between 7-11%, depending on the size of shared_buffers (and some runtime
variations). That's just on my laptop, with an i7-6820HQ / Haswell CPU.
I'd be surprised if there weren't adversarial loads with bigger
slowdowns - what gets more expensive with the mitigations is syscalls.

Yikes. This all in warm shared buffers, right? So effectively this
is the cost of recvfrom() and sendto() going up? Did you use -M
prepared? If not, there would also be a couple of lseek(SEEK_END)
calls in between for planning... I wonder how many more
syscall-taxing mitigations we need before relation size caching pays
off.

--
Thomas Munro
https://enterprisedb.com

Hi,

On 2019-05-15 12:52:47 +1200, Thomas Munro wrote:

On Wed, May 15, 2019 at 10:31 AM Andres Freund <andres@anarazel.de> wrote:

*Without* disabling SMT, for readonly pgbench, I'm seeing regressions
between 7-11%, depending on the size of shared_buffers (and some runtime
variations). That's just on my laptop, with an i7-6820HQ / Haswell CPU.
I'd be surprised if there weren't adversarial loads with bigger
slowdowns - what gets more expensive with the mitigations is syscalls.

Yikes. This all in warm shared buffers, right?

Not initially, but it ought to warm up quite quickly. I ran something
boiling down to pgbench -q -i -s 200; psql -c 'vacuum (freeze, analyze,
verbose)'; pgbench -n -S -c 32 -j 32 -S -M prepared -T 100 -P1. As both
pgbench -i's COPY and VACUUM use ringbuffers, initially s_b will
effectively be empty.

So effectively this is the cost of recvfrom() and sendto() going up?

Plus epoll_wait(). And read(), for the cases where s_b was smaller than
the data.

Did you use -M prepared?

Yes.

If not, there would also be a couple of lseek(SEEK_END) calls in
between for planning... I wonder how many more syscall-taxing
mitigations we need before relation size caching pays off.

Yea, I suspect we're going to have to go there soon for a number of
reasons.

- Andres

Hi,

On 2019-05-14 15:30:52 -0700, Andres Freund wrote:

There's a new set of CPU vulnerabilities, so far only affecting intel
CPUs. Cribbing from the linux-kernel announcement I'm referring to
https://xenbits.xen.org/xsa/advisory-297.html
for details.

The "fix" is for the OS to perform some extra mitigations:
https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html
https://www.kernel.org/doc/html/latest/x86/mds.html#mds

*And* SMT/hyperthreading needs to be disabled, to be fully safe.

Fun.

I've run a quick pgbench benchmark:

*Without* disabling SMT, for readonly pgbench, I'm seeing regressions
between 7-11%, depending on the size of shared_buffers (and some runtime
variations). That's just on my laptop, with an i7-6820HQ / Haswell CPU.
I'd be surprised if there weren't adversarial loads with bigger
slowdowns - what gets more expensive with the mitigations is syscalls.

The profile after the mitigations looks like:

+    3.62%  postgres         [kernel.vmlinux]         [k] do_syscall_64
+    2.99%  postgres         postgres                 [.] _bt_compare
+    2.76%  postgres         postgres                 [.] hash_search_with_hash_value
+    2.33%  postgres         [kernel.vmlinux]         [k] entry_SYSCALL_64
+    1.69%  pgbench          [kernel.vmlinux]         [k] do_syscall_64
+    1.61%  postgres         postgres                 [.] AllocSetAlloc
     1.41%  postgres         postgres                 [.] PostgresMain
+    1.22%  pgbench          [kernel.vmlinux]         [k] entry_SYSCALL_64
+    1.11%  postgres         postgres                 [.] LWLockAcquire
+    0.86%  postgres         postgres                 [.] PinBuffer
+    0.80%  postgres         postgres                 [.] LockAcquireExtended
+    0.78%  postgres         [kernel.vmlinux]         [k] psi_task_change
     0.76%  pgbench          pgbench                  [.] threadRun
     0.69%  postgres         postgres                 [.] LWLockRelease
+    0.69%  postgres         postgres                 [.] SearchCatCache1
     0.66%  postgres         postgres                 [.] LockReleaseAll
+    0.65%  postgres         postgres                 [.] GetSnapshotData
+    0.58%  postgres         postgres                 [.] hash_seq_search
     0.54%  postgres         postgres                 [.] hash_search
+    0.53%  postgres         [kernel.vmlinux]         [k] __switch_to
+    0.53%  postgres         postgres                 [.] hash_any
     0.52%  pgbench          libpq.so.5.12            [.] pqParseInput3
     0.50%  pgbench          [kernel.vmlinux]         [k] do_raw_spin_lock

where do_syscall_64 show this instruction profile:

│ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool branch)
│ {
│ asm_volatile_goto("1:"
1.58 │ ↓ jmpq bd
│ mds_clear_cpu_buffers():
│ * Works with any segment selector, but a valid writable
│ * data segment is the fastest variant.
│ *
│ * "cc" clobber is required because VERW modifies ZF.
│ */
│ asm volatile("verw %[ds]" : : [ds] "m" (ds) : "cc");
77.38 │ verw 0x13fea53(%rip) # ffffffff82400ee0 <ds.4768>
│ do_syscall_64():
│ }
│
│ syscall_return_slowpath(regs);
│ }
13.18 │ bd: pop %rbx
0.08 │ pop %rbp
│ ← retq
│ nr = syscall_trace_enter(regs);
│ c0: mov %rbp,%rdi
│ → callq syscall_trace_enter

Where verw is the instruction that was recycled to now have the
side-effect of flushing CPU buffers.

Greetings,

Andres Freund

On Wed, May 15, 2019 at 1:13 PM Andres Freund <andres@anarazel.de> wrote:

I've run a quick pgbench benchmark:

*Without* disabling SMT, for readonly pgbench, I'm seeing regressions
between 7-11%, depending on the size of shared_buffers (and some runtime
variations). That's just on my laptop, with an i7-6820HQ / Haswell CPU.
I'd be surprised if there weren't adversarial loads with bigger
slowdowns - what gets more expensive with the mitigations is syscalls.

This stuff landed in my FreeBSD 13.0-CURRENT kernel, so I was curious
to measure it with and without the earlier mitigations. On my humble
i7-8550U laptop with the new 1.22 microcode installed, with my usual
settings of PTI=on and IBRS=off, so far MDS=VERW gives me ~1.5% loss
of TPS with a single client, up to 4.3% loss of TPS for 16 clients,
but it didn't go higher when I tried 32 clients. This was a tiny
scale 10 database, though in a quick test it didn't look like it was
worse with scale 100.

With all three mitigations activated, my little dev machine has gone
from being able to do ~11.8 million baseline syscalls per second to
~1.6 million, or ~1.4 million with the AVX variant of the mitigation.

Raw getuid() syscalls per second:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 11798658 4764159 3274043
off on 2652564 1941606 1655356
on off 4973053 2932906 2339779
on on 1988527 1556922 1378798

pgbench read-only transactions per second, 1 client thread:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 19393 18949 18615
off on 17946 17586 17323
on off 19381 19015 18696
on on 18045 17709 17418

pgbench -M prepared read-only transactions per second, 1 client thread:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 35020 34049 33200
off on 31658 30902 30229
on off 35445 34353 33415
on on 32415 31599 30712

pgbench -M prepared read-only transactions per second, 4 client threads:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 79515 76898 76465
off on 63608 62220 61952
on off 77863 75431 74847
on on 62709 60790 60575

pgbench -M prepared read-only transactions per second, 16 client threads:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 125984 121164 120468
off on 112884 108346 107984
on off 121032 116156 115462
on on 108889 104636 104027

time gmake -s check:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 16.78 16.85 17.03
off on 18.19 18.81 19.08
on off 16.67 16.86 17.33
on on 18.58 18.83 18.99

--
Thomas Munro
https://enterprisedb.com

Missatge de Thomas Munro <thomas.munro@gmail.com> del dia dj., 16 de
maig 2019 a les 13:09:

On Wed, May 15, 2019 at 1:13 PM Andres Freund <andres@anarazel.de> wrote:

I've run a quick pgbench benchmark:

*Without* disabling SMT, for readonly pgbench, I'm seeing regressions
between 7-11%, depending on the size of shared_buffers (and some runtime
variations). That's just on my laptop, with an i7-6820HQ / Haswell CPU.
I'd be surprised if there weren't adversarial loads with bigger
slowdowns - what gets more expensive with the mitigations is syscalls.

This stuff landed in my FreeBSD 13.0-CURRENT kernel, so I was curious
to measure it with and without the earlier mitigations. On my humble
i7-8550U laptop with the new 1.22 microcode installed, with my usual
settings of PTI=on and IBRS=off, so far MDS=VERW gives me ~1.5% loss
of TPS with a single client, up to 4.3% loss of TPS for 16 clients,
but it didn't go higher when I tried 32 clients. This was a tiny
scale 10 database, though in a quick test it didn't look like it was
worse with scale 100.

With all three mitigations activated, my little dev machine has gone
from being able to do ~11.8 million baseline syscalls per second to

Did you mean "1.8"?

~1.6 million, or ~1.4 million with the AVX variant of the mitigation.

Raw getuid() syscalls per second:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 11798658 4764159 3274043
off on 2652564 1941606 1655356
on off 4973053 2932906 2339779
on on 1988527 1556922 1378798

pgbench read-only transactions per second, 1 client thread:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 19393 18949 18615
off on 17946 17586 17323
on off 19381 19015 18696
on on 18045 17709 17418

pgbench -M prepared read-only transactions per second, 1 client thread:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 35020 34049 33200
off on 31658 30902 30229
on off 35445 34353 33415
on on 32415 31599 30712

pgbench -M prepared read-only transactions per second, 4 client threads:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 79515 76898 76465
off on 63608 62220 61952
on off 77863 75431 74847
on on 62709 60790 60575

pgbench -M prepared read-only transactions per second, 16 client threads:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 125984 121164 120468
off on 112884 108346 107984
on off 121032 116156 115462
on on 108889 104636 104027

time gmake -s check:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 16.78 16.85 17.03
off on 18.19 18.81 19.08
on off 16.67 16.86 17.33
on on 18.58 18.83 18.99

--
Thomas Munro
https://enterprisedb.com

--
Albert Cervera i Areny
http://www.NaN-tic.com
Tel. 93 553 18 03

On 5/16/19 12:24 PM, Albert Cervera i Areny wrote:

Missatge de Thomas Munro <thomas.munro@gmail.com> del dia dj., 16 de
maig 2019 a les 13:09:

With all three mitigations activated, my little dev machine has gone
from being able to do ~11.8 million baseline syscalls per second to

Did you mean "1.8"?

Not in what I thought I saw:

~1.6 million, or ~1.4 million ...

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 11798658 4764159 3274043

^^^^^^^^

off on 2652564 1941606 1655356
on off 4973053 2932906 2339779
on on 1988527 1556922 1378798

^^^^^^^ ^^^^^^^

-Chap

On Fri, May 17, 2019 at 5:26 AM Chapman Flack <chap@anastigmatix.net> wrote:

On 5/16/19 12:24 PM, Albert Cervera i Areny wrote:

Missatge de Thomas Munro <thomas.munro@gmail.com> del dia dj., 16 de
maig 2019 a les 13:09:

With all three mitigations activated, my little dev machine has gone
from being able to do ~11.8 million baseline syscalls per second to

Did you mean "1.8"?

Not in what I thought I saw:

~1.6 million, or ~1.4 million ...

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 11798658 4764159 3274043

^^^^^^^^

off on 2652564 1941606 1655356
on off 4973053 2932906 2339779
on on 1988527 1556922 1378798

^^^^^^^ ^^^^^^^

Right. Actually it's worse than that -- after I posted I realised
that I had some debug stuff enabled in my kernel that was slowing
things down a bit, so I reran the tests overnight with a production
kernel and here is what I see this morning. It's actually ~17.8
million syscalls/sec -> ~1.7 million syscalls/sec, if you go from all
mitigations off to all mitigations on, or -> ~3.2 million for just PTI
+ MDS. And the loss of TPS is ~5% for the case I was most interested
in, just turning on MDS=VERW if you already had PTI on and IBRS off.

Raw getuid() syscalls per second:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 17771744 5372032 3575035
off on 3060923 2166527 1817052
on off 5622591 3150883 2463934
on on 2213190 1687748 1475605

pgbench read-only transactions per second, 1 client thread:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 22414 22103 21571
off on 21298 20817 20418
on off 22473 22080 21550
on on 21286 20850 20386

pgbench -M prepared read-only transactions per second, 1 client thread:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 43508 42476 41123
off on 40729 39483 38555
on off 44110 42989 42012
on on 41143 39990 38798

pgbench -M prepared read-only transactions per second, 4 client threads:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 100735 97689 96662
off on 80142 77804 77064
on off 100540 97010 95827
on on 79492 76976 76226

pgbench -M prepared read-only transactions per second, 16 client threads:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 161015 152978 152556
off on 145605 139438 139179
on off 155359 147691 146987
on on 140976 134978 134177

pgbench -M prepared read-only transactions per second, 16 client threads:

PTI IBRS MDS=off MDS=VERW MDS=AVX
===== ===== ======== ======== ========
off off 157986 150132 149436
off on 142618 136220 135901
on off 153482 146214 145839
on on 138650 133074 132142

--
Thomas Munro
https://enterprisedb.com

On Fri, May 17, 2019 at 9:42 AM Thomas Munro <thomas.munro@gmail.com> wrote:

pgbench -M prepared read-only transactions per second, 16 client threads:

(That second "16 client threads" line should read "32 client threads".)

--
Thomas Munro
https://enterprisedb.com