Using a stock openssl BIO
Hi,
Thomas^WA bad person recently nerdsniped me (with the help of an accidental use
of an SSL connection in a benchmark leading to poor results) into checking what
would be needed to benefit from SSL/TLS hardware acceleration (available with
suitable hardware, OS support (linux and freebsd) and openssl 3). One problem
turns out to be the custom BIO we use.
Which made me look at why we use those.
In the backend the first reason is:
* Private substitute BIO: this does the sending and receiving using send() and
* recv() instead. This is so that we can enable and disable interrupts
* just while calling recv(). We cannot have interrupts occurring while
* the bulk of OpenSSL runs, because it uses malloc() and possibly other
* non-reentrant libc facilities.
I think this part has been obsolete for a while now (primarily [1]https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=387da18874afa17156ee3af63766f17efb53c4b9). These days
we always operate the backend sockets in nonblocking mode, and handle blocking
at a higher level. Which is where we then can handle interrupts etc correctly
(which we couldn't really before, because it still wasn't ok to jump out of
openssl).
The second part is
* We also need to call send() and recv()
* directly so it gets passed through the socket/signals layer on Win32.
And the not stated need to set/unset pgwin32_noblock around the recv/send
calls.
I don't think the signal handling stuff is still needed with nonblocking
sockets? It seems we could just ensure that there's a pgwin32_poll_signals()
somewhere higher up in secure_read/write()? E.g. in
ProcessClientReadInterrupt()/ProcessClientWriteInterrupt() or with an explicit
call.
And the pgwin32_noblock handling could just be done outside the SSL_read/write().
On the client side it looks like things would be a bit harder. The main problem
seems to be dealing with SIGPIPE. We could possibly deal with that by moving
the handling of that a layer up. That's a thick nest of ugly stuff :(.
A problem shared by FE & BE openssl is that openssl internally uses
BIO_set_data() inside the BIO routines we reuse. Oops. I hacked around that
using BIO_set_callback_arg()/BIO_get_callback_arg(), but that's not
particularly pretty either, although it probably is more reliable, since
callbacks are intended to be set separately from the BIO implementation.
A better approach might be to move the code using per-bio custom data from
pqsecure_raw_read() up to pqsecure_read() etc.
If we wanted to be able to use TLS acceleration while continuing to use our
custom socket routines we'd have to copy a good bit more functionality from
openssl into our BIO implementations:(.
FWIW, I don't think hardware tls acceleration is a particularly crucial thing
for now. Outside of backups it's rare to have the symmetric encryption part of
tls be the problem these days thanks, to the AES etc functions in most of the
common CPUs.
I don't plan to work on this, but Thomas encouraged me to mention this on the
list when I mention it to him.
Greetings,
Andres Freund
On Wed, Jul 14, 2021 at 07:17:47PM -0700, Andres Freund wrote:
FWIW, I don't think hardware tls acceleration is a particularly crucial thing
for now. Outside of backups it's rare to have the symmetric encryption part of
tls be the problem these days thanks, to the AES etc functions in most of the
common CPUs.I don't plan to work on this, but Thomas encouraged me to mention this on the
list when I mention it to him.
So, I am aware of CPU AES acceleration and I assume PG uses that. It is
the public key certificate verification part of TLS that we don't use
hardware acceleration for, right?
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
Hi,
On 2021-07-15 13:59:26 -0400, Bruce Momjian wrote:
On Wed, Jul 14, 2021 at 07:17:47PM -0700, Andres Freund wrote:
FWIW, I don't think hardware tls acceleration is a particularly crucial thing
for now. Outside of backups it's rare to have the symmetric encryption part of
tls be the problem these days thanks, to the AES etc functions in most of the
common CPUs.I don't plan to work on this, but Thomas encouraged me to mention this on the
list when I mention it to him.So, I am aware of CPU AES acceleration and I assume PG uses that.
Yes, it does so via openssl. But that still happens on the CPU. And
what's more, there's a lot of related work in TLS that's fairly
expensive (chunking up the data into TLS records etc). Some of the
better NICs can do that work in the happy path, so the CPU doesn't have
to do encryption nor framing. In some cases that can avoid the
to-be-sent data ever being pulled into the CPU caches, but instead it
can be DMA directly to the NIC.
In PG's case that's particularly interesting when sending out file data
in bulk, say in basebackup.c or walsender.c - the data can be sent via
sendfile(), so it never goes to userspace.
Here's an overview of the kernel TLS / TLS offload
https://legacy.netdevconf.info/1.2/papers/ktls.pdf
It is the public key certificate verification part of TLS that we
don't use hardware acceleration for, right?
That's true, but separate from what I was talking about. For most
database workloads the public key stuff shouldn't be a major piece,
because connection establishment shouldn't be that frequent...
Greetings,
Andres Freund
On 15 Jul 2021, at 04:17, Andres Freund <andres@anarazel.de> wrote:
Thomas^WA bad person recently nerdsniped me (with the help of an accidental use
of an SSL connection in a benchmark leading to poor results) into checking what
would be needed to benefit from SSL/TLS hardware acceleration (available with
suitable hardware, OS support (linux and freebsd) and openssl 3).
Now why does that sounds so familiar.. =)
In the backend the first reason is:
* Private substitute BIO: this does the sending and receiving using send() and
* recv() instead. This is so that we can enable and disable interrupts
* just while calling recv(). We cannot have interrupts occurring while
* the bulk of OpenSSL runs, because it uses malloc() and possibly other
* non-reentrant libc facilities.I think this part has been obsolete for a while now
I concur.
The second part is
* We also need to call send() and recv()
* directly so it gets passed through the socket/signals layer on Win32.And the not stated need to set/unset pgwin32_noblock around the recv/send
calls.I don't think the signal handling stuff is still needed with nonblocking
sockets? It seems we could just ensure that there's a pgwin32_poll_signals()
somewhere higher up in secure_read/write()? E.g. in
ProcessClientReadInterrupt()/ProcessClientWriteInterrupt() or with an explicit
call.And the pgwin32_noblock handling could just be done outside the SSL_read/write().
I hadn't yet looked at the pgwin32 parts in detail, but this is along what I
was thinking (just more refined).
On the client side it looks like things would be a bit harder. The main problem
seems to be dealing with SIGPIPE. We could possibly deal with that by moving
the handling of that a layer up. That's a thick nest of ugly stuff :(.
My initial plan was to keep this for the backend, as the invasiveness of the
frontend patch is unlikely to be justified by the returns of the acceleration.
FWIW, I don't think hardware tls acceleration is a particularly crucial thing
for now.
Agreed, it will most likely be of limited use to most. It might however make
sense to "get in on the ground floor" to be ready in case it's expanded on in
kernel+OpenSSL with postgres automatically just reaping the benefits. Either
way I was hoping to get to a patch which is close enough to what it would need
to look like so we can decide with the facts at hand.
I don't plan to work on this, but Thomas encouraged me to mention this on the
list when I mention it to him.
I still have it on my TODO for after the vacation, and hope to reach that part
of the list soon.
--
Daniel Gustafsson https://vmware.com/
Hi,
On 2021-07-22 00:21:16 +0200, Daniel Gustafsson wrote:
On 15 Jul 2021, at 04:17, Andres Freund <andres@anarazel.de> wrote:
Thomas^WA bad person recently nerdsniped me (with the help of an accidental use
of an SSL connection in a benchmark leading to poor results) into checking what
would be needed to benefit from SSL/TLS hardware acceleration (available with
suitable hardware, OS support (linux and freebsd) and openssl 3).Now why does that sounds so familiar.. =)
:)
On the client side it looks like things would be a bit harder. The main problem
seems to be dealing with SIGPIPE. We could possibly deal with that by moving
the handling of that a layer up. That's a thick nest of ugly stuff :(.My initial plan was to keep this for the backend, as the invasiveness of the
frontend patch is unlikely to be justified by the returns of the acceleration.
There's two main reasons I'd prefer not to do that:
1) It makes it surprisingly hard to benchmark the single connection TLS
throughput, because there's no client that can pull data quick enough.
2) The main case for wanting offload imo is bulk data stuff (basebackup,
normal backup). For that you also want to be able to receive the data
fast. Outside of situations like that I don't think the gains are likely to
be meaningful given AES-NI and other similar cpu level acceleration.
FWIW, I don't think hardware tls acceleration is a particularly crucial thing
for now.Agreed, it will most likely be of limited use to most. It might however make
sense to "get in on the ground floor" to be ready in case it's expanded on in
kernel+OpenSSL with postgres automatically just reaping the benefits. Either
way I was hoping to get to a patch which is close enough to what it would need
to look like so we can decide with the facts at hand.
Yea. I also just think getting rid of the bio stuff is good for
maintainability / robustness. Relying on our own socket functions being
compatible with the openssl bio doesn't sound very future proof... Especially
not combined with our use of the data field, which of course the other bio
functions may use (as they do when ktls is enabled!).
I don't plan to work on this, but Thomas encouraged me to mention this on the
list when I mention it to him.I still have it on my TODO for after the vacation, and hope to reach that part
of the list soon.
Cool!
Greetings,
Andres Freund