Disable OpenSSL compression

On Tue, Nov 8, 2011 at 14:59, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:

I ran into a performance problem described in this thread:

http://archives.postgresql.org/pgsql-performance/2011-10/msg00249.php
continued here:
http://archives.postgresql.org/pgsql-performance/2011-11/msg00045.php

OpenSSL compresses data by default, and that causes a
performance penalty of 100% and more, at least when
SELECTing larger bytea objects.
The backend process becomes CPU bound.

From OpenSSL version 1.0.0. on, compression can be
disabled.  The attached patch does that, and with that
patch I see dramatic performance improvements:

Unpatched:

samples %       image name         symbol name
6754    83.7861 libz.so.1.2.3      /lib64/libz.so.1.2.3
618      7.6665 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0
534      6.6245 postgres           hex_encode
95       1.1785 libc-2.12.so       memcpy

Patched:

samples %       image name         symbol name
751     50.1670 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0
594     39.6794 postgres           hex_encode
83       5.5444 libc-2.12.so       memcpy

(the test case is selecting one 27 MB bytea in text
mode over a localhost connection)

Are there any objections to this?

This should probably be made an option.

And doesn't it, at least in a lot of cases, make more sense to control
this from the client side? It might typically be good to use
comopression if you are connecting over a slow link such as mobile or
satellite. And typically the client knows that, not the server. So
either client, or pg_hba driven, perhaps?

It is possible that this could cause a performance
regression for people who SELECT lots of compressible
data over really slow network connections, but is that
a realistic scenario?

Turning it off unconditionally can certainly create such a regression.
I don't think it's at all unrealstic.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 11/08/2011 03:59 PM, Albe Laurenz wrote:

If there are concerns about that, maybe a GUC variable like
ssl_compression (defaulting to off) would be a solution.

I'd vote for a libpq connect option instead. Something like
sslcompress=yes|no accompanied by PGSSLCOMPRESS environment
variable. And defaulting to "yes", as not to break any
backward compatibilty. For instance we expect SSL to provide
compression, wouldn't even use it without it.

Regards,
Martin

On Tue, Nov 8, 2011 at 3:59 PM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:

It is possible that this could cause a performance
regression for people who SELECT lots of compressible
data over really slow network connections, but is that
a realistic scenario?

Yes, it's a realistic scenario. Please make it a option.

Also, high-security links may prefer compression.

--
marko

Marko Kreen <markokr@gmail.com> writes:

On Tue, Nov 8, 2011 at 3:59 PM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:

It is possible that this could cause a performance
regression for people who SELECT lots of compressible
data over really slow network connections, but is that
a realistic scenario?

Yes, it's a realistic scenario. Please make it a option.

I distinctly recall us getting bashed a few years ago because there
wasn't any convenient way to turn SSL compression *on*. Now that SSL
finally does the sane thing by default, you want to turn it off?

The fact of the matter is that in most situations where you want SSL,
ie links across insecure WANs, compression is a win. Testing a local
connection, as you seem to have done, is just about 100% irrelevant to
performance in the real world.

There might be some argument for providing a client option to disable
compression, but it should not be forced, and it shouldn't even be the
default. But before adding YA connection option, I'd want to see some
evidence that it's useful over non-local connections.

regards, tom lane

On Tue, Nov 8, 2011 at 2:34 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

The fact of the matter is that in most situations where you want SSL,
ie links across insecure WANs, compression is a win.  Testing a local
connection, as you seem to have done, is just about 100% irrelevant to
performance in the real world.

I would disagree with that. Deployments in the cloud may have fast,
but untrustworthy network connections.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 11/08/2011 09:34 AM, Tom Lane wrote:

Marko Kreen<markokr@gmail.com> writes:

On Tue, Nov 8, 2011 at 3:59 PM, Albe Laurenz<laurenz.albe@wien.gv.at> wrote:

It is possible that this could cause a performance
regression for people who SELECT lots of compressible
data over really slow network connections, but is that
a realistic scenario?

Yes, it's a realistic scenario. Please make it a option.

I distinctly recall us getting bashed a few years ago because there
wasn't any convenient way to turn SSL compression *on*. Now that SSL
finally does the sane thing by default, you want to turn it off?

The fact of the matter is that in most situations where you want SSL,
ie links across insecure WANs, compression is a win. Testing a local
connection, as you seem to have done, is just about 100% irrelevant to
performance in the real world.

There might be some argument for providing a client option to disable
compression, but it should not be forced, and it shouldn't even be the
default. But before adding YA connection option, I'd want to see some
evidence that it's useful over non-local connections.

I can certainly conceive of situations where one wants SSL on a high
speed/bandwidth network. I don't think we should assume that all or even
most real world SSL use will be across slow networks.

Here's another data point:
<http://journal.paul.querna.org/articles/2011/04/05/openssl-memory-use/&gt;

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:

On 11/08/2011 09:34 AM, Tom Lane wrote:

There might be some argument for providing a client option to disable
compression, but it should not be forced, and it shouldn't even be the
default. But before adding YA connection option, I'd want to see some
evidence that it's useful over non-local connections.

I can certainly conceive of situations where one wants SSL on a high
speed/bandwidth network. I don't think we should assume that all or even
most real world SSL use will be across slow networks.

Even for that use-case, I don't believe that testing on a local loopback
connection should be considered representative.

regards, tom lane

On Tue, Nov 8, 2011 at 9:58 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Andrew Dunstan <andrew@dunslane.net> writes:

On 11/08/2011 09:34 AM, Tom Lane wrote:

There might be some argument for providing a client option to disable
compression, but it should not be forced, and it shouldn't even be the
default.  But before adding YA connection option, I'd want to see some
evidence that it's useful over non-local connections.

I can certainly conceive of situations where one wants SSL on a high
speed/bandwidth network. I don't think we should assume that all or even
most real world SSL use will be across slow networks.

Even for that use-case, I don't believe that testing on a local loopback
connection should be considered representative.

Probably not, but I think we ought to provide the option to disable
compression for those who want to do that. I also agree with you that
we should leave the default as-is.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Nov 8, 2011 at 4:34 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Marko Kreen <markokr@gmail.com> writes:

On Tue, Nov 8, 2011 at 3:59 PM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:

It is possible that this could cause a performance
regression for people who SELECT lots of compressible
data over really slow network connections, but is that
a realistic scenario?

Yes, it's a realistic scenario.  Please make it a option.

I distinctly recall us getting bashed a few years ago because there
wasn't any convenient way to turn SSL compression *on*.  Now that SSL
finally does the sane thing by default, you want to turn it off?

The fact of the matter is that in most situations where you want SSL,
ie links across insecure WANs, compression is a win.  Testing a local
connection, as you seem to have done, is just about 100% irrelevant to
performance in the real world.

There might be some argument for providing a client option to disable
compression, but it should not be forced, and it shouldn't even be the
default.  But before adding YA connection option, I'd want to see some
evidence that it's useful over non-local connections.

+1 for keeping current default.

But I can imagine scenarios where having option to turn compression
off could be useful:
- when minimal latency is required
- when "normal" latency is required, but data is big
- when serving big non-compressible blobs - zlib can be very slow
- when serving lots of connections and want
to minimize unnecessary cpu and memory load

Depending on how zlib is used by openssl, some of
them may not happen in practice.

--
marko

Tom Lane wrote:

I distinctly recall us getting bashed a few years ago because there
wasn't any convenient way to turn SSL compression *on*. Now that SSL
finally does the sane thing by default, you want to turn it off?

The fact of the matter is that in most situations where you want SSL,
ie links across insecure WANs, compression is a win. Testing a local
connection, as you seem to have done, is just about 100% irrelevant to
performance in the real world.

Maybe that's paranoia, but we use SSL via the company's LAN to keep
potentially sensitive data from crossing the network unencrypted.

There might be some argument for providing a client option to disable
compression, but it should not be forced, and it shouldn't even be the
default. But before adding YA connection option, I'd want to see some
evidence that it's useful over non-local connections.

I will try to provide test results via remote connection; I thought
that localhost was a good enough simulation for a situation where
you are not network bound.

I agree with you that a client option would make more sense.
The big problem I personally have with that is that it only works
if you use libpq. When using the JDBC driver or Npgsql, a client
option wouldn't help me at all.

Yours,
Laurenz Albe

Tom Lane wrote:

There might be some argument for providing a client option to disable
compression, but it should not be forced, and it shouldn't even be the
default. But before adding YA connection option, I'd want to see some
evidence that it's useful over non-local connections.

Here are numbers from a test via LAN.
The client machine has OpenSSL 0.9.8e, the server OpenSSL 1.0.0.

The client command run was

echo 'select ...' | time psql "host=..." -o /dev/null

and \timing was turned on in .psqlrc

In addition to the oprofile data I collected three times:
- the duration as shown in the server log
- the duration as shown by \timing
- the duration of the psql command as measured by "time"

Without patch:

duration: 5730.996 ms (log), 5975.093 ms (\timing), 22.87 s (time)

samples % image name symbol name
4428 80.2029 libz.so.1.2.3 /lib64/libz.so.1.2.3
559 10.1250 postgres hex_encode
361 6.5387 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0
83 1.5034 libc-2.12.so memcpy

With patch:

duration: 3001.009 ms (log), 3243.690 ms (\timing), 20.27 s (time)

samples % image name symbol name
1072 58.0401 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0
587 31.7813 postgres hex_encode
105 5.6849 libc-2.12.so memcpy

I think this makes a good case for disabling compression.

Yours,
Laurenz Albe

On Tue, Nov 08, 2011 at 04:59:02PM +0100, Albe Laurenz wrote:

Tom Lane wrote:

There might be some argument for providing a client option to disable
compression, but it should not be forced, and it shouldn't even be the
default. But before adding YA connection option, I'd want to see some
evidence that it's useful over non-local connections.

Here are numbers from a test via LAN.
The client machine has OpenSSL 0.9.8e, the server OpenSSL 1.0.0.

The client command run was

echo 'select ...' | time psql "host=..." -o /dev/null

and \timing was turned on in .psqlrc

In addition to the oprofile data I collected three times:
- the duration as shown in the server log
- the duration as shown by \timing
- the duration of the psql command as measured by "time"

Without patch:

duration: 5730.996 ms (log), 5975.093 ms (\timing), 22.87 s (time)

samples % image name symbol name
4428 80.2029 libz.so.1.2.3 /lib64/libz.so.1.2.3
559 10.1250 postgres hex_encode
361 6.5387 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0
83 1.5034 libc-2.12.so memcpy

With patch:

duration: 3001.009 ms (log), 3243.690 ms (\timing), 20.27 s (time)

samples % image name symbol name
1072 58.0401 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0
587 31.7813 postgres hex_encode
105 5.6849 libc-2.12.so memcpy

I think this makes a good case for disabling compression.

Yours,
Laurenz Albe

Certainly a good case for providing the option to disable compression.

Regards,
Ken

On Tue, Nov 08, 2011 at 04:19:02PM +0100, Albe Laurenz wrote:

Tom Lane wrote:

I distinctly recall us getting bashed a few years ago because there
wasn't any convenient way to turn SSL compression *on*. Now that SSL
finally does the sane thing by default, you want to turn it off?

The fact of the matter is that in most situations where you want SSL,
ie links across insecure WANs, compression is a win. Testing a local
connection, as you seem to have done, is just about 100% irrelevant to
performance in the real world.

Maybe that's paranoia, but we use SSL via the company's LAN to keep
potentially sensitive data from crossing the network unencrypted.

There might be some argument for providing a client option to disable
compression, but it should not be forced, and it shouldn't even be the
default. But before adding YA connection option, I'd want to see some
evidence that it's useful over non-local connections.

I will try to provide test results via remote connection; I thought
that localhost was a good enough simulation for a situation where
you are not network bound.

I agree with you that a client option would make more sense.
The big problem I personally have with that is that it only works
if you use libpq. When using the JDBC driver or Npgsql, a client
option wouldn't help me at all.

Yours,
Laurenz Albe

I think that JDBC and Npgsql should also support disabling compression.

Regards,
Ken

On Tue, Nov 8, 2011 at 11:06 AM, ktm@rice.edu <ktm@rice.edu> wrote:

I think that JDBC and Npgsql should also support disabling compression.

That's the *real* problem here...

You're quite right that if we allow controlling this on the libpq
side, it is surely desirable to allow controlling this via JDBC,
Npgsql, and other mechanisms people may have around. (There are
native protocol implementations for Common Lisp and Go, for instance.
They may not be particularly important, )

Unfortunately, each protocol implementation is independent, which
really is the nature of the beast, which includes:
a) The code of the implementation,
b) Release of the implementation,
c) Packaging of releases into software distributions.

With that series of complications, I wonder if maybe the right place
to control this is pg_hba.conf. That has the merit of centralizing
control in such a way that it would apply commonly to
libpq/JDBC/Npgsql/..., though with the demerit that the control does
not take place on the client side, which is desirable.

I wonder how many SSL parameters there are which would be worth trying
to have available. I expect we'd benefit from looking at all the
relevant ones at once, so as to not have the problem of hacking "one
more" into place and perhaps doing it a bit differently each time.
--
When confronted by a difficult problem, solve it by reducing it to the
question, "How would the Lone Ranger handle this?"

On 2011-11-08 22:59, Albe Laurenz wrote:

In addition to the oprofile data I collected three times:
- the duration as shown in the server log
- the duration as shown by \timing
- the duration of the psql command as measured by "time"

[...]

I think this makes a good case for disabling compression.

It's a few data points, but is it enough to make a good case? As I
understand it, compression can save time not only on transport but also
on the amount of data that needs to go through encryption -- probably
depending on choice of cypher, hardware support, machine word width,
compilation details etc. Would it make sense to run a wider experiment,
e.g. in the buld farm?

Another reason why I believe compression is often used with encryption
is to maximize information content per byte of data: harder to guess,
harder to crack. Would that matter?

Jeroen

Jeroen Vermeulen <jtv@xs4all.nl> writes:

Another reason why I believe compression is often used with encryption
is to maximize information content per byte of data: harder to guess,
harder to crack. Would that matter?

Yes, it would. There's a reason why the OpenSSL default is what it is.

regards, tom lane

Tom,

* Tom Lane (tgl@sss.pgh.pa.us) wrote:

The fact of the matter is that in most situations where you want SSL,
ie links across insecure WANs, compression is a win. Testing a local
connection, as you seem to have done, is just about 100% irrelevant to
performance in the real world.

I'm mystified by the idea that SSL shouldn't be used on local networks.
If the only things talking to the database are other servers on
physically secure networks, perhaps, but when you've got databases
exposed (even through firewalls) to client networks (which are in the
same building), and any data that's even remotely sensetive, you should
be using SSL or IPSEC. The chances of eaves-dropping on a typiacal WAN
physical/dedicated link (not over the Internet..) are actually much less
than some disgruntled employee spoofing the local switches to monitor
someone else's traffic. For starters, you're going to need some pretty
specialized gear to eavesdrop on a T1 or similar link and once it's past
the last mile and into the fibre network... Well, there's some folks
who can manage that, but it's not very many.

There might be some argument for providing a client option to disable
compression, but it should not be forced, and it shouldn't even be the
default. But before adding YA connection option, I'd want to see some
evidence that it's useful over non-local connections.

I agree that it should be an option and that it should be on by default.
It's going to typically be a win.

Thanks,

Stephen

Christopher Browne wrote:

I think that JDBC and Npgsql should also support disabling

compression.

That's the *real* problem here...

You're quite right that if we allow controlling this on the libpq
side, it is surely desirable to allow controlling this via JDBC,
Npgsql, and other mechanisms people may have around.

[...]

With that series of complications, I wonder if maybe the right place
to control this is pg_hba.conf.

I think that wouldn't work, because to query pg_hba.conf, you have to
know user and database, which come from the client side.
But the SSL negotiation takes place earlier, namely when the
connection is established.

I wonder how many SSL parameters there are which would be worth trying
to have available. I expect we'd benefit from looking at all the
relevant ones at once, so as to not have the problem of hacking "one
more" into place and perhaps doing it a bit differently each time.

Sure, if anybody can think of any. A quick look at
"man SSL_CTX_set_options" didn't show me any, but then OpenSSL's
documentation is very bad (the page does not even mention
SSL_OP_NO_COMPRESSION) and I am no SSL expert.

Is the following proposal acceptable:

- Add a GUC ssl_compression, defaulting to "on".
- Add a client option "sslcompression" and an environment variable
PGSSLCOMPRESSION, defaulting to "1".

Compression will be disabled if either side refuses.

That way you can control the setting per client, but you can also
force it on clients that do not use libpq if you want.

Yours,
Laurenz Albe

On Wednesday, November 9, 2011, Albe Laurenz wrote:

Christopher Browne wrote:

I think that JDBC and Npgsql should also support disabling

compression.

That's the *real* problem here...

You're quite right that if we allow controlling this on the libpq
side, it is surely desirable to allow controlling this via JDBC,
Npgsql, and other mechanisms people may have around.

[...]

With that series of complications, I wonder if maybe the right place
to control this is pg_hba.conf.

I think that wouldn't work, because to query pg_hba.conf, you have to
know user and database, which come from the client side.
But the SSL negotiation takes place earlier, namely when the
connection is established.

Oh, right, that's going to be a problem doing it there.

I wonder how many SSL parameters there are which would be worth trying
to have available. I expect we'd benefit from looking at all the
relevant ones at once, so as to not have the problem of hacking "one
more" into place and perhaps doing it a bit differently each time.

Sure, if anybody can think of any. A quick look at
"man SSL_CTX_set_options" didn't show me any, but then OpenSSL's
documentation is very bad (the page does not even mention
SSL_OP_NO_COMPRESSION) and I am no SSL expert.

Oh yeah, their docs are. Um. Yeah, let's leave it at that.

I think the other one is to control which encryption options are available
- but we already have a guc for that.

Is the following proposal acceptable:

- Add a GUC ssl_compression, defaulting to "on".
- Add a client option "sslcompression" and an environment variable
PGSSLCOMPRESSION, defaulting to "1".

Seems like the reasonable thing, yes.

Compression will be disabled if either side refuses.

I assume OpenSSL takes care of this for us, right? We just have to set the
flags on the connection?

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/