Recent vendor SSL renegotiation patches break PostgreSQL
Greetings, hackers!
The flurry of patches that vendors have recently been making to OpenSSL to address the potential man-in-the-middle attack during SSL renegotiation have disabled SSL renegotiation altogether in the OpenSSL libraries. Applications that make use of SSL renegotiation, such as PostgreSQL, start failing.
I’ve noticed such failures on Mac OS X 10.6.2 after installing Security Update 2010-001 (which is when Apple distributed their OpenSSL patch):
http://support.apple.com/kb/HT4004
OpenSSL
CVE-ID: CVE-2009-3555
Available for: Mac OS X v10.5.8, Mac OS X Server v10.5.8, Mac OS X v10.6.2, Mac OS X Server v10.6.2
Impact: An attacker with a privileged network position may capture data or change the operations performed in sessions protected by SSL
Description: A man-in-the-middle vulnerability exists in the SSL and TLS protocols. Further information is available at http://www.phonefactor.com/sslgap A change to the renegotiation protocol is underway within the IETF. This update disables renegotiation in OpenSSL as a preventive security measure.
After installing Security Update 2010-001, any libpq connection to the server that exchanges more than 512MB of data (the RENEGOTIATION_LIMIT defined in src/backend/libpq/be-secure.c) will trigger an SSL renegotiation, which fails, which disconnects the client. I observed the problem on both PostgreSQL 8.1.19 and PostgreSQL 8.4.2 (those are the only versions I have in production).
I have been working around the problem by disabling SSL renegotiation entirely in my PostgreSQL servers, commenting out lines 316-339 in src/backend/libpq/be-secure.c.
There have been reports of such SSL-related breakage on other platforms, too:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=560205
Thanks! Happy hacking!
- Chris
On Wed, Feb 3, 2010 at 6:24 AM, Chris Campbell <chris_campbell@mac.com> wrote:
The flurry of patches that vendors have recently been making to OpenSSL to address
the potential man-in-the-middle attack during SSL renegotiation have disabled SSL
renegotiation altogether in the OpenSSL libraries. Applications that make use of SSL
renegotiation, such as PostgreSQL, start failing.
Should we think about adding a GUC to disable renegotiation until this
blows over?
...Robert
Robert Haas wrote:
On Wed, Feb 3, 2010 at 6:24 AM, Chris Campbell <chris_campbell@mac.com> wrote:
The flurry of patches that vendors have recently been making to OpenSSL to address
the potential man-in-the-middle attack during SSL renegotiation have disabled SSL
renegotiation altogether in the OpenSSL libraries. Applications that make use of SSL
renegotiation, such as PostgreSQL, start failing.Should we think about adding a GUC to disable renegotiation until this
blows over?
hmm I wonder if we should not go as far as removing the whole
renegotiation code, from the field it seems that there are very very few
daemons actually doing that kind forced renegotiation.
Stefan
On Feb 3, 2010, at 10:16 AM, Stefan Kaltenbrunner wrote:
Robert Haas wrote:
On Wed, Feb 3, 2010 at 6:24 AM, Chris Campbell <chris_campbell@mac.com> wrote:
The flurry of patches that vendors have recently been making to OpenSSL to address
the potential man-in-the-middle attack during SSL renegotiation have disabled SSL
renegotiation altogether in the OpenSSL libraries. Applications that make use of SSL
renegotiation, such as PostgreSQL, start failing.Should we think about adding a GUC to disable renegotiation until this
blows over?hmm I wonder if we should not go as far as removing the whole renegotiation code, from the field it seems that there are very very few daemons actually doing that kind forced renegotiation.
There was a discussion about the relevance and consequences of SSL renegotiation on this list back in 2003:
http://archives.postgresql.org/pgsql-interfaces/2003-04/msg00075.php
Personally, my production servers have been patched to remove renegotiation completely, and I’m comfortable with the consequences of that for my usage.
- Chris
Robert Haas <robertmhaas@gmail.com> writes:
Should we think about adding a GUC to disable renegotiation until this
blows over?
Bad idea: once set, it'll never get unset, thus leaving installations
with a weakened security posture even after they've installed fixed
versions of openssl.
regards, tom lane
On Wed, Feb 3, 2010 at 10:21 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Robert Haas <robertmhaas@gmail.com> writes:
Should we think about adding a GUC to disable renegotiation until this
blows over?Bad idea: once set, it'll never get unset, thus leaving installations
with a weakened security posture even after they've installed fixed
versions of openssl.
That's a problem, but our current posture of holding our breath
doesn't seem to be working either. If we insist on shipping code that
doesn't work with currently-distributed versions of OpenSSL, people
will do things like, say, shut SSL off. Or packagers of PostgreSQL
will apply patches that disable it unconditionally, leaving us with no
control.
...Robert
Is there a way to detect when the SSL library has renegotiation disabled? (Either at compile-time or runtime, although runtime would definitely be better because we’ll change our behavior if/when the user updates their SSL library.)
If so, we could skip renegotiation when it’s disabled in the library, but otherwise perform renegotiation like we normally do (every 512 MB, I think it is).
Also, the official OpenSSL patch provides a way for the application to re-enable renegotiation. I don’t think all implementations will do so, though (e.g., some vendors might have patched it differently).
- Chris
On Wed, Feb 3, 2010 at 10:21 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Bad idea: once set, it'll never get unset, thus leaving installations
with a weakened security posture even after they've installed fixed
versions of openssl.regards, tom lane
One might argue that the current method is already weakened as it is
measured by the amount of data sent instead of of a length of time. A
session could live a long time under the 512MB threshold depending on
the queries that are being performed.
Michael
Michael Ledford <mledford@gmail.com> writes:
One might argue that the current method is already weakened as it is
measured by the amount of data sent instead of of a length of time. A
session could live a long time under the 512MB threshold depending on
the queries that are being performed.
Renegotiation after X amount of data is the recommended method AFAIK,
because it limits the volume of data available to cryptanalysis.
What makes you think that elapsed time is relevant at all?
regards, tom lane
On Wed, Feb 3, 2010 at 11:09 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Renegotiation after X amount of data is the recommended method AFAIK,
because it limits the volume of data available to cryptanalysis.
What makes you think that elapsed time is relevant at all?regards, tom lane
You are correct. In that volume of data also matters. It depends on
what kind of attack you are trying to minimize here. In my particular
use case I fluctuate between idle and busy but mostly low bandwidth.
You have four different primary cases that you are possible:
1) timed method on an idle link (or low usage)
2) timed method on a busy link (or high usage)
3) data limit on an idle link (or low usage)
4) data limit on a busy link (or high usage)
The timed method is more optimal for case 1.
The data limit is more optimal for case 4.
Case 2 gives an attacker more data to do crypto-analysis.
Case 3 gives an attacker more time to work with the current secret key
on a live link.
Depending on your use case, being able to work with a live link is
worse than working with the data postmortem. There is I'm sure some
hybrid in the middle between these where optimal lives.
Michael
Chris Campbell <chris_campbell@mac.com> writes:
Is there a way to detect when the SSL library has renegotiation disabled?
Probably not. The current set of emergency security patches would
certainly not have exposed any new API that would help us tell this :-(
If said patches were done properly they'd have also turned an
application-level renegotiation request into a no-op, instead of
breaking apps by making it fail --- but apparently they were not done
properly.
regards, tom lane
On Wed, Feb 3, 2010 at 11:52 AM, Michael Ledford <mledford@gmail.com> wrote:
On Wed, Feb 3, 2010 at 11:09 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Renegotiation after X amount of data is the recommended method AFAIK,
because it limits the volume of data available to cryptanalysis.
What makes you think that elapsed time is relevant at all?You are correct. In that volume of data also matters. It depends on
what kind of attack you are trying to minimize here. In my particular
use case I fluctuate between idle and busy but mostly low bandwidth.You have four different primary cases that you are possible:
This may all be true, but I think we're getting off track. If we
force ANY negotiation (whether based on time or bytes transferred), we
will, apparently, break things. So I think that means we should have
a way to disable that behavior, for fear of dissuading people from
using SSL (or PostgreSQL) altogether, or hacking their own copies of
the source in ways that may be even uglier.
...Robert
Tom Lane wrote:
Chris Campbell <chris_campbell@mac.com> writes:
Is there a way to detect when the SSL library has renegotiation disabled?
Probably not. The current set of emergency security patches would
certainly not have exposed any new API that would help us tell this :-(If said patches were done properly they'd have also turned an
application-level renegotiation request into a no-op, instead of
breaking apps by making it fail --- but apparently they were not done
properly.
Yea, and also keep in mind any SSL library checks need to be done at
run-time (because I believe openssl is usually linked as a shared
object), which even further limits our options.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Tom Lane escribi�:
Michael Ledford <mledford@gmail.com> writes:
One might argue that the current method is already weakened as it is
measured by the amount of data sent instead of of a length of time. A
session could live a long time under the 512MB threshold depending on
the queries that are being performed.Renegotiation after X amount of data is the recommended method AFAIK,
because it limits the volume of data available to cryptanalysis.
What makes you think that elapsed time is relevant at all?
FWIW I think there's another problem with streaming replication here,
which is that most data flows from client to server, so it would take
quite some time for the threshold to be reached. Note that there's no
size check in the libpq frontend code. Normally this is not an issue
because the bulk of data is expected to flow in the other direction.
--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.
Alvaro Herrera <alvherre@commandprompt.com> writes:
FWIW I think there's another problem with streaming replication here,
which is that most data flows from client to server, so it would take
quite some time for the threshold to be reached. Note that there's no
size check in the libpq frontend code. Normally this is not an issue
because the bulk of data is expected to flow in the other direction.
Huh? I thought the slaves connect to the master, rather than the other
way round?
It's true that libpq doesn't contain any such code, but that seems like
a fortunate thing right at the moment, as it limits the number of places
we might have to hack something.
regards, tom lane
Tom Lane wrote:
Alvaro Herrera <alvherre@commandprompt.com> writes:
FWIW I think there's another problem with streaming replication here,
which is that most data flows from client to server, so it would take
quite some time for the threshold to be reached. Note that there's no
size check in the libpq frontend code. Normally this is not an issue
because the bulk of data is expected to flow in the other direction.Huh? I thought the slaves connect to the master, rather than the other
way round?
Correct, slave connects to the master.
Alvaro is pointing out that most data flows from client to server, like
in COPY FROM. But the server counts both in- and out-going data against
the renegotiation limit, so the server will initiate renegotiation just
fine in that case too.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
Tom Lane wrote:
Chris Campbell <chris_campbell@mac.com> writes:
Is there a way to detect when the SSL library has renegotiation disabled?
Probably not. The current set of emergency security patches would
certainly not have exposed any new API that would help us tell this :-(If said patches were done properly they'd have also turned an
application-level renegotiation request into a no-op, instead of
breaking apps by making it fail --- but apparently they were not done
properly.
Is there anything remaining to do on this issue?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
PG East: http://www.enterprisedb.com/community/nav-pg-east-2010.do
+ If your life is a hard drive, Christ can be your backup. +Bruce Momjian <bruce@momjian.us> writes:
Tom Lane wrote:
Chris Campbell <chris_campbell@mac.com> writes:
Is there a way to detect when the SSL library has renegotiation disabled?
Probably not. The current set of emergency security patches would
certainly not have exposed any new API that would help us tell this :-(If said patches were done properly they'd have also turned an
application-level renegotiation request into a no-op, instead of
breaking apps by making it fail --- but apparently they were not done
properly.
Is there anything remaining to do on this issue?
I'm not sure. My impression is that by the time we had anything in the
field, there will be real fixes for the SSL renegotiation problem.
So all we'd be accomplishing is to weaken security for people who have
those fixes, to cater to people who are using copies of openssl they'd
obtained in the past couple of months and then not updated to latest.
However, if anyone thinks that the SSL problem isn't going to get fixed
promptly, maybe it needs more consideration.
regards, tom lane
2010/2/20 Tom Lane <tgl@sss.pgh.pa.us>:
Bruce Momjian <bruce@momjian.us> writes:
Tom Lane wrote:
Chris Campbell <chris_campbell@mac.com> writes:
Is there a way to detect when the SSL library has renegotiation disabled?
Probably not. The current set of emergency security patches would
certainly not have exposed any new API that would help us tell this :-(If said patches were done properly they'd have also turned an
application-level renegotiation request into a no-op, instead of
breaking apps by making it fail --- but apparently they were not done
properly.Is there anything remaining to do on this issue?
I'm not sure. My impression is that by the time we had anything in the
field, there will be real fixes for the SSL renegotiation problem.
So all we'd be accomplishing is to weaken security for people who have
those fixes, to cater to people who are using copies of openssl they'd
obtained in the past couple of months and then not updated to latest.
However, if anyone thinks that the SSL problem isn't going to get fixed
promptly, maybe it needs more consideration.
The problem with this is, I think, that there are "semi-patched"
versions of OpenSSL out there, that simply *break*. Instead of not
doing renegotiation when we ask for it, they break it and return an
error, thus canceling the connection. Correct?
If so, shouldn't we try to disable renegotiation for all versions
*before* it was properly fixed?
Which today means all versions released. The proper fix is in 0.9.8m,
which is currently in beta. At least that's my understanding.
The way I read it, the code now does:
* If the user has an old version of openssl, exposes the user to the
security issue and makes the connection insecure
* OpenSSL 0.9.8l, breaks the connection whenever renegotiation
happens, but doesn't compromise security
* OpenSSL 0.9.8m-beta, does the right thing.
Do we have any idea of what people like RHEL are doing wrt
backpatching these things?
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
Magnus Hagander <magnus@hagander.net> writes:
If so, shouldn't we try to disable renegotiation for all versions
*before* it was properly fixed?
If we could tell that, sure. But I don't believe there is any way to
identify whether a given installation of openssl has this patched.
Please don't suggest looking at the version number --- Red Hat and
other vendors are in the habit of back-patching security fixes without
changing the version number.
Which today means all versions released. The proper fix is in 0.9.8m,
which is currently in beta. At least that's my understanding.
Red Hat's already shipping the patch. Dunno about other vendors.
The real bottom line here is that this isn't our bug. It's unfortunate
that we're affected by it, but that doesn't mean that we should be
installing kluges to work around it.
regards, tom lane
