BUG #12769: SSL-Renegotiation failures
The following bug has been logged on the website:
Bug reference: 12769
Logged by: Arne Scheffer
Email address: pilum.70@uni-muenster.de
PostgreSQL version: 9.3.6
Operating system: CentOS6, openssl version OpenSSL 1.0.1e-fips
Description:
I get ssl renegotiation failures with streaming standbys. Sometimes the
connection breaks and is reconnected afterwards. However, if I use
pg_basebackup (same libpq connection string), I don't get any of these
failures, allthough the transferred data ist far beyond 512 MB
So I don't think it's the
ssl renegotiation bug (openssl of a yum update patched centos6)
If I disable ssl_renegotiation_limit to 0, there are no errors any more,
but that is only a workaround, no solution.
Steps to reproduce with testdata:
Tested on patched centos6, Postgresql 9.3.6
bash-4.1$ openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013
Install Repository
yum install postgresql93 postgresql93-server postgresql93-contrib
postgresql93-devel postgresql93-libs
/usr/pgsql-9.3/bin/initdb -D /tmp/data1
Not necessary but comfortable: create two postgresql-93test Init-Scripts (cp
/etc/init.d/postgresql-9.3 postgresql-93test)
- changing data-Directory to /tmp/data1 und /tmp/data2
- changing PGPORT to yourport and yourport2
Start server in data1
On server in data1:
create user replicateme with replication encrypted password "insecure";
<#####
postgresql.conf:
ssl = on
listen_addresses = 'yourserver.domain.com'
wal_level = hot_standby
# 4GB fuer WAL-Segmente:
wal_keep_segments = 256
max_wal_senders = 2
#####>
Certificates installed
<##### only 1 line:
pg_hba.conf:
hostssl replication ...
#####>
usr/pgsql-9.3/bin/pg_basebackup -D /tmp/data2 -v -P -Fp -l testsslbackup -x
-d "host=yourserver.domain.com port=yourport user=replicator
sslmode=verify-full"
<####
recovery.conf:
recovery_target_timeline = 'latest'
standby_mode = 'on'
primary_conninfo = 'host=yourserver.domain.com port=yourport user=replicator
sslmode=verify-full'
#####>
Start standby server in data2
On server in data1:
create database test
\c test
create table test2 (a int, b text);
insert into test2 (select generate_series(0,5000000,1));
update test2 set
b='sljfdlksjaofjsaolfjsdlkofjdolasjfoldsjfoldsjaolfjdloksjfolajsdflkjsdlokfjslojdalkfjoldsjfoslafjolasljfdloas';
tail -f /tmp/data1/pg_log/postgresql-Fri.log
[...]
< 2015-02-13 18:09:55.964 CET >LOG: SSL renegotiation failure
[...]
< 2015-02-13 18:10:11.819 CET >LOG: SSL renegotiation failure
[...]
usr/pgsql-9.3/bin/pg_basebackup -D /tmp/data3 -v -P -Fp -l testsslbackup2 -x
-d "host=yourserver.domain.com port=yourport user=replicator
sslmode=verify-full"
tail -f /tmp/data1/pg_log/postgresql-Fri.log
--> nothing!
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
Hi Arne,
On 2015-02-13 18:52:02 +0000, pilum.70@uni-muenster.de wrote:
The following bug has been logged on the website:
Bug reference: 12769
Logged by: Arne Scheffer
Email address: pilum.70@uni-muenster.de
PostgreSQL version: 9.3.6
Operating system: CentOS6, openssl version OpenSSL 1.0.1e-fips
Description:I get ssl renegotiation failures with streaming standbys. Sometimes the
connection breaks and is reconnected afterwards. However, if I use
pg_basebackup (same libpq connection string), I don't get any of these
failures, allthough the transferred data ist far beyond 512 MB
So I don't think it's the
ssl renegotiation bug (openssl of a yum update patched centos6)
If I disable ssl_renegotiation_limit to 0, there are no errors any more,
but that is only a workaround, no solution.
Heikki and me have recently investigated problems around SSL
renegotiation. See
/messages/by-id/20150126101405.GA31719@awork2.anarazel.de
.
Greetings,
Andres Freund
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
On 02/13/2015 10:59 PM, Andres Freund wrote:
On 2015-02-13 18:52:02 +0000, pilum.70@uni-muenster.de wrote:
I get ssl renegotiation failures with streaming standbys. Sometimes the
connection breaks and is reconnected afterwards. However, if I use
pg_basebackup (same libpq connection string), I don't get any of these
failures, allthough the transferred data ist far beyond 512 MB
So I don't think it's the
ssl renegotiation bug (openssl of a yum update patched centos6)
If I disable ssl_renegotiation_limit to 0, there are no errors any more,
but that is only a workaround, no solution.Heikki and me have recently investigated problems around SSL
renegotiation. See
/messages/by-id/20150126101405.GA31719@awork2.anarazel.de
.
I wasn't able to reproduce exactly the same error you saw, Arne, so it
would be good if you could test the patches I've been developing, to see
if they fix your problem too. That is, patches 0001 and 0002 from
/messages/by-id/54DE6FAF.6050005@vmware.com. Could
you do that?
- Heikki
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
On the same machine (CentOS6)
Cloned from 9.5devel,
./configure --with-perl --with-openssl --with-python --with-tcl --with-pam --with-ldap --enable-thread-safety --enable-debug
make make install make clean
Repeated the procedure attached in the mail
Got a different error (also twice at exected renegotiation times):
< 2015-02-15 16:40:45.438 CET >LOG: SSL error: session id context uninitialized
< 2015-02-15 16:40:45.439 CET >LOG: could not receive data from client: Connection reset by peer
< 2015-02-15 16:40:45.439 CET >LOG: unexpected EOF on standby connection
Tried
git checkout -b ssl_patch
patch -p1 <../0001-Fix-sslv3-alert-unexpected-message-errors-in-SSL-ren.patch
(got applied on 2 files)
patch -p1 <../0002-Also-drain-input-buffer-in-non-blocking-mode-if-send.patch
(got applied on 1 file)
Repeated make make install make clean
Repeated the procedure attached in the mail.
(Both twice.)
Got the same errors.
Perhaps I did something wrong. Could you add a temporary debug line, so that I can
see, that the patch is really applied in my environment?
Even tried 0003, but no change.
Patch expectedly doesn't apply on 9.3.6:
[root@zivwebapp13 postgresql-9.3.6patched]# patch -p1
<../0001-Fix-sslv3-alert-unexpected-message-errors-in-SSL-ren.patch
patching file src/interfaces/libpq/fe-misc.c
Hunk #1 succeeded at 919 (offset -1 lines).
can't find file to patch at input line 45
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/src/interfaces/libpq/fe-secure-openssl.c
b/src/interfaces/libpq/fe-secure-openssl.c
|index a32af34..93b8184 100644
|--- a/src/interfaces/libpq/fe-secure-openssl.c
|+++ b/src/interfaces/libpq/fe-secure-openssl.c
--------------------------
I would also test backpatched patch code once it's made.
VlG
Arne
On Sat, 14 Feb 2015, Heikki Linnakangas wrote:
On 02/13/2015 10:59 PM, Andres Freund wrote:
On 2015-02-13 18:52:02 +0000, pilum.70@uni-muenster.de wrote:
I get ssl renegotiation failures with streaming standbys. Sometimes the
connection breaks and is reconnected afterwards. However, if I use
pg_basebackup (same libpq connection string), I don't get any of these
failures, allthough the transferred data ist far beyond 512 MB
So I don't think it's the
ssl renegotiation bug (openssl of a yum update patched centos6)
If I disable ssl_renegotiation_limit to 0, there are no errors any more,
but that is only a workaround, no solution.Heikki and me have recently investigated problems around SSL
renegotiation. See
/messages/by-id/20150126101405.GA31719@awork2.anarazel.de
.I wasn't able to reproduce exactly the same error you saw, Arne, so it would
be good if you could test the patches I've been developing, to see if they
fix your problem too. That is, patches 0001 and 0002 from
/messages/by-id/54DE6FAF.6050005@vmware.com. Could you
do that?- Heikki
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
Retried my test suite after commit
1c2b7c0879d83ff79e4adf2c0a883df92b713da4 Restore the SSL_set_session_id_context() call to OpenSS...
Cloned again from 9.5devel master.
Now the two patches work as expected (Means: errors without patch, no errors
with patch (I can't see, whether the code is really performing renegotiation)).
That seems promising.
As already mentioned:
I would also test the two patches for 9.3
in a production near environment to prove them fixing the reported Bug.
If that isn't desired,
I will clone them from REL9_3_STABLE branch once they are committed.
VlG-Arne
On Sun, 15 Feb 2015, Arne Scheffer wrote:
On the same machine (CentOS6)
Cloned from 9.5devel,
./configure --with-perl --with-openssl --with-python --with-tcl --with-pam
--with-ldap --enable-thread-safety --enable-debug
make make install make cleanRepeated the procedure attached in the mail
Got a different error (also twice at exected renegotiation times):< 2015-02-15 16:40:45.438 CET >LOG: SSL error: session id context
uninitialized
< 2015-02-15 16:40:45.439 CET >LOG: could not receive data from client:
Connection reset by peer
< 2015-02-15 16:40:45.439 CET >LOG: unexpected EOF on standby connectionTried
git checkout -b ssl_patch
patch -p1 <../0001-Fix-sslv3-alert-unexpected-message-errors-in-SSL-ren.patch
(got applied on 2 files)
patch -p1 <../0002-Also-drain-input-buffer-in-non-blocking-mode-if-send.patch
(got applied on 1 file)Repeated make make install make clean
Repeated the procedure attached in the mail.
(Both twice.)
Got the same errors.
Perhaps I did something wrong. Could you add a temporary debug line, so that
I can
see, that the patch is really applied in my environment?Even tried 0003, but no change.
Patch expectedly doesn't apply on 9.3.6:
[root@zivwebapp13 postgresql-9.3.6patched]# patch -p1
<../0001-Fix-sslv3-alert-unexpected-message-errors-in-SSL-ren.patch patching
file src/interfaces/libpq/fe-misc.c
Hunk #1 succeeded at 919 (offset -1 lines).
can't find file to patch at input line 45
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/src/interfaces/libpq/fe-secure-openssl.c
b/src/interfaces/libpq/fe-secure-openssl.c
|index a32af34..93b8184 100644
|--- a/src/interfaces/libpq/fe-secure-openssl.c
|+++ b/src/interfaces/libpq/fe-secure-openssl.c
--------------------------I would also test backpatched patch code once it's made.
VlG
Arne
On Sat, 14 Feb 2015, Heikki Linnakangas wrote:
On 02/13/2015 10:59 PM, Andres Freund wrote:
On 2015-02-13 18:52:02 +0000, pilum.70@uni-muenster.de wrote:
I get ssl renegotiation failures with streaming standbys. Sometimes the
connection breaks and is reconnected afterwards. However, if I use
pg_basebackup (same libpq connection string), I don't get any of these
failures, allthough the transferred data ist far beyond 512 MB
So I don't think it's the
ssl renegotiation bug (openssl of a yum update patched centos6)
If I disable ssl_renegotiation_limit to 0, there are no errors any more,
but that is only a workaround, no solution.Heikki and me have recently investigated problems around SSL
renegotiation. See
/messages/by-id/20150126101405.GA31719@awork2.anarazel.de
.I wasn't able to reproduce exactly the same error you saw, Arne, so it
would be good if you could test the patches I've been developing, to see if
they fix your problem too. That is, patches 0001 and 0002 from
/messages/by-id/54DE6FAF.6050005@vmware.com. Could you
do that?- Heikki
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
On 02/19/2015 05:06 PM, Arne Scheffer wrote:
Retried my test suite after commit
1c2b7c0879d83ff79e4adf2c0a883df92b713da4 Restore the SSL_set_session_id_context() call to OpenSS...
Cloned again from 9.5devel master.
Now the two patches work as expected (Means: errors without patch, no errors
with patch (I can't see, whether the code is really performing renegotiation)).
That seems promising.As already mentioned:
I would also test the two patches for 9.3
in a production near environment to prove them fixing the reported Bug.
If that isn't desired,
I will clone them from REL9_3_STABLE branch once they are committed.
Thanks, and sorry for not responding sooner. I investigated this on
Tuesday, and committed that fix. I was able to reproduce the problem you
saw on a Centos 6 VM, and also concluded that it now works on master
with those two patches applied. I tried backpatching them to 9.4, but I
still saw some errors on 9.4 even with the patches. I ran out of time to
dig deepeer, but it seems that some of the other changes done in master
are needed to fully fix this. There has been a lot of changes between
9.4 and master, and some changes between 9.3 and 9.4 as well, so it's
probably going to take some time to come up with correct fixes for all
the back-branches.
- Heikki
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
Hi,
On Thu, 19 Feb 2015, Heikki Linnakangas wrote:
On 02/19/2015 05:06 PM, Arne Scheffer wrote:
Retried my test suite after commit
1c2b7c0879d83ff79e4adf2c0a883df92b713da4 Restore the
SSL_set_session_id_context() call to OpenSS...Cloned again from 9.5devel master.
Now the two patches work as expected (Means: errors without patch, no
errors
with patch (I can't see, whether the code is really performing
renegotiation)).
That seems promising.As already mentioned:
I would also test the two patches for 9.3
in a production near environment to prove them fixing the reported Bug.
If that isn't desired,
I will clone them from REL9_3_STABLE branch once they are committed.Thanks, and sorry for not responding sooner. I investigated this on Tuesday,
and committed that fix. I was able to reproduce the problem you saw on a
Centos 6 VM, and also concluded that it now works on master with those two
patches applied. I tried backpatching them to 9.4, but I still saw some
errors on 9.4 even with the patches. I ran out of time to dig deepeer, but it
seems that some of the other changes done in master are needed to fully fix
this. There has been a lot of changes between 9.4 and master, and some
changes between 9.3 and 9.4 as well, so it's probably going to take some time
to come up with correct fixes for all the back-branches.
Any progress fixing this bug?
If not, wouldn't it make sense to (at least) include the Fixes into 9.5?
VlG
Arne Scheffer
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs