postgres disconnects on master after setting up replication

Started by Andy Hallover 4 years ago5 messagesgeneral
Jump to latest
#1Andy Hall
andyjohnhall@gmail.com

we have an odd issue where a java app using JDBC which has been
working fine has suddenly started disconnecting after we setup
replication from the database it was connecting to ( which is now the
primary in a replicating pair ) with errors such as the following...

CLIENT SIDE: DBNAME/logs/20210815-200601.20966.log:SessionManagerImpl
WARN 20210816-00:01:00.252 - A session lost connection with the
database. Disabling all sessions
SERVER SIDE: 2021-08-16 01:48:27 BST FATAL: unsupported frontend
protocol 1234.5680: server supports 1.0 to 3.0

( these are different times but you get the idea )

I appreciate this seems like a JDBC driver issue but everything has
been fine and we observe the errors only once we setup replication.

could there be some issue with streaming / WAL replication that could
be causing this ? we are running 9.6 by the way thanks.

#2Tom Lane
tgl@sss.pgh.pa.us
In reply to: Andy Hall (#1)
Re: postgres disconnects on master after setting up replication

Andy Hall <andyjohnhall@gmail.com> writes:

we have an odd issue where a java app using JDBC which has been
working fine has suddenly started disconnecting after we setup
replication from the database it was connecting to ( which is now the
primary in a replicating pair ) with errors such as the following...

CLIENT SIDE: DBNAME/logs/20210815-200601.20966.log:SessionManagerImpl
WARN 20210816-00:01:00.252 - A session lost connection with the
database. Disabling all sessions
SERVER SIDE: 2021-08-16 01:48:27 BST FATAL: unsupported frontend
protocol 1234.5680: server supports 1.0 to 3.0

Hmm. That "protocol version" is a GSS request:

#define NEGOTIATE_GSS_CODE PG_PROTOCOL(1234,5680)

So

1. You are using a server version that predates our GSS support.

2. On the other hand, the client knows GSS and is trying to use it.
The client should fall back to a non-GSS connection upon receiving
this error response. But ...

3. Something --- it's not very clear what --- is seeing the error
and going into an unwarranted panic.

I'd try to figure out what's issuing the "Disabling all sessions"
message and then filing a complaint with the authors of that.

regards, tom lane

#3Andy Hall
andyjohnhall@gmail.com
In reply to: Tom Lane (#2)
Re: postgres disconnects on master after setting up replication

thanks for the very clear explanation much appreciated shall take this
back to the team.

Show quoted text

On Mon, 16 Aug 2021 at 17:14, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Andy Hall <andyjohnhall@gmail.com> writes:

we have an odd issue where a java app using JDBC which has been
working fine has suddenly started disconnecting after we setup
replication from the database it was connecting to ( which is now the
primary in a replicating pair ) with errors such as the following...

CLIENT SIDE: DBNAME/logs/20210815-200601.20966.log:SessionManagerImpl
WARN 20210816-00:01:00.252 - A session lost connection with the
database. Disabling all sessions
SERVER SIDE: 2021-08-16 01:48:27 BST FATAL: unsupported frontend
protocol 1234.5680: server supports 1.0 to 3.0

Hmm. That "protocol version" is a GSS request:

#define NEGOTIATE_GSS_CODE PG_PROTOCOL(1234,5680)

So

1. You are using a server version that predates our GSS support.

2. On the other hand, the client knows GSS and is trying to use it.
The client should fall back to a non-GSS connection upon receiving
this error response. But ...

3. Something --- it's not very clear what --- is seeing the error
and going into an unwarranted panic.

I'd try to figure out what's issuing the "Disabling all sessions"
message and then filing a complaint with the authors of that.

regards, tom lane

#4Dave Cramer
pg@fastcrypt.com
In reply to: Andy Hall (#3)
Re: postgres disconnects on master after setting up replication

Dave Cramer
www.postgres.rocks

On Mon, 16 Aug 2021 at 12:32, Andy Hall <andyjohnhall@gmail.com> wrote:

thanks for the very clear explanation much appreciated shall take this
back to the team.

On Mon, 16 Aug 2021 at 17:14, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Andy Hall <andyjohnhall@gmail.com> writes:

we have an odd issue where a java app using JDBC which has been
working fine has suddenly started disconnecting after we setup
replication from the database it was connecting to ( which is now the
primary in a replicating pair ) with errors such as the following...

CLIENT SIDE: DBNAME/logs/20210815-200601.20966.log:SessionManagerImpl
WARN 20210816-00:01:00.252 - A session lost connection with the
database. Disabling all sessions
SERVER SIDE: 2021-08-16 01:48:27 BST FATAL: unsupported frontend
protocol 1234.5680: server supports 1.0 to 3.0

Hmm. That "protocol version" is a GSS request:

#define NEGOTIATE_GSS_CODE PG_PROTOCOL(1234,5680)

So

1. You are using a server version that predates our GSS support.

2. On the other hand, the client knows GSS and is trying to use it.
The client should fall back to a non-GSS connection upon receiving
this error response. But ...

3. Something --- it's not very clear what --- is seeing the error
and going into an unwarranted panic.

I'd try to figure out what's issuing the "Disabling all sessions"
message and then filing a complaint with the authors of that.

What version of the JDBC driver are you using? There was one version that
unfortunately blindly sent out the GSS request.
That has been rectified in later versions, I believe somewhere around
42.2.20 or so.

Although as Tom mentioned the driver does know how to deal with this,
however there are some cloud providers that chose to just drop the
connection

Dave

#5Andy Hall
andyjohnhall@gmail.com
In reply to: Dave Cramer (#4)
Re: postgres disconnects on master after setting up replication

yep we think this is exactly the issue...we have been testing a new
build with a later driver in UAT this morning and no recurrence of the
issue so looking most probable. thanks all for the very prompt and
clear responses it's most appreciated.

Show quoted text

On Tue, 17 Aug 2021 at 13:09, Dave Cramer <davecramer@postgres.rocks> wrote:

Dave Cramer
www.postgres.rocks

On Mon, 16 Aug 2021 at 12:32, Andy Hall <andyjohnhall@gmail.com> wrote:

thanks for the very clear explanation much appreciated shall take this
back to the team.

On Mon, 16 Aug 2021 at 17:14, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Andy Hall <andyjohnhall@gmail.com> writes:

we have an odd issue where a java app using JDBC which has been
working fine has suddenly started disconnecting after we setup
replication from the database it was connecting to ( which is now the
primary in a replicating pair ) with errors such as the following...

CLIENT SIDE: DBNAME/logs/20210815-200601.20966.log:SessionManagerImpl
WARN 20210816-00:01:00.252 - A session lost connection with the
database. Disabling all sessions
SERVER SIDE: 2021-08-16 01:48:27 BST FATAL: unsupported frontend
protocol 1234.5680: server supports 1.0 to 3.0

Hmm. That "protocol version" is a GSS request:

#define NEGOTIATE_GSS_CODE PG_PROTOCOL(1234,5680)

So

1. You are using a server version that predates our GSS support.

2. On the other hand, the client knows GSS and is trying to use it.
The client should fall back to a non-GSS connection upon receiving
this error response. But ...

3. Something --- it's not very clear what --- is seeing the error
and going into an unwarranted panic.

I'd try to figure out what's issuing the "Disabling all sessions"
message and then filing a complaint with the authors of that.

What version of the JDBC driver are you using? There was one version that unfortunately blindly sent out the GSS request.
That has been rectified in later versions, I believe somewhere around 42.2.20 or so.

Although as Tom mentioned the driver does know how to deal with this, however there are some cloud providers that chose to just drop the connection

Dave