Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing

Started by Lukas Meisegeierabout 5 years ago8 messages
#1Lukas Meisegeier
MeisegeierLukas@gmx.de
1 attachment(s)

Hi,

I try to host multiple postgresql-servers on the same ip and the same
port through SNI-based load-balancing.
Currently this is not possible because of two issues:
1. The psql client won't set the tls-sni-extension correctly
(/messages/by-id/20181211145240.GL20222@redhat.com)
2. The psql connection protocol implements a SSLRequest in plain text
before actually opening a connection.

The first issue is easily solvable by calling
`SSL_set_tlsext_host_name(conn->ssl,
conn->connhost[conn->whichhost].host)` before opening the connection.

The second issue is also solvable through a new parameter
"ssltermination" which if set to "proxy" will skip the initial
SSLRequest and connects directly through ssl.
The default value would be "server" which changes nothing on the
existing behaviour.

I compiled the psql-client with these changes and was able to connect to
2 different databases through the same ip and port just by changing the
hostname.

This fix is important to allow multiple postgres instances on one ip
without having to add a port number.

I implemented this change on a fork of the postgres mirror on github:
https://github.com/klg71/mayope_postgres

The affected files are:
- src/interfaces/libpq/fe-connect.c (added ssltermination parameter)
- src/interfaces/libpq/libpq-int.h (added ssltermination parameter)
- src/interfaces/libpq/fe-secure-openssl.c (added tls-sni-extension)

I appended the relevant diff.

Best Regards
Lukas

Attachments:

difftext/plain; charset=UTF-16LE; name=diffDownload
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 7d04d3664e..43fcfc2274 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -131,6 +131,7 @@ static int	ldapServiceLookup(const char *purl, PQconninfoOption *options,
 #define DefaultTargetSessionAttrs	"any"
 #ifdef USE_SSL
 #define DefaultSSLMode "prefer"
+#define DefaultSSLTermination "server"
 #else
 #define DefaultSSLMode	"disable"
 #endif
@@ -293,6 +294,11 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
 		"SSL-Mode", "", 12,		/* sizeof("verify-full") == 12 */
 	offsetof(struct pg_conn, sslmode)},
 
+	{"ssltermination", "PGSSLTERMINATION", DefaultSSLMode, NULL,
+		"SSL-Termination-Mode", "", 6,		/* sizeof("server") == 6 */
+	offsetof(struct pg_conn, ssltermination)},
+
+
 	{"sslcompression", "PGSSLCOMPRESSION", "0", NULL,
 		"SSL-Compression", "", 1,
 	offsetof(struct pg_conn, sslcompression)},
@@ -1278,6 +1284,16 @@ connectOptions2(PGconn *conn)
 			return false;
 		}
 
+		if (strcmp(conn->ssltermination, "server") != 0
+			&& strcmp(conn->ssltermination, "proxy") != 0)
+		{
+			conn->status = CONNECTION_BAD;
+			printfPQExpBuffer(&conn->errorMessage,
+							  libpq_gettext("invalid %s value: \"%s\"\n"),
+							  "ssltermination", conn->ssltermination);
+			return false;
+		}
+
 #ifndef USE_SSL
 		switch (conn->sslmode[0])
 		{
@@ -2915,6 +2931,13 @@ keep_going:						/* We will come back to here until there is
 				if (conn->allow_ssl_try && !conn->wait_ssl_try &&
 					!conn->ssl_in_use)
 				{
+					/*
+					* SSL termination is handled by proxy/load-balancer, no need to send SSLRequest
+					*/
+					if(conn->ssltermination[0]=='p'){
+						conn->status = CONNECTION_SSL_STARTUP;
+						return PGRES_POLLING_WRITING;
+					}
 					ProtocolVersion pv;
 
 					/*
@@ -2995,77 +3018,89 @@ keep_going:						/* We will come back to here until there is
 				if (!conn->ssl_in_use)
 				{
 					/*
-					 * We use pqReadData here since it has the logic to
-					 * distinguish no-data-yet from connection closure. Since
-					 * conn->ssl isn't set, a plain recv() will occur.
+					 * Skip SSLRequest package and initialize ssl directly with proxy
 					 */
-					char		SSLok;
-					int			rdresult;
-
-					rdresult = pqReadData(conn);
-					if (rdresult < 0)
-					{
-						/* errorMessage is already filled in */
-						goto error_return;
-					}
-					if (rdresult == 0)
-					{
-						/* caller failed to wait for data */
-						return PGRES_POLLING_READING;
-					}
-					if (pqGetc(&SSLok, conn) < 0)
-					{
-						/* should not happen really */
-						return PGRES_POLLING_READING;
-					}
-					if (SSLok == 'S')
+					if (conn->ssltermination[0]=='p')
 					{
 						/* mark byte consumed */
 						conn->inStart = conn->inCursor;
 						/* Set up global SSL state if required */
 						if (pqsecure_initialize(conn) != 0)
 							goto error_return;
-					}
-					else if (SSLok == 'N')
-					{
-						/* mark byte consumed */
-						conn->inStart = conn->inCursor;
-						/* OK to do without SSL? */
-						if (conn->sslmode[0] == 'r' ||	/* "require" */
-							conn->sslmode[0] == 'v')	/* "verify-ca" or
-														 * "verify-full" */
+					} else {
+						/*
+						* We use pqReadData here since it has the logic to
+						* distinguish no-data-yet from connection closure. Since
+						* conn->ssl isn't set, a plain recv() will occur.
+						*/
+						char		SSLok;
+						int			rdresult;
+
+						rdresult = pqReadData(conn);
+						if (rdresult < 0)
 						{
-							/* Require SSL, but server does not want it */
-							appendPQExpBufferStr(&conn->errorMessage,
-												 libpq_gettext("server does not support SSL, but SSL was required\n"));
+							/* errorMessage is already filled in */
+							goto error_return;
+						}
+						if (rdresult == 0)
+						{
+							/* caller failed to wait for data */
+							return PGRES_POLLING_READING;
+						}
+						if (pqGetc(&SSLok, conn) < 0)
+						{
+							/* should not happen really */
+							return PGRES_POLLING_READING;
+						}
+						if (SSLok == 'S')
+						{
+							/* mark byte consumed */
+							conn->inStart = conn->inCursor;
+							/* Set up global SSL state if required */
+							if (pqsecure_initialize(conn) != 0)
+								goto error_return;
+						}
+						else if (SSLok == 'N')
+						{
+							/* mark byte consumed */
+							conn->inStart = conn->inCursor;
+							/* OK to do without SSL? */
+							if (conn->sslmode[0] == 'r' ||	/* "require" */
+								conn->sslmode[0] == 'v')	/* "verify-ca" or
+															* "verify-full" */
+							{
+								/* Require SSL, but server does not want it */
+								appendPQExpBufferStr(&conn->errorMessage,
+													libpq_gettext("server does not support SSL, but SSL was required\n"));
+								goto error_return;
+							}
+							/* Otherwise, proceed with normal startup */
+							conn->allow_ssl_try = false;
+							conn->status = CONNECTION_MADE;
+							return PGRES_POLLING_WRITING;
+						}
+						else if (SSLok == 'E')
+						{
+							/*
+							* Server failure of some sort, such as failure to
+							* fork a backend process.  We need to process and
+							* report the error message, which might be formatted
+							* according to either protocol 2 or protocol 3.
+							* Rather than duplicate the code for that, we flip
+							* into AWAITING_RESPONSE state and let the code there
+							* deal with it.  Note we have *not* consumed the "E"
+							* byte here.
+							*/
+							conn->status = CONNECTION_AWAITING_RESPONSE;
+							goto keep_going;
+						}
+						else
+						{
+							appendPQExpBuffer(&conn->errorMessage,
+											libpq_gettext("received invalid response to SSL negotiation: %c\n"),
+											SSLok);
 							goto error_return;
 						}
-						/* Otherwise, proceed with normal startup */
-						conn->allow_ssl_try = false;
-						conn->status = CONNECTION_MADE;
-						return PGRES_POLLING_WRITING;
-					}
-					else if (SSLok == 'E')
-					{
-						/*
-						 * Server failure of some sort, such as failure to
-						 * fork a backend process.  We need to process and
-						 * report the error message, which might be formatted
-						 * according to either protocol 2 or protocol 3.
-						 * Rather than duplicate the code for that, we flip
-						 * into AWAITING_RESPONSE state and let the code there
-						 * deal with it.  Note we have *not* consumed the "E"
-						 * byte here.
-						 */
-						conn->status = CONNECTION_AWAITING_RESPONSE;
-						goto keep_going;
-					}
-					else
-					{
-						appendPQExpBuffer(&conn->errorMessage,
-										  libpq_gettext("received invalid response to SSL negotiation: %c\n"),
-										  SSLok);
-						goto error_return;
 					}
 				}
 
diff --git a/src/interfaces/libpq/fe-secure-openssl.c b/src/interfaces/libpq/fe-secure-openssl.c
index d609a38bbe..d05d8a78f4 100644
--- a/src/interfaces/libpq/fe-secure-openssl.c
+++ b/src/interfaces/libpq/fe-secure-openssl.c
@@ -1044,6 +1044,7 @@ initialize_SSL(PGconn *conn)
 	 * it doesn't really matter.)
 	 */
 	if (!(conn->ssl = SSL_new(SSL_context)) ||
+	    !SSL_set_tlsext_host_name(conn->ssl, conn->connhost[conn->whichhost].host) ||
 		!SSL_set_app_data(conn->ssl, conn) ||
 		!my_SSL_set_fd(conn, conn->sock))
 	{
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 1de91ae295..7f5b03d518 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -356,6 +356,7 @@ struct pg_conn
 	char	   *keepalives_count;	/* maximum number of TCP keepalive
 									 * retransmits */
 	char	   *sslmode;		/* SSL mode (require,prefer,allow,disable) */
+	char	   *ssltermination;		/* SSL termination (proxy,server) */
 	char	   *sslcompression; /* SSL compression (0 or 1) */
 	char	   *sslkey;			/* client key filename */
 	char	   *sslcert;		/* client certificate filename */
#2Heikki Linnakangas
hlinnaka@iki.fi
In reply to: Lukas Meisegeier (#1)
Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing

On 10/12/2020 17:49, Lukas Meisegeier wrote:

I try to host multiple postgresql-servers on the same ip and the same
port through SNI-based load-balancing.
Currently this is not possible because of two issues:
1. The psql client won't set the tls-sni-extension correctly
(/messages/by-id/20181211145240.GL20222@redhat.com)
2. The psql connection protocol implements a SSLRequest in plain text
before actually opening a connection.

The first issue is easily solvable by calling
`SSL_set_tlsext_host_name(conn->ssl,
conn->connhost[conn->whichhost].host)` before opening the connection.

The second issue is also solvable through a new parameter
"ssltermination" which if set to "proxy" will skip the initial
SSLRequest and connects directly through ssl.
The default value would be "server" which changes nothing on the
existing behaviour.

Don't you need backend changes as well? The backend will still expect
the client to send an SSLRequest. Or is the connection from the proxy to
the actual server unencrypted?

It's not very nice that the client needs to set special options,
depending on whether the server is behind a proxy or not. Could you
teach the proxy to deal with the SSLRequest message?

Perhaps we should teach the backend to accept a TLS ClientHello
directly, without the SSLRequest message. That way, the client could
send the ClientHello without SSLRequest, and the proxy wouldn't need to
care about SSLRequest. It would also eliminate one round-trip from the
protocol handshake, which would be nice. A long deprecation/transition
period would be needed before we could make that the default behavior,
but that's ok.

- Heikki

#3Lukas Meisegeier
MeisegeierLukas@gmx.de
In reply to: Heikki Linnakangas (#2)
Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing

Hey Heikki,

thanks for providing feedback :)
The traffic between proxy and psql-server is unencrypted thats why I
don't need to patch the server.
I tried returning a fixed response on the first plain SSLRequest
forwarding it to a psql-server with ssl enabled an tried to switch then
on the ssl connection startup but that didn't work out. I guess its
because the psql-server won't accept an ssl connection if its not
requested via SSLRequest.
I would definitly appreciate if the psql-server could accept the
TLS-client hello directly but we would still need to set the
tls-sni-extension correctly.
Perhaps we could rename the parameter to "sslplainrequest(yes/no)" and
start with making the plain SSLRequest optional in the psql-server.

Best Regards
Lukas

Am 11-Dec-20 um 14:26 schrieb Heikki Linnakangas:

Show quoted text

On 10/12/2020 17:49, Lukas Meisegeier wrote:

I try to host multiple postgresql-servers on the same ip and the same
port through SNI-based load-balancing.
Currently this is not possible because of two issues:
1. The psql client won't set the tls-sni-extension correctly
(/messages/by-id/20181211145240.GL20222@redhat.com)

2. The psql connection protocol implements a SSLRequest in plain text
before actually opening a connection.

The first issue is easily solvable by calling
`SSL_set_tlsext_host_name(conn->ssl,
conn->connhost[conn->whichhost].host)` before opening the connection.

The second issue is also solvable through a new parameter
"ssltermination" which if set to "proxy" will skip the initial
SSLRequest and connects directly through ssl.
The default value would be "server" which changes nothing on the
existing behaviour.

Don't you need backend changes as well? The backend will still expect
the client to send an SSLRequest. Or is the connection from the proxy to
the actual server unencrypted?

It's not very nice that the client needs to set special options,
depending on whether the server is behind a proxy or not. Could you
teach the proxy to deal with the SSLRequest message?

Perhaps we should teach the backend to accept a TLS ClientHello
directly, without the SSLRequest message. That way, the client could
send the ClientHello without SSLRequest, and the proxy wouldn't need to
care about SSLRequest. It would also eliminate one round-trip from the
protocol handshake, which would be nice. A long deprecation/transition
period would be needed before we could make that the default behavior,
but that's ok.

- Heikki

#4Heikki Linnakangas
hlinnaka@iki.fi
In reply to: Lukas Meisegeier (#3)
Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing

On 11/12/2020 16:46, Lukas Meisegeier wrote:

Hey Heikki,

thanks for providing feedback :)
The traffic between proxy and psql-server is unencrypted thats why I
don't need to patch the server.

Ok.

I tried returning a fixed response on the first plain SSLRequest
forwarding it to a psql-server with ssl enabled an tried to switch then
on the ssl connection startup but that didn't work out. I guess its
because the psql-server won't accept an ssl connection if its not
requested via SSLRequest.

Your proxy could receive the client's SSLRequest message, and respond
with a single byte 'S'. You don't need to forward that to the real
PostgreSQL server, since the connection to the PostgreSQL server is
unencrypted. Then perform the TLS handshake, and forward all traffic to
the real server only after that.

Client: -> SSLRequest
Proxy: <- 'S'
Client: -> TLS ClientHello
Proxy: [finish TLS handshake]

- Heikki

#5Lukas Meisegeier
MeisegeierLukas@gmx.de
In reply to: Heikki Linnakangas (#4)
Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing

Thanks for the provided ideas :)
I use HaProxy for my load-balancing and unfortunately I can't define
that I want to listen on a port for both ssl and non ssl requests.
That means if I try to return a fixed response 'S' on the SSLRequest it
fails with an SSL-Handshake failure cause the server expects a ssl message.

I searched for some way to forward to a default backend on ssl failure
but this seems to be no use-case for haproxy and isn't supported.

I also didn't find any other tcp-loadbalancer, which could handle this
type of ssl-failure fallback.

My only option would therefore be to write a custom loadbalancer for
this usecase, which is not really feasible given the amount of features
of haproxy I plan to use.

I have to say the psql ssl handshake procedure is really unique and
challenging :D

The tool stunnel is capable of this protocol but I can't do sni-based
loadbalancing with it so this is kind of a dead end here.

Lukas

Am 11-Dec-20 um 16:44 schrieb Heikki Linnakangas:

Show quoted text

On 11/12/2020 16:46, Lukas Meisegeier wrote:

Hey Heikki,

thanks for providing feedback :)
The traffic between proxy and psql-server is unencrypted thats why I
don't need to patch the server.

Ok.

I tried returning a fixed response on the first plain SSLRequest
forwarding it to a psql-server with ssl enabled an tried to switch then
on the ssl connection startup but that didn't work out. I guess its
because the psql-server won't accept an ssl connection if its not
requested via SSLRequest.

Your proxy could receive the client's SSLRequest message, and respond
with a single byte 'S'. You don't need to forward that to the real
PostgreSQL server, since the connection to the PostgreSQL server is
unencrypted. Then perform the TLS handshake, and forward all traffic to
the real server only after that.

Client: -> SSLRequest
 Proxy: <- 'S'
Client: -> TLS ClientHello
 Proxy: [finish TLS handshake]

- Heikki

#6Heikki Linnakangas
hlinnaka@iki.fi
In reply to: Lukas Meisegeier (#5)
Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing

On 12/12/2020 13:52, Lukas Meisegeier wrote:

Thanks for the provided ideas :)
I use HaProxy for my load-balancing and unfortunately I can't define
that I want to listen on a port for both ssl and non ssl requests.

Could you configure HaProxy to listen on separate ports for SSL and
non-SSL connections, then? And forward both to the same Postgres server.

That means if I try to return a fixed response 'S' on the SSLRequest it
fails with an SSL-Handshake failure cause the server expects a ssl message.

That doesn't sound right to me, or perhaps I have misunderstood what you
mean. If you don't send the SSLRequest to the Postgres server, but "eat"
it in the proxy, the Postgres server will not try to do an SSL handshake.

I have to say the psql ssl handshake procedure is really unique and
challenging :D

Yeah. IMAP and SMTP can use "STARTTLS" to switch an unencrypted
connection to encrypted, though. That's pretty similar to the
'SSLRequest' message used in the postgres protocol.

- Heikki

#7Lukas Meisegeier
MeisegeierLukas@gmx.de
In reply to: Heikki Linnakangas (#6)
Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing

I liked the idea with separate ports for ssl and non ssl requests and
tried it with haproxy.
The psql-client connects with haproxy and receives the fixed 'S' byte
response. After that he tried to continue on the same connection and
doens't open a new one. This crashes the connection because haproxy
expects a new tcp connection.

psqlClient: opens connection (ARP: SYN)
haproxy: accepts connection (ARP: SYN ACK)
psqlClient: confirmes the connection (ARP: ACK)
psqlClient: sends SSLRequest
haproxy: sends confirmation (ARP: ACK)
haproxy: sends fixed byte response ('S')
haproxy: closes connection (ARP: FIN, ACK)
psqlclient: confirmed fixed byte response (ARP: ACK)
psqlclient: sends ssl hello request --> error connection already
closed("psql: error: SSL SYSCALL error: No error (0x00000000/0))

In my eyes the problem lies in upgrading the connection rather then
opening a new one.

Am 14-Dec-20 um 14:50 schrieb Heikki Linnakangas:

Show quoted text

On 12/12/2020 13:52, Lukas Meisegeier wrote:

Thanks for the provided ideas :)
I use HaProxy for my load-balancing and unfortunately I can't define
that I want to listen on a port for both ssl and non ssl requests.

Could you configure HaProxy to listen on separate ports for SSL and
non-SSL connections, then? And forward both to the same Postgres server.

That means if I try to return a fixed response 'S' on the SSLRequest it
fails with an SSL-Handshake failure cause the server expects a ssl
message.

That doesn't sound right to me, or perhaps I have misunderstood what you
mean. If you don't send the SSLRequest to the Postgres server, but "eat"
it in the proxy, the Postgres server will not try to do an SSL handshake.

I have to say the psql ssl handshake procedure is really unique and
challenging :D

Yeah. IMAP and SMTP can use "STARTTLS" to switch an unencrypted
connection to encrypted, though. That's pretty similar to the
'SSLRequest' message used in the postgres protocol.

- Heikki

#8Lukas Meisegeier
MeisegeierLukas@gmx.de
In reply to: Heikki Linnakangas (#4)
Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing

Hey,

whats the state of this? Can we start working out a plan to remove the
inital SSLRequest from the connection protocol or is there any reason to
keep it?

I would start by removing the need of the SSLRequest in the psql-server
if its started with a special parameter(ssl-only or so).
Simultaneously I would add a parameter to this disable the SSLRequest in
the client as well.

Later we could make this behaviour default for psql-server with
ssl-enabled and clients and some far time ahead we could remove the
implementation of SSLRequest in both server and client.

What are your thaughts about this?

Best Regards