Practical impediment to supporting multiple SSL libraries

-----Original Message-----
From: pgsql-hackers-owner@postgresql.org
[mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of
Martijn van Oosterhout
Sent: 12 April 2006 16:48
To: pgsql-hackers@postgresql.org
Subject: [HACKERS] Practical impediment to supporting
multiple SSL libraries

Just quickly going through what might be needed to support
multiple SSL libraries revealed one big problem in libpq-fe.h.

#ifdef USE_SSL
/* Get the SSL structure associated with a connection */
extern SSL *PQgetssl(PGconn *conn); #else extern void
*PQgetssl(PGconn *conn); #endif

The return type of the function changes depending on whether
SSL is compiled in or not. :( So, libpq exposes to its users
the underlying SSL library, which seems wrong. Now, options include:

1. Changing it to always return (void*), irrespective of SSL
2. Creating a PGsslcontext type that varies depending on what
library you use (or not).
3. Removing the function entirely because the only user
appears to be psql (in tree anyway).
4. Only declare the function if the user has #included
openssl themselves.

Or alternatively we could do nothing because:

5. It's not a problem
6. It's a backward incompatable change

The next version of psqlODBC (that has just gone into CVS tip after
months of work and debate) uses it, and would break almost completely
should it be removed, therefore any backwards incompatible change should
be avoided imho. And 2 or 4 could cause chaos for Windows users if
different DLL builds get mixed up.

Regards, Dave.

On Wed, Apr 12, 2006 at 05:03:32PM +0100, Dave Page wrote:

<about the declaration of PQgetssl>

The next version of psqlODBC (that has just gone into CVS tip after
months of work and debate) uses it, and would break almost completely
should it be removed, therefore any backwards incompatible change should
be avoided imho. And 2 or 4 could cause chaos for Windows users if
different DLL builds get mixed up.

Hmm, may I ask what it uses it for? Just to get information, or
something more substantial?

Thanks in advance,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

-----Original Message-----
From: Martijn van Oosterhout [mailto:kleptog@svana.org]
Sent: 12 April 2006 17:15
To: Dave Page
Cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Practical impediment to supporting
multiple SSL libraries

On Wed, Apr 12, 2006 at 05:03:32PM +0100, Dave Page wrote:

<about the declaration of PQgetssl>

The next version of psqlODBC (that has just gone into CVS tip after
months of work and debate) uses it, and would break almost

completely

should it be removed, therefore any backwards incompatible change
should be avoided imho. And 2 or 4 could cause chaos for

Windows users

if different DLL builds get mixed up.

Hmm, may I ask what it uses it for? Just to get information,
or something more substantial?

The driver implements all versions of the wire protocol itself, but if
libpq is available at runtime (it will dynamically load it on platforms
that support it) it can use it for connection setup so features like SSL
can be provided easily. I'm still not overly familiar with how it works
yet, but I'm sure Hiroshi (CC'd) can provide further details if you need
them.

Regards, Dave.

Martijn van Oosterhout <kleptog@svana.org> writes:

1. Changing it to always return (void*), irrespective of SSL
...
Personally, I'm in favour of 1, because then we can get rid of the
#include for openssl, so users don't have to have openssl headers
installed to compile postgresql programs.

I like that too. I've never been very happy about having libpq-fe.h
depending on USE_SSL.

There is a more serious issue here though: if we allow more than one SSL
library, what exactly can an application safely do with the returned
pointer? It strikes me as very dangerous for the app to assume it knows
which SSL library is underneath libpq. It's not at all hard to imagine
an app getting an OpenSSL struct pointer and trying to pass it to GnuTLS
or vice versa. To the extent that there are apps out there that depend
on doing something with this function, I think that even contemplating
supporting multiple SSL libraries is a threat.

regards, tom lane

Tom Lane wrote:

Martijn van Oosterhout <kleptog@svana.org> writes:

1. Changing it to always return (void*), irrespective of SSL
...
Personally, I'm in favour of 1, because then we can get rid of the
#include for openssl, so users don't have to have openssl headers
installed to compile postgresql programs.

I like that too. I've never been very happy about having libpq-fe.h
depending on USE_SSL.

There is a more serious issue here though: if we allow more than one SSL
library, what exactly can an application safely do with the returned
pointer? It strikes me as very dangerous for the app to assume it knows
which SSL library is underneath libpq. It's not at all hard to imagine
an app getting an OpenSSL struct pointer and trying to pass it to GnuTLS
or vice versa. To the extent that there are apps out there that depend
on doing something with this function, I think that even contemplating
supporting multiple SSL libraries is a threat.

I wonder if there are apps that actually use the ssl pointer, beyond
detection of encrypted connections. So interpreting the result as bool
would be sufficient.

Regards,
Andreas

* Tom Lane (tgl@sss.pgh.pa.us) wrote:

Martijn van Oosterhout <kleptog@svana.org> writes:

1. Changing it to always return (void*), irrespective of SSL
...
Personally, I'm in favour of 1, because then we can get rid of the
#include for openssl, so users don't have to have openssl headers
installed to compile postgresql programs.

I like that too. I've never been very happy about having libpq-fe.h
depending on USE_SSL.

I'm all in favor of dropping the dependency on OpenSSL headers from
libpq, just to throw my 2 cents in there.

There is a more serious issue here though: if we allow more than one SSL
library, what exactly can an application safely do with the returned
pointer? It strikes me as very dangerous for the app to assume it knows
which SSL library is underneath libpq. It's not at all hard to imagine
an app getting an OpenSSL struct pointer and trying to pass it to GnuTLS
or vice versa. To the extent that there are apps out there that depend
on doing something with this function, I think that even contemplating
supporting multiple SSL libraries is a threat.

I'm afraid the way to do this would probably be to have it return a
Postgres-defined structure (without depending on if it's compiled with
SSL or not) which then indicates if the connection is SSL-enabled or not
and then probably other 'common' information (remote DN, remote CA,
ASN.1 formatted certificate perhaps, etc...).

Thanks,

Stephen

* Andreas Pflug (pgadmin@pse-consulting.de) wrote:

I wonder if there are apps that actually use the ssl pointer, beyond
detection of encrypted connections. So interpreting the result as bool
would be sufficient.

I'm not sure if there are apps out there which use it for anything but a
bool but there's certainly a potential for apps to want to do things
like get the DN of the remote server...

Thanks,

Stephen

On Wed, Apr 12, 2006 at 01:42:51PM -0400, Stephen Frost wrote:

* Andreas Pflug (pgadmin@pse-consulting.de) wrote:

I wonder if there are apps that actually use the ssl pointer, beyond
detection of encrypted connections. So interpreting the result as bool
would be sufficient.

I'm not sure if there are apps out there which use it for anything but a
bool but there's certainly a potential for apps to want to do things
like get the DN of the remote server...

Strangly enough, the SSL code in libpq has stored the peer DN and CN
except it doesn't appear to be available to the client...

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

On Wed, Apr 12, 2006 at 12:32:01PM -0400, Tom Lane wrote:

There is a more serious issue here though: if we allow more than one SSL
library, what exactly can an application safely do with the returned
pointer? It strikes me as very dangerous for the app to assume it knows
which SSL library is underneath libpq. It's not at all hard to imagine
an app getting an OpenSSL struct pointer and trying to pass it to GnuTLS
or vice versa. To the extent that there are apps out there that depend
on doing something with this function, I think that even contemplating
supporting multiple SSL libraries is a threat.

The only real way to a solution is to work out why people want the
pointer. So far I've found two reasons:

- People want to hijack the connection after libpq has set it up to do
their own processing.

- People want to examine the certificates more closely.

The first would be easily handled by providing a formal interface for
libpq to hijack the connection with, providing read/write and maybe a
few others. The latter is tricker. You're invariably going to run into
the problem where the app uses one lib and libpq the other.

Other than DN and CN, what else would people want?
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

There is a more serious issue here though: if we allow more

than one

SSL library, what exactly can an application safely do with the
returned pointer? It strikes me as very dangerous for the app to
assume it knows which SSL library is underneath libpq. It's not at
all hard to imagine an app getting an OpenSSL struct pointer and
trying to pass it to GnuTLS or vice versa. To the extent

that there

are apps out there that depend on doing something with this

function,

I think that even contemplating supporting multiple SSL

libraries is a threat.

The only real way to a solution is to work out why people
want the pointer. So far I've found two reasons:

- People want to hijack the connection after libpq has set it
up to do their own processing.

- People want to examine the certificates more closely.

The first would be easily handled by providing a formal
interface for libpq to hijack the connection with, providing
read/write and maybe a few others. The latter is tricker.
You're invariably going to run into the problem where the app
uses one lib and libpq the other.

Other than DN and CN, what else would people want?

Issuer (name and certificate), validity dates, basic constraints, key
usage, posslby fingerprint.

//Magnus

On Wed, Apr 12, 2006 at 08:14:58PM +0200, Magnus Hagander wrote:

Other than DN and CN, what else would people want?

Issuer (name and certificate), validity dates, basic constraints, key
usage, posslby fingerprint.

GnuTLS handles this with just one function:

gnutls_x509_crt_get_dn_by_oid( cert, oid, index, raw, &data, &length )

And a whole pile of #defines

#define GNUTLS_OID_X520_COUNTRY_NAME "2.5.4.6"
#define GNUTLS_OID_X520_ORGANIZATION_NAME "2.5.4.10"
#define GNUTLS_OID_X520_ORGANIZATIONAL_UNIT_NAME "2.5.4.11"

etc...

Which is nice because then end users can code in the attributes they
want and we don't have to deal with the endless variations. I don't
however know enough to know if this (with a function to get OIDs by
index) is sufficient to extract all the information from the
certificate.

Presumably OpenSSL can do this too...
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

"Magnus Hagander" <mha@sollentuna.net> writes:

Other than DN and CN, what else would people want?

Issuer (name and certificate), validity dates, basic constraints, key
usage, posslby fingerprint.

I think that way madness lies --- do we really want to commit to
re-inventing an SSL API that will cover anything someone might want
to do with either underlying library? Moreover, this does not fix
the problem: an existing app that thinks it can pass the returned
pointer to an OpenSSL routine will still crash the moment a GnuTLS
version of libpq is put under it. Case in point: psql, as currently
coded.

An idea that just occurred to me is to define PQgetssl as "return SSL*
if we are using OpenSSL for this connection; else return NULL". Then
add a parallel routine (maybe PQgetgnussl?) defined as returning the
equivalent GnuTLS handle, only if we are using GnuTLS for this
connection. (Presumably, in any one build of libpq, one of the pair of
routines would be an always-returns-null stub.)

The advantage of this is that an app knows what it'll get, and an app
that's only familiar with one of the two SSL libraries will not be
given a pointer it can't use.

I'd still want to adopt Martijn's idea of declaring both of 'em as
returning void *, to avoid depending on other packages' include files.

regards, tom lane

Martijn van Oosterhout wrote:

On Wed, Apr 12, 2006 at 05:03:32PM +0100, Dave Page wrote:

<about the declaration of PQgetssl>

The next version of psqlODBC (that has just gone into CVS tip after
months of work and debate) uses it, and would break almost completely
should it be removed, therefore any backwards incompatible change should
be avoided imho. And 2 or 4 could cause chaos for Windows users if
different DLL builds get mixed up.

Hmm, may I ask what it uses it for? Just to get information, or
something more substantial?

In case of SSL mode, the driver gets the communication path using
PQsocket() or PQgetssl() after calling PQconnectdb(). The driver
comunicates with the server by itself using the path. In case of
non-SSL mode, the driver never calls libpq API at all.

regards,
Hiroshi Inoue

On Wed, Apr 12, 2006 at 05:00:17PM -0400, Tom Lane wrote:

Issuer (name and certificate), validity dates, basic constraints, key
usage, posslby fingerprint.

I think that way madness lies --- do we really want to commit to
re-inventing an SSL API that will cover anything someone might want
to do with either underlying library?

Indeed. There's also the issue that the underlying system may not be
using what you think it is. e.g. GnuTLS can authenticate on PGP keys
rather than x509 certificates. There's still the mystery regarding
libpq extracting peer DN and CN but passing it to the user.

An idea that just occurred to me is to define PQgetssl as "return SSL*
if we are using OpenSSL for this connection; else return NULL". Then
add a parallel routine (maybe PQgetgnussl?) defined as returning the
equivalent GnuTLS handle, only if we are using GnuTLS for this
connection. (Presumably, in any one build of libpq, one of the pair of
routines would be an always-returns-null stub.)

Alternatively, create a new function PQgetsslinfo() that returns both
the library name and a (void) pointer. In any case the old interface
can never return anything other than a pointer for OpenSSL.

I'd still want to adopt Martijn's idea of declaring both of 'em as
returning void *, to avoid depending on other packages' include files.

Ack, at least we can get that out of the way. It doesn't change
anything from the user's point of view, other than they know for sure
what the signiture is.

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

On Wed, Apr 12, 2006 at 05:25:47PM +0100, Dave Page wrote:

The driver implements all versions of the wire protocol itself, but if
libpq is available at runtime (it will dynamically load it on platforms
that support it) it can use it for connection setup so features like SSL
can be provided easily. I'm still not overly familiar with how it works
yet, but I'm sure Hiroshi (CC'd) can provide further details if you need
them.

Right, so what you're basically doing is setting up the connection via
libpq then grabbing the SSL pointer and using that to continue
communicating. If it's not SSL you use PQsocket get the socket and
continue from there.

Unorthodox usage, but it should work.

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

-----Original Message-----
From: Martijn van Oosterhout [mailto:kleptog@svana.org]
Sent: 13 April 2006 07:58
To: Dave Page
Cc: pgsql-hackers@postgresql.org; Hiroshi Inoue
Subject: Re: [HACKERS] Practical impediment to supporting
multiple SSL libraries

On Wed, Apr 12, 2006 at 05:25:47PM +0100, Dave Page wrote:

The driver implements all versions of the wire protocol

itself, but if

libpq is available at runtime (it will dynamically load it on
platforms that support it) it can use it for connection setup so
features like SSL can be provided easily. I'm still not overly
familiar with how it works yet, but I'm sure Hiroshi (CC'd) can
provide further details if you need them.

Right, so what you're basically doing is setting up the
connection via libpq then grabbing the SSL pointer and using
that to continue communicating. If it's not SSL you use
PQsocket get the socket and continue from there.

Yup.

Unorthodox usage, but it should work.

Well, we had a pure custom implementation of the protocol, had a pure
libpq based version and after much discussion decided that the best
version of all was the hybrid as it allowed us to hijack features like
SSL, Kerberos, pgpass et al, yet not be constrained by the limitations
of libpq, or copy query results about so much.

Regards, Dave

On Thu, Apr 13, 2006 at 08:48:54AM +0100, Dave Page wrote:

Well, we had a pure custom implementation of the protocol, had a pure
libpq based version and after much discussion decided that the best
version of all was the hybrid as it allowed us to hijack features like
SSL, Kerberos, pgpass et al, yet not be constrained by the limitations
of libpq, or copy query results about so much.

Right. Would you see value in a more formal libpq "hijack-me" interface
that would support making the initial connection and then handing off
the rest to something else?

I'm wondering because obviously with the current setup, if libpq is
compiled with SSL support, psqlODBC must also be. Are there any points
where you have to fight libpq over control of the socket?

I'm thinking that such an interface would need to provide the
following:

read (sync/async)
write (sync/async)
getfd (for select/poll)
ispending (is there stuff to do)
release (for when you're finished)

Is there anything else you might need?
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

-----Original Message-----
From: Martijn van Oosterhout [mailto:kleptog@svana.org]
Sent: 13 April 2006 09:15
To: Dave Page
Cc: pgsql-hackers@postgresql.org; Hiroshi Inoue
Subject: Re: [HACKERS] Practical impediment to supporting
multiple SSL libraries

On Thu, Apr 13, 2006 at 08:48:54AM +0100, Dave Page wrote:

Well, we had a pure custom implementation of the protocol,

had a pure

libpq based version and after much discussion decided that the best
version of all was the hybrid as it allowed us to hijack

features like

SSL, Kerberos, pgpass et al, yet not be constrained by the

limitations

of libpq, or copy query results about so much.

Right. Would you see value in a more formal libpq "hijack-me"
interface that would support making the initial connection
and then handing off the rest to something else?

I'm wondering because obviously with the current setup, if
libpq is compiled with SSL support, psqlODBC must also be.
Are there any points where you have to fight libpq over
control of the socket?

I'm thinking that such an interface would need to provide the
following:

read (sync/async)
write (sync/async)
getfd (for select/poll)
ispending (is there stuff to do)
release (for when you're finished)

Is there anything else you might need?

I'll have to let Hiroshi comment on that as he wrote the code. I've only
skimmed over it a few times so far.

Regards, Dave.

* Martijn van Oosterhout (kleptog@svana.org) wrote:

On Thu, Apr 13, 2006 at 08:48:54AM +0100, Dave Page wrote:

Well, we had a pure custom implementation of the protocol, had a pure
libpq based version and after much discussion decided that the best
version of all was the hybrid as it allowed us to hijack features like
SSL, Kerberos, pgpass et al, yet not be constrained by the limitations
of libpq, or copy query results about so much.

Right. Would you see value in a more formal libpq "hijack-me" interface
that would support making the initial connection and then handing off
the rest to something else?

I'm wondering because obviously with the current setup, if libpq is
compiled with SSL support, psqlODBC must also be. Are there any points
where you have to fight libpq over control of the socket?

[...]

Is there anything else you might need?

Instead of having it hijack the libpq connection and implement the
wireline protocol itself, why don't we work on fixing the problems (such
as the double-copying that libpq requires) in libpq to allow the driver
(and others!) to use it in the 'orthodox' way?

I would have spoken up on the ODBC list if I understood that 'hybrid'
really meant 'just using libpq for connection/authentication'. I really
think it's a bad idea to have the ODBC driver reimplement the wireline
protocol because that protocol does change from time to time and someone
using libpq will hopefully have fewer changes (and thus makes the code
easier to maintain) than someone implementing the wireline protocol
themselves (just causing more busy-work that, at least we saw in the
past with the ODBC driver, may end up taking *forever* for someone to
be able to commit the extra required time to implement).

Thanks,

Stephen