SSL over Unix-domain sockets

On Fri, Jan 04, 2008 at 05:13:21PM +0100, Peter Eisentraut wrote:

Using the attached patch, SSL will act over Unix-domain sockets. AFAICT, this
just works. I didn't find a way to sniff a Unix-domain socket, however.

Looks clear enough. You should be able to test if it works by using
strace to check the data it's sending.

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

Peter Eisentraut wrote:

Using the attached patch, SSL will act over Unix-domain sockets. AFAICT, this
just works. I didn't find a way to sniff a Unix-domain socket, however.

How should we proceed with this?

I am confused by the shortness of this patch. Right now pg_hba.conf
has:

# host DATABASE USER CIDR-ADDRESS METHOD [OPTION]
# hostssl DATABASE USER CIDR-ADDRESS METHOD [OPTION]
# hostnossl DATABASE USER CIDR-ADDRESS METHOD [OPTION]

These are all for TCP connections. How do we handle 'local' SSL
connection specification? Do we want to provide similar functionality
for local connections?

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Am Freitag, 4. Januar 2008 schrieb Bruce Momjian:

Peter Eisentraut wrote:

Using the attached patch, SSL will act over Unix-domain sockets. AFAICT,
this just works. I didn't find a way to sniff a Unix-domain socket,
however.

How should we proceed with this?

I am confused by the shortness of this patch. Right now pg_hba.conf
has:

# host DATABASE USER CIDR-ADDRESS METHOD [OPTION]
# hostssl DATABASE USER CIDR-ADDRESS METHOD [OPTION]
# hostnossl DATABASE USER CIDR-ADDRESS METHOD [OPTION]

These are all for TCP connections. How do we handle 'local' SSL
connection specification? Do we want to provide similar functionality
for local connections?

Yes, we might want to add that as well. That and some documentation updates
would probably cover everything.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:

Am Freitag, 4. Januar 2008 schrieb Bruce Momjian:

Peter Eisentraut wrote:

Using the attached patch, SSL will act over Unix-domain sockets. AFAICT,
this just works. I didn't find a way to sniff a Unix-domain socket,
however.

How should we proceed with this?

I am confused by the shortness of this patch. Right now pg_hba.conf
has:

# host DATABASE USER CIDR-ADDRESS METHOD [OPTION]
# hostssl DATABASE USER CIDR-ADDRESS METHOD [OPTION]
# hostnossl DATABASE USER CIDR-ADDRESS METHOD [OPTION]

These are all for TCP connections. How do we handle 'local' SSL
connection specification? Do we want to provide similar functionality
for local connections?

Yes, we might want to add that as well. That and some documentation updates
would probably cover everything.

OK. Right now the documentation about spoofing says to use directory
permissions for the socket, and that works. I am thinking this is
something for 8.4.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:

Peter Eisentraut wrote:

Am Freitag, 4. Januar 2008 schrieb Bruce Momjian:

Peter Eisentraut wrote:

Using the attached patch, SSL will act over Unix-domain sockets. AFAICT,
this just works. I didn't find a way to sniff a Unix-domain socket,
however.

How should we proceed with this?

I am confused by the shortness of this patch. Right now pg_hba.conf
has:

# host DATABASE USER CIDR-ADDRESS METHOD [OPTION]
# hostssl DATABASE USER CIDR-ADDRESS METHOD [OPTION]
# hostnossl DATABASE USER CIDR-ADDRESS METHOD [OPTION]

These are all for TCP connections. How do we handle 'local' SSL
connection specification? Do we want to provide similar functionality
for local connections?

Yes, we might want to add that as well. That and some documentation updates
would probably cover everything.

OK. Right now the documentation about spoofing says to use directory
permissions for the socket, and that works. I am thinking this is
something for 8.4.

Actually, if you just commit that patch *without* pg_hba modifications,
it still solves the problem stated, no? Because the client can be
configured to require ssl and to require server certificate validation,
and that's the hole we're trying to plug here...

//Magnus

Magnus Hagander wrote:

Bruce Momjian wrote:

Peter Eisentraut wrote:

Am Freitag, 4. Januar 2008 schrieb Bruce Momjian:

Peter Eisentraut wrote:

Using the attached patch, SSL will act over Unix-domain sockets. AFAICT,
this just works. I didn't find a way to sniff a Unix-domain socket,
however.

How should we proceed with this?

I am confused by the shortness of this patch. Right now pg_hba.conf
has:

# host DATABASE USER CIDR-ADDRESS METHOD [OPTION]
# hostssl DATABASE USER CIDR-ADDRESS METHOD [OPTION]
# hostnossl DATABASE USER CIDR-ADDRESS METHOD [OPTION]

These are all for TCP connections. How do we handle 'local' SSL
connection specification? Do we want to provide similar functionality
for local connections?

Yes, we might want to add that as well. That and some documentation updates
would probably cover everything.

OK. Right now the documentation about spoofing says to use directory
permissions for the socket, and that works. I am thinking this is
something for 8.4.

Actually, if you just commit that patch *without* pg_hba modifications,
it still solves the problem stated, no? Because the client can be
configured to require ssl and to require server certificate validation,
and that's the hole we're trying to plug here...

Yes, it would plug the hole without fully implementing SSL control on
local sockets. However, the hole is already plugged by using directory
permissions so I question the need for a partial solution at this point
in 8.3.

At this point in 8.3 I think we have to ask if we would make such a
change in a minor release, and I don't think we would.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

* Bruce Momjian <bruce@momjian.us> [080104 13:00]:

Actually, if you just commit that patch *without* pg_hba modifications,
it still solves the problem stated, no? Because the client can be
configured to require ssl and to require server certificate validation,
and that's the hole we're trying to plug here...

Yes, it would plug the hole without fully implementing SSL control on
local sockets. However, the hole is already plugged by using directory
permissions so I question the need for a partial solution at this point
in 8.3.

Yet we have respected people warning us that the *only* place we can
have the socket is /tmp, because that's where everybody (for varying
definitions of everybody) looks. Moving the socket from /tmp actually
makes the problem of a spoofed postmaster bigger.

If you have a scheme to "move" or protect the unix socket, make sure you
still provide the one in /tmp. A simple test looks like the
/tmp/.s.PGSQL.XXXX can be a symlink the socket in the protected dir, so
it may be enough for concerned admins to create this symlink (or the
actual socket with correct owner/permissions) on system startup,
preventing an "outsider" from taking this file before postgresql (and
make sure that no tmpwatch or anything removes it again).

But if PostgreSQL is started before your "untrusted user processes",
then your untrusted user processes should never get the chance to spoof
the server unless they get to mv/delete the postgres-user owned socket
in /tmp, in which case, you've got larger problems to worry about...

a.

--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.

Bruce Momjian <bruce@momjian.us> writes:

Yes, it would plug the hole without fully implementing SSL control on
local sockets. However, the hole is already plugged by using directory
permissions so I question the need for a partial solution at this point
in 8.3.

As already noted, "fix /tmp's directory permissions" isn't a very
helpful suggestion.

At this point in 8.3 I think we have to ask if we would make such a
change in a minor release, and I don't think we would.

It depends on whether you call it a new feature or a bug fix.
If it is a bug fix, wouldn't we also back-patch it?

Given the smallness of Peter's patch, I don't think that treating
it as a bug fix is unreasonable, if that (and the docs) are all we
change. Now adding "localssl" etc to pg_hba.conf's options seems
more like a new feature, and that I think should wait for 8.4.

One question is whether patching this without adding localssl
(and therefore, without providing a way for the DBA to enforce
SSL use) is actually very helpful. You could be secure but you'd
be depending on the client side to get it right. OTOH that's true
anyway if we have no way to enforce that the client verify the
postmaster's certificate.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

Yes, it would plug the hole without fully implementing SSL control on
local sockets. However, the hole is already plugged by using directory
permissions so I question the need for a partial solution at this point
in 8.3.

As already noted, "fix /tmp's directory permissions" isn't a very
helpful suggestion.

Right, I was saying moving to a another directory, not chaning /tmp,
but you are right that some clients might still look in /tmp, and that
allows spoofing even when the postmaster is running.

At this point in 8.3 I think we have to ask if we would make such a
change in a minor release, and I don't think we would.

It depends on whether you call it a new feature or a bug fix.
If it is a bug fix, wouldn't we also back-patch it?

Given the smallness of Peter's patch, I don't think that treating
it as a bug fix is unreasonable, if that (and the docs) are all we
change. Now adding "localssl" etc to pg_hba.conf's options seems
more like a new feature, and that I think should wait for 8.4.

One question is whether patching this without adding localssl
(and therefore, without providing a way for the DBA to enforce
SSL use) is actually very helpful. You could be secure but you'd
be depending on the client side to get it right. OTOH that's true
anyway if we have no way to enforce that the client verify the
postmaster's certificate.

Well, if we are relying on the client we might as well tell clients to
use a new non-tmp socket location, and even with SSL we can't require
the client to check the postmaster's certificate, as you said.

If we trust the client, a new socket directory will work, but if we
don't, even SSL doesn't help us, right? SSL was used for TCP only
because it allowed trusted clients to work. We already have a unix
socket solution for trusted clients, namely a different directory.

The problem with adding SSL to local sockets is this slippery slope
where we only do part of the job, but it isn't clear where to draw the
line.

Should we wait and do the full job in 8.3.1?

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Fri, Jan 04, 2008 at 02:37:03PM -0500, Bruce Momjian wrote:

The problem with adding SSL to local sockets is this slippery slope
where we only do part of the job, but it isn't clear where to draw the
line.

I don't think "part of the job" for a patch is a slippery slope. It's what
you do with patches for issues discovered too late to make full release:
pick the least invasive thing you can do, release that patch, and implement
the full feature later.

A

Bruce Momjian wrote:

Peter Eisentraut wrote:

Using the attached patch, SSL will act over Unix-domain sockets. AFAICT,
this just works. I didn't find a way to sniff a Unix-domain socket,
however.

How should we proceed with this?

I am confused by the shortness of this patch. Right now pg_hba.conf
has:

# host DATABASE USER CIDR-ADDRESS METHOD [OPTION]
# hostssl DATABASE USER CIDR-ADDRESS METHOD [OPTION]
# hostnossl DATABASE USER CIDR-ADDRESS METHOD [OPTION]

These are all for TCP connections. How do we handle 'local' SSL
connection specification? Do we want to provide similar functionality
for local connections?

Here is a patch that implements "localssl" as well. It is quite simple.
(Note that the code in hba.c is all copy and paste.)

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e@gmx.net> writes:

Here is a patch that implements "localssl" as well. It is quite simple.

The other area that would need some thought before we could consider
this "done" is the behavior of libpq's sslmode parameter. With the
patch as given, an SSL-capable libpq will *default* to using SSL over
sockets, which might be thought overkill; it is almost certainly
going to result in a performance penalty. Is this a reasonable default
behavior? Should sslmode be extended to allow specification of
different behaviors for sockets vs. TCP?

regards, tom lane

Tom Lane wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

Here is a patch that implements "localssl" as well. It is quite simple.

The other area that would need some thought before we could consider
this "done" is the behavior of libpq's sslmode parameter. With the
patch as given, an SSL-capable libpq will *default* to using SSL over
sockets, which might be thought overkill; it is almost certainly
going to result in a performance penalty. Is this a reasonable default
behavior? Should sslmode be extended to allow specification of
different behaviors for sockets vs. TCP

Does the patch handle patched clients connecting to unpatched servers
and vice versa?

I am undecided whether I will use this proposed functionality or not. I
would like to tighten up security (only a few people have access to the
machine, but even a few may be a few too many?). Cryptographic
authentication and encrypted data stream cost is high compared to no
cryptographic authentication or encrypted data streams. I don't know if
it would impact me or not. Peter: Have you tried running a benchmark of
localssl vs localnossl?

Cheers,
mark

--
Mark Mielke <mark@mielke.cc>

Mark Mielke wrote:

Does the patch handle patched clients connecting to unpatched servers
and vice versa?

Yes, it is all compatible.

Cryptographic
authentication and encrypted data stream cost is high compared to no
cryptographic authentication or encrypted data streams. I don't know if
it would impact me or not. Peter: Have you tried running a benchmark of
localssl vs localnossl?

Good point. I tried this

time for x in $(seq 1 1000); do
pg-install/bin/psql -c "select 1" >/dev/null
done

With ssl=off it looks about like this:

real 0m12.909s
user 0m3.712s
sys 0m3.056s

With ssl=on it looks about like this:

real 1m4.741s
user 0m26.638s
sys 0m4.328s

It has been reported that the data transmission overhead is much less than the
connection establishing overhead, which is measured here. But this is
certainly not an encouraging measurement, if we want to put this close to the
default path of use.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e@gmx.net> writes:

It has been reported that the data transmission overhead is much less
than the connection establishing overhead, which is measured here.
But this is certainly not an encouraging measurement, if we want to
put this close to the default path of use.

I did some more experiments to confirm Peter's results. My test case
for measuring connection overhead is
pgbench -c 1 -t 1000 -S -n -C bench
(ie, single client, SELECT-only transaction, connecting again for each
transaction). This is marginally more realistic than Peter's test
since the client executes a SQL command per connection. I get

$ PGSSLMODE=prefer time pgbench -c 1 -t 1000 -S -n -C bench
transaction type: SELECT only
scaling factor: 10
number of clients: 1
number of transactions per client: 1000
number of transactions actually processed: 1000/1000
tps = 33.078772 (including connections establishing)
tps = 33.078772 (excluding connections establishing)
10.45user 0.68system 0:30.26elapsed 36%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+48465minor)pagefaults 0swaps

$ PGSSLMODE=disable time pgbench -c 1 -t 1000 -S -n -C bench
transaction type: SELECT only
scaling factor: 10
number of clients: 1
number of transactions per client: 1000
number of transactions actually processed: 1000/1000
tps = 156.237184 (including connections establishing)
tps = 156.237208 (excluding connections establishing)
0.20user 0.18system 0:06.41elapsed 6%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+2500minor)pagefaults 0swaps

$ PGSSLMODE=prefer time pgbench -c 1 -t 1000 -S -n -C -h localhost bench
transaction type: SELECT only
scaling factor: 10
number of clients: 1
number of transactions per client: 1000
number of transactions actually processed: 1000/1000
tps = 32.320773 (including connections establishing)
tps = 32.320774 (excluding connections establishing)
10.54user 1.01system 0:30.97elapsed 37%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+49807minor)pagefaults 0swaps

$ PGSSLMODE=disable time pgbench -c 1 -t 1000 -S -n -C -h localhost bench
transaction type: SELECT only
scaling factor: 10
number of clients: 1
number of transactions per client: 1000
number of transactions actually processed: 1000/1000
tps = 144.859620 (including connections establishing)
tps = 144.859641 (excluding connections establishing)
0.32user 0.62system 0:06.91elapsed 13%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+4512minor)pagefaults 0swaps

I also did some tests to measure the encryption overhead for bulk data,
in the form of pg_dumping a medium-size table (which is in fact just
the data from the regression test's tenk1 table, repeated 128 times):

[tgl@rh2 ~]$ PGSSLMODE=prefer time pg_dump -t foo regression | wc
2.71user 0.36system 0:25.09elapsed 12%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+1093minor)pagefaults 0swaps
1280054 20480136 85863449

[tgl@rh2 ~]$ PGSSLMODE=disable time pg_dump -t foo regression | wc
0.64user 0.30system 0:09.63elapsed 9%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+793minor)pagefaults 0swaps
1280054 20480136 85863449

[tgl@rh2 ~]$ PGSSLMODE=prefer time pg_dump -t foo -h localhost regression | wc
3.06user 0.45system 0:25.82elapsed 13%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+1105minor)pagefaults 0swaps
1280054 20480136 85863449

[tgl@rh2 ~]$ PGSSLMODE=disable time pg_dump -t foo -h localhost regression | wc
0.66user 0.42system 0:09.91elapsed 10%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+806minor)pagefaults 0swaps
1280054 20480136 85863449

Note that these times are for SSL enabled, but without any root.crt
files, so no actual authentication is happening --- I'm not sure how
much more connection-time overhead that would incur. Presumably the
bulk transfer rate wouldn't change though. All these numbers are
stable to within a percent or three over repeated trials.

Conclusions:

* SSL, even without real authentication, is *way* too expensive to
enable by default.

* The extra cost of going across a local TCP connection is measurable,
but it's insignificant compared to the cost of turning on SSL. (This
is on a Fedora 8 kernel BTW ... that result might vary on other
platforms.)

So you could make a pretty good case that the answer for DBAs who
want to prevent spoofing is to disable socket connections in pg_hba.conf
and force even local connections to come through "hostssl" connections.

If we do want to apply Peter's patch, I think it needs to be extended so
that the default behavior on sockets is the same as before, ie, no SSL.
This could be done by giving libpq an additional connection parameter,
say "socketsslmode", having the same alternatives as sslmode but
defaulting to "allow" instead of "prefer".

regards, tom lane

Tom Lane wrote:

Conclusions:

* SSL, even without real authentication, is *way* too expensive to
enable by default.

* The extra cost of going across a local TCP connection is measurable,
but it's insignificant compared to the cost of turning on SSL. (This
is on a Fedora 8 kernel BTW ... that result might vary on other
platforms.)

So you could make a pretty good case that the answer for DBAs who
want to prevent spoofing is to disable socket connections in pg_hba.conf
and force even local connections to come through "hostssl" connections.

Yea, I figured using protected directories for the socket was the
zero-cost solution, and if you have to do SSL, might as well just use
TCP too. (If you moved the socket file to a protected directory I think
you could use external_pid_file='/tmp/.s.PGSQL.5432' to prevent a spoof
socket file in /tmp. Should we document that idea?)

If we do want to apply Peter's patch, I think it needs to be extended so
that the default behavior on sockets is the same as before, ie, no SSL.
This could be done by giving libpq an additional connection parameter,
say "socketsslmode", having the same alternatives as sslmode but
defaulting to "allow" instead of "prefer".

That seems like it is going to be added confusion; just using the
protected socket diretory or TCP & SSL seems less error-prone.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <bruce@momjian.us> writes:

Yea, I figured using protected directories for the socket was the
zero-cost solution, and if you have to do SSL, might as well just use
TCP too. (If you moved the socket file to a protected directory I think
you could use external_pid_file='/tmp/.s.PGSQL.5432' to prevent a spoof
socket file in /tmp. Should we document that idea?)

Umm ... two questions about that:

* will the postmaster fail if there's a socket where it tries to write
the external_pid_file? (If it does fail, does that really fix
anything? The spoofer already owns the socket.)

* if there's a plain file where a client expects to find the socket,
what happens? (Probably nothing very good, since the first thing the
client will do is write on it.)

If we do want to apply Peter's patch, I think it needs to be extended so
that the default behavior on sockets is the same as before, ie, no SSL.

That seems like it is going to be added confusion; just using the
protected socket diretory or TCP & SSL seems less error-prone.

Yeah, all of this is about confusion and error-proneness. I still think
that the real problem is that we don't have full control over
client-side code, and therefore can't just write off the problem of a
client deciding to connect to /tmp/.s.PGSQL.5432 even if the local DBA
thinks the socket would be safer elsewhere.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

Yea, I figured using protected directories for the socket was the
zero-cost solution, and if you have to do SSL, might as well just use
TCP too. (If you moved the socket file to a protected directory I think
you could use external_pid_file='/tmp/.s.PGSQL.5432' to prevent a spoof
socket file in /tmp. Should we document that idea?)

Umm ... two questions about that:

* will the postmaster fail if there's a socket where it tries to write
the external_pid_file? (If it does fail, does that really fix
anything? The spoofer already owns the socket.)

I figured it would prevent someone from spoofing while the server was
up, which is a _new_ problem when moving the socket. :-(

My feeling on the moving of sockets risk is that you are probably going
to have all your clients using the new socket directory before anyone
tries to put something in /tmp, especially if you have the lock file in
/tmp as outlined above. To spoof in such a situation you would need to
do the attack while the server is down _and_ against a client that
doesn't know the right socket location.

* if there's a plain file where a client expects to find the socket,
what happens? (Probably nothing very good, since the first thing the
client will do is write on it.)

We would have to test that.

If we do want to apply Peter's patch, I think it needs to be extended so
that the default behavior on sockets is the same as before, ie, no SSL.

That seems like it is going to be added confusion; just using the
protected socket diretory or TCP & SSL seems less error-prone.

Yeah, all of this is about confusion and error-proneness. I still think
that the real problem is that we don't have full control over
client-side code, and therefore can't just write off the problem of a
client deciding to connect to /tmp/.s.PGSQL.5432 even if the local DBA
thinks the socket would be safer elsewhere.

Right. I think the lock file in /tmp does help somewhat.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <bruce@momjian.us> writes:

Tom Lane wrote:

Yeah, all of this is about confusion and error-proneness. I still think
that the real problem is that we don't have full control over
client-side code, and therefore can't just write off the problem of a
client deciding to connect to /tmp/.s.PGSQL.5432 even if the local DBA
thinks the socket would be safer elsewhere.

Right. I think the lock file in /tmp does help somewhat.

Even if it happens to work (on some platforms) it seems like a kluge.

It strikes me that given the postmaster's infrastructure for listening
on multiple sockets, it would be a pretty small matter of programming
to teach it to listen on socket files in multiple directories not only
one. If we had that, the postmaster could listen in both /tmp and
your-more-secure-directory-of-choice. Surely an actual socket file
would be a more useful "blocker" in /tmp than a dead-weight PID file.

regards, tom lane