Spoofing as the postmaster
A few months ago a security concern was sent to core. We have discussed
it but see little we can do to address it in the code so I am posting to
hackers in case there is something we didn't think of or if
documentation additions are necessary.
Most users understand that if they are connecting to the postmaster over
an insecure network that it is possible for a middle-man to intercept
passwords, queries, and results. SSL certificates are designed to avoid
this problem.
The new attack vector reported involves a local user on the same machine
as the postmaster. When the postmaster is running it has bound port
5432 and created the unix-domain socket file in /tmp. The new attack
involves cases where the postmaster is stopped.
The attacker can bind to TCP port 5432 or create a socket file in /tmp
and get passwords and queries. PGDATA is secure so results cannot be
returned while the postmaster is down. They can also prevent the real
server from starting.
It is possible for the attacker to use one of the interfaces (tcp or
unix domain) and wait for the postmaster to start. The postmaster will
fail to start on the interface in use but will start on the other
interface and the attacker could route queries to the active postmaster
interface.
So, what solutions exist? We could require the use of port numbers less
than 1024 which typically require root and then become a non-root user,
but that requires root to start the server. We could put the unix
domain socket file in a secure directory (but where?) but the client has
to know that location.
An interesting idea would be for the unix domain client to check that
the ownership of the socket file is the same as PGDATA, but clients
typically don't know PGDATA, nor do they know who should be running the
postmaster.
I suppose we could create a poor-man's SSL for unix domain sockets that
just checks the ownership of the socket file, but users can already do
that by specifying the socket file in a directory that only has write
permission for the postmaster user.
Could we create some kind of lock mode that keeps these interfaces
locked when the postmaster is down?
Conclusion
----------
The fundamental problem is that because we don't require root, any user's
postmaster or pretend postmaster is as legitimate as anyone else's. SSL
certificates add legitimacy checks for TCP, but not for unix domain sockets.
This is not a Postgres-specific problem. It is probably shared by any
server that doesn't need root permissions, but our handling of passwords
makes it a larger problem.
I think at a minimum we need to add documentation that states if you
don't trust the local users on the postmaster server you should:
o create unix domain socket files in a non-world-writable
directory
o require SSL server certificates for TCP connections
Ideas?
Remember, as long as your postmaster is running you are safe. It is
only postmaster downtime that has this risk.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
On Sat, 22 Dec 2007 09:25:05 -0500 (EST)
Bruce Momjian <bruce@momjian.us> wrote:
I think at a minimum we need to add documentation that states if you
don't trust the local users on the postmaster server you should:o create unix domain socket files in a non-world-writable
directory
o require SSL server certificates for TCP connectionsIdeas?
It's generally a bad idea to put your database on a public server
anyway but if you do you should definitely disable unix domain sockets
and connect over TCP to localhost. That has been our rule for years.
It's certainly a corner case. I would think that warnings, perhaps in
the config file itself, would be sufficient.
--
D'Arcy J.M. Cain <darcy@druid.net> | Democracy is three wolves
http://www.druid.net/darcy/ | and a sheep voting on
+1 416 425 1212 (DoD#0082) (eNTP) | what's for dinner.
Bruce Momjian wrote:
The fundamental problem is that because we don't require root, any user's
postmaster or pretend postmaster is as legitimate as anyone else's. �SSL
certificates add legitimacy checks for TCP, but not for unix domain
sockets.
Wouldn't SSL work over Unix-domain sockets as well? The API only deals with
file descriptors.
--
Peter Eisentraut
http://developer.postgresql.org/~petere/
Peter Eisentraut wrote:
Bruce Momjian wrote:
The fundamental problem is that because we don't require root, any user's
postmaster or pretend postmaster is as legitimate as anyone else's. SSL
certificates add legitimacy checks for TCP, but not for unix domain
sockets.Wouldn't SSL work over Unix-domain sockets as well? The API only deals with
file descriptors.
But we don't check the SSL cert's credentials in the client, AFAIK. That
means that postmaster spoofer could just as easily spoof SSL.
Communications between the client and the endpoint will be protected,
but there is no protection from a man in the middle attack, which is
what this is.
cheers
andrew
Andrew Dunstan wrote:
But we don't check the SSL cert's credentials in the client, AFAIK.
We do if you configure it so. But I must admit that this fact is not well
advertised. It is documented, but you have to look carefully.
--
Peter Eisentraut
http://developer.postgresql.org/~petere/
Andrew Dunstan wrote:
Peter Eisentraut wrote:
Bruce Momjian wrote:
The fundamental problem is that because we don't require root, any
user's
postmaster or pretend postmaster is as legitimate as anyone else's. SSL
certificates add legitimacy checks for TCP, but not for unix domain
sockets.Wouldn't SSL work over Unix-domain sockets as well? The API only
deals with file descriptors.But we don't check the SSL cert's credentials in the client, AFAIK. That
means that postmaster spoofer could just as easily spoof SSL.
Communications between the client and the endpoint will be protected,
but there is no protection from a man in the middle attack, which is
what this is.
We do if you put the CA cert on the client.
//Magnus
Peter Eisentraut <peter_e@gmx.net> writes:
Wouldn't SSL work over Unix-domain sockets as well? The API only deals with
file descriptors.
Hmm ... we've always thought of SSL as being primarily comm security
and thus useless on a Unix socket, but the mutual authentication aspect
could come in handy as an answer for this type of threat. Anyone want
to try this and see if it really works or not?
Does OpenSSL have a mode where it only does mutual auth and not
encryption? The encryption would be wasted cycles in this scenario,
so being able to turn it off would be nice.
regards, tom lane
On Dec 22, 2007 1:04 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Peter Eisentraut <peter_e@gmx.net> writes:
Wouldn't SSL work over Unix-domain sockets as well? The API only deals with
file descriptors.Hmm ... we've always thought of SSL as being primarily comm security
and thus useless on a Unix socket, but the mutual authentication aspect
could come in handy as an answer for this type of threat. Anyone want
to try this and see if it really works or not?Does OpenSSL have a mode where it only does mutual auth and not
encryption? The encryption would be wasted cycles in this scenario,
so being able to turn it off would be nice.
miker@whirly:~$ openssl ciphers -v 'NULL'
NULL-SHA SSLv3 Kx=RSA Au=RSA Enc=None Mac=SHA1
NULL-MD5 SSLv3 Kx=RSA Au=RSA Enc=None Mac=MD5
I see no way to turn off the message digest, but maybe that's just an
added benefit.
--miker
On 12/22/07, Peter Eisentraut <peter_e@gmx.net> wrote:
Bruce Momjian wrote:
The fundamental problem is that because we don't require root, any user's
postmaster or pretend postmaster is as legitimate as anyone else's. SSL
certificates add legitimacy checks for TCP, but not for unix domain
sockets.Wouldn't SSL work over Unix-domain sockets as well? The API only deals with
file descriptors.
For Unix sockets it should be enough to just check server
process uid, no?
(FYI - Debian already puts unix socket to directory writable
only to postgres user, so they dont have the problem. Maybe
we should encourage distros to move away from /tmp?)
--
marko
"Mike Rylander" <mrylander@gmail.com> writes:
On Dec 22, 2007 1:04 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Hmm ... we've always thought of SSL as being primarily comm security
and thus useless on a Unix socket, but the mutual authentication aspect
could come in handy as an answer for this type of threat. Anyone want
to try this and see if it really works or not?Does OpenSSL have a mode where it only does mutual auth and not
encryption?
miker@whirly:~$ openssl ciphers -v 'NULL'
Cool. I took a quick look through the code, and I think that a smoke
test could be made just by diking out these lines in
src/interfaces/libpq/fe-connect.c:
if (IS_AF_UNIX(conn->raddr.addr.ss_family))
{
/* Don't bother requesting SSL over a Unix socket */
conn->allow_ssl_try = false;
}
Actual support would require rather more effort --- for instance,
I doubt that the default behavior should be to try to do SSL over a
socket, so "sslmode" would need some extension, and we'd want to extend
the pg_hba.conf keywords --- but I think this would be enough to allow
verifying whether it will work.
regards, tom lane
"Marko Kreen" <markokr@gmail.com> writes:
(FYI - Debian already puts unix socket to directory writable
only to postgres user, so they dont have the problem. Maybe
we should encourage distros to move away from /tmp?)
No, we shouldn't, and if I had any authority over them I would make
Debian stop doing that. It amounts to a unilateral distro-specific
change in the protocol, and I think it makes things *less* secure,
because any clients who are expecting the socket to be in /tmp will be
easy pickings for a spoofer. Debian cannot hope to prevent that
scenario, because there are non-libpq-based client implementations.
regards, tom lane
On Dec 22, 2007 6:25 AM, Bruce Momjian <bruce@momjian.us> wrote:
It is possible for the attacker to use one of the interfaces (tcp or
unix domain) and wait for the postmaster to start. The postmaster will
fail to start on the interface in use but will start on the other
interface and the attacker could route queries to the active postmaster
interface.
I am not very conversant with networking, but I see a possibly simple
solution. Why not refuse to start the postmaster if we are unable to bind
with any of the interfaces (all that are specified in the conf file).
This way, if the attacker has control of even one interface (and
optionally the local socket) that the clients are expected to connect to,
the postmaster wouldn't start and the attacker won't have any traffic to
peek into.
Best regards,
--
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com
EnterpriseDB http://www.enterprisedb.com
17° 29' 34.37"N, 78° 30' 59.76"E - Hyderabad
18° 32' 57.25"N, 73° 56' 25.42"E - Pune
37° 47' 19.72"N, 122° 24' 1.69" W - San Francisco *
Mail sent from my BlackLaptop device
Gurjeet Singh wrote:
On Dec 22, 2007 6:25 AM, Bruce Momjian <bruce@momjian.us> wrote:
It is possible for the attacker to use one of the interfaces (tcp or
unix domain) and wait for the postmaster to start. The postmaster will
fail to start on the interface in use but will start on the other
interface and the attacker could route queries to the active postmaster
interface.I am not very conversant with networking, but I see a possibly simple
solution. Why not refuse to start the postmaster if we are unable to bind
with any of the interfaces (all that are specified in the conf file).This way, if the attacker has control of even one interface (and
optionally the local socket) that the clients are expected to connect to,
the postmaster wouldn't start and the attacker won't have any traffic to
peek into.
Yes, that would fix the problem I mentioned but at that point the
attacker already has passwords so they can just connect themselves.
Having the server fail if it can't get one interface makes the server
less reliable.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
On Dec 23, 2007 12:20 PM, Bruce Momjian <bruce@momjian.us> wrote:
Gurjeet Singh wrote:
On Dec 22, 2007 6:25 AM, Bruce Momjian <bruce@momjian.us> wrote:
This way, if the attacker has control of even one interface (and
optionally the local socket) that the clients are expected to connect to,
the postmaster wouldn't start and the attacker won't have any traffic to
peek into.Yes, that would fix the problem I mentioned but at that point the
attacker already has passwords so they can just connect themselves.
Having the server fail if it can't get one interface makes the server
less reliable.
It doesn't solve the spoofing attack problem, but isn't Gurjeet's idea
a good one in any case?
If the postmaster can't bind on one of the specified interfaces, then
at the least, haven't you got got a serious configuration error the
sysadmin would want to know about? Having postmaster fail seems like
a sensible response.
"I can't start with the configuration you've given me, so I won't
start at all" is fairly normal behaviour for a server process, no?
Regards,
BJ
Brendan Jurd wrote:
On Dec 23, 2007 12:20 PM, Bruce Momjian <bruce@momjian.us> wrote:
Gurjeet Singh wrote:
On Dec 22, 2007 6:25 AM, Bruce Momjian <bruce@momjian.us> wrote:
This way, if the attacker has control of even one interface (and
optionally the local socket) that the clients are expected to connect to,
the postmaster wouldn't start and the attacker won't have any traffic to
peek into.Yes, that would fix the problem I mentioned but at that point the
attacker already has passwords so they can just connect themselves.
Having the server fail if it can't get one interface makes the server
less reliable.It doesn't solve the spoofing attack problem, but isn't Gurjeet's idea
a good one in any case?If the postmaster can't bind on one of the specified interfaces, then
at the least, haven't you got got a serious configuration error the
sysadmin would want to know about? Having postmaster fail seems like
a sensible response."I can't start with the configuration you've given me, so I won't
start at all" is fairly normal behaviour for a server process, no?
Yes, we have talked about this in the past and there were concerns that
that the server might have some network problem that would prevent
binding on all interfaces, particularly IPv6.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Bruce Momjian wrote:
I think at a minimum we need to add documentation that states if you
don't trust the local users on the postmaster server you should:o create unix domain socket files in a non-world-writable
directory
o require SSL server certificates for TCP connections
I have written documentation for this item:
http://momjian.us/tmp/pgsql/server-shutdown.html#SERVER-SPOOFING
Comments?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
On Dec 23, 2007 1:25 PM, Bruce Momjian <bruce@momjian.us> wrote:
I have written documentation for this item:
http://momjian.us/tmp/pgsql/server-shutdown.html#SERVER-SPOOFING
Comments?
I thought the content made sense, but the location didn't. I wouldn't
expect to find instructions on configuring Postgres for secure
operation under a section about how to shut the server down.
I realise that in order for the exploit to occur, the server must be
shut down (or not yet started), but unless a user already knows about
the way the exploit works, how will they know to look for info about
it here?
IMO by putting this guidance under "Shutting Down" you're going to
hurt the chances of anyone stumbling across it. I doubt you'd get
many users reading "Shutting Down" at all because in most cases, it's
an easy or obvious thing to do (initscripts provided by package and
pg_ctl are self-explanatory).
Regards,
BJ
Brendan Jurd wrote:
On Dec 23, 2007 1:25 PM, Bruce Momjian <bruce@momjian.us> wrote:
I have written documentation for this item:
http://momjian.us/tmp/pgsql/server-shutdown.html#SERVER-SPOOFING
Comments?
I thought the content made sense, but the location didn't. I wouldn't
expect to find instructions on configuring Postgres for secure
operation under a section about how to shut the server down.I realise that in order for the exploit to occur, the server must be
shut down (or not yet started), but unless a user already knows about
the way the exploit works, how will they know to look for info about
it here?IMO by putting this guidance under "Shutting Down" you're going to
hurt the chances of anyone stumbling across it. I doubt you'd get
many users reading "Shutting Down" at all because in most cases, it's
an easy or obvious thing to do (initscripts provided by package and
pg_ctl are self-explanatory).
Agreed. I moved it up to its own section:
http://momjian.us/tmp/pgsql/preventing-server-spoofing.html
I improved the wording slightly too.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Brendan Jurd wrote:
It doesn't solve the spoofing attack problem, but isn't Gurjeet's idea
a good one in any case?
What makes it good? It solves no problems. It prevents the server from
coming up when it otherwise might still be able to.
If the postmaster can't bind on one of the specified interfaces, then
at the least, haven't you got got a serious configuration error the
sysadmin would want to know about? Having postmaster fail seems like
a sensible response.
I don't think it really matters what it does in the grand scheme of
things, as it's not solving a real problem.
"I can't start with the configuration you've given me, so I won't
start at all" is fairly normal behaviour for a server process, no
None of my servers work this way. If possible, I try to make my servers
auto-recover at a later time while they are still up. It means an
administrator does not need to login to a machine at the data center to
solve the problem. "Self healing" is a term that is used to describe
approaches such as this.
Cheers,
mark
--
Mark Mielke <mark@mielke.cc>
Mark Mielke <mark@mark.mielke.cc> writes:
Brendan Jurd wrote:
It doesn't solve the spoofing attack problem, but isn't Gurjeet's idea
a good one in any case?What makes it good? It solves no problems. It prevents the server from
coming up when it otherwise might still be able to.
The primary reason things work like that is that there are boatloads of
machines that are marginally misconfigured. For instance, userland
thinks there is IPv6 support when the kernel thinks not (or vice versa).
If we made the postmaster abort every time it couldn't latch onto every
address that the listen_addresses setting suggested it might be able to
latch onto, what we'd mostly accomplish is to drive away a lot of
potential users.
Given that everyone agrees that this change wouldn't actually fix
anything w.r.t. spoofing, I don't think there's grounds for making it.
regards, tom lane
