Compromised postgresql instances
I've noticed a steady trickle of reports of postgresql servers being compromised via being left available to the internet with insecure or default configuration, or brute-forced credentials. The symptoms are randomly named binaries being uploaded to the data directory and executed with the permissions of the postgresql user, apparently via an extension or an untrusted PL.
Is anyone tracking or investigating this?
Cheers,
Steve
On 06/08/2018 04:34 PM, Steve Atkins wrote:
I've noticed a steady trickle of reports of postgresql servers being compromised via being left available to the internet with insecure or default configuration, or brute-forced credentials. The symptoms are randomly named binaries being uploaded to the data directory and executed with the permissions of the postgresql user, apparently via an extension or an untrusted PL.
Is anyone tracking or investigating this?
Please cite actual instances of such reports. Vague queries like this
help nobody.
Furthermore, security concerns are best addressed to the security
mailing list.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
On 06/08/2018 04:34 PM, Steve Atkins wrote:
I've noticed a steady trickle of reports of postgresql servers being compromised via being left available to the internet with insecure or default configuration, or brute-forced credentials. The symptoms are randomly named binaries being uploaded to the data directory and executed with the permissions of the postgresql user, apparently via an extension or an untrusted PL.
Is anyone tracking or investigating this?
Please cite actual instances of such reports. Vague queries like this
help nobody.
I imagine Steve is reacting to this report from today:
/messages/by-id/CANozSKLGgWDpzfua2L=OGFN=Dg3Po98UjqJJ18gBVFR1-yK5+A@mail.gmail.com
I recall something similar being reported a few weeks ago, but am
too lazy to trawl the archives right now.
Furthermore, security concerns are best addressed to the security
mailing list.
Unless there's some evidence that these attacks are getting in through
a heretofore unknown PG security vulnerability, rather than user
misconfiguration (such as weak/no password), I'm not sure what the
security list would have to offer. Right now it seems like Steve's move
to try to gather more evidence is quite the right thing to do.
regards, tom lane
On Jun 8, 2018, at 1:47 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
On 06/08/2018 04:34 PM, Steve Atkins wrote:
I've noticed a steady trickle of reports of postgresql servers being compromised via being left available to the internet with insecure or default configuration, or brute-forced credentials. The symptoms are randomly named binaries being uploaded to the data directory and executed with the permissions of the postgresql user, apparently via an extension or an untrusted PL.
Is anyone tracking or investigating this?
Please cite actual instances of such reports. Vague queries like this
help nobody.I imagine Steve is reacting to this report from today:
/messages/by-id/CANozSKLGgWDpzfua2L=OGFN=Dg3Po98UjqJJ18gBVFR1-yK5+A@mail.gmail.comI recall something similar being reported a few weeks ago,
/messages/by-id/020901d3f14c$512a46d0$f37ed470$@gmail.com
but am
too lazy to trawl the archives right now.
Yes, plus I recall a couple of discussions on IRC with similar behaviour, and
a few more details about how the binaries were being uploaded.
Furthermore, security concerns are best addressed to the security
mailing list.Unless there's some evidence that these attacks are getting in through
a heretofore unknown PG security vulnerability, rather than user
misconfiguration (such as weak/no password), I'm not sure what the
security list would have to offer. Right now it seems like Steve's move
to try to gather more evidence is quite the right thing to do.
Yeah. It's not a security issue with postgresql itself, I don't believe, so not
really something that has to go to the security alias. It's more of an ops
issue, but I thought I'd ask here to see if anyone was already looking at it,
and to raise a flag if they weren't.
Cheers,
Steve
On 06/08/2018 04:54 PM, Steve Atkins wrote:
On Jun 8, 2018, at 1:47 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
On 06/08/2018 04:34 PM, Steve Atkins wrote:
I've noticed a steady trickle of reports of postgresql servers being compromised via being left available to the internet with insecure or default configuration, or brute-forced credentials. The symptoms are randomly named binaries being uploaded to the data directory and executed with the permissions of the postgresql user, apparently via an extension or an untrusted PL.
Is anyone tracking or investigating this?
Please cite actual instances of such reports. Vague queries like this
help nobody.I imagine Steve is reacting to this report from today:
/messages/by-id/CANozSKLGgWDpzfua2L=OGFN=Dg3Po98UjqJJ18gBVFR1-yK5+A@mail.gmail.comI recall something similar being reported a few weeks ago,
OK, those appeared on other mailing lists I don't subscribe to, so I was
missing context.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
"Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
Please cite actual instances of such reports. Vague queries like
this help nobody.
We do also get them on the IRC channel every once in a while, not
very frequently but enough to notice (maybe 2-3 so far this year?).
Tom> Unless there's some evidence that these attacks are getting in
Tom> through a heretofore unknown PG security vulnerability, rather
Tom> than user misconfiguration (such as weak/no password), I'm not
Tom> sure what the security list would have to offer. Right now it
Tom> seems like Steve's move to try to gather more evidence is quite
Tom> the right thing to do.
Right. All the instances on IRC that I'm personally aware of have
followed this pattern: either the user has used "host all all 0.0.0.0/0
trust", or they used "host all all 0.0.0.0/0 md5" where the password for
the postgres user has been one where it's plausible that a simple
automated dictionary attack could have guessed it (e.g. "654321" in one
report).
Excuses for doing this have varied (but I'm pretty sure I've heard "we
put that in while trying to fix a problem and forgot to take it out"
more than once - so it's worth a reminder that one should never, ever,
suggest this on any help forum).
The reports are similar enough and generic enough that it seems pretty
certain that the scanning and subsequent compromise is all automated;
some reports have included a (failed) attempt to use local root exploits
to escalate from the postgres user to root access. Compromised systems
have been reportedly used as DDoS traffic sources and for cryptocurrency
mining, but obviously other uses can't be ruled out. I don't know of any
reports of data being exfiltrated or modified, but that doesn't mean it
doesn't happen.
I have (private) logs of the channel going back a while, but I haven't
made any attempt to track how often it happens - the above is basically
all from memory.
--
Andrew (irc:RhodiumToad)
On 06/08/2018 06:13 PM, Andrew Gierth wrote:
"Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes:
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
Please cite actual instances of such reports. Vague queries like
this help nobody.We do also get them on the IRC channel every once in a while, not
very frequently but enough to notice (maybe 2-3 so far this year?).Tom> Unless there's some evidence that these attacks are getting in
Tom> through a heretofore unknown PG security vulnerability, rather
Tom> than user misconfiguration (such as weak/no password), I'm not
Tom> sure what the security list would have to offer. Right now it
Tom> seems like Steve's move to try to gather more evidence is quite
Tom> the right thing to do.Right. All the instances on IRC that I'm personally aware of have
followed this pattern: either the user has used "host all all 0.0.0.0/0
trust", or they used "host all all 0.0.0.0/0 md5" where the password for
the postgres user has been one where it's plausible that a simple
automated dictionary attack could have guessed it (e.g. "654321" in one
report).Excuses for doing this have varied (but I'm pretty sure I've heard "we
put that in while trying to fix a problem and forgot to take it out"
more than once - so it's worth a reminder that one should never, ever,
suggest this on any help forum).The reports are similar enough and generic enough that it seems pretty
certain that the scanning and subsequent compromise is all automated;
some reports have included a (failed) attempt to use local root exploits
to escalate from the postgres user to root access. Compromised systems
have been reportedly used as DDoS traffic sources and for cryptocurrency
mining, but obviously other uses can't be ruled out. I don't know of any
reports of data being exfiltrated or modified, but that doesn't mean it
doesn't happen.I have (private) logs of the channel going back a while, but I haven't
made any attempt to track how often it happens - the above is basically
all from memory.
Well, I guess every service is going to be vulnerable to
misconfiguration. The fact that they are automated (oresumably knowing
how to speak the wire protocol or using a library that does) is a bit
scary, but not surprising.
My view is that generally DB servers should not be reachable from the
internet in the first place, certainly not without a client cert, but
ideally not at all. Then misconfigurations like those Andrew refers to
above would not be so lethal.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Please cite actual instances of such reports. Vague queries like this help
nobody.
There were several questions on SO
https://stackoverflow.com/questions/49815460
https://stackoverflow.com/questions/47499766
https://stackoverflow.com/questions/47741077
https://dba.stackexchange.com/questions/184540
And a blog post going into details on how that specific attack works.
--
Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html
"Thomas" == Thomas Kellerer <spam_eater@gmx.net> writes:
Thomas> And a blog post going into details on how that specific attack works.
*headdesk*
*headdesk*
*headdesk*
FOR THE LOVE OF LITTLE APPLES, why, in an article as comprehensive as
this, did they not list in the "quick tips" at the end, the quickest and
most absolutely basic and essential tip of all, which is "don't open up
your database for superuser access from the whole world" ???
To become vulnerable to this attack, you have to do ALL of these:
- give your db a public IP
- allow access (or forget to prevent access) to it through any
firewall
- configure pg to listen on the public IP
- explicitly add an entry to pg_hba.conf that allows access from
0.0.0.0/0 for all users or at least the postgres user
- AND have a guessable password on the postgres user or explicitly
use "trust" on the above hba entry
*headdesk*
--
Andrew (irc:RhodiumToad)
On 06/09/2018 03:27 AM, Andrew Gierth wrote:
"Thomas" == Thomas Kellerer <spam_eater@gmx.net> writes:
Thomas> And a blog post going into details on how that specific attack works.
*headdesk*
*headdesk*
*headdesk*
FOR THE LOVE OF LITTLE APPLES, why, in an article as comprehensive as
this, did they not list in the "quick tips" at the end, the quickest and
most absolutely basic and essential tip of all, which is "don't open up
your database for superuser access from the whole world" ???To become vulnerable to this attack, you have to do ALL of these:
- give your db a public IP
- allow access (or forget to prevent access) to it through any
firewall
- configure pg to listen on the public IP
- explicitly add an entry to pg_hba.conf that allows access from
0.0.0.0/0 for all users or at least the postgres user
- AND have a guessable password on the postgres user or explicitly
use "trust" on the above hba entry*headdesk*
Against stupidity the Gods themselves contend in vain.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services