posgresql.log

On 05/21/2018 02:40 PM, Bartosz Dmytrak wrote:

Hi Gurus,

Looking into my postgresql.log on one of my test servers I found scary
entry:

Is there a Web app running on this server?

The log entries below are from the Postgres logs in?:

/var/log/postgresql/

--2018-05-19 05:28:21--  http://207.148.79.161/post0514/post

Connecting to 207.148.79.161:80... connected.

HTTP request sent, awaiting response... 200 OK

Length: 1606648 (1.5M) [application/octet-stream]

Hmm, the below says it downloaded 12.5M.

Saving to: ‘/var/lib/postgresql/10/main/postgresq1’

The postgresq1 file is actually there?

If so have you looked at the file:

file postgresq1

to get an idea of what it is?

0K .......... .......... .......... .......... ..........  3% 71.0K 21s

    50K .......... .......... .......... .......... ..........  6%
106K 17s

   100K .......... .......... .......... .......... ..........  9%
213K 13s

   150K .......... .......... .......... .......... .......... 12%
213K 11s

   200K .......... .......... .......... .......... .......... 15% 16.3M 9s

   250K .......... .......... .......... .......... .......... 19%  215K 8s

   300K .......... .......... .......... .......... .......... 22% 15.6M 7s

   350K .......... .......... .......... .......... .......... 25% 11.7M 6s

   400K .......... .......... .......... .......... .......... 28%  219K 5s

   450K .......... .......... .......... .......... .......... 31% 12.1M 5s

   500K .......... .......... .......... .......... .......... 35% 11.7M 4s

   550K .......... .......... .......... .......... .......... 38% 12.2M 3s

   600K .......... .......... .......... .......... .......... 41% 12.1M 3s

   650K .......... .......... .......... .......... .......... 44%  228K 3s

   700K .......... .......... .......... .......... .......... 47% 12.2M 3s

   750K .......... .......... .......... .......... .......... 50% 12.1M 2s

   800K .......... .......... .......... .......... .......... 54% 11.7M 2s

   850K .......... .......... .......... .......... .......... 57% 12.1M 2s

   900K .......... .......... .......... .......... .......... 60% 11.8M 2s

   950K .......... .......... .......... .......... .......... 63% 12.1M 1s

  1000K .......... .......... .......... .......... .......... 66% 12.0M 1s

  1050K .......... .......... .......... .......... .......... 70%  243K 1s

  1100K .......... .......... .......... .......... .......... 73% 12.1M 1s

  1150K .......... .......... .......... .......... .......... 76% 12.1M 1s

  1200K .......... .......... .......... .......... .......... 79% 11.7M 1s

  1250K .......... .......... .......... .......... .......... 82% 12.1M 1s

  1300K .......... .......... .......... .......... .......... 86% 12.1M 0s

  1350K .......... .......... .......... .......... .......... 89% 11.8M 0s

  1400K .......... .......... .......... .......... .......... 92% 12.1M 0s

  1450K .......... .......... .......... .......... .......... 95% 12.1M 0s

  1500K .......... .......... .......... .......... .......... 98% 11.8M 0s

  1550K .......... ........ 100% 12.5M=2.6s

2018-05-19 05:28:25 (598 KB/s) -
‘/var/lib/postgresql/10/main/postgresq1’ saved [1606648/1606648]

Downloaded file is not posgresql but postgresq1(one).

It was pure pg instalation without any contrib modules addons etc,
istalled on ubuntu box by apt manager using repos:

http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main

http://apt.postgresql.org/pub/repos/apt xenial-pgdg

I have never seen such entry on other my other servers…

Could you be so kind and explain me what is it? I am afraid my postgres
has been hacekd.

Best Regards

*/Bartosz Dmytrak/*

--
Adrian Klaver
adrian.klaver@aklaver.com

On Mon, May 21, 2018 at 2:40 PM Bartosz Dmytrak <bdmytrak@gmail.com> wrote:

Hi Gurus,

Looking into my postgresql.log on one of my test servers I found scary
entry:

--2018-05-19 05:28:21-- http://207.148.79.161/post0514/post

Connecting to 207.148.79.161:80... connected.

HTTP request sent, awaiting response... 200 OK

Length: 1606648 (1.5M) [application/octet-stream]

Saving to: ‘/var/lib/postgresql/10/main/postgresq1’

0K .......... .......... .......... .......... .......... 3% 71.0K
21s

50K .......... .......... .......... .......... .......... 6% 106K
17s

100K .......... .......... .......... .......... .......... 9% 213K
13s

150K .......... .......... .......... .......... .......... 12% 213K
11s

200K .......... .......... .......... .......... .......... 15% 16.3M 9s

250K .......... .......... .......... .......... .......... 19% 215K 8s

300K .......... .......... .......... .......... .......... 22% 15.6M 7s

350K .......... .......... .......... .......... .......... 25% 11.7M 6s

400K .......... .......... .......... .......... .......... 28% 219K 5s

450K .......... .......... .......... .......... .......... 31% 12.1M 5s

500K .......... .......... .......... .......... .......... 35% 11.7M 4s

550K .......... .......... .......... .......... .......... 38% 12.2M 3s

600K .......... .......... .......... .......... .......... 41% 12.1M 3s

650K .......... .......... .......... .......... .......... 44% 228K 3s

700K .......... .......... .......... .......... .......... 47% 12.2M 3s

750K .......... .......... .......... .......... .......... 50% 12.1M 2s

800K .......... .......... .......... .......... .......... 54% 11.7M 2s

850K .......... .......... .......... .......... .......... 57% 12.1M 2s

900K .......... .......... .......... .......... .......... 60% 11.8M 2s

950K .......... .......... .......... .......... .......... 63% 12.1M 1s

1000K .......... .......... .......... .......... .......... 66% 12.0M 1s

1050K .......... .......... .......... .......... .......... 70% 243K 1s

1100K .......... .......... .......... .......... .......... 73% 12.1M 1s

1150K .......... .......... .......... .......... .......... 76% 12.1M 1s

1200K .......... .......... .......... .......... .......... 79% 11.7M 1s

1250K .......... .......... .......... .......... .......... 82% 12.1M 1s

1300K .......... .......... .......... .......... .......... 86% 12.1M 0s

1350K .......... .......... .......... .......... .......... 89% 11.8M 0s

1400K .......... .......... .......... .......... .......... 92% 12.1M 0s

1450K .......... .......... .......... .......... .......... 95% 12.1M 0s

1500K .......... .......... .......... .......... .......... 98% 11.8M 0s

1550K .......... ........ 100%
12.5M=2.6s

2018-05-19 05:28:25 (598 KB/s) - ‘/var/lib/postgresql/10/main/postgresq1’
saved [1606648/1606648]

Downloaded file is not posgresql but postgresq1(one).

It was pure pg instalation without any contrib modules addons etc,
istalled on ubuntu box by apt manager using repos:

http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main

http://apt.postgresql.org/pub/repos/apt xenial-pgdg

I have never seen such entry on other my other servers…

Could you be so kind and explain me what is it? I am afraid my postgres
has been hacekd.

Best Regards

*Bartosz Dmytrak*

If this is a test server and you can take it offline for forensics I would
do so, especially if it could provide a path to other internal or critical
resources. If you can image it for safekeeping and forensics, even better.

That appears to be output from wget but the intrusion, if any, could be
through any number of vectors (web, ssh, local attack, etc.) not directly
related to PostgreSQL. Check in your other logs starting with a search for
anything related to that IP address.

Verify the usual. Patches up to date, ports appropriately firewalled off,
no default passwords, etc.

IP comes back to vultr.com which is a cloud company (i.e. could be anyone)
but if it is an attack perhaps contact their abuse department.

Unless you are positive the server was not attacked, don't trust it unless
you can be absolutely certain it is clean. Best bet is to backup any
critical data (and check it for trustworthiness), wipe and rebuild.

Only you (well, OK, maybe them, now) know what data was on this server but
depending on its importance, internal policies, legal requirements and
agreements with third-parties you may have notification requirements and
could need to engage forensics experts.

Good luck,
Steve

On May 21, 2018, at 3:21 PM, Steve Crawford <scrawford@pinpointresearch.com> wrote:

If this is a test server and you can take it offline for forensics I would do so, especially if it could provide a path to other internal or critical resources. If you can image it for safekeeping and forensics, even better.

+1

It's compromised. Image it if possible; save the compromise payload you know about if not.

Treat it as compromised and unsafe to attach to a network until you completely wipe and reinstall it.

That appears to be output from wget but the intrusion, if any, could be through any number of vectors (web, ssh, local attack, etc.) not directly related to PostgreSQL. Check in your other logs starting with a search for anything related to that IP address.

It's probably a compromise via postgresql open to the network with insecure settings. I've seen several of those reported recently, and this one is saving it's payload to the postgresql data directory - somewhere no other user or app will have access to, but which a compromised postgresql can easily write to.

Check the pg_hba.conf and packet filter / firewall settings and see what the issue may be. Do the same checks on all your other postgresql servers, test and production. If there's a configuration mistake that let one server be compromised it's may well be there on others too.

Verify the usual. Patches up to date, ports appropriately firewalled off, no default passwords, etc.

IP comes back to vultr.com which is a cloud company (i.e. could be anyone) but if it is an attack perhaps contact their abuse department.

The C&C server there is already down; It can't hurt to notify them, but I doubt Choopa would be particularly interested beyond that point unless a subpoena or search warrant were involved.

Unless you are positive the server was not attacked, don't trust it unless you can be absolutely certain it is clean. Best bet is to backup any critical data (and check it for trustworthiness), wipe and rebuild.

+1.

Only you (well, OK, maybe them, now) know what data was on this server but depending on its importance, internal policies, legal requirements and agreements with third-parties you may have notification requirements and could need to engage forensics experts.

Cheers,
Steve

On 05/21/2018 04:40 PM, Bartosz Dmytrak wrote:

Hi Gurus,

Looking into my postgresql.log on one of my test servers I found scary entry:

--2018-05-19 05:28:21-- http://207.148.79.161/post0514/post

Connecting to 207.148.79.161:80... connected.

HTTP request sent, awaiting response... 200 OK

Length: 1606648 (1.5M) [application/octet-stream]

Saving to: ‘/var/lib/postgresql/10/main/postgresq1’

0K .......... .......... .......... .......... ..........  3% 71.0K 21s

    50K .......... .......... .......... .......... ..........  6% 106K 17s

   100K .......... .......... .......... .......... ..........  9% 213K 13s

   150K .......... .......... .......... .......... .......... 12% 213K 11s

[snip]

1500K .......... .......... .......... .......... .......... 98% 11.8M 0s

  1550K .......... ........ 100% 12.5M=2.6s

2018-05-19 05:28:25 (598 KB/s) - ‘/var/lib/postgresql/10/main/postgresq1’
saved [1606648/1606648]

Downloaded file is not posgresql but postgresq1(one).

It was pure pg instalation without any contrib modules addons etc,
istalled on ubuntu box by apt manager using repos:

http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main

http://apt.postgresql.org/pub/repos/apt xenial-pgdg

I have never seen such entry on other my other servers…

Could you be so kind and explain me what is it? I am afraid my postgres
has been hacekd.

This looks like what happens when the adobe flash player package downloads
the closed-source binary installer.  Thus, I wouldn't be surprised if the
repository package isn't downloading the installation binaries from
http://207.148.79.161/post0514/post.

--
Angular momentum makes the world go 'round.

-----Original Message-----
From: Adrian Klaver [mailto:adrian.klaver@aklaver.com]
Sent: Tuesday, May 22, 2018 12:03 AM
To: Bartosz Dmytrak <bdmytrak@gmail.com>; pgsql-general@postgresql.org
Subject: Re: posgresql.log

On 05/21/2018 02:40 PM, Bartosz Dmytrak wrote:

Hi Gurus,

Looking into my postgresql.log on one of my test servers I found scary
entry:

Is there a Web app running on this server?

The log entries below are from the Postgres logs in?:

/var/log/postgresql/

--2018-05-19 05:28:21-- http://207.148.79.161/post0514/post

Connecting to 207.148.79.161:80... connected.

HTTP request sent, awaiting response... 200 OK

Length: 1606648 (1.5M) [application/octet-stream]

Hmm, the below says it downloaded 12.5M.

Saving to: ‘/var/lib/postgresql/10/main/postgresq1’

The postgresq1 file is actually there?

If so have you looked at the file:

file postgresq1

to get an idea of what it is?

0K .......... .......... .......... .......... .......... 3% 71.0K
21s

50K .......... .......... .......... .......... .......... 6%
106K 17s

100K .......... .......... .......... .......... .......... 9%
213K 13s

150K .......... .......... .......... .......... .......... 12%
213K 11s

200K .......... .......... .......... .......... .......... 15%
16.3M 9s

250K .......... .......... .......... .......... .......... 19%
215K 8s

300K .......... .......... .......... .......... .......... 22%
15.6M 7s

350K .......... .......... .......... .......... .......... 25%
11.7M 6s

400K .......... .......... .......... .......... .......... 28%
219K 5s

450K .......... .......... .......... .......... .......... 31%
12.1M 5s

500K .......... .......... .......... .......... .......... 35%
11.7M 4s

550K .......... .......... .......... .......... .......... 38%
12.2M 3s

600K .......... .......... .......... .......... .......... 41%
12.1M 3s

650K .......... .......... .......... .......... .......... 44%
228K 3s

700K .......... .......... .......... .......... .......... 47%
12.2M 3s

750K .......... .......... .......... .......... .......... 50%
12.1M 2s

800K .......... .......... .......... .......... .......... 54%
11.7M 2s

850K .......... .......... .......... .......... .......... 57%
12.1M 2s

900K .......... .......... .......... .......... .......... 60%
11.8M 2s

950K .......... .......... .......... .......... .......... 63%
12.1M 1s

1000K .......... .......... .......... .......... .......... 66%
12.0M 1s

1050K .......... .......... .......... .......... .......... 70%
243K 1s

1100K .......... .......... .......... .......... .......... 73%
12.1M 1s

1150K .......... .......... .......... .......... .......... 76%
12.1M 1s

1200K .......... .......... .......... .......... .......... 79%
11.7M 1s

1250K .......... .......... .......... .......... .......... 82%
12.1M 1s

1300K .......... .......... .......... .......... .......... 86%
12.1M 0s

1350K .......... .......... .......... .......... .......... 89%
11.8M 0s

1400K .......... .......... .......... .......... .......... 92%
12.1M 0s

1450K .......... .......... .......... .......... .......... 95%
12.1M 0s

1500K .......... .......... .......... .......... .......... 98%
11.8M 0s

1550K .......... ........ 100% 12.5M=2.6s

2018-05-19 05:28:25 (598 KB/s) -
‘/var/lib/postgresql/10/main/postgresq1’ saved [1606648/1606648]

Downloaded file is not posgresql but postgresq1(one).

It was pure pg instalation without any contrib modules addons etc,
istalled on ubuntu box by apt manager using repos:

http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main

http://apt.postgresql.org/pub/repos/apt xenial-pgdg

I have never seen such entry on other my other servers…

Could you be so kind and explain me what is it? I am afraid my
postgres has been hacekd.

Best Regards

*/Bartosz Dmytrak/*

--
Adrian Klaver
adrian.klaver@aklaver.com

HI, thanks for response,
Yes - there is also webapp running on the server, but still it's rather odd to find it's logs in postgresql.log file (located in /var/log/postgresql, where my log exists). postgresq1 file exists in /var/lib/postgresql/10/main and it's binary file, I've also noticed there is a n596tx.so which is not a part of standard installation.
Fortunately there is no important data on this server so, a according to other advice, I'll rebuilt it with more aggressive security settings and I'll apply them to other servers too.

Best regards,
Bartek

-----Original Message-----
From: Steve Atkins [mailto:steve@blighty.com]
Sent: Tuesday, May 22, 2018 12:44 AM
To: PG-General Mailing List <pgsql-general@postgresql.org>
Cc: bdmytrak@gmail.com
Subject: Re: posgresql.log

On May 21, 2018, at 3:21 PM, Steve Crawford <scrawford@pinpointresearch.com> wrote:

If this is a test server and you can take it offline for forensics I would do so, especially if it could provide a path to other internal or critical resources. If you can image it for safekeeping and forensics, even better.

+1

It's compromised. Image it if possible; save the compromise payload you know about if not.

Treat it as compromised and unsafe to attach to a network until you completely wipe and reinstall it.

That appears to be output from wget but the intrusion, if any, could be through any number of vectors (web, ssh, local attack, etc.) not directly related to PostgreSQL. Check in your other logs starting with a search for anything related to that IP address.

It's probably a compromise via postgresql open to the network with insecure settings. I've seen several of those reported recently, and this one is saving it's payload to the postgresql data directory - somewhere no other user or app will have access to, but which a compromised postgresql can easily write to.

Check the pg_hba.conf and packet filter / firewall settings and see what the issue may be. Do the same checks on all your other postgresql servers, test and production. If there's a configuration mistake that let one server be compromised it's may well be there on others too.

Verify the usual. Patches up to date, ports appropriately firewalled off, no default passwords, etc.

IP comes back to vultr.com which is a cloud company (i.e. could be anyone) but if it is an attack perhaps contact their abuse department.

The C&C server there is already down; It can't hurt to notify them, but I doubt Choopa would be particularly interested beyond that point unless a subpoena or search warrant were involved.

Unless you are positive the server was not attacked, don't trust it unless you can be absolutely certain it is clean. Best bet is to backup any critical data (and check it for trustworthiness), wipe and rebuild.

+1.

Only you (well, OK, maybe them, now) know what data was on this server but depending on its importance, internal policies, legal requirements and agreements with third-parties you may have notification requirements and could need to engage forensics experts.

Cheers,
Steve

Hi Steve,
Thanks a lot. That’s virtual server, so I'll do the backup. I suppose it was hacked, but fortunately there are no important data and I can wipe it out with no hurt. According to your advice I'll check firewall rules and block new IP ranges (many of them are blocked now). There are no default passwords, but will work on more secure approach. This server has to be accessible from internet, but not necessary for every IP.

Thanks again for your help and advice. Hope this case will let other keep their servers more secure.

Regards,
Bartek

Hi Bartek

It is quite significant that your postgres log file has these entries.
Normally if a web application gets compromised allowing remote code
execution, the attacker will be able to run scripts (often via cron
and at) as the user running the web application. Typically www-data,
tomcat8 etc. If the attacker has managed to execute something as the
postgres user you need to check:
(i) that the web application server (eg tomcat) is not running as
root. I see this often enough. Then when the web app gets attacked,
root is gone on the machine;
(ii) the web application has minimally configured access vi
pg_hba.conf. For example not as the postgres user.
Just some thoughts to consider as you put your stuff back together on
the new machine.

Regards
Bob