High CPU usage

On 2022-10-20 15:59, ertan.kucukoglu@1nar.com.tr wrote:

Hello,

I am using PostgreSQL v14.5 on Linux Debian 11.5. I recently observe very
high CPU usage on my Linux system as below

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
2357756 postgres 20 0 2441032 2,3g 4 S 298,7 67,9 2114:58
Tspjzj2Z

I could not find any file named Tspjzj2Z on the file system. I could not
find PID number using below SQL

Hi,

I'm not an expert in PostgreSQL but that looks like a rogue app, if
you're lucky just a miner running as the prostgres user, likely the
result of a postgres RCE exploited successfully... The more worring case
would be a program exfiltrating and/or encrypting the database in a
ransomware attack.

The executable has most likely been removed to hide traces, or cleaned
up automatically from ex. /tmp, however if the process is still running
you should be able to cat the executable, and any other open files,
directly from /proc/<pid>/ (look for exe and fd/*).

I strongly recommend you check other postgres servers you have, make a
copy of any process file found (for later investigation), then isolate
or shutdown these servers and proceed with a proper investigation from a
livecd or revovery OS.

There is no replication of any kind. This is a single instance server which
alows certification login only.

Is is even available from the outside world? Else you should likely
audit any internal hosts that could have accessed your postgresql
server. If you have firewall logs looks for unusual connection attempts,
any evidence of scanning, etc.

Hackers will often spend quite some time once inside to gather as much
information as possible before doing any real damage, although if this
is effectively a miner it would be less likely to be that kind of attack
as they would probably not risk getting discovered with something that
will at best make them pennies...

If you see the process having any open database files, it's possible
it's either compressing them to exfiltrate the data or encrypting them,
or both....

Hope this helps...

--
Thomas

On 10/20/22 12:59, ertan.kucukoglu@1nar.com.tr wrote:

Hello,

I am using PostgreSQL v14.5 on Linux Debian 11.5. I recently observe very
high CPU usage on my Linux system as below

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
2357756 postgres 20 0 2441032 2,3g 4 S 298,7 67,9 2114:58
Tspjzj2Z

The USER is the system user postgres which could be running any program
not necessarily the Postgres server. In this case that would seem to be
the program Tspjzj2Z.

Try running:

top -U postgres -c

to get the full command line under COMMAND.

I could not find any file named Tspjzj2Z on the file system. I could not
find PID number using below SQL

I doubt this has anything to do with the Postgres server itself.

Is this machine accessible to the Internet?

There is the possibility you have had an intrusion.

SELECT * FROM pg_stat_activity

I also see another suggestion like below to identify long running queries

SELECT max(now() - xact_start) FROM pg_stat_activity
WHERE state IN ('idle in transaction',
'active');

I get no long running query at all.

There is no replication of any kind. This is a single instance server which
alows certification login only.

I appreciate any help to figure this out.

Thanks & Regards,
Ertan

--
Adrian Klaver
adrian.klaver@aklaver.com

ertan.kucukoglu@1nar.com.tr schrieb am 20.10.2022 um 21:59:

Hello,

I am using PostgreSQL v14.5 on Linux Debian 11.5. I recently observe very
high CPU usage on my Linux system as below

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
2357756 postgres 20 0 2441032 2,3g 4 S 298,7 67,9 2114:58
Tspjzj2Z

The program name looks suspiciously as if you have been hacked.

See e.g.

https://stackoverflow.com/questions/50149034
https://dba.stackexchange.com/questions/191282/
https://www.imperva.com/blog/deep-dive-database-attacks-scarlett-johanssons-picture-used-for-crypto-mining-on-postgre-database/