100% cpu usage on some postmaster processes kill the complete database

On 01/03/12 16:41, Paul Dunkler wrote:

Hi List,

we are currently running a rather large postgresql-installation with approximately 4k Transactions and 50k index scans per second.

In the last days on some times of the day (irregular - 3-4 times a day), some of the postmaster processes are running with 100% cpu usage. That leads to a totally breakdown of the query execution. We see tons of statements which are correctly automatically aborted by our statement_timeout set to 15 seconds. I tried to search, but do not really recognize what the problem could be there...

Some things i have checked:
- We are not running any bulk jobs or maintenance scripts at this time
- No system errors in any logs during that slowdowns
- I/O Performance seems fine. No high IO Wait amount... But IO Write totally drops in that times because it seems that no postgres process can perform any update

I just installed a script, which prints me out the top and ps axf information for facing out the problem. I will post a snippet of the top here:

Combine that with this:
SELECT * FROM pg_stat_activity;

That will let you line up pids from top with active queries.

--
Richard Huxton
Archonet Ltd

I did that now - and analyzed the situation a bit. There are only queries running which will process very fast under high load (only index scans, very low rates of sequential scans). I found a remarkable number of Insert statements...

And sometimes when that happens, the CPU Utilization is going up to nearby 100% too and 98% is system usage...

Am 01.03.2012 um 18:02 schrieb Richard Huxton:

On 01/03/12 16:41, Paul Dunkler wrote:

Hi List,

we are currently running a rather large postgresql-installation with approximately 4k Transactions and 50k index scans per second.

In the last days on some times of the day (irregular - 3-4 times a day), some of the postmaster processes are running with 100% cpu usage. That leads to a totally breakdown of the query execution. We see tons of statements which are correctly automatically aborted by our statement_timeout set to 15 seconds. I tried to search, but do not really recognize what the problem could be there...

Some things i have checked:
- We are not running any bulk jobs or maintenance scripts at this time
- No system errors in any logs during that slowdowns
- I/O Performance seems fine. No high IO Wait amount... But IO Write totally drops in that times because it seems that no postgres process can perform any update

I just installed a script, which prints me out the top and ps axf information for facing out the problem. I will post a snippet of the top here:

Combine that with this:
SELECT * FROM pg_stat_activity;

That will let you line up pids from top with active queries.

--
Richard Huxton
Archonet Ltd

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

--
Mit freundlichen Grüßen

Paul Dunkler

-----------------------------------------------------
XYRALITY GmbH • Lerchenstraße 28a • 22767 Hamburg
Paul Dunkler • Softwareentwickler
Mail: paul.dunkler@xyrality.com
Tel: +49 (0) 40 23 51 78 97
Mobil: +49 (0) 151 252 228 42
Fax: +49 (0) 40 23 51 78 98
Web: http://www.xyrality.com/
Registergericht: Hamburg HRB 115332
Geschäftsführer: Sven Ossenbrüggen & Alexander Spohr
-----------------------------------------------------

On 01/03/12 19:41, Paul Dunkler wrote:

I did that now - and analyzed the situation a bit. There are only queries
running which will process very fast under high load (only index scans, very low
rates of sequential scans). I found a remarkable number of Insert statements...

And sometimes when that happens, the CPU Utilization is going up to nearby 100%
too and 98% is system usage...

You're running on a box larger than I'm used to, so this is only
speculation. I'm wondering whether you're hitting problems with lock
contention or some such. It looks like you've got 48 cores there all at
about 100% possibly none of them getting much chance to do any work.

Oddly, the totals you posted in your top output show 6.3% user cpu
usage, which I can't make match with 50-odd processes all approaching
100% cpu.

Perhaps have a look at vmstat output too - see if context-switches spike
unusually high during these periods (sorry - no idea what an unusually
high number would be on a machine like yours).

Reducing the number of concurrent backends might help, but that rather
depends on whether my guess is right.

If no-one more experienced than me comes along shortly, try reposting to
the performance list. There are people there who are used to machines of
this size.

--
Richard Huxton
Archonet Ltd

Hi,

You're running on a box larger than I'm used to, so this is only speculation. I'm wondering whether you're hitting problems with lock contention or some such. It looks like you've got 48 cores there all at about 100% possibly none of them getting much chance to do any work.

Yes. That is what i see too...

Oddly, the totals you posted in your top output show 6.3% user cpu usage, which I can't make match with 50-odd processes all approaching 100% cpu.

Sometimes the Cpu is only 7% used in this times but at other peak times, the cpu is used 100% (97% system load) as i posted before.

Perhaps have a look at vmstat output too - see if context-switches spike unusually high during these periods (sorry - no idea what an unusually high number would be on a machine like yours).

Thanks. i will have a look at it.

Reducing the number of concurrent backends might help, but that rather depends on whether my guess is right.

Yes... already thought about setting up a connection pool.

If no-one more experienced than me comes along shortly, try reposting to the performance list. There are people there who are used to machines of this size.

Thanks. I will wait a time and consider re-posting it to the perfornance list.

--
Richard Huxton
Archonet Ltd

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

--
Mit freundlichen Grüßen

Paul Dunkler

-----------------------------------------------------
XYRALITY GmbH • Lerchenstraße 28a • 22767 Hamburg
Paul Dunkler • Softwareentwickler
Mail: paul.dunkler@xyrality.com
Tel: +49 (0) 40 23 51 78 97
Mobil: +49 (0) 151 252 228 42
Fax: +49 (0) 40 23 51 78 98
Web: http://www.xyrality.com/
Registergericht: Hamburg HRB 115332
Geschäftsführer: Sven Ossenbrüggen & Alexander Spohr
-----------------------------------------------------

On Thu, Mar 1, 2012 at 1:52 PM, Paul Dunkler <paul.dunkler@xyrality.com> wrote:

Hi,

You're running on a box larger than I'm used to, so this is only speculation. I'm wondering whether you're hitting problems with lock contention or some such. It looks like you've got 48 cores there all at about 100% possibly none of them getting much chance to do any work.

Yes. That is what i see too...

Oddly, the totals you posted in your top output show 6.3% user cpu usage, which I can't make match with 50-odd processes all approaching 100% cpu.

Sometimes the Cpu is only 7% used in this times but at other peak times, the cpu is used 100% (97% system load) as i posted before.

Perhaps have a look at vmstat output too - see if context-switches spike unusually high during these periods (sorry - no idea what an unusually high number would be on a machine like yours).

Thanks. i will have a look at it.

Reducing the number of concurrent backends might help, but that rather depends on whether my guess is right.

Yes... already thought about setting up a connection pool.

If no-one more experienced than me comes along shortly, try reposting to the performance list. There are people there who are used to machines of this size.

Thanks. I will wait a time and consider re-posting it to the perfornance list.

I'd look at vmstat and iostat output (vmstat 10, iostat -xd 10) for a
few minutes. In vmstat look for high (>100k) ints or cs numbers, in
iostat look at io utilization.