Weird CPU utilization patterns with Postgres

Hi,

We are having a really interesting problem with our Postgres 9.3 instance
in our infrastructure.

Few days ago our box started to show huge CPU spikes while the IO Wait is
negligible on the box. After a while I have installed perf and started to
monitor the Postgres master process and here is what I have found:

Samples: 372K of event 'cycles', Event count (approx.): 110095222173,
ThreaSamples: 372K of event 'cycles', Event count (approx.): 1100 93.65%
libc-2.12.so [.] __strcoll_l
0.97% libc-2.12.so [.] memcpy
0.90% postgres [.] slot_getattr
0.88% postgres [.] nocachegetattr
0.64% postgres [.] varstr_cmp
0.52% libc-2.12.so [.] __strcmp_sse42
0.43% postgres [.] hash_any
0.32% postgres [.] pg_detoast_datum_packed
0.31% libc-2.12.so [.] __strlen_sse2
0.22% postgres [.] bttextcmp
0.18% postgres [.] ExecStoreTuple
0.14% postgres [.] MemoryContextReset
0.09% postgres [.] pgstat_end_function_usage
0.08% libc-2.12.so [.] strcoll
0.08% postgres [.] heap_hot_search_buffer
0.07% postgres [.] lc_collate_is_c
0.06% [kernel] [k] sys_semtimedop
0.06% postgres [.] heap_page_prune_opt
0.05% postgres [.] slot_getsomeattrs
0.05% postgres [.] heap_fill_tuple
0.04% postgres [.] hash_search
0.03% postgres [.] GetMemoryChunkSpace
0.03% postgres [.] heap_form_minimal_tuple
0.03% [kernel] [k] update_queue
0.02% postgres [.] ReadBufferExtended
0.02% postgres [.] memcpy@plt

It seems that the box is using __strcoll a lot. The query performance is
down, while previously the box was able to sustain with ~20 clients right
now it is hardly able to keep up with 5.

I am wondering why the root cause might be here.

Let me know if anybody has seen this before.

Regards,
Istvan

--
the sun shines for all