Really out of memory?
I have a linux postgres server in the field. Its version is:
PostgreSQL 8.2.4 on i686-redhat-linux-gnu, compiled by GCC gcc (GCC) 4.1.1 20070105 (Red Hat 4.1.1-51)
(aka postgresql-8.2.4-1PGDG)
A few days ago, its log started showing this:
May 31 02:59:40 sfmelwss postgres[30103]: [1-1] ERROR: out of memory
May 31 02:59:40 sfmelwss postgres[30103]: [1-2] DETAIL: Failed on request of size 16777212.
May 31 03:02:40 sfmelwss postgres[31490]: [1-1] ERROR: out of memory
May 31 03:02:40 sfmelwss postgres[31490]: [1-2] DETAIL: Failed on request of size 16777212.
May 31 03:05:40 sfmelwss postgres[31913]: [1-1] ERROR: out of memory
May 31 03:05:40 sfmelwss postgres[31913]: [1-2] DETAIL: Failed on request of size 16777212.
That seems pretty self-explainitory. But I'm not so sure, because SAR
says:
02:30:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
02:40:01 AM 13332 1003316 98.69 130448 198188 1034572 13996 1.33 32
02:50:01 AM 17116 999532 98.32 128708 196384 1034596 13972 1.33 44
03:00:01 AM 16372 1000276 98.39 129128 196388 1034596 13972 1.33 44
03:10:01 AM 17220 999428 98.31 128268 196828 1034736 13832 1.32 132
03:20:01 AM 14416 1002232 98.58 130464 197348 1035224 13344 1.27 152
03:30:01 AM 16292 1000356 98.40 127604 196684 1035700 12868 1.23 168
...which indicates there was still plenty of space left in swap. Now, I
realize I don't want to be actually using my swap, but I'm wondering if
the out of memory messages are a red herring. Should I be looking at
something else, like the number of processes, open files, or shared memory
segments?
FWIW, I have disabled the OOM killer (but not, as I understand it, my
swap space) by setting:
vm.overcommit_memory = 2
vm.overcommit_ratio = 100
On Tue, Jun 02, 2009 at 11:10:04AM -0700, Ben Chobot wrote:
I have a linux postgres server in the field. Its version is:
PostgreSQL 8.2.4 on i686-redhat-linux-gnu, compiled by GCC gcc (GCC) 4.1.1 20070105 (Red Hat 4.1.1-51)
(aka postgresql-8.2.4-1PGDG)
A few days ago, its log started showing this:
May 31 02:59:40 sfmelwss postgres[30103]: [1-1] ERROR: out of memory
May 31 02:59:40 sfmelwss postgres[30103]: [1-2] DETAIL: Failed on request of size 16777212.
Add even more swap. By turning overcommit off you make the kernel
really pessimistic about how much memory is in use.
...which indicates there was still plenty of space left in swap. Now, I
realize I don't want to be actually using my swap, but I'm wondering if
the out of memory messages are a red herring. Should I be looking at
something else, like the number of processes, open files, or shared
memory segments?
You got as much swap as memory, try doubling it.
Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/
Show quoted text
Please line up in a tree and maintain the heap invariant while
boarding. Thank you for flying nlogn airlines.
Ben Chobot wrote:
May 31 02:59:40 sfmelwss postgres[30103]: [1-1] ERROR: out of memory
May 31 02:59:40 sfmelwss postgres[30103]: [1-2] DETAIL: Failed on
request of size 16777212.
Thats a 16MB request is that your work_mem size or something by any chance?
02:30:01 AM kbmemfree kbmemused %memused kbbuffers kbcached
kbswpfree kbswpused %swpused kbswpcad
02:40:01 AM 13332 1003316 98.69 130448 198188
1034572 13996 1.33 32
so you only have 13MB memory free. you have -do- have free swap, however.
hey, is any ULIMIT in effect for the postgres process?
On Tue, 2 Jun 2009, Martijn van Oosterhout wrote:
On Tue, Jun 02, 2009 at 11:10:04AM -0700, Ben Chobot wrote:
May 31 02:59:40 sfmelwss postgres[30103]: [1-1] ERROR: out of memory
May 31 02:59:40 sfmelwss postgres[30103]: [1-2] DETAIL: Failed on request of size 16777212.Add even more swap. By turning overcommit off you make the kernel
really pessimistic about how much memory is in use.
Is it so pessimistic that it won't try to swap out 16MB into almost 1GB of
free swap? That seems surprising to me.
Ben Chobot <bench@silentmedia.com> writes:
May 31 02:59:40 sfmelwss postgres[30103]: [1-1] ERROR: out of memory
May 31 02:59:40 sfmelwss postgres[30103]: [1-2] DETAIL: Failed on request of size 16777212.
So the kernel isn't letting PG have any more memory.
That seems pretty self-explainitory. But I'm not so sure, because SAR
says:
...
...which indicates there was still plenty of space left in swap.
Which the kernel isn't letting us use. Check the "ulimit" settings
that the postmaster is being started with. On a Linux box, any of
the -d -m or -v settings might cause this.
It's possible you are running out of 32-bit address space in the backend
process, but what seems more likely is that the per-process ulimit is
unreasonably small.
regards, tom lane
On Tue, 2 Jun 2009, John R Pierce wrote:
Ben Chobot wrote:
May 31 02:59:40 sfmelwss postgres[30103]: [1-1] ERROR: out of memory
May 31 02:59:40 sfmelwss postgres[30103]: [1-2] DETAIL: Failed on request of size 16777212.Thats a 16MB request is that your work_mem size or something by any chance?
work_mem is 1MB, but maintenance_work_mem is 16MB. So it's probably
autovacuum kicking off most of these messages.
02:30:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree
kbswpused %swpused kbswpcad
02:40:01 AM 13332 1003316 98.69 130448 198188 1034572
13996 1.33 32so you only have 13MB memory free. you have -do- have free swap, however.
hey, is any ULIMIT in effect for the postgres process?
Not that I can tell. There's nothing special in /etc/init.d/postgresql or
/etc/sysconfig/pgsql/postgresql, and ulimit -a shows:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16127
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
max rt priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16127
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Is there a way to see what the limits are for a given pid? I don't see
anything obviously relevant in /proc/<pid>/....
On Tue, 2 Jun 2009, Tom Lane wrote:
It's possible you are running out of 32-bit address space in the backend
process, but what seems more likely is that the per-process ulimit is
unreasonably small.
I only have 1GB in the machine, and another 1GB of swap, so running out of
32-bit address space seems unlikely. Is there any way to rule it out?
Ben Chobot <bench@silentmedia.com> writes:
hey, is any ULIMIT in effect for the postgres process?
Not that I can tell. There's nothing special in /etc/init.d/postgresql or
/etc/sysconfig/pgsql/postgresql, and ulimit -a shows:
That tells you the limits for your interactive shell, but a daemon might
be started under some other set of limits.
Is there a way to see what the limits are for a given pid? I don't see
anything obviously relevant in /proc/<pid>/....
You don't have /proc/<pid>/limits ?
regards, tom lane
On Tue, 2 Jun 2009, Tom Lane wrote:
Is there a way to see what the limits are for a given pid? I don't see
anything obviously relevant in /proc/<pid>/....You don't have /proc/<pid>/limits ?
Nope. I'd like to believe I would consider that "obviously relevant." :)
This server is running 2.6.20-1.2962.fc6, but should be upgraded to
2.6.26.8-57.fc8 in a month or two, which does provide that file. I was
hoping to not have to wait till then to understand what's going wrong
though.
Ben Chobot <bench@silentmedia.com> writes:
On Tue, 2 Jun 2009, Tom Lane wrote:
You don't have /proc/<pid>/limits ?
Nope. I'd like to believe I would consider that "obviously relevant." :)
Next best thing I can think of is to stick "ulimit -a >/tmp/mylimits"
into the postgres initscript and restart.
If the initscript is starting postgres via "su -l", it might be better
to add the command in postgres' ~/.bashrc or some such place. You
have to consider the possibility that the su is changing the ulimit
environment.
regards, tom lane
On Tue, Jun 02, 2009 at 11:45:11AM -0700, Ben Chobot wrote:
On Tue, 2 Jun 2009, Martijn van Oosterhout wrote:
On Tue, Jun 02, 2009 at 11:10:04AM -0700, Ben Chobot wrote:
May 31 02:59:40 sfmelwss postgres[30103]: [1-1] ERROR: out of memory
May 31 02:59:40 sfmelwss postgres[30103]: [1-2] DETAIL: Failed on request of size 16777212.Add even more swap. By turning overcommit off you make the kernel
really pessimistic about how much memory is in use.Is it so pessimistic that it won't try to swap out 16MB into almost 1GB
of free swap? That seems surprising to me.
It's got nothing to do with how much swap is in use. It's preventing
you from allocating memory that *hypothetically* might not be available
if every byte of allocated memory were actually used.
For example, on my desktop I have 1GB of RAM of which about 600MB is
free, yet there is 1.4GB committed. With overcommit off my machine
may not boot. As you can see, only 25% of committed memory is actually
needed, because lots of pages are blank or shared. Ofcourse, all those
copies of libc are realistically never not going to be shared so it's a
good bet.
But with overcommit off you can see that you might want to have double
or triple the amount of swap to handle the hypothetical case.
I'm not saying this is necessarily the case for you, but it's the first
thing that came to mind and relatively easy to check.
Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/
Show quoted text
Please line up in a tree and maintain the heap invariant while
boarding. Thank you for flying nlogn airlines.
On Tue, 2 Jun 2009, Martijn van Oosterhout wrote:
It's got nothing to do with how much swap is in use. It's preventing
you from allocating memory that *hypothetically* might not be available
if every byte of allocated memory were actually used.For example, on my desktop I have 1GB of RAM of which about 600MB is
free, yet there is 1.4GB committed. With overcommit off my machine
may not boot. As you can see, only 25% of committed memory is actually
needed, because lots of pages are blank or shared. Ofcourse, all those
copies of libc are realistically never not going to be shared so it's a
good bet.But with overcommit off you can see that you might want to have double
or triple the amount of swap to handle the hypothetical case.
No, sorry, I don't see why I would need more swap when I've disabled
memory overcommit. As I understand it, the kernel should be able to
allocate (swap + (physical * overcommit_ratio)), which in my case is just
swap+physical, and it seems to not want to do that.