Setting oom_adj on linux?
I realize this is a very platform-specific thing, but should we
consider setting the value of /proc/<pid>/oom_adj when running on
linux? See:
section 3.1.
To get the best benefit, I think it needs to be done in cooperation
between the startup scripts and PostgreSQL. We'd want -17 ("never oom
kill") on the postmaster, but some different value on regular backends
(since if it has to kill someone, it's better to pick a regular
backend so we can do a controlled restart). Only root can drop the
value, so the startup script needs to be part of it. But if we then
want to increase it for sub-processes, that'd require something in the
backend itself...
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
Magnus Hagander wrote:
I realize this is a very platform-specific thing, but should we
consider setting the value of /proc/<pid>/oom_adj when running on
linux? See:
http://archives.postgresql.org/message-id/20080201223336.GC24780%40alvh.no-ip.org
--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.
On Mon, Jan 4, 2010 at 16:45, Alvaro Herrera <alvherre@commandprompt.com> wrote:
Magnus Hagander wrote:
I realize this is a very platform-specific thing, but should we
consider setting the value of /proc/<pid>/oom_adj when running on
linux? See:http://archives.postgresql.org/message-id/20080201223336.GC24780%40alvh.no-ip.org
Grr. I had zero recollectoin of that :S
Can't find a useful consensus though?
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
On Mon, 4 Jan 2010 16:57:28 +0100, Magnus Hagander <magnus@hagander.net>
wrote:
On Mon, Jan 4, 2010 at 16:45, Alvaro Herrera
<alvherre@commandprompt.com>
wrote:
Magnus Hagander wrote:
I realize this is a very platform-specific thing, but should we
consider setting the value of /proc/<pid>/oom_adj when running on
linux? See:
http://archives.postgresql.org/message-id/20080201223336.GC24780%40alvh.no-ip.org
Grr. I had zero recollectoin of that :S
Can't find a useful consensus though?
I don't think we should set a setting like that automatically. Perhaps a
warning on startup?
Joshua D. Drake
--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997
Magnus Hagander wrote:
On Mon, Jan 4, 2010 at 16:45, Alvaro Herrera <alvherre@commandprompt.com> wrote:
Magnus Hagander wrote:
I realize this is a very platform-specific thing, but should we
consider setting the value of /proc/<pid>/oom_adj when running on
linux? See:http://archives.postgresql.org/message-id/20080201223336.GC24780%40alvh.no-ip.org
Grr. I had zero recollectoin of that :S
Can't find a useful consensus though?
It is probably worth trying to protect the postmaster in the init
script. Beyond that things probably start to get fairly difficult.
cheers
andrew
On Mon, Jan 4, 2010 at 17:07, Andrew Dunstan <andrew@dunslane.net> wrote:
Magnus Hagander wrote:
On Mon, Jan 4, 2010 at 16:45, Alvaro Herrera <alvherre@commandprompt.com>
wrote:Magnus Hagander wrote:
I realize this is a very platform-specific thing, but should we
consider setting the value of /proc/<pid>/oom_adj when running on
linux? See:http://archives.postgresql.org/message-id/20080201223336.GC24780%40alvh.no-ip.org
Grr. I had zero recollectoin of that :S
Can't find a useful consensus though?
It is probably worth trying to protect the postmaster in the init script.
Beyond that things probably start to get fairly difficult.
Right. But AFAICS (though I haven't tested with -17), it will become
inherited to children, which is something we'd want to *undo*, no?
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
Magnus Hagander wrote:
On Mon, Jan 4, 2010 at 16:45, Alvaro Herrera <alvherre@commandprompt.com> wrote:
Magnus Hagander wrote:
I realize this is a very platform-specific thing, but should we
consider setting the value of /proc/<pid>/oom_adj when running on
linux? See:http://archives.postgresql.org/message-id/20080201223336.GC24780%40alvh.no-ip.org
Can't find a useful consensus though?
In http://archives.postgresql.org/pgsql-hackers/2008-02/msg00049.php Tom
points out that while you could make this adjustment in the init scripts
for PostgreSQL, actually doing so is quite questionable as a packaging
decision. That's where that thread ended as far as I was concerned.
The best I think anyone could do here is to add such a capability into
some of the init scripts, but it would probably need to be disabled by
default. Since that sort of defeats the purpose of the change, I'm not
sure what the benefit there is--if you have to turn it on, you might as
well do something at a higher level instead.
--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.com
Magnus Hagander wrote:
On Mon, Jan 4, 2010 at 17:07, Andrew Dunstan <andrew@dunslane.net> wrote:
Magnus Hagander wrote:
On Mon, Jan 4, 2010 at 16:45, Alvaro Herrera <alvherre@commandprompt.com>
wrote:Magnus Hagander wrote:
I realize this is a very platform-specific thing, but should we
consider setting the value of /proc/<pid>/oom_adj when running on
linux? See:http://archives.postgresql.org/message-id/20080201223336.GC24780%40alvh.no-ip.org
Grr. I had zero recollectoin of that :S
Can't find a useful consensus though?
It is probably worth trying to protect the postmaster in the init script.
Beyond that things probably start to get fairly difficult.Right. But AFAICS (though I haven't tested with -17), it will become
inherited to children, which is something we'd want to *undo*, no?
[experiments]
Yes, darnit, you're right. But it looks like the oom_adj file can be set
to the default by the process owner:
[andrew@sophia ~]$ ls -l /proc/6520/oom_adj
-rw-r--r-- 1 andrew andrew 0 2010-01-04 12:37 /proc/6520/oom_adj
[andrew@sophia ~]$ cat /proc/6520/oom_adj
0
[andrew@sophia ~]$ id
uid=500(andrew) gid=500(andrew) groups=10(wheel),500(andrew)
[andrew@sophia ~]$ echo -17 > /proc/6520/oom_adj
-bash: echo: write error: Permission denied
[andrew@sophia ~]$ echo 0 > /proc/6520/oom_adj
[andrew@sophia ~]$ echo -17 > /proc/6520/oom_adj
-bash: echo: write error: Permission denied
[andrew@sophia ~]$
But that would be a pain to have to do.
OTOH, disabling the OOM killer is not always an option. I recently tried
it on one system and had to revert it rapidly because the system stopped
working in minutes. Some software just doesn't live well in such
environments, sadly.
cheers
andrew
On Mon, Jan 4, 2010 at 17:40, Andrew Dunstan <andrew@dunslane.net> wrote:
Magnus Hagander wrote:
On Mon, Jan 4, 2010 at 17:07, Andrew Dunstan <andrew@dunslane.net> wrote:
Magnus Hagander wrote:
On Mon, Jan 4, 2010 at 16:45, Alvaro Herrera
<alvherre@commandprompt.com>
wrote:Magnus Hagander wrote:
I realize this is a very platform-specific thing, but should we
consider setting the value of /proc/<pid>/oom_adj when running on
linux? See:http://archives.postgresql.org/message-id/20080201223336.GC24780%40alvh.no-ip.org
Grr. I had zero recollectoin of that :S
Can't find a useful consensus though?
It is probably worth trying to protect the postmaster in the init script.
Beyond that things probably start to get fairly difficult.Right. But AFAICS (though I haven't tested with -17), it will become
inherited to children, which is something we'd want to *undo*, no?[experiments]
Yes, darnit, you're right. But it looks like the oom_adj file can be set to
the default by the process owner:[andrew@sophia ~]$ ls -l /proc/6520/oom_adj
-rw-r--r-- 1 andrew andrew 0 2010-01-04 12:37 /proc/6520/oom_adj
[andrew@sophia ~]$ cat /proc/6520/oom_adj
0
[andrew@sophia ~]$ id
uid=500(andrew) gid=500(andrew) groups=10(wheel),500(andrew)
[andrew@sophia ~]$ echo -17 > /proc/6520/oom_adj
-bash: echo: write error: Permission denied
[andrew@sophia ~]$ echo 0 > /proc/6520/oom_adj
[andrew@sophia ~]$ echo -17 > /proc/6520/oom_adj
-bash: echo: write error: Permission denied
[andrew@sophia ~]$But that would be a pain to have to do.
OTOH, disabling the OOM killer is not always an option. I recently tried it
on one system and had to revert it rapidly because the system stopped
working in minutes. Some software just doesn't live well in such
environments, sadly.
Right. Which is why I like the idea of disabling the OOM killer for
the *postmaster*, but not the regular backends. Gives it a chance to
recover. It's not nice, but it's better than nothing.
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
Greg Smith <greg@2ndquadrant.com> writes:
In http://archives.postgresql.org/pgsql-hackers/2008-02/msg00049.php Tom
points out that while you could make this adjustment in the init scripts
for PostgreSQL, actually doing so is quite questionable as a packaging
decision.
I just wondered if it would be questioned, I didn't say there was a
problem.
However, the long and the short of this is that we can't do anything
without the close cooperation of an init script. I think that moves
it out of the realm of what Postgres as a project should be doing.
It seems more like a patch that the Linux-based packagers should be
carrying.
Memo to self: get off duff and prepare such a patch for the Red Hat/Fedora
packages.
regards, tom lane
Magnus Hagander wrote:
Right. Which is why I like the idea of disabling the OOM killer for
the *postmaster*, but not the regular backends. Gives it a chance to
recover. It's not nice, but it's better than nothing.
It doesn't sound like the init script can reenable the killer for the
child processes though. So, if there's anything that the core code
ought to do, is re-enable OOM-killer for postmaster children, after
being disabled by the initscript.
BTW, is it possible for pg_ctl to disable OOM-killer? I guess not,
since it's not run by root ...
Is there a way to disable memory overcommit for particular processes?
That would be very useful -- just disable overcommit for all Postgres
processes, and there shouldn't be much need to enable the killer for
backends.
--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.
On Mon, Jan 4, 2010 at 17:55, Alvaro Herrera <alvherre@commandprompt.com> wrote:
Magnus Hagander wrote:
Right. Which is why I like the idea of disabling the OOM killer for
the *postmaster*, but not the regular backends. Gives it a chance to
recover. It's not nice, but it's better than nothing.It doesn't sound like the init script can reenable the killer for the
child processes though. So, if there's anything that the core code
ought to do, is re-enable OOM-killer for postmaster children, after
being disabled by the initscript.
Yeah, that's why the backend code would need to be involved.
BTW, is it possible for pg_ctl to disable OOM-killer? I guess not,
since it's not run by root ...
No, it has to run as root.
Is there a way to disable memory overcommit for particular processes?
That would be very useful -- just disable overcommit for all Postgres
processes, and there shouldn't be much need to enable the killer for
backends.
Not that I've been able to find.
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
Alvaro Herrera <alvherre@commandprompt.com> writes:
Is there a way to disable memory overcommit for particular processes?
I would think not --- the very essence of overcommit is that you're
promising more total memory than the system has got, and that's
inherently a global proposition.
regards, tom lane
Magnus Hagander wrote:
On Mon, Jan 4, 2010 at 17:55, Alvaro Herrera <alvherre@commandprompt.com> wrote:
BTW, is it possible for pg_ctl to disable OOM-killer? �I guess not,
since it's not run by root ...No, it has to run as root.
We could at least make it work on Windows, since it is often run as
Administrator and drops privileges afterwards ...
... oh, wait ...
--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.
Magnus Hagander <magnus@hagander.net> writes:
I realize this is a very platform-specific thing, but should we
consider setting the value of /proc/<pid>/oom_adj when running on
linux? See:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/proc.txt;h=220cc6376ef80e0c9bcfec162d45552e729cdf5a;hb=45d28b097280a78893ce25a5d0db41e6a2717853
One interesting thing I read there is:
Swapped out tasks are killed first. Half of each child's memory size is
added to the parent's score if they do not share the same memory.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This suggests that PG's shared memory ought not be counted in the
postmaster's OOM score, which would mean that the problem shouldn't be
quite as bad as we've believed. I wonder if that is a recent change?
Or maybe it's supposed to be that way and is not implemented correctly?
BTW, the given link shows only chapter 1, see
for the whole file.
regards, tom lane
Tom Lane <tgl@sss.pgh.pa.us> wrote:
This suggests that PG's shared memory ought not be counted in the
postmaster's OOM score, which would mean that the problem
shouldn't be quite as bad as we've believed. I wonder if that is
a recent change? Or maybe it's supposed to be that way and is not
implemented correctly?
I've wondered about that based on my experience. When I found that
memory leak back in 8.2devel, running on a SLES 9 SP 3 system, the
OOM killer killed the offending backend rather than the postmaster,
although it took out a couple Java middle tier processes before
starting in on PostgreSQL.
-Kevin
Tom Lane wrote:
Magnus Hagander <magnus@hagander.net> writes:
...oom_adj...
One interesting thing I read there is:
Swapped out tasks are killed first. Half of each child's memory size is
added to the parent's score if they do not share the same memory.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This suggests that PG's shared memory ought not be counted in the
postmaster's OOM score, which would mean that the problem shouldn't be
quite as bad as we've believed. I wonder if that is a recent change?
Or maybe it's supposed to be that way and is not implemented correctly?
The code for oom_kill.c looks fairly readable (link below [1]http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=mm/oom_kill.c;h=f52481b1c1e5442c9a5b16b06b22221b75b9bb7c;hb=HEAD):
96 points = mm->total_vm;
....
117 list_for_each_entry(child, &p->children, sibling) {
118 task_lock(child);
119 if (child->mm != mm && child->mm)
120 points += child->mm->total_vm/2 + 1;
121 task_unlock(child);
122 }
Which seems to add points for each child who doesn't share the
same mm structure as the parent. Which I think is a quite a bit
stricter interpretation of "if they do not share the same memory".
On Mon, Jan 4, 2010 at 09:55, Alvaro Herrera <alvherre@commandprompt.com> wrote:
Magnus Hagander wrote:
Right. Which is why I like the idea of disabling the OOM killer for
the *postmaster*, but not the regular backends. Gives it a chance to
recover. It's not nice, but it's better than nothing.It doesn't sound like the init script can reenable the killer for the
child processes though. So, if there's anything that the core code
ought to do, is re-enable OOM-killer for postmaster children, after
being disabled by the initscript.
Exactly.
FWIW here is the patch I run. Stupid as the patch may be, count it as
a +1 for people in the field doing this. Hence a reason to think
about doing something in core. maybe.
This has some oddities like it does not reset oom to 0 for the (wal)
writer process. Plus assuming you do oom, the stats collector has a
good chance of being hit. Although normal backends will probably have
a higher score.
[ oom_adj gets set to -17 in the startup script. I run this on top of
disabling overcommit, color me paranoid ]
*** a/src/backend/postmaster/autovacuum.c
--- b/src/backend/postmaster/autovacuum.c
***************
*** 362,367 **** StartAutoVacLauncher(void)
--- 362,370 ----
#ifndef EXEC_BACKEND
case 0:
/* in postmaster child ... */
+
+ oom_adjust();
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(false);*** a/src/backend/postmaster/fork_process.c
--- b/src/backend/postmaster/fork_process.c
***************
*** 65,68 **** fork_process(void)
--- 65,84 ----
return result;
}+ void
+ oom_adjust(void)
+ {
+ /* adjust oom */
+ FILE *oom = fopen("/proc/self/oom_adj", "w");
+
+ /*
+ * ignore errors we dont really care
+ */
+ if (oom)
+ {
+ fprintf(oom, "0\n");
+ fclose(oom);
+ }
+ }
+
#endif /* ! WIN32 */
*** a/src/backend/postmaster/pgarch.c
--- b/src/backend/postmaster/pgarch.c
***************
*** 161,166 **** pgarch_start(void)
--- 161,169 ----
#ifndef EXEC_BACKEND
case 0:
/* in postmaster child ... */
+
+ oom_adjust();
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(false);*** a/src/backend/postmaster/pgstat.c
--- b/src/backend/postmaster/pgstat.c
***************
*** 622,627 **** pgstat_start(void)
--- 622,630 ----
#ifndef EXEC_BACKEND
case 0:
/* in postmaster child ... */
+
+ oom_adjust();
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(false);*** a/src/backend/postmaster/postmaster.c
--- b/src/backend/postmaster/postmaster.c
***************
*** 3056,3061 **** BackendStartup(Port *port)
--- 3056,3063 ----
{
free(bn);+ oom_adjust();
+
/*
* Let's clean up ourselves as the postmaster child, and close the
* postmaster's listen sockets. (In EXEC_BACKEND case this is all
*** a/src/backend/postmaster/syslogger.c
--- b/src/backend/postmaster/syslogger.c
***************
*** 530,535 **** SysLogger_Start(void)
--- 530,538 ----
#ifndef EXEC_BACKEND
case 0:
/* in postmaster child ... */
+
+ oom_adjust();
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(true);*** a/src/include/postmaster/fork_process.h
--- b/src/include/postmaster/fork_process.h
***************
*** 13,17 ****
--- 13,18 ----
#define FORK_PROCESS_H extern pid_t fork_process(void);
+ extern void oom_adjust(void);
#endif /* FORK_PROCESS_H */
Attachments:
oom_adj.patchtext/x-patch; charset=US-ASCII; name=oom_adj.patchDownload+31-0
Alex Hunsaker <badalex@gmail.com> writes:
FWIW here is the patch I run. Stupid as the patch may be, count it as
a +1 for people in the field doing this. Hence a reason to think
about doing something in core. maybe.
Thanks for the patch --- it's certainly a fine starting point.
We can either drop this in core (with a lot of #ifdef LINUX added)
or expect Linux packagers to carry it as a patch. Given that the
packagers would also have to modify their init scripts to go with,
the patch route is not unreasonable. Comments?
This has some oddities like it does not reset oom to 0 for the (wal)
writer process.
FWIW, I think that's probably a feature --- I'd vote for only resetting
in regular backends and possibly autovac workers.
regards, tom lane
On Thu, Jan 7, 2010 at 20:26, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Alex Hunsaker <badalex@gmail.com> writes:
We can either drop this in core (with a lot of #ifdef LINUX added)
Any thoughts on doing something like (in fork_process.c)
#ifdef LINUX
void oom_adjust()
{
...
}
#else
void oom_adjust() {}
#endif
So there is only one #ifdef? It still leaves the ugly calls to the function...
or expect Linux packagers to carry it as a patch. Given that the
packagers would also have to modify their init scripts to go with,
the patch route is not unreasonable. Comments?
Id plus +1 for core. The problem certainly does not look to be going
away soon (if ever).
This has some oddities like it does not reset oom to 0 for the (wal)
writer process.FWIW, I think that's probably a feature --- I'd vote for only resetting
in regular backends and possibly autovac workers.
I think that makes sense +1. In-fact thats why the patch has it as a
separate function instead of hacked into fork_process(). However its
mainly odd because IIRC I greped for all instances of fork_process()
and added the oom_adjusting to the callers. Given that it seems the
wall writer procs should also be set to 0. My guess is its a race
with my startup script launching postgres and then setting oom_adj. Or
maybe I missed a caller? Maybe they don't use fork_process()? Ill
check it out.
