Fix doc: explanation of how postgres works when OOM killer is invoked
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
This thread has been committed, so CI has stopped here. Anything below is the last result it produced.
You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:
docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t254099psql -h localhost -U postgresBuilt from patchset v7 (message #7), October 16, 2026 at 08:41 AM.
Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:
git clone --branch t254099_7 https://github.com/hackorum-dev/postgres.gitIn a checkout you already have, add the fork once:
git remote add hackorum https://github.com/hackorum-dev/postgres.gitthen, for this patchset and every later one:
git fetch hackorum t254099_7 && git checkout t254099_7Patchset v7 (message #7) is on t254099_7
Hi postgres-hackers
While reviewing a doc patch[1]/messages/by-id/CABH8dKwkp1_TzR7nxfvSpRSLD1b33xw7N_VLMLy3MiaEjAEPNw@mail.gmail.com , I noticed that current explanation
about how postgres behaves when an OOM killer is invoked is not quite
correct.
Postgres doc [2]https://www.postgresql.org/docs/devel/kernel-resources.html#LINUX-MEMORY-OVERCOMMIT says:
This indicates that the <filename>postgres</filename> process
has been terminated due to memory pressure.
Although existing database connections will continue to function
normally, no new connections will be accepted. To recover,
<productname>PostgreSQL</productname> will need to be restarted.
But I believe all postgres processes will be terminated in this case.
When a backend process is terminated by an OOM killer, the whole
postgres will be restarted or stop depending on restart_after_crash
setting. In a special case where postmaster is killed, each backend
proccesses will also eventually be terminated. The backend process will
be stopped when it notices postmaster is terminated. In this case, the
postgres will just stop and a user needs to restart it. I tested above
two cases and checked the relevant codes.
The statement was very old and committed at 02cd2b96acdb in 2003. I
assume it was correct at the time but I believe it's not as of now
though I haven't checked.
Could you check attached patch?
The original document mentions to postmaster explicitly. So, in my patch
I explain how postmaster and other processes work when OOM killer is
invoked.
[1]: /messages/by-id/CABH8dKwkp1_TzR7nxfvSpRSLD1b33xw7N_VLMLy3MiaEjAEPNw@mail.gmail.com
/messages/by-id/CABH8dKwkp1_TzR7nxfvSpRSLD1b33xw7N_VLMLy3MiaEjAEPNw@mail.gmail.com
[2]: https://www.postgresql.org/docs/devel/kernel-resources.html#LINUX-MEMORY-OVERCOMMIT
https://www.postgresql.org/docs/devel/kernel-resources.html#LINUX-MEMORY-OVERCOMMIT
Thank you,
Takeshi Ideriha
Hi Takeshi,
But I believe all postgres processes will be terminated in this case.
[...] The backend process will be stopped when it notices postmaster
is terminated.
I looked into this as well, and the patch matches what I see. In the
reaper (postmaster.c) any child that exits abnormally goes through
HandleChildCrash, so there is no special case for an OOM SIGKILL. I
reproduced it: killing a client backend, the checkpointer, the walwriter,
the archiver, or the autovacuum launcher each ends with the postmaster
logging "all server processes terminated; reinitializing" (with
restart_after_crash on), i.e. every connection is gone. Killing the
postmaster instead leaves the backends up until they notice its death and
exit, and new connections are refused until a manual restart. So the old
"existing database connections will continue to function normally" no
longer holds for the usual case.
Two small things:
- With restart_after_crash off, the server does not stay down passively:
it logs 'shutting down because "restart_after_crash" is off' and shuts
the whole cluster down, so a manual restart is needed. The patch gives
the "on" case; it might be worth saying what "off" does too.
- A little further down, this same section recommends protecting the
postmaster with a low oom_score_adj so the OOM killer picks a child.
With that in place the victim is a backend -- your "another server
process" case -- while "the postmaster is terminated" is mostly what
happens without that protection. Relating the two might help the
reader, though it may be more than this fix intends.
The wording reads well to me otherwise.
Regards,
Manu
Hi Manu
Thank you for the comments.
I registered this to CF https://commitfest.postgresql.org/patch/7407/ .
I looked into this as well, and the patch matches what I see.
I checked the almost same thing as you did. Thank you for the confirmation.
Two small things:
- With restart_after_crash off, the server does not stay down passively:
it logs 'shutting down because "restart_after_crash" is off' and shuts
the whole cluster down, so a manual restart is needed. The patch gives
the "on" case; it might be worth saying what "off" does too.- A little further down, this same section recommends protecting the
postmaster with a low oom_score_adj so the OOM killer picks a child.
With that in place the victim is a backend -- your "another server
process" case -- while "the postmaster is terminated" is mostly what
happens without that protection. Relating the two might help the
reader, though it may be more than this fix intends.
Both of the points make sense. I added them in v2 patch.
Regards,
Takeshi Ideriha
Hi Takeshi,
Both of your points make sense. I added them in v2 patch.
v2 reads correctly to me -- both the restart_after_crash=off case and the
pointer to the postmaster oom_score_adj are now covered, and the wording
matches what I reproduced.
One thing that might further help the reader, if it is in scope: the
section shows the kernel message but not what PostgreSQL itself logs, and
since every process is named "postgres" the reader cannot tell from
"Killed process 12345 (postgres)" alone which of your two cases applies.
The server log is what separates them. When a child is killed, the
postmaster logs (with restart_after_crash on):
LOG: client backend (PID 769432) was terminated by signal 9: Killed
LOG: terminating any other active server processes
LOG: all server processes terminated; reinitializing
LOG: database system is ready to accept connections
When the postmaster itself is killed there is no such reinitialization
sequence -- the log simply ends and the backends exit as they notice it
is gone. That mapping also answers the "where to look" the kernel-message
note leaves open: the kernel log for the OOM line, the server log for the
effect. Might be more than this fix intends, so feel free to leave it for
a follow-up.
Regards,
Manu
Hi Manu
Both of your points make sense. I added them in v2 patch.
v2 reads correctly to me -- both the restart_after_crash=off case and the
pointer to the postmaster oom_score_adj are now covered, and the wording
matches what I reproduced.
Thank you.
One thing that might further help the reader, if it is in scope: the
section shows the kernel message but not what PostgreSQL itself logs, and
since every process is named "postgres" the reader cannot tell from
"Killed process 12345 (postgres)" alone which of your two cases applies.
The server log is what separates them. When a child is killed, the
postmaster logs (with restart_after_crash on):LOG: client backend (PID 769432) was terminated by signal 9: Killed
LOG: terminating any other active server processes
LOG: all server processes terminated; reinitializing
LOG: database system is ready to accept connectionsWhen the postmaster itself is killed there is no such reinitialization
sequence -- the log simply ends and the backends exit as they notice it
is gone. That mapping also answers the "where to look" the kernel-message
note leaves open: the kernel log for the OOM line, the server log for the
effect. Might be more than this fix intends, so feel free to leave it for
a follow-up.
Thank you for the suggestion. I agree that mapping the kernel message to
the server log would be useful for readers.
But I'd like to focus on correcting the wrong sentence for now.
Current "Linux Memory Overcommit" section is somewhat long and these
logs are easy to understand so I'd like to make this out of scope.
Regards,
Takeshi Ideriha
On Wed, Oct 14, 2026 at 5:33 PM Takeshi Ideriha
<iderihatakeshi@gmail.com> wrote:
But I'd like to focus on correcting the wrong sentence for now.
+1
So, IMO, it's better to omit the OOM score description because it's not
needed to correct the existing text and is already explained later in
the same section.
+ If a child process is terminated, all connections
+ will be terminated and <productname>PostgreSQL</productname> will
+ automatically restart when
+ <link linkend="guc-restart-after-crash"><varname>restart_after_crash</varname></link>
+ is on. If it is off, <productname>PostgreSQL</productname> will stop and
+ must be restarted manually.
This description seems misleading because not all child process
terminations have this effect. For example, terminating the syslogger
does not terminate existing connections. But, to address this,
listing all the exceptions here would be overkill, though. So, how about
using "typically" instead, as follows?
---------------
If a child process is terminated, PostgreSQL will typically terminate
all remaining connections and either recover automatically, if
restart_after_crash is enabled, or shut down. If PostgreSQL shuts
down, it will need to be restarted to recover.
---------------
Regards,
--
Fujii Masao
Hi
On 2026/10/14 20:36, Fujii Masao wrote:
On Wed, Oct 14, 2026 at 5:33 PM Takeshi Ideriha
<iderihatakeshi@gmail.com> wrote:But I'd like to focus on correcting the wrong sentence for now.
+1
So, IMO, it's better to omit the OOM score description because it's not
needed to correct the existing text and is already explained later in
the same section.
On second thought, I agree with you. I'd like to keep the postgres doc
concise as possible. Fixed in v3 patch.
+ If a child process is terminated, all connections + will be terminated and <productname>PostgreSQL</productname> will + automatically restart when + <link linkend="guc-restart-after-crash"><varname>restart_after_crash</varname></link> + is on. If it is off, <productname>PostgreSQL</productname> will stop and + must be restarted manually.This description seems misleading because not all child process
terminations have this effect. For example, terminating the syslogger
does not terminate existing connections. But, to address this,
listing all the exceptions here would be overkill, though. So, how about
using "typically" instead, as follows?---------------
If a child process is terminated, PostgreSQL will typically terminate
all remaining connections and either recover automatically, if
restart_after_crash is enabled, or shut down. If PostgreSQL shuts
down, it will need to be restarted to recover.
---------------
Oh, syslogger. It's true. Thank you for pointing this out and great
suggestion. Your suggestion looks much more correct and simple. I use
your wordings in v3 patch as is.
Regards,
Takeshi Ideriha
On Thu, Oct 15, 2026 at 5:27 PM Takeshi Ideriha
<iderihatakeshi@gmail.com> wrote:
On second thought, I agree with you. I'd like to keep the postgres doc
concise as possible. Fixed in v3 patch.
I made a few minor adjustments and pushed the patch. Thanks!
Regards,
--
Fujii Masao
On 2026/10/16 22:50, Fujii Masao wrote:
On Thu, Oct 15, 2026 at 5:27 PM Takeshi Ideriha
<iderihatakeshi@gmail.com> wrote:On second thought, I agree with you. I'd like to keep the postgres doc
concise as possible. Fixed in v3 patch.I made a few minor adjustments and pushed the patch. Thanks!
Clarify that the OOM killer can terminate either the postmaster or a
child process.
Thank you for additional clarification and push!
Regards,
Takeshi Ideriha