Add doc advice about systemd RemoveIPC
Here is a patch to add some information about the systemd RemoveIPC
issue to the documentation, sort of in the spirit of the OOM discussion
nearby.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-doc-Add-advice-about-systemd-RemoveIPC.patchtext/x-patch; name=0001-doc-Add-advice-about-systemd-RemoveIPC.patchDownload
From eaf0eda3f4c402a2e8b7f2f2395a8536f38aafbc Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter_e@gmx.net>
Date: Tue, 27 Dec 2016 12:00:00 -0500
Subject: [PATCH] doc: Add advice about systemd RemoveIPC
---
doc/src/sgml/runtime.sgml | 82 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 82 insertions(+)
diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml
index 787cfce987..fee0d65d90 100644
--- a/doc/src/sgml/runtime.sgml
+++ b/doc/src/sgml/runtime.sgml
@@ -1165,6 +1165,88 @@ <title><systemitem class="osname">System V</> <acronym>IPC</> Parameters</title>
</sect2>
+ <sect2 id="systemd-removeipc">
+ <title>systemd RemoveIPC</title>
+
+ <indexterm>
+ <primary>systemd</primary>
+ <secondary>RemoveIPC</secondary>
+ </indexterm>
+
+ <caution>
+ <para>
+ If <productname>systemd</productname> is in use, the advice in this
+ section must be followed, or your PostgreSQL server will be very
+ unreliable.
+ </para>
+ </caution>
+
+ <para>
+ If <productname>systemd</productname> is in use, special care must be
+ taken that IPC resources (shared memory and semaphores) are not
+ prematurely removed by the operating system. Although the issues
+ described here are most commonly known to happen if the PostgreSQL server
+ is started through a <productname>systemd</productname> service unit, they
+ can also happen in other setups.
+ </para>
+
+ <para>
+ The setting <literal>RemoveIPC</literal>
+ in <filename>logind.conf</filename> controls whether IPC objects are
+ removed when a user fully logs out. System users are exempt. This
+ setting defaults to on in stock <productname>systemd</productname>, but
+ some operating system distributions default it to off.
+ </para>
+
+ <para>
+ A typical observed effect when this setting is on is that the semaphore
+ objects used by a PostgreSQL server are removed at apparently random
+ times, leading to the server crashing with log messages like
+<screen>
+LOG: semctl(1234567890, 0, IPC_RMID, ...) failed: Invalid argument
+</screen>
+ Different types of IPC objects (shared memory vs. semaphores, System V
+ vs. POSIX) are treated slightly differently
+ by <productname>systemd</productname>, so one might observe that some IPC
+ resources are not removed in the same way as others. But it is not
+ advisable to rely on these subtle differences.
+ </para>
+
+ <para>
+ A <quote>user logging out</quote> might happen as part of a maintenance
+ job or manually when an administrator logs in as
+ the <literal>postgres</literal> user or similar, so it is hard to prevent
+ in general.
+ </para>
+
+ <para>
+ What is a <quote>system user</quote> is determined
+ at <productname>systemd</productname> compile time from
+ the <symbol>SYS_UID_MAX</symbol> setting
+ in <filename>/etc/login.defs</filename>.
+ </para>
+
+ <para>
+ It is recommended to set
+<programlisting>
+RemoveIPC=no
+</programlisting>
+ on all server hosts used for PostgreSQL.
+ </para>
+
+ <para>
+ Also, packaging and deployment scripts should be careful to create
+ the <literal>postgres</literal> user as a system user by
+ using <literal>useradd -r</literal>, <literal>adduser --system</literal>,
+ or equivalent.
+ </para>
+
+ <para>
+ <emphasis>At least one of these two things have to be ensured, or the
+ PostgreSQL server will be very unreliable.</emphasis>
+ </para>
+ </sect2>
+
<sect2>
<title>Resource Limits</title>
--
2.11.0
On Wed, Dec 28, 2016 at 4:34 AM, Peter Eisentraut <
peter.eisentraut@2ndquadrant.com> wrote:
Here is a patch to add some information about the systemd RemoveIPC
issue to the documentation, sort of in the spirit of the OOM discussion
nearby.
I wonder if I missed part of the discussions around this, so maybe my
understanding of the cases where this occurs is wrong, but isn't it the
case of pretty much all (or actually) all the packaged versions of
postgresql out there (debian, redhat etc) that they do the right thing, as
in that they create "postgres" as a system user?
I like the text in general, but if the above is true, then I think we
should put a note at the beginning of it with something along the line (not
using those words) of "if you have installed postgresql using packages, the
packager should have taken care of this already"? So as not to scare people
unnecessarily?
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
On 30 December 2016 at 16:59, Magnus Hagander <magnus@hagander.net> wrote:
On Wed, Dec 28, 2016 at 4:34 AM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:Here is a patch to add some information about the systemd RemoveIPC
issue to the documentation, sort of in the spirit of the OOM discussion
nearby.I wonder if I missed part of the discussions around this, so maybe my
understanding of the cases where this occurs is wrong, but isn't it the case
of pretty much all (or actually) all the packaged versions of postgresql out
there (debian, redhat etc) that they do the right thing, as in that they
create "postgres" as a system user?
Yes.
The postgres docs do tend to ignore the reality of most actual
postgres users, though, and talk as if you installed it from source
code under your own user account. I see people bewildered by this
regularly, since we have no discussion at all of common things like
"sudo -u postgres psql" on default packaged installs. Sure, there are
many platforms, but still.
I like the text in general, but if the above is true, then I think we should
put a note at the beginning of it with something along the line (not using
those words) of "if you have installed postgresql using packages, the
packager should have taken care of this already"? So as not to scare people
unnecessarily?
You need to have not only installed it with packages, but be running
it under the package-provided postgres user account. This is not
always the case. I see installs from packages that are then manually
initdb'd in /srv/wtf/why all the time, sadly, and often launched by
manual pg_ctl invocations under surprising user accounts.
"If you have installed postgres from distribution or
postgresql.org-provided packages and use the scripts or commands
provided by the packages to start and stop PostgreSQL, this issue is
unlikely to affect you."
?
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 12/30/16 3:59 AM, Magnus Hagander wrote:
I wonder if I missed part of the discussions around this, so maybe my
understanding of the cases where this occurs is wrong, but isn't it the
case of pretty much all (or actually) all the packaged versions of
postgresql out there (debian, redhat etc) that they do the right thing,
as in that they create "postgres" as a system user?
If you install a package but the user already exists, then the package
will just use that user. So just using a package is not a guarantee
that everything will be alright.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Sat, Dec 31, 2016 at 6:30 AM, Peter Eisentraut <
peter.eisentraut@2ndquadrant.com> wrote:
On 12/30/16 3:59 AM, Magnus Hagander wrote:
I wonder if I missed part of the discussions around this, so maybe my
understanding of the cases where this occurs is wrong, but isn't it the
case of pretty much all (or actually) all the packaged versions of
postgresql out there (debian, redhat etc) that they do the right thing,
as in that they create "postgres" as a system user?If you install a package but the user already exists, then the package
will just use that user. So just using a package is not a guarantee
that everything will be alright.
Good point.
I still think that some wording in the direction of the fact that the
majority of all users won't actually have this problem is the right thing
to do (regardless of our previous history in the area as pointed out by
Craig)
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
Magnus Hagander <magnus@hagander.net> writes:
I still think that some wording in the direction of the fact that the
majority of all users won't actually have this problem is the right thing
to do (regardless of our previous history in the area as pointed out by
Craig)
+1. The use-a-system-user solution is the one that's in place on the
ground for most current PG users on affected platforms. We should explain
that one first and make clear that platform-specific packages attempt to
set it up that way for you. The RemoveIPC technique should be documented
as a fallback to be used if you can't/won't use a system userid.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 12/31/16 11:43 AM, Tom Lane wrote:
Magnus Hagander <magnus@hagander.net> writes:
I still think that some wording in the direction of the fact that the
majority of all users won't actually have this problem is the right thing
to do (regardless of our previous history in the area as pointed out by
Craig)+1. The use-a-system-user solution is the one that's in place on the
ground for most current PG users on affected platforms. We should explain
that one first and make clear that platform-specific packages attempt to
set it up that way for you. The RemoveIPC technique should be documented
as a fallback to be used if you can't/won't use a system userid.
How about this version, which shifts the emphasis a bit, as suggested?
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
v2-0001-doc-Add-advice-about-systemd-RemoveIPC.patchtext/x-patch; name=v2-0001-doc-Add-advice-about-systemd-RemoveIPC.patchDownload
From 5b4ba436d3882bfa2ce0e6243b9ab2ece66a4da4 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter_e@gmx.net>
Date: Fri, 10 Feb 2017 16:34:20 -0500
Subject: [PATCH v2] doc: Add advice about systemd RemoveIPC
---
doc/src/sgml/runtime.sgml | 77 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 77 insertions(+)
diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml
index 130c386462..25c57192db 100644
--- a/doc/src/sgml/runtime.sgml
+++ b/doc/src/sgml/runtime.sgml
@@ -1165,6 +1165,83 @@ <title><systemitem class="osname">System V</> <acronym>IPC</> Parameters</title>
</sect2>
+ <sect2 id="systemd-removeipc">
+ <title>systemd RemoveIPC</title>
+
+ <indexterm>
+ <primary>systemd</primary>
+ <secondary>RemoveIPC</secondary>
+ </indexterm>
+
+ <para>
+ If <productname>systemd</productname> is in use, some care must be taken
+ that IPC resources (shared memory and semaphores) are not prematurely
+ removed by the operating system. This is especially of concern when
+ installing PostgreSQL from source. Users of distribution packages of
+ PostgreSQL are less likely to be affected.
+ </para>
+
+ <para>
+ The setting <literal>RemoveIPC</literal>
+ in <filename>logind.conf</filename> controls whether IPC objects are
+ removed when a user fully logs out. System users are exempt. This
+ setting defaults to on in stock <productname>systemd</productname>, but
+ some operating system distributions default it to off.
+ </para>
+
+ <para>
+ A typical observed effect when this setting is on is that the semaphore
+ objects used by a PostgreSQL server are removed at apparently random
+ times, leading to the server crashing with log messages like
+<screen>
+LOG: semctl(1234567890, 0, IPC_RMID, ...) failed: Invalid argument
+</screen>
+ Different types of IPC objects (shared memory vs. semaphores, System V
+ vs. POSIX) are treated slightly differently
+ by <productname>systemd</productname>, so one might observe that some IPC
+ resources are not removed in the same way as others. But it is not
+ advisable to rely on these subtle differences.
+ </para>
+
+ <para>
+ A <quote>user logging out</quote> might happen as part of a maintenance
+ job or manually when an administrator logs in as
+ the <literal>postgres</literal> user or similar, so it is hard to prevent
+ in general.
+ </para>
+
+ <para>
+ What is a <quote>system user</quote> is determined
+ at <productname>systemd</productname> compile time from
+ the <symbol>SYS_UID_MAX</symbol> setting
+ in <filename>/etc/login.defs</filename>.
+ </para>
+
+ <para>
+ Packaging and deployment scripts should be careful to create
+ the <literal>postgres</literal> user as a system user by
+ using <literal>useradd -r</literal>, <literal>adduser --system</literal>,
+ or equivalent.
+ </para>
+
+ <para>
+ Alternatively, if the user account was created incorrectly or cannot be
+ changed, it is recommended to set
+<programlisting>
+RemoveIPC=no
+</programlisting>
+ in <filename>/etc/systemd/logind.conf</filename> or another appropriate
+ configuration file.
+ </para>
+
+ <caution>
+ <para>
+ At least one of these two things have to be ensured, or the PostgreSQL
+ server will be very unreliable.
+ </para>
+ </caution>
+ </sect2>
+
<sect2>
<title>Resource Limits</title>
--
2.11.1
On Fri, Feb 10, 2017 at 10:36 PM, Peter Eisentraut <
peter.eisentraut@2ndquadrant.com> wrote:
On 12/31/16 11:43 AM, Tom Lane wrote:
Magnus Hagander <magnus@hagander.net> writes:
I still think that some wording in the direction of the fact that the
majority of all users won't actually have this problem is the rightthing
to do (regardless of our previous history in the area as pointed out by
Craig)+1. The use-a-system-user solution is the one that's in place on the
ground for most current PG users on affected platforms. We shouldexplain
that one first and make clear that platform-specific packages attempt to
set it up that way for you. The RemoveIPC technique should be documented
as a fallback to be used if you can't/won't use a system userid.How about this version, which shifts the emphasis a bit, as suggested?
Looks much better.
+ <para>
+ If <productname>systemd</productname> is in use, some care must be
taken
+ that IPC resources (shared memory and semaphores) are not prematurely
+ removed by the operating system. This is especially of concern when
+ installing PostgreSQL from source. Users of distribution packages of
+ PostgreSQL are less likely to be affected.
+ </para>I would add "are less likely to be affected as the postgres user is
normally created as a system user" or something like that -- to indicate
*why* they are less likely to be affected (and it also tells people that if
they change the user, then they might become affected again).
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
On 2/14/17 11:49 AM, Magnus Hagander wrote:
+ <para> + If <productname>systemd</productname> is in use, some care must be taken + that IPC resources (shared memory and semaphores) are not prematurely + removed by the operating system. This is especially of concern when + installing PostgreSQL from source. Users of distribution packages of + PostgreSQL are less likely to be affected. + </para>I would add "are less likely to be affected as the postgres user is
normally created as a system user" or something like that -- to indicate
*why* they are less likely to be affected (and it also tells people that
if they change the user, then they might become affected again).
Committed with that addition, thanks!
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers