Doc tweak for huge_pages?

On Fri, Dec 01, 2017 at 04:01:24PM +1300, Thomas Munro wrote:

Hi hackers,

The manual implies that only Linux can use huge pages. That is not
true: FreeBSD, Illumos and probably others support larger page sizes
using transparent page coalescing algorithms. On my FreeBSD box
procstat -v often shows PostgreSQL shared buffers in "S"-flagged
memory. I think we should adjust the manual to make clear that it's
the *explicit request for huge pages* that is supported only on Linux
(and hopefully soon Windows). Am I being too pedantic?

I suggest to remove "other" and include Linux in the enumeration, since it also
supports "transparent" hugepages.

Justin

On Fri, Dec 1, 2017 at 5:04 PM, Justin Pryzby <pryzby@telsasoft.com> wrote:

On Fri, Dec 01, 2017 at 04:01:24PM +1300, Thomas Munro wrote:

Hi hackers,

The manual implies that only Linux can use huge pages. That is not
true: FreeBSD, Illumos and probably others support larger page sizes
using transparent page coalescing algorithms. On my FreeBSD box
procstat -v often shows PostgreSQL shared buffers in "S"-flagged
memory. I think we should adjust the manual to make clear that it's
the *explicit request for huge pages* that is supported only on Linux
(and hopefully soon Windows). Am I being too pedantic?

I suggest to remove "other" and include Linux in the enumeration, since it also
supports "transparent" hugepages.

Hmm. Yeah, it does, but apparently it's not so transparent. So if we
mention that we'd better indicate in the same paragraph that you
probably don't actually want to use it. How about the attached?

--
Thomas Munro
http://www.enterprisedb.com

On 11/30/17 23:35, Thomas Munro wrote:

On Fri, Dec 1, 2017 at 5:04 PM, Justin Pryzby <pryzby@telsasoft.com> wrote:

On Fri, Dec 01, 2017 at 04:01:24PM +1300, Thomas Munro wrote:

Hi hackers,

The manual implies that only Linux can use huge pages. That is not
true: FreeBSD, Illumos and probably others support larger page sizes
using transparent page coalescing algorithms. On my FreeBSD box
procstat -v often shows PostgreSQL shared buffers in "S"-flagged
memory. I think we should adjust the manual to make clear that it's
the *explicit request for huge pages* that is supported only on Linux
(and hopefully soon Windows). Am I being too pedantic?

I suggest to remove "other" and include Linux in the enumeration, since it also
supports "transparent" hugepages.

Hmm. Yeah, it does, but apparently it's not so transparent. So if we
mention that we'd better indicate in the same paragraph that you
probably don't actually want to use it. How about the attached?

Part of the confusion is that the huge_pages setting is only for shared
memory, whereas the kernel settings affect all memory. Is the same true
for the proposed Windows patch?

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Sat, Dec 2, 2017 at 4:08 AM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:

On 11/30/17 23:35, Thomas Munro wrote:

On Fri, Dec 1, 2017 at 5:04 PM, Justin Pryzby <pryzby@telsasoft.com> wrote:

On Fri, Dec 01, 2017 at 04:01:24PM +1300, Thomas Munro wrote:

Hi hackers,

The manual implies that only Linux can use huge pages. That is not
true: FreeBSD, Illumos and probably others support larger page sizes
using transparent page coalescing algorithms. On my FreeBSD box
procstat -v often shows PostgreSQL shared buffers in "S"-flagged
memory. I think we should adjust the manual to make clear that it's
the *explicit request for huge pages* that is supported only on Linux
(and hopefully soon Windows). Am I being too pedantic?

I suggest to remove "other" and include Linux in the enumeration, since it also
supports "transparent" hugepages.

Hmm. Yeah, it does, but apparently it's not so transparent. So if we
mention that we'd better indicate in the same paragraph that you
probably don't actually want to use it. How about the attached?

Part of the confusion is that the huge_pages setting is only for shared
memory, whereas the kernel settings affect all memory.

Right. And more specifically, just the main shared memory area, not
DSM segments. Updated to make this point.

(I have wondered whether DSM segments should respect this GUC: it
seems plausible that they should when the size is a multiple of the
huge page size, so that very large DSA areas finish up mostly backed
by huge pages, so that very large shared hash tables would benefit
from lower TLB miss rates. I have only read in an academic paper that
this is a good idea, I haven't investigated whether that would really
help us in practice, and the first problem is that Linux shm_open
doesn't support huge pages anyway so you've need one of the other DSM
implementation options which are currently non-default.)

Is the same true
for the proposed Windows patch?

Yes. It adds a flag to the request for the main shared memory area
(after jumping through various permissions hoops).

--
Thomas Munro
http://www.enterprisedb.com

On 12/01/2017 05:35 AM, Thomas Munro wrote:

since it also
supports "transparent" hugepages.

Hmm. Yeah, it does, but apparently it's not so transparent.

+1. We saw performance drop with transparent_hugepage enabled on server with
more than 256GB RAM. Access to the cache where slow down when kernel try to
defragment pages.

When that happens, we saw the function isolate_freepages_block appearing in most
consuming function with perf top.

Thanks to Marc Cousin analysis, putting "madvise" to
/sys/kernel/mm/transparent_hugepage/enabled solved the problem. He also notice
that THP only works for "anonymous memory mappings"[1] (shared_buffers are not
anonymous).

1: https://www.kernel.org/doc/Documentation/vm/transhuge.txt

Regards,

--
Adrien NAYRAT

On Mon, Dec 04, 2017 at 09:48:44AM +0100, Adrien Nayrat wrote:

On 12/01/2017 05:35 AM, Thomas Munro wrote:

since it also supports "transparent" hugepages.

Hmm. Yeah, it does, but apparently it's not so transparent.

+1. We saw performance drop with transparent_hugepage enabled on server with
more than 256GB RAM. Access to the cache where slow down when kernel try to
defragment pages.
consuming function with perf top.

Thanks to Marc Cousin analysis, putting "madvise" to
/sys/kernel/mm/transparent_hugepage/enabled solved the problem. He also notice
that THP only works for "anonymous memory mappings"[1] (shared_buffers are not
anonymous).

Note PG since 9.3 can and prefers to use anonymous mmap for shared buffers instead
of sysv shm.

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=b0fc0df9364d2d2d17c0162cf3b8b59f6cb09f67
./src/backend/port/sysv_shmem.c and
src/include/portability/mem.h:#define PG_MMAP_FLAGS (MAP_SHARED|MAP_ANONYMOUS|MAP_HASSEMAPHORE)

Justin

On Fri, Dec 1, 2017 at 10:09 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:

On 11/30/17 23:35, Thomas Munro wrote:

Hmm. Yeah, it does, but apparently it's not so transparent. So if we
mention that we'd better indicate in the same paragraph that you
probably don't actually want to use it. How about the attached?

Here's a review for v3.

I find that the first paragraph is an improvement as it's more precise.

What I didn't like about the second paragraph is that it pointed out
Linux transparent huge pages too favorably while they are actually
known to cause big (huge?, pardon the pun) issues (as witnessed in
this thread as well). v3 basically says "in Linux it can be
transparent or explicit and explicit is faster than transparent".
Reading that, and seeing that explicit needs tweaking of kernel
parameters and so on, one might very well conclude "I'll use the
slightly-slower-but-still-better-than-nothing transparent version".

So I tried to redo the second paragraph and ended up with the
attached. Rationale for the changes:
* changed "this feature" to "explicitly requesting huge pages" to
contrast with the automatic one described below
* made the wording of Linux THP more negative (but still with some
wiggle room for future kernel versions which might improve THP),
contrasting with the positive explicit request from this GUC
* integrated your mention of other OSes with automatic huge pages
* moved the new text to the last paragraph to lower its importance

What do you think?

On Tue, Jan 9, 2018 at 6:24 AM, Catalin Iacob <iacobcatalin@gmail.com> wrote:

On Fri, Dec 1, 2017 at 10:09 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:

On 11/30/17 23:35, Thomas Munro wrote:

Hmm. Yeah, it does, but apparently it's not so transparent. So if we
mention that we'd better indicate in the same paragraph that you
probably don't actually want to use it. How about the attached?

Here's a review for v3.

Thanks!

I find that the first paragraph is an improvement as it's more precise.

What I didn't like about the second paragraph is that it pointed out
Linux transparent huge pages too favorably while they are actually
known to cause big (huge?, pardon the pun) issues (as witnessed in
this thread as well). v3 basically says "in Linux it can be
transparent or explicit and explicit is faster than transparent".
Reading that, and seeing that explicit needs tweaking of kernel
parameters and so on, one might very well conclude "I'll use the
slightly-slower-but-still-better-than-nothing transparent version".

So I tried to redo the second paragraph and ended up with the
attached. Rationale for the changes:
* changed "this feature" to "explicitly requesting huge pages" to
contrast with the automatic one described below
* made the wording of Linux THP more negative (but still with some
wiggle room for future kernel versions which might improve THP),
contrasting with the positive explicit request from this GUC
* integrated your mention of other OSes with automatic huge pages
* moved the new text to the last paragraph to lower its importance

What do you think?

I don't know enough about this to make such a strong recommendation
myself, which is why I was only trying to report that bad performance
had been observed on some version, not that you shouldn't do it. Any
other views on this stronger statement?

--
Thomas Munro
http://www.enterprisedb.com

On 12/1/17 10:08, Peter Eisentraut wrote:

Part of the confusion is that the huge_pages setting is only for shared
memory, whereas the kernel settings affect all memory. Is the same true
for the proposed Windows patch?

Btw., I'm kind of hoping that the Windows patch would be committed
first, so that we don't have to rephrase this whole thing again after that.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Fri, Jan 12, 2018 at 1:12 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:

On Tue, Jan 9, 2018 at 6:24 AM, Catalin Iacob <iacobcatalin@gmail.com> wrote:

So I tried to redo the second paragraph and ended up with the
attached. Rationale for the changes:
* changed "this feature" to "explicitly requesting huge pages" to
contrast with the automatic one described below
* made the wording of Linux THP more negative (but still with some
wiggle room for future kernel versions which might improve THP),
contrasting with the positive explicit request from this GUC
* integrated your mention of other OSes with automatic huge pages
* moved the new text to the last paragraph to lower its importance

What do you think?

I don't know enough about this to make such a strong recommendation
myself, which is why I was only trying to report that bad performance
had been observed on some version, not that you shouldn't do it. Any
other views on this stronger statement?

Now that the Windows huge pages patch has landed, here is a rebase. I
took your alternative and tweaked it a tiny bit more. Thoughts?

--
Thomas Munro
http://www.enterprisedb.com

On Mon, Jan 22, 2018 at 03:54:26PM +1300, Thomas Munro wrote:

On Fri, Jan 12, 2018 at 1:12 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:

On Tue, Jan 9, 2018 at 6:24 AM, Catalin Iacob <iacobcatalin@gmail.com> wrote:
I don't know enough about this to make such a strong recommendation
myself, which is why I was only trying to report that bad performance
had been observed on some version, not that you shouldn't do it. Any
other views on this stronger statement?

Now that the Windows huge pages patch has landed, here is a rebase. I
took your alternative and tweaked it a tiny bit more. Thoughts?

+       <para>
+        Note that, besides explicitly requesting huge pages via
+        <varname>huge_pages</varname>,
=> I would just say:
"Note that, besides huge pages requested explicitly, ..."

+ In Linux this automatic use is
=> ON Linux comma?

+        called "transparent huge pages" and is not enabled by default in
+        popular distributions as of the time of writing, but since transparent

=> really ? I don't know if I've ever seen it not enabled. In any case,
that's a strong statement to make (to be disabled in ALL popular distributions).

I checked all our servers, including centos6 and ubuntu t-LTS and x-LTS. On a
limited few where it was disabled, I'd explicitly done so.

On a server on which I just installed ubuntu-x LTS, with 4.13.0-26-generic:
pryzbyj@gta-ubuntu:~$ cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never

https://github.com/torvalds/linux/commit/13ece886d99cd668483113f7238e419d5331af26
=> the compile time default is to disable, but (if enabled at compile time),
the runtime default is "always".

On centos7
Linux template0 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
$ grep TRANS /boot/config-3.10.0-693.11.6.el7.x86_64
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set

https://blog.nelhage.com/post/transparent-hugepages/
=> It is enabled (”enabled=always”) by default in most Linux distributions.

Justin

On Mon, Jan 22, 2018 at 6:30 PM, Justin Pryzby <pryzby@telsasoft.com> wrote:

On Mon, Jan 22, 2018 at 03:54:26PM +1300, Thomas Munro wrote:

On Fri, Jan 12, 2018 at 1:12 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:

On Tue, Jan 9, 2018 at 6:24 AM, Catalin Iacob <iacobcatalin@gmail.com> wrote:
I don't know enough about this to make such a strong recommendation
myself, which is why I was only trying to report that bad performance
had been observed on some version, not that you shouldn't do it. Any
other views on this stronger statement?

Now that the Windows huge pages patch has landed, here is a rebase. I
took your alternative and tweaked it a tiny bit more. Thoughts?

+       <para>
+        Note that, besides explicitly requesting huge pages via
+        <varname>huge_pages</varname>,
=> I would just say:
"Note that, besides huge pages requested explicitly, ..."

+1

+ In Linux this automatic use is
=> ON Linux comma?

+1

+        called "transparent huge pages" and is not enabled by default in
+        popular distributions as of the time of writing, but since transparent

=> really ? I don't know if I've ever seen it not enabled. In any case,
that's a strong statement to make (to be disabled in ALL popular distributions).

Argh.

https://blog.nelhage.com/post/transparent-hugepages/
=> It is enabled (”enabled=always”) by default in most Linux distributions.

Sorry, right, that was 100% wrong. It would probably be correct to
remove the "not", but let's just remove that bit. New version
attached.

Thanks.

--
Thomas Munro
http://www.enterprisedb.com

On Mon, Jan 22, 2018 at 07:10:33PM +1300, Thomas Munro wrote:

On Mon, Jan 22, 2018 at 6:30 PM, Justin Pryzby <pryzby@telsasoft.com> wrote:

On Mon, Jan 22, 2018 at 03:54:26PM +1300, Thomas Munro wrote:

On Fri, Jan 12, 2018 at 1:12 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:

On Tue, Jan 9, 2018 at 6:24 AM, Catalin Iacob <iacobcatalin@gmail.com> wrote:
I don't know enough about this to make such a strong recommendation
myself, which is why I was only trying to report that bad performance
had been observed on some version, not that you shouldn't do it. Any
other views on this stronger statement?

Now that the Windows huge pages patch has landed, here is a rebase. I
took your alternative and tweaked it a tiny bit more. Thoughts?

Sorry, right, that was 100% wrong. It would probably be correct to
remove the "not", but let's just remove that bit. New version
attached.

+        <productname>PostgreSQL</productname>. On Linux, this is called
+        "transparent huge pages", but since that feature is known to cause
+        performance degradation with
+        <productname>PostgreSQL</productname> on current Linux versions
+        (unlike explicit use of <varname>huge_pages</varname>), its use is
+        discouraged.

Consider this shorter, less-severe sounding alternative:
"... (but note that this feature can degrade performance of some
<productname>PostgreSQL</productname> workloads)."

Justin

On Mon, Jan 22, 2018 at 7:23 AM, Justin Pryzby <pryzby@telsasoft.com> wrote:

Consider this shorter, less-severe sounding alternative:
"... (but note that this feature can degrade performance of some
<productname>PostgreSQL</productname> workloads)."

I think the patch looks good now.

As Justin mentions, as far as I see the only arguable piece is how
strong the language should be against Linux THP.

On one hand it can be argued that warning about THP issues is not the
job of this patch. On the other hand this patch does say more about
THP and Googling does bring up a lot of trouble and advice to disable
THP, including:

/messages/by-id/CANQNgOrD02f8mR3Y8Pi=zFsoL14RqNQA8hwz1r4rSnDLr1b2Cw@mail.gmail.com
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-memory-transhuge

The RedHat article above says "However, THP is not recommended for
database workloads."

I'll leave this to the committer and switch this patch to Ready for Committer.

By the way, Fedora 27 does disable THP by default, they deviate from
upstream in this regard:

[catalin@fedie scripts]$ cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never
[catalin@fedie scripts]$ grep TRANSPARENT /boot/config-4.14.13-300.fc27.x86_64
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD=y
CONFIG_TRANSPARENT_HUGEPAGE=y
# CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS is not set
CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y
CONFIG_TRANSPARENT_HUGE_PAGECACHE=y

When I have some time I'll try to do some digging into history of the
Fedora kernel package to see if they provide a rationale for changing
the default. That might hint whether it's likely that future RHEL will
change as well.

On Tue, Jan 23, 2018 at 7:13 PM, Catalin Iacob <iacobcatalin@gmail.com> wrote:

By the way, Fedora 27 does disable THP by default, they deviate from
upstream in this regard:

When I have some time I'll try to do some digging into history of the
Fedora kernel package to see if they provide a rationale for changing
the default. That might hint whether it's likely that future RHEL will
change as well.

I see Peter assigned himself as committer, some more information below
for him to decide on the strength of the anti THP message.

commit 9a031d5070d9f8f5916c48637bd0c237cd52eaf9
Author: Josh Boyer <jwboyer@redhat.com>
Date: Thu Mar 27 18:31:16 2014 -0400

Switch to CONFIG_TRANSPARENT_HUGEPAGE_MADVISE instead of always on

The benefit of THP has been somewhat questionable overall for a while,
and it's been known to cause performance issues with some workloads.
Upstream also considers it to be overly complicated and really not worth
it on machines with memory in the amounts found on typical desktops/SMB
servers.

Switch to using it via madvise, which most applications that care about
it should likely already be doing.

Debian 9 also seems to default to madvise instead of always.

Digging more into it, there were changes in the 4.6 kernel (released
May 2016) that should improve THP, more precisely:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=444eb2a449ef36fe115431ed7b71467c4563c7f1

This also lead Debian to change their default in September 2017 (so
for the future Debian release) back to always, referencing the 44eb2a
improvements:
https://anonscm.debian.org/cgit/kernel/linux.git/commit/debian/changelog?id=611a8e67260e8b8190ab991206a3867681d6df91

Ben Hutchings <ben@decadent.org.uk>2017-09-29 14:32:09 (GMT)
thp: Enable TRANSPARENT_HUGEPAGE_ALWAYS instead of TRANSPARENT_HUGEPAGE_MADVISE
As advised by Andrea Arcangeli - since commit 444eb2a449ef "mm: thp:
set THP defrag by default to madvise and add a stall-free defrag
option" this will generally be best for performance.

So maybe we should weaken the language against THP. Maybe present the
known facts so far, even if the post 4.6 situation is vague/unknown:
before Linux 4.6 there were repeated reports of THP problems with
Postgres, Linux >= 4.6 might improve things but this isn't confirmed.
And it would be good if somebody could run benchmarks on pre 4.6 and
post 4.6 kernels. I would love to but have no access to big (or
medium) hardware.

On Wed, Jan 24, 2018 at 07:46:41AM +0100, Catalin Iacob wrote:

I see Peter assigned himself as committer, some more information below
for him to decide on the strength of the anti THP message.

Thanks for digging this up!

And it would be good if somebody could run benchmarks on pre 4.6 and
post 4.6 kernels. I would love to but have no access to big (or
medium) hardware.

I should be able to do this, since I have a handful of kernels upgrades on my
todo list. Can you recommend a test ? Otherwise I'll come up with something
for pgbench.

But I think any test should be independant of and not influence the doc change
(I don't know anywhere else in the docs which talks about behaviors of specific
kernel versions, which often have vendor patches backpatched anyway).

So maybe we should weaken the language against THP. Maybe present the
known facts so far, even if the post 4.6 situation is vague/unknown:
before Linux 4.6 there were repeated reports of THP problems with
Postgres, Linux >= 4.6 might improve things but this isn't confirmed.
And it would be good if somebody could run benchmarks on pre 4.6 and
post 4.6 kernels. I would love to but have no access to big (or
medium) hardware.

I think all the details should go elsewhere in the docs; config.sgml already
references this:
https://www.postgresql.org/docs/current/static/kernel-resources.html#LINUX-HUGE-PAGES
..but it doesn't currently mention "transparent" hugepages.

Justin

On 1/22/18 01:10, Thomas Munro wrote:

Sorry, right, that was 100% wrong. It would probably be correct to
remove the "not", but let's just remove that bit. New version
attached.

Committed that.

I reordered some of the existing material because it seemed to have
gotten a bit out of order with repeated patching.

I also softened the advice against THP just a bit, since that is
apparently still changing all the time.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services