heavy swapping, not sure why

On August 29, 2011 01:36:07 PM Lonni J Friedman wrote:

I have several Linux-x68_64 based dedicated PostgreSQL servers where
I'm experiencing significant swap usage growth over time.

It's the Linux kernel that does it, not PostgreSQL. Set vm.swappiness=0
(usually in /etc/sysctl.conf) and put that into effect.

On Mon, Aug 29, 2011 at 1:46 PM, Alan Hodgson <ahodgson@simkin.ca> wrote:

On August 29, 2011 01:36:07 PM Lonni J Friedman wrote:

I have several Linux-x68_64 based dedicated PostgreSQL servers where
I'm experiencing significant swap usage growth over time.

It's the Linux kernel that does it, not PostgreSQL. Set vm.swappiness=0
(usually in /etc/sysctl.conf) and put that into effect.

I understand that the kernel determines what is swapped out, however
postgres is what is using nearly all the RAM, and then overflowing
into swap. I guess I should have noted that this isn't a case of a
significant amount of RAM not being used, and swapping occurring
anyway. Most of the RAM is already consumed when the heavy swapping
is happening. So, I'd be surprised if setting vm.swappiness=0 will
make a significant difference, however I can certainly try.

On Mon, Aug 29, 2011 at 2:57 PM, Lonni J Friedman <netllama@gmail.com> wrote:

On Mon, Aug 29, 2011 at 1:46 PM, Alan Hodgson <ahodgson@simkin.ca> wrote:

On August 29, 2011 01:36:07 PM Lonni J Friedman wrote:

I have several Linux-x68_64 based dedicated PostgreSQL servers where
I'm experiencing significant swap usage growth over time.

It's the Linux kernel that does it, not PostgreSQL. Set vm.swappiness=0
(usually in /etc/sysctl.conf) and put that into effect.

I understand that the kernel determines what is swapped out, however
postgres is what is using nearly all the RAM, and then overflowing
into swap.  I guess I should have noted that this isn't a case of a
significant amount of RAM not being used, and swapping occurring
anyway.  Most of the RAM is already consumed when the heavy swapping
is happening.  So, I'd be surprised if setting vm.swappiness=0 will
make a significant difference, however I can certainly try.

You haven't shown us how you determined this, it would be nice to see
some copy and paste of things like top, free, or whatever. How much
free AND cache is left over when the machine starts to run out of
memory etc.

Your settings for shared_memory are HUGE. I run a machine witih 128G
of ram and my shared_memory is 8G and that's quite large. No testing
anyone has done has shown anything over 10G being useful.

On Mon, Aug 29, 2011 at 3:46 PM, Alan Hodgson <ahodgson@simkin.ca> wrote:

On August 29, 2011 01:36:07 PM Lonni J Friedman wrote:

I have several Linux-x68_64 based dedicated PostgreSQL servers where
I'm experiencing significant swap usage growth over time.

It's the Linux kernel that does it, not PostgreSQL. Set vm.swappiness=0
(usually in /etc/sysctl.conf) and put that into effect.

that won't help and, in almost all cases, is a bad idea.

merlin

On Mon, Aug 29, 2011 at 3:36 PM, Lonni J Friedman <netllama@gmail.com> wrote:

I have several Linux-x68_64 based dedicated PostgreSQL servers where
I'm experiencing significant swap usage growth over time.  All of them
have fairly substantial amounts of RAM (not including swap), yet the
amount of swap that postgres is using ramps up over time and
eventually hurts performance badly.  In every case, simply restarting
postgresql frees up all the swap in use (until it ramps up again
later).  I'm assuming that I have at least one postgresql.conf
parameter set wrong, but I'm not sure which.  I read that
(max_connections * work_mem) should never exceed physical RAM, and if
that's accurate, then I suspect that's the root of my problem on
systemA (below).  However, I'd like confirmation before I start
tweaking things, as one of these servers is in production, and I can't
easily tweak settings to experiment (plus this problem takes a few
weeks before swapping gets bad enough to impact performance).

using any C code in the backend? this includes 3rd party libraries
which link in C, including postgis, pljava, xml2, etc. Any features
being used not included in the standard core distribution are
interesting.

How long do your database connections stay open? forever? If yes, is
memory distributed semi-evenly across all postgres processes or only
to particular ones? If no, do you see excessive consumption with the
non user backends like the stats collector, etc?

Anything else running on these boxes? Application server? Anything
written in java?

merlin

On Mon, Aug 29, 2011 at 2:24 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:

On Mon, Aug 29, 2011 at 2:57 PM, Lonni J Friedman <netllama@gmail.com> wrote:

On Mon, Aug 29, 2011 at 1:46 PM, Alan Hodgson <ahodgson@simkin.ca> wrote:

On August 29, 2011 01:36:07 PM Lonni J Friedman wrote:

I have several Linux-x68_64 based dedicated PostgreSQL servers where
I'm experiencing significant swap usage growth over time.

It's the Linux kernel that does it, not PostgreSQL. Set vm.swappiness=0
(usually in /etc/sysctl.conf) and put that into effect.

I understand that the kernel determines what is swapped out, however
postgres is what is using nearly all the RAM, and then overflowing
into swap.  I guess I should have noted that this isn't a case of a
significant amount of RAM not being used, and swapping occurring
anyway.  Most of the RAM is already consumed when the heavy swapping
is happening.  So, I'd be surprised if setting vm.swappiness=0 will
make a significant difference, however I can certainly try.

You haven't shown us how you determined this, it would be nice to see
some copy and paste of things like top, free, or whatever.   How much
free AND cache is left over when the machine starts to run out of
memory etc.

Sorry, I was looking at the output from 'free' (plus we have some
generic monitoring tools which generate pretty graphs that also
illustrate the problem). I restarted postgres this morning, so
everything is in a good state right now:
total used free shared buffers cached
Mem: 56481 55486 995 0 15 53298
-/+ buffers/cache: 2172 54309
Swap: 1099 18 1081

total used free shared buffers cached
Mem: 121177 111603 9573 0 0 101007
-/+ buffers/cache: 10596 110581
Swap: 1498 10 1488

Based on past results, it'll be about two weeks before a few hundred
MB of swap is in use, and perf is noticeably poor. Although it will
creep up over time, so even in a day or 2, it will be worse than right
now.

I could post the pretty graph somewhere (or send it to the list, if
you'd prefer) if you want to see something right now (filesize is less
than 40KB).

Your settings for shared_memory are HUGE.  I run a machine witih 128G
of ram and my shared_memory is 8G and that's quite large.  No testing
anyone has done has shown anything over 10G being useful.

Do you mean shared_buffers?

thanks

On Mon, Aug 29, 2011 at 2:38 PM, Merlin Moncure <mmoncure@gmail.com> wrote:

On Mon, Aug 29, 2011 at 3:36 PM, Lonni J Friedman <netllama@gmail.com> wrote:

I have several Linux-x68_64 based dedicated PostgreSQL servers where
I'm experiencing significant swap usage growth over time.  All of them
have fairly substantial amounts of RAM (not including swap), yet the
amount of swap that postgres is using ramps up over time and
eventually hurts performance badly.  In every case, simply restarting
postgresql frees up all the swap in use (until it ramps up again
later).  I'm assuming that I have at least one postgresql.conf
parameter set wrong, but I'm not sure which.  I read that
(max_connections * work_mem) should never exceed physical RAM, and if
that's accurate, then I suspect that's the root of my problem on
systemA (below).  However, I'd like confirmation before I start
tweaking things, as one of these servers is in production, and I can't
easily tweak settings to experiment (plus this problem takes a few
weeks before swapping gets bad enough to impact performance).

using any C code in the backend? this includes 3rd party libraries
which link in C, including postgis, pljava, xml2, etc.  Any features
being used not included in the standard core distribution are
interesting.

Nope, nothing like that. They're fairly generic setups, with nothing
added that isn't part of the core distribution.

How long do your database connections stay open? forever? If yes, is
memory distributed semi-evenly across all postgres processes or only
to particular ones? If no, do you see excessive consumption with the
non user backends like the stats collector, etc?

Nope, nothing is forever, everything is a fairly brief connection (a
few seconds, tops, with most under 1s). Although I do have pgbouncer
sitting in front of systemA to serve as a connection pooler.

Anything else running on these boxes?  Application server?  Anything
written in java?

Nothing. They're all 100% dedicated to postgres, and don't run
anything else beyond the basic OS level stuff (crond, etc).

On Mon, Aug 29, 2011 at 3:38 PM, Lonni J Friedman <netllama@gmail.com> wrote:

On Mon, Aug 29, 2011 at 2:24 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:

On Mon, Aug 29, 2011 at 2:57 PM, Lonni J Friedman <netllama@gmail.com> wrote:

On Mon, Aug 29, 2011 at 1:46 PM, Alan Hodgson <ahodgson@simkin.ca> wrote:

On August 29, 2011 01:36:07 PM Lonni J Friedman wrote:

I have several Linux-x68_64 based dedicated PostgreSQL servers where
I'm experiencing significant swap usage growth over time.

It's the Linux kernel that does it, not PostgreSQL. Set vm.swappiness=0
(usually in /etc/sysctl.conf) and put that into effect.

I understand that the kernel determines what is swapped out, however
postgres is what is using nearly all the RAM, and then overflowing
into swap.  I guess I should have noted that this isn't a case of a
significant amount of RAM not being used, and swapping occurring
anyway.  Most of the RAM is already consumed when the heavy swapping
is happening.  So, I'd be surprised if setting vm.swappiness=0 will
make a significant difference, however I can certainly try.

You haven't shown us how you determined this, it would be nice to see
some copy and paste of things like top, free, or whatever.   How much
free AND cache is left over when the machine starts to run out of
memory etc.

Sorry, I was looking at the output from 'free' (plus we have some
generic monitoring tools which generate pretty graphs that also
illustrate the problem).  I restarted postgres this morning, so
everything is in a good state right now:
            total       used       free     shared    buffers     cached
Mem:         56481      55486        995          0         15      53298
-/+ buffers/cache:       2172      54309
Swap:         1099         18       1081

            total       used       free     shared    buffers     cached
Mem:        121177     111603       9573          0          0     101007
-/+ buffers/cache:      10596     110581
Swap:         1498         10       1488

Based on past results, it'll be about two weeks before a few hundred
MB of swap is in use, and perf is noticeably poor.

That's all a few hundred Megs? That shouldn't make any real
difference. Now a few dozen gigs that would make a difference. Use
top or htop or some other method that shows you the VIRT RES and SHR
memory usage of the processes.

 Although it will
creep up over time, so even in a day or 2, it will be worse than right
now.

I could post the pretty graph somewhere (or send it to the list, if
you'd prefer) if you want to see something right now (filesize is less
than 40KB).

Your settings for shared_memory are HUGE.  I run a machine witih 128G
of ram and my shared_memory is 8G and that's quite large.  No testing
anyone has done has shown anything over 10G being useful.

Do you mean shared_buffers?

Yeah

On Mon, Aug 29, 2011 at 2:51 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:

On Mon, Aug 29, 2011 at 3:38 PM, Lonni J Friedman <netllama@gmail.com> wrote:

On Mon, Aug 29, 2011 at 2:24 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:

On Mon, Aug 29, 2011 at 2:57 PM, Lonni J Friedman <netllama@gmail.com> wrote:

On Mon, Aug 29, 2011 at 1:46 PM, Alan Hodgson <ahodgson@simkin.ca> wrote:

On August 29, 2011 01:36:07 PM Lonni J Friedman wrote:

I have several Linux-x68_64 based dedicated PostgreSQL servers where
I'm experiencing significant swap usage growth over time.

It's the Linux kernel that does it, not PostgreSQL. Set vm.swappiness=0
(usually in /etc/sysctl.conf) and put that into effect.

I understand that the kernel determines what is swapped out, however
postgres is what is using nearly all the RAM, and then overflowing
into swap.  I guess I should have noted that this isn't a case of a
significant amount of RAM not being used, and swapping occurring
anyway.  Most of the RAM is already consumed when the heavy swapping
is happening.  So, I'd be surprised if setting vm.swappiness=0 will
make a significant difference, however I can certainly try.

You haven't shown us how you determined this, it would be nice to see
some copy and paste of things like top, free, or whatever.   How much
free AND cache is left over when the machine starts to run out of
memory etc.

Sorry, I was looking at the output from 'free' (plus we have some
generic monitoring tools which generate pretty graphs that also
illustrate the problem).  I restarted postgres this morning, so
everything is in a good state right now:
            total       used       free     shared    buffers     cached
Mem:         56481      55486        995          0         15      53298
-/+ buffers/cache:       2172      54309
Swap:         1099         18       1081

            total       used       free     shared    buffers     cached
Mem:        121177     111603       9573          0          0     101007
-/+ buffers/cache:      10596     110581
Swap:         1498         10       1488

Based on past results, it'll be about two weeks before a few hundred
MB of swap is in use, and perf is noticeably poor.

That's all a few hundred Megs?  That shouldn't make any real
difference.  Now a few dozen gigs that would make a difference.  Use
top or htop or some other method that shows you the VIRT RES and SHR
memory usage of the processes.

Yes, a few hundred MB of swap, and its definitely making a huge
difference. Upon restarting postgres, its all freed up, and then perf
is good again. Also, this box only has 1GB of swap total, so its
never going to get up a few dozen GB.

Anyway, here's some of top output for systemA right now:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5582 postgres 20 0 13.5g 8.9g 8.9g R 97.7 16.2 2:51.61 postmaster
6554 postgres 20 0 13.5g 1.9g 1.9g D 63.8 3.4 1:50.50
postmaster
6052 postgres 20 0 13.5g 1.3g 1.2g S 22.6 2.3 0:26.33
postmaster
2751 postgres 20 0 13.5g 1.6g 1.6g S 21.6 2.8 0:52.32 postmaster
31221 postgres 20 0 13.5g 2.0g 2.0g S 10.0 3.6 1:19.05
postmaster
1721 postgres 20 0 13.5g 6.7g 6.7g S 3.0 12.2 2:19.21
postmaster
6050 postgres 20 0 13.5g 879m 867m S 8.3 1.6 0:06.89 postmaster

I can certainly grab more in a few days once swap usage has started to
creep up a bit.

 Although it will
creep up over time, so even in a day or 2, it will be worse than right
now.

I could post the pretty graph somewhere (or send it to the list, if
you'd prefer) if you want to see something right now (filesize is less
than 40KB).

Your settings for shared_memory are HUGE.  I run a machine witih 128G
of ram and my shared_memory is 8G and that's quite large.  No testing
anyone has done has shown anything over 10G being useful.

Do you mean shared_buffers?

Yeah

OK, I'll reduce it to 10GB and see if there's any noticable change in
performance. thanks

On Mon, Aug 29, 2011 at 4:45 PM, Lonni J Friedman <netllama@gmail.com> wrote:

using any C code in the backend? this includes 3rd party libraries
which link in C, including postgis, pljava, xml2, etc.  Any features
being used not included in the standard core distribution are
interesting.

Nope, nothing like that.  They're fairly generic setups, with nothing
added that isn't part of the core distribution.

How long do your database connections stay open? forever? If yes, is
memory distributed semi-evenly across all postgres processes or only
to particular ones? If no, do you see excessive consumption with the
non user backends like the stats collector, etc?

Nope, nothing is forever, everything is a fairly brief connection (a
few seconds, tops, with most under 1s). Although I do have pgbouncer
sitting in front of systemA to serve as a connection pooler.

hm. well iirc pgbouncer tries to dump server connections older than
an hour or so. this plus your other statements makes me very
suspension the problem is in postgres itself. since postgres process
dies when the connection dies, long term memory accumulation is just
not possible except in the processes that aren't serving client
sessions (the very first thing you need to do is rule those processes
out). pgbouncer itself could be the issue, but i doubt it.
obviously, a full memory profile during your problem times is a
critical piece of evidence (a 'top' sorted by memory usage should to
the trick nicely).

it's possible you've unhappily tripped a leak in the o/s -- is
everything properly updated? running any funky hardware (like fiber
san drivers)? anything else interesting or out of the ordinary to
report?

merlin

On Mon, Aug 29, 2011 at 3:17 PM, Merlin Moncure <mmoncure@gmail.com> wrote:

On Mon, Aug 29, 2011 at 4:45 PM, Lonni J Friedman <netllama@gmail.com> wrote:

using any C code in the backend? this includes 3rd party libraries
which link in C, including postgis, pljava, xml2, etc.  Any features
being used not included in the standard core distribution are
interesting.

Nope, nothing like that.  They're fairly generic setups, with nothing
added that isn't part of the core distribution.

How long do your database connections stay open? forever? If yes, is
memory distributed semi-evenly across all postgres processes or only
to particular ones? If no, do you see excessive consumption with the
non user backends like the stats collector, etc?

Nope, nothing is forever, everything is a fairly brief connection (a
few seconds, tops, with most under 1s). Although I do have pgbouncer
sitting in front of systemA to serve as a connection pooler.

hm.  well iirc pgbouncer tries to dump server connections older than
an hour or so.  this plus your other statements makes me very
suspension the problem is in postgres itself.  since postgres process
dies when the connection dies, long term memory accumulation is just
not possible except in the processes that aren't serving client
sessions (the very first thing you need to do is rule those processes
out).  pgbouncer itself could be the issue, but i doubt it.
obviously, a full memory profile during your problem times is a
critical piece of evidence (a 'top' sorted by memory usage should to
the trick nicely).

OK, I'll get that top output for everyone in a week or so once swap
usage has grown noticeably above its current level.

it's possible you've unhappily tripped a leak in the o/s -- is
everything properly updated?  running any funky hardware (like fiber
san drivers)? anything else interesting or out of the ordinary to
report?

No funky HW. Fairly standard 1U server with SATA disks, the OS is up to date.

On August 29, 2011 02:34:26 PM you wrote:

On Mon, Aug 29, 2011 at 3:46 PM, Alan Hodgson <ahodgson@simkin.ca> wrote:

On August 29, 2011 01:36:07 PM Lonni J Friedman wrote:

I have several Linux-x68_64 based dedicated PostgreSQL servers where
I'm experiencing significant swap usage growth over time.

It's the Linux kernel that does it, not PostgreSQL. Set vm.swappiness=0
(usually in /etc/sysctl.conf) and put that into effect.

that won't help and, in almost all cases, is a bad idea.

Overly aggressive swapping with the default settings has frequently caused me
performance issues. Using this prevents those problems.

Lonni J Friedman <netllama@gmail.com> writes:

I have several Linux-x68_64 based dedicated PostgreSQL servers where
I'm experiencing significant swap usage growth over time. All of them
have fairly substantial amounts of RAM (not including swap), yet the
amount of swap that postgres is using ramps up over time and
eventually hurts performance badly. In every case, simply restarting
postgresql frees up all the swap in use (until it ramps up again
later).

If you're certain that it's restarting *postgres* that does it, and not
restarting your application or pgbouncer or some other code, then it
seems like you must have uncovered a memory leak someplace. We haven't
got nearly enough info here to diagnose it though.

First thing I'd want to know is which process(es) exactly are bloating.
The top output you showed us is unhelpful for that since it just shows
them all as "postmaster" --- you'll need to match up the problem PIDs
with "ps auxww" output. Keep in mind also that top is pretty awful
about distinguishing a process's actual memory use (private memory)
from the portion of PG's shared memory that it happens to have touched.
What you need to pay attention to is RES minus SHR, not either number
alone. With shared buffers set as high as you've got it, you'll
probably not be able to be sure that a process is bloating until it's
eaten hundreds of megs of private space.

Where we go from there will depend on what you find out ... but if
possible, don't restart the server right away, or it'll probably be
another couple weeks before you can do the next round of investigation.

regards, tom lane

On Mon, Aug 29, 2011 at 5:42 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Lonni J Friedman <netllama@gmail.com> writes:

I have several Linux-x68_64 based dedicated PostgreSQL servers where
I'm experiencing significant swap usage growth over time.  All of them
have fairly substantial amounts of RAM (not including swap), yet the
amount of swap that postgres is using ramps up over time and
eventually hurts performance badly.  In every case, simply restarting
postgresql frees up all the swap in use (until it ramps up again
later).

If you're certain that it's restarting *postgres* that does it, and not
restarting your application or pgbouncer or some other code, then it
seems like you must have uncovered a memory leak someplace.  We haven't
got nearly enough info here to diagnose it though.

I'm 100% certain its hte postgres restart that frees up all the swap.

First thing I'd want to know is which process(es) exactly are bloating.
The top output you showed us is unhelpful for that since it just shows
them all as "postmaster" --- you'll need to match up the problem PIDs
with "ps auxww" output.  Keep in mind also that top is pretty awful
about distinguishing a process's actual memory use (private memory)
from the portion of PG's shared memory that it happens to have touched.
What you need to pay attention to is RES minus SHR, not either number
alone.  With shared buffers set as high as you've got it, you'll
probably not be able to be sure that a process is bloating until it's
eaten hundreds of megs of private space.

ok, I'll do my best to capture this data, and then reply back.

Where we go from there will depend on what you find out ... but if
possible, don't restart the server right away, or it'll probably be
another couple weeks before you can do the next round of investigation.

understood

thanks!

You should monitor PageTables value in /proc/meminfo.if the value larger than 1G,I Suggest enable hugepages .

To monitor PageTables:
# cat /proc/meminfo |grep -i pagetables

On Mon, Aug 29, 2011 at 5:01 PM, Alan Hodgson <ahodgson@simkin.ca> wrote:

On August 29, 2011 02:34:26 PM you wrote:

On Mon, Aug 29, 2011 at 3:46 PM, Alan Hodgson <ahodgson@simkin.ca> wrote:

On August 29, 2011 01:36:07 PM Lonni J Friedman wrote:

I have several Linux-x68_64 based dedicated PostgreSQL servers where
I'm experiencing significant swap usage growth over time.

It's the Linux kernel that does it, not PostgreSQL. Set vm.swappiness=0
(usually in /etc/sysctl.conf) and put that into effect.

that won't help and, in almost all cases, is a bad idea.

Overly aggressive swapping with the default settings has frequently caused me
performance issues. Using this prevents those problems.

On a machine with lots of memory, I've run into pathological behaviour
with both the RHEL 5 and Ubuntu 10.04 kernels where the kswapd starts
eating up CPU and swap io like mad, while doing essentially nothing.
Setting swappiness to 0 delayed this behaviour but did not stop it.
Given that I'm on a machine with 128G ram, I just put "/sbin/swapoff
-a" in /etc/rc.local and viola, problem solved.

On 08/29/2011 06:12 PM, Lonni J Friedman wrote:

OK, I'll reduce it to 10GB and see if there's any noticable change in
performance. thanks

I've never heard a report of a Linux system using more than 8GB of
shared_buffers usefully, and peak performance on systems I've tested has
sometimes been far less than that even. (I have one server that's stuck
at 512MB!) The only report of even 10GB helping came from a Solaris test.

I doubt this has anything to do with your problem, just pointing this
out as future guidance. Until there's a breakthrough in the PostgreSQL
buffer cache code, there really is no reason to give more than 8GB of
dedicated memory to the database on Linux via shared_buffers. You're
better off letting the OS do caching with it instead.

--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

* Scott Marlowe:

On a machine with lots of memory, I've run into pathological behaviour
with both the RHEL 5 and Ubuntu 10.04 kernels where the kswapd starts
eating up CPU and swap io like mad, while doing essentially nothing.
Setting swappiness to 0 delayed this behaviour but did not stop it.
Given that I'm on a machine with 128G ram, I just put "/sbin/swapoff
-a" in /etc/rc.local and viola, problem solved.

Was this NUMA machine? Some older kernels can only migrate pages
between nodes through swap.

--
Florian Weimer <fweimer@bfk.de>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99

On Tue, Aug 30, 2011 at 12:54 AM, Florian Weimer <fweimer@bfk.de> wrote:

* Scott Marlowe:

On a machine with lots of memory, I've run into pathological behaviour
with both the RHEL 5 and Ubuntu 10.04 kernels where the kswapd starts
eating up CPU and swap io like mad, while doing essentially nothing.
Setting swappiness to 0 delayed this behaviour but did not stop it.
Given that I'm on a machine with 128G ram, I just put "/sbin/swapoff
-a" in /etc/rc.local and viola, problem solved.

Was this NUMA machine?  Some older kernels can only migrate pages
between nodes through swap.

48 core AMD Magny Cours. I also had to turn off zone reclaim mode on
it or it would slow to a crawl as the OS started to cache more and
more data.