PostgreSQL database and other daemons crashing
I have a Ruby on Rails web site that is running on a CentOS 5 server using
Apache 2 with Passenger and PostgreSQL 8.4. Over the past two weeks I am
seeing my PostgreSQL daemon, postfix, and rsyslog daemons all die at the
same time, this happens at least 5 times per day at random times. Apache
however, does not die.
I cannot associate it with any specific URL on the site or database query,
it doesn’t match up with any scheduled tasks, and hardly reports anything to
most logs. The only relevant log entry I can find is below from my
PostgreSQL logs. I’ve searched through many PostgreSQL list archives and
found little information. Only the suggestion that something might be
sending SIGTERM commands to PostgreSQL or that system limits may be
reached. The server has plenty of free disk space and 1 GB of ram with only
250MB in use. It’s a dedicated web server with no other processes running
really, just the base CentOS processes and the ones I mentioned at the start
of this post.
It seems to be the result of database query within my Ruby code. The
processes do not die when apache isn’t running, but I cannot pin point it.
I’ve even rebuilt the server from scratch and am having the same results.
Has anyone seen anything like this before?
2010-05-19 14:40:45 MST | 127.0.0.1FATAL: terminating connection due to
administrator command
2010-05-19 14:40:45 MST | 127.0.0.1DEBUG: shmem_exit(1): 6 callbacks to
make
2010-05-19 14:40:45 MST | 127.0.0.1DEBUG: proc_exit(1): 3 callbacks to make
2010-05-19 14:40:45 MST | 127.0.0.1DEBUG: exit(1)
2010-05-19 14:40:45 MST | 127.0.0.1LOG: could not send data to client: Bad
file descriptor
2010-05-19 14:40:45 MST | 127.0.0.1DEBUG: shmem_exit(-1): 0 callbacks to
make
2010-05-19 14:40:45 MST | 127.0.0.1DEBUG: proc_exit(-1): 0 callbacks to
make
Thanks for any help,
SpencerPostgreSQL database and other daemons
crashing<http://railsforum.com/viewtopic.php?id=39113>
On Thu, 2010-05-20 at 15:10 -0500, Spencer Anderson wrote:
I have a Ruby on Rails web site that is running on a CentOS 5 server
using Apache 2 with Passenger and PostgreSQL 8.4. Over the past two
weeks I am seeing my PostgreSQL daemon, postfix, and rsyslog daemons
all die at the same time, this happens at least 5 times per day at
random times. Apache however, does not die.
All three? What does /var/log/messages or /var/log/syslog say?
I wonder if the oom killer is acting up.
Joshua D. Drake
--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
/var/log/messages doesn't really show anything (see below). Just my
repeated manual restarts because of the daemon crashing. Yes, all three
crash each time, I can't figure out why rsyslog and postfix die because of a
database issue. I'm not familiar with the oom killer. I'll google around.
Do you have any advice on how to test that as a cause?
May 20 09:59:28 kernel: rklogd 2.0.6, log source = /proc/kmsg started.
May 20 10:22:51 rsyslogd: [origin software="rsyslogd" swVersion="2.0.6"
x-pid="18289" x-info="http://www.rsyslog.com"][x-configInfo
udpReception="No" udpPort="514" tcpReception="No" tcpPort="0"] restart
May 20 10:22:51 kernel: rklogd 2.0.6, log source = /proc/kmsg started.
May 20 11:26:26 rsyslogd: [origin software="rsyslogd" swVersion="2.0.6"
x-pid="17507" x-info="http://www.rsyslog.com"][x-configInfo
udpReception="No" udpPort="514" tcpReception="No" tcpPort="0"] restart
On Thu, May 20, 2010 at 3:21 PM, Joshua D. Drake <jd@commandprompt.com>wrote:
On Thu, 2010-05-20 at 15:10 -0500, Spencer Anderson wrote:
I have a Ruby on Rails web site that is running on a CentOS 5 server
using Apache 2 with Passenger and PostgreSQL 8.4. Over the past two
weeks I am seeing my PostgreSQL daemon, postfix, and rsyslog daemons
all die at the same time, this happens at least 5 times per day at
random times. Apache however, does not die.All three? What does /var/log/messages or /var/log/syslog say?
I wonder if the oom killer is acting up.
Joshua D. Drake
--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
--
Spencer Anderson
spencer@minneapp.com
(424) 901-1363
On Thu, May 20, 2010 at 4:21 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
I wonder if the oom killer is acting up.
that'd be my guess. postfix *never* dies. something external is
killing your processes.
That may in fact be the cause. I ran, free -lm -s 0.5, to watch my memory
use and saw the output below. My processes didn't die this time when it hit
zero bytes free, but it only lasted for 1 second. Some process must be
occasionally using very excessive memory, and every once in a while, I get
the OOM kill scenario. Guess I need to find that process..
total used free shared buffers cached
Mem: 1024 176 847 0 0 0
Low: 1024 176 847
High: 0 0 0
-/+ buffers/cache: 176 847
Swap: 0 0 0
total used free shared buffers cached
Mem: 1024 176 847 0 0 0
Low: 1024 176 847
High: 0 0 0
-/+ buffers/cache: 176 847
Swap: 0 0 0
total used free shared buffers cached
Mem: 1024 1024 0 0 0 0
Low: 1024 1024 0
High: 0 0 0
-/+ buffers/cache: 1024 0
Swap: 0 0 0
total used free shared buffers cached
Mem: 1024 1024 0 0 0 0
Low: 1024 1024 0
High: 0 0 0
-/+ buffers/cache: 1024 0
Swap: 0 0 0
total used free shared buffers cached
Mem: 1024 176 847 0 0 0
Low: 1024 176 847
High: 0 0 0
-/+ buffers/cache: 176 847
Swap: 0 0 0
total used free shared buffers cached
Mem: 1024 176 847 0 0 0
Low: 1024 176 847
High: 0 0 0
-/+ buffers/cache: 176 847
Swap: 0 0 0
On Thu, May 20, 2010 at 3:29 PM, Vick Khera <vivek@khera.org> wrote:
On Thu, May 20, 2010 at 4:21 PM, Joshua D. Drake <jd@commandprompt.com>
wrote:I wonder if the oom killer is acting up.
that'd be my guess. postfix *never* dies. something external is
killing your processes.--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
--
Spencer Anderson
spencer@minneapp.com
(424) 901-1363
Vick Khera <vivek@khera.org> writes:
On Thu, May 20, 2010 at 4:21 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
I wonder if the oom killer is acting up.
that'd be my guess. postfix *never* dies. something external is
killing your processes.
Either that or his hardware is failing, eg RAM dropping critical bits in
the memory of several processes at the same time.
regards, tom lane
On Thu, May 20, 2010 at 3:00 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Vick Khera <vivek@khera.org> writes:
On Thu, May 20, 2010 at 4:21 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
I wonder if the oom killer is acting up.
that'd be my guess. postfix *never* dies. something external is
killing your processes.Either that or his hardware is failing, eg RAM dropping critical bits in
the memory of several processes at the same time.
Defintely worth cranking up an afternoon of memtest86+ to see.