OperationalError: FATAL: could not open file <filename>: Too many open files in system

Started by Ben Welshalmost 17 years ago2 messagesgeneral
Jump to latest
#1Ben Welsh
ben.welsh@gmail.com

Hello,

I'm managing a pgsql->django->mod_python->apache machine that serves no
media, but handles postgres, memcached and apache all by itself.

Recently I've begun drawing "too many open files in the system" errors, as
seen in the subject line.

Browsing the postgres documentation, I suspect that I need to lower my
`max_files_per_process` from the default 100. But the support team at my
host is pointing the finger at memcached.

Before I do anything, I thought I'd consult the experts here on the list and
see whether you can had any opinions. Here are some of the basic config
settings that, from cruising around the web and talking to listservs, I've
come to suspect are crucial to this sort of problem. Really, any advice is
appreciated, because I've been getting competiting advice from different
sources and I'm not sure exactly where to go with this.

And, of course, thanks listers and developers for such a great product and
community.

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 137215
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

postgres.conf (max_files_per_process is commented out)
max_connections = 100
shared_buffers = 1000

$ httpd -l

Compiled in modules:
core.c
prefork.c
http_core.c
mod_so.c

httpd.conf...

Timeout 20
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 5
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>

$ cat /proc/sys/fs/file-nr
73320 0 262144

$ lsof | awk '{print $1}' | sort | uniq -c
1 COMMAND
10 awk
49 bash
17 crond
5154 httpd
9 init
25 lsof
39 memcached
807 postmaste
62 saslauthd
69 sendmail
28 sh
7 sort
183 sshd
13 syslogd
8 uniq
19 xinetd

#2Erik Jones
ejones@engineyard.com
In reply to: Ben Welsh (#1)
Re: OperationalError: FATAL: could not open file <filename>: Too many open files in system

On Apr 23, 2009, at 1:39 PM, Ben Welsh wrote:

Hello,

I'm managing a pgsql->django->mod_python->apache machine that serves
no media, but handles postgres, memcached and apache all by itself.

Recently I've begun drawing "too many open files in the system"
errors, as seen in the subject line.

Browsing the postgres documentation, I suspect that I need to lower
my `max_files_per_process` from the default 100. But the support
team at my host is pointing the finger at memcached.

Before I do anything, I thought I'd consult the experts here on the
list and see whether you can had any opinions. Here are some of the
basic config settings that, from cruising around the web and talking
to listservs, I've come to suspect are crucial to this sort of
problem. Really, any advice is appreciated, because I've been
getting competiting advice from different sources and I'm not sure
exactly where to go with this.

And, of course, thanks listers and developers for such a great
product and community.

<snip>

$ lsof | awk '{print $1}' | sort | uniq -c
1 COMMAND
10 awk
49 bash
17 crond
5154 httpd
9 init
25 lsof
39 memcached
807 postmaste
62 saslauthd
69 sendmail
28 sh
7 sort
183 sshd
13 syslogd
8 uniq
19 xinetd

I'd start by investigating the 5K+ file descriptors held by Apache.

Erik Jones, Database Administrator
Engine Yard
Support, Scalability, Reliability
866.518.9273 x 260
Location: US/Pacific
IRC: mage2k