Thousands of errors...what happened?

Started by Jerry LeVanabout 12 years ago5 messagesgeneral
Jump to latest
#1Jerry LeVan
jerry.levan@gmail.com

The other day I attempted to connect to my 9.3.2 postgresql data base and my connection
attempts kept failing.

I found about 100000 lines in the log file that looked like:

ERROR: could not seek to end of file "global/12292": Too many open files
LOG: out of file descriptors: Too many open files; release and retry
ERROR: could not seek to end of file "global/12292": Too many open files
LOG: out of file descriptors: Too many open files; release and retry
ERROR: could not seek to end of file "global/12292": Too many open files
LOG: out of file descriptors: Too many open files; release and retry
ERROR: could not seek to end of file "global/12292": Too many open files
LOG: out of file descriptors: Too many open files; release and retry
ERROR: could not seek to end of file "global/12292": Too many open files
LOG: out of file descriptors: Too many open files; release and retry
ERROR: could not seek to end of file "global/12292": Too many open files
LOG: out of file descriptors: Too many open files; release and retry
ERROR: could not seek to end of file "global/12292": Too many open files
LOG: out of file descriptors: Too many open files; release and retry
ERROR: could not seek to end of file "global/12292": Too many open files
LOG: out of file descriptors: Too many open files; release and retry
ERROR: could not seek to end of file "global/12292": Too many open files
LOG: out of file descriptors: Too many open files; release and retry
ERROR: could not seek to end of file "global/12292": Too many open files
LOG: database system was interrupted; last known up at 2014-03-01 07:49:58 CST
LOG: database system was not properly shut down; automatic recovery in progress
LOG: record with zero length at 0/1BD6680
LOG: redo is not required
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
LOG: database system was interrupted; last known up at 2014-03-15 07:35:29 CDT
LOG: database system was not properly shut down; automatic recovery in progress
LOG: record with zero length at 0/1BE41F4
LOG: redo is not required
LOG: database system is ready to accept connections
LOG: autovacuum launcher started

I could not stop the postgresql server so I had to reboot the system (:

I am the only user of the db and I use the rascal spareinly.

Any idea what the problem could have been?

Jerry

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

#2Sergey Konoplev
gray.ru@gmail.com
In reply to: Jerry LeVan (#1)
Re: Thousands of errors...what happened?

On Mon, Mar 24, 2014 at 9:14 AM, Jerry Levan <jerry.levan@gmail.com> wrote:

ERROR: could not seek to end of file "global/12292": Too many open files
LOG: out of file descriptors: Too many open files; release and retry

[...]

Any idea what the problem could have been?

Well, it tells for itself. It is out of file descriptors. Assuming you
are on Linux, the ways to fix it (to increase the limit) are described
by the link [1]http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/. Chose one that suits you better.

[1]: http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

--
Kind regards,
Sergey Konoplev
PostgreSQL Consultant and DBA

http://www.linkedin.com/in/grayhemp
+1 (415) 867-9984, +7 (901) 903-0499, +7 (988) 888-1979
gray.ru@gmail.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

#3Jerry LeVan
jerry.levan@gmail.com
In reply to: Sergey Konoplev (#2)
Re: Thousands of errors...what happened?

On Mar 24, 2014, at 11:30 AM, Sergey Konoplev <gray.ru@gmail.com> wrote:

On Mon, Mar 24, 2014 at 9:14 AM, Jerry Levan <jerry.levan@gmail.com> wrote:

ERROR: could not seek to end of file "global/12292": Too many open files
LOG: out of file descriptors: Too many open files; release and retry

[...]

Any idea what the problem could have been?

Well, it tells for itself. It is out of file descriptors. Assuming you
are on Linux, the ways to fix it (to increase the limit) are described
by the link [1]. Chose one that suits you better.

This is on the latest MacOSX and generally I have very few apps open.

This is the first time this has happened in over fourteen years of using
Postgresql.

Jerry

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

#4Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Jerry LeVan (#1)
Re: Thousands of errors...what happened?

Jerry Levan wrote:

The other day I attempted to connect to my 9.3.2 postgresql data base and my connection
attempts kept failing.

I found about 100000 lines in the log file that looked like:

ERROR: could not seek to end of file "global/12292": Too many open files
LOG: out of file descriptors: Too many open files; release and retry

I think this means there is a file descriptor leak somewhere; maybe a
third-party module by Apple. It might be useful to see what files are
open by the offending process; in Linux you would just see
ls -l /proc/{pid}/fd
but I don't know if this works on Mac OS X.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

#5john gale
john@smadness.com
In reply to: Alvaro Herrera (#4)
Re: Thousands of errors...what happened?

On Mar 24, 2014, at 9:43 AM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

Jerry Levan wrote:

The other day I attempted to connect to my 9.3.2 postgresql data base and my connection
attempts kept failing.

I found about 100000 lines in the log file that looked like:

ERROR: could not seek to end of file "global/12292": Too many open files
LOG: out of file descriptors: Too many open files; release and retry

I think this means there is a file descriptor leak somewhere; maybe a
third-party module by Apple. It might be useful to see what files are
open by the offending process; in Linux you would just see
ls -l /proc/{pid}/fd
but I don't know if this works on Mac OS X.

% sudo lsof

is the tool. It doesn't have to be a leak; if you've ignored the postgres configuration, you could have a potential max clients that exceeds the standard open file limit of the system, which you may have just been lucky in never reaching until a surge in traffic generated enough postgres children to breach the limit. Adjust open file limits for launchd-based agents through /etc/launchd.conf:

https://www.google.com/search?client=safari&amp;rls=en&amp;q=launchd.conf+open+files

~ john

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general