Unexpected data beyond EOF during heavy writes
Hello everyone,
We are seeing the following error message occasionally in the postgres logs:
2010-05-13 23:49:03 PDT ERROR: unexpected data beyond EOF in block 4106698 of relation "custom_discoveryprofile"
2010-05-13 23:49:03 PDT HINT: This has been seen to occur with buggy kernels; consider updating your system.
I have read several threads on this, including:
http://archives.postgresql.org/pgsql-general/2007-03/msg01535.php
and
http://archives.postgresql.org/pgsql-general/2009-07/msg01011.php
I am wondering if anyone has ever placed a bug report to any of the linux vendors or open source sites?
I could not find a bug report that matched this at http://bugzilla.redhat.com/ and am trying to track down what is needed to resolve the error.
We are running a an IBM Blade Center server 64 bit Red Hat 5.3 kernel with 16 cores doing heavy I/O to the database.
uname -a
Linux 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux x86_64 x86_64 x86_64 GNU/Linux
Thanks for any pointers,
Tony Sullivan
Excerpts from Tony Sullivan's message of mar may 18 13:19:13 -0400 2010:
Hello everyone,
We are seeing the following error message occasionally in the postgres logs:
2010-05-13 23:49:03 PDT ERROR: unexpected data beyond EOF in block 4106698 of relation "custom_discoveryprofile"
2010-05-13 23:49:03 PDT HINT: This has been seen to occur with buggy kernels; consider updating your system.
What's your storage?
--
Hello everyone,
We are seeing the following error message occasionally in the postgres logs:
2010-05-13 23:49:03 PDT ERROR: unexpected data beyond EOF in block 4106698 of relation "custom_discoveryprofile"
2010-05-13 23:49:03 PDT HINT: This has been seen to occur with buggy kernels; consider updating your system.
What's your storage?
--
It is NetApp storage - a FAS3070 running Data ONTAP 7.3.2
Here are the mount options
<server>:/vol/sw on /x/eng/sw type nfs
(rw,intr,hard,rsize=32768,wsize=32768,nfsvers=3,timeo=600,tcp,nointr,addr=xx.xx.
xx.xx)
Thanks,
Tony Sullivan
Import Notes
Resolved by subject fallback
Excerpts from Tony Sullivan's message of jue may 20 16:54:17 -0400 2010:
Hello everyone,
We are seeing the following error message occasionally in the postgres logs:
2010-05-13 23:49:03 PDT ERROR: unexpected data beyond EOF in block 4106698 of relation "custom_discoveryprofile"
2010-05-13 23:49:03 PDT HINT: This has been seen to occur with buggy kernels; consider updating your system.
What's your storage?
This was added here
http://archives.postgresql.org/message-id/20060925220110.76B6A9FB251@postgresql.org
in response to these two:
http://thread.gmane.org/gmane.comp.db.postgresql.admin/18807
http://thread.gmane.org/gmane.comp.db.postgresql.general/74532
We (at Command Prompt) researched this recently for another setup and
the common point you both have is NetApp. I then wondered about a bug
in NetApp driver or NFS client implementation.
--
On Thu, May 20, 2010 at 3:19 PM, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
We (at Command Prompt) researched this recently for another setup and
the common point you both have is NetApp. I then wondered about a bug
in NetApp driver or NFS client implementation.
It's definitely not (just) NetApp, though it may be their NFS -- or
NFS in general; I couldn't say. I can't speak to their NFS
implementation, beyond having generally heard good things about it,
but I've run PostgreSQL on filers for years, and have never seen that
message. Granted, I've only been iSCSI- or fibre-attached (or had the
storage path abstracted away by some form of virtualization), so I
haven't seen every possible use-case.
In general, though, I'd be pretty wary of running postgres on an NFS
mount. I know a lot of people run Oracle that way, but at the
filesystem level, there are some vast differences between the two.
Has anyone ever seen this message on non-NetApp NFS?
rls
--
:wq
Rosser Schwarz <rosser.schwarz@gmail.com> writes:
Has anyone ever seen this message on non-NetApp NFS?
It's been seen on non-NFS storage:
http://archives.postgresql.org/pgsql-admin/2006-09/msg00096.php
I don't believe we implicated NFS in the other original report,
either. However, it's certainly possible that there's a similar
bug in the NFS stack too on some platforms.
regards, tom lane