ERROR: unexpected data beyond EOF

Joshua D. Drake wrote:

Alright folks,

So I have this error:

postgres[21118]: [8-1] ERROR: unexpected data beyond EOF
in block 9 of relation base/430666195/430666206

Which produces this lovely hint:

postgres[21118]: [8-2] HINT: This has been seen to occur with buggy
kernels; consider updating your system.

However, the problem is, all relevant information we can find is Linux/NFS
based.

I have vague recollections of seeing this relatively recently on tables
that were truncated often. Is that the case here?

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 04/30/2015 10:28 AM, Alvaro Herrera wrote:

Joshua D. Drake wrote:

Alright folks,

So I have this error:

postgres[21118]: [8-1] ERROR: unexpected data beyond EOF
in block 9 of relation base/430666195/430666206

Which produces this lovely hint:

postgres[21118]: [8-2] HINT: This has been seen to occur with buggy
kernels; consider updating your system.

However, the problem is, all relevant information we can find is Linux/NFS
based.

I have vague recollections of seeing this relatively recently on tables
that were truncated often. Is that the case here?

Just dug into the table a bit and yes, it appears to be a transform
table. Further questions/thoughts?

JD

--
Command Prompt, Inc. - http://www.commandprompt.com/ 503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Announcing "I'm offended" is basically telling the world you can't
control your own emotions, so everyone else should do it for you.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

I take that back, it appears this table is heavily deleted from and also
uses the lo_manage() triggers.

--
Command Prompt, Inc. - http://www.commandprompt.com/ 503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Announcing "I'm offended" is basically telling the world you can't
control your own emotions, so everyone else should do it for you.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Joshua D. Drake wrote:

I take that back, it appears this table is heavily deleted from and also
uses the lo_manage() triggers.

Well, if it's heavily deleted, then it's probably also heavily vacuumed
and from time to time empty pages at the tail are removed by vacuum. It
might also be the case I was remembering, rather than regular TRUNCATE.

I don't think the vacuumlo stuff would have much to do with it issue; I
think it would only scan the table, then delete stuff from
pg_largeobject. It doesn't modify the table itself (unless I'm
misremembering)

Anyway, I don't remember that we reached any useful conclusion. Andres
suspected a PG bug, but we didn't find anything.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 04/30/2015 12:09 PM, Alvaro Herrera wrote:

Joshua D. Drake wrote:

I take that back, it appears this table is heavily deleted from and also
uses the lo_manage() triggers.

Well, if it's heavily deleted, then it's probably also heavily vacuumed
and from time to time empty pages at the tail are removed by vacuum. It
might also be the case I was remembering, rather than regular TRUNCATE.

I don't think the vacuumlo stuff would have much to do with it issue; I
think it would only scan the table, then delete stuff from
pg_largeobject. It doesn't modify the table itself (unless I'm
misremembering)

Anyway, I don't remember that we reached any useful conclusion. Andres
suspected a PG bug, but we didn't find anything.

Well it certainly causes an outage unfortunately. Once the error occurs
postgresql will stop accepting requests (which is correct but still).

JD

--
Command Prompt, Inc. - http://www.commandprompt.com/ 503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Announcing "I'm offended" is basically telling the world you can't
control your own emotions, so everyone else should do it for you.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Hi,

On 2015-04-30 10:05:33 -0700, Joshua D. Drake wrote:

postgres[21118]: [8-1] ERROR: unexpected data beyond EOF
in block 9 of relation base/430666195/430666206

FreeBSD 10
ZFS
iSCSI
RAID 50 (don't start, I didn't spec it).
fsync on, full_page_writes on

The restart of PostgreSQL makes the error go away and things progress
normally. We don't experience further errors etc...

A couple questions:
1) Do you know what 430666206 refers to? Something like SELECT
oid::regclass, relkind FROM pg_class WHERE pg_relation_filenode(430666206) in the
correct database should answer.

2) Any chance you have zero_damaged_pages enabled?

3) Does the issue still occur? Is there any chance of manual
investigation before restaring the database?

Andres

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers