does this look more like a possible bug or more like a possible hardware problem...? (long)

Frank van Vugt <ftm.van.vugt@foxi.nl> writes:

At one point, I arrived at the following situation:

psql:/home/data/megadump.sql:5169: WARNING: specified item offset is too
large
psql:/home/data/megadump.sql:5169: PANIC: failed to add item to the page for
"pg_attribute_relid_attnum_index"

Looks like a bug to me. Can you create a reproducible test case?

regards, tom lane

Hi Tom,

At one point, I arrived at the following situation:
psql:/home/data/megadump.sql:5169: WARNING: specified item offset is too
large
psql:/home/data/megadump.sql:5169: PANIC: failed to add item to the page
for "pg_attribute_relid_attnum_index"

Looks like a bug to me. Can you create a reproducible test case?

Nope.....

Trying the same script on a newly created database doesn't show the problem.
However, I do still have the database that came up with this message, which
is now failing a vacuum full verbose analyse with:

(psql)
INFO: vacuuming "public.user_pref"
INFO: "user_pref": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
Nonremovable row versions range from 0 to 0 bytes long.
There were 0 unused item pointers.
Total free space (including removable row versions) is 0 bytes.
0 pages are or will become empty, including 0 at the end of the table.
0 pages containing 0 free bytes are potential move destinations.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO: index "user_pref_pkey" now contains 0 row versions in 1 pages
DETAIL: 0 index pages have been deleted, 0 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.03 sec.
INFO: index "user_pref_idx1" now contains 0 row versions in 1 pages
DETAIL: 0 index pages have been deleted, 0 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.01 sec.
INFO: analyzing "public.user_pref"
INFO: "user_pref": 0 pages, 0 rows sampled, 0 estimated total rows
ERROR: catalog is missing 6 attribute(s) for relid 8349771

(log)
2004-03-10 10:10:00 [8207] LOG: statement: vacuum full verbose analyse;
2004-03-10 10:11:43 [8207] ERROR: catalog is missing 6 attribute(s) for relid
8349771

The server itself has no particular messages in its log that hint on what went
on, except for the earlier mentioned 'possible clash' of a running
pg_autovacuum. By the way, I have stopped pg_autovacuum by now to avoid any
more changes on this particular database.

--
Best,

Frank.

Frank van Vugt <ftm.van.vugt@foxi.nl> writes:

At one point, I arrived at the following situation:
psql:/home/data/megadump.sql:5169: WARNING: specified item offset is too
large
psql:/home/data/megadump.sql:5169: PANIC: failed to add item to the page
for "pg_attribute_relid_attnum_index"

Trying the same script on a newly created database doesn't show the problem.
However, I do still have the database that came up with this message, which
is now failing a vacuum full verbose analyse with:

ERROR: catalog is missing 6 attribute(s) for relid 8349771

This is consistent with the idea that pg_attribute_relid_attnum_index is
corrupted. I would suggest saving a copy of that file for postmortem
analysis and then trying to REINDEX pg_attribute. (Depending on which
PG version you are running, that may require running a standalone
backend. See the REINDEX man page.)

If REINDEX makes the problem go away, would you send me the corrupted
index file off-list?

regards, tom lane

Hi,

ERROR: catalog is missing 6 attribute(s) for relid 8349771

This is consistent with the idea that pg_attribute_relid_attnum_index is
corrupted. I would suggest saving a copy of that file for postmortem
analysis and then trying to REINDEX pg_attribute.

Uhm...... looking for a file named 8349771 located in the directory base/<nr>
where nr corresponds to the proper db, using oid2name.....

We seem to have a small problem : there is no such file.....

I'll hold reindexing in case you first want to look elsewhere.

If REINDEX makes the problem go away, would you send me the corrupted
index file off-list?

I happily would, but you already know its content ;-)

Here's a log of what I did:

base/7847604# ls 82*
8240054 8240056 8240058 8240059 8240600 8240602 8261384 8261386
8261388 8261398

base/7847604# ls 83*
8351687 8352194 8352623 8352754 8353080 8353478 8353581 8353741
<cut>

The last few existing entries from the 82* range seem to be regular
user-tables and corresponding indexes, likewise for the first few in the 83*
range

But when I try the particular oid:

# /usr/local/pgsql/bin/oid2name -d megafox_trial -U ... -P ... -o 8349771
Tablename of oid 8349771 from database "megafox_trial":
---------------------------------
8349771 = article_property_tree

and its revers:

# /usr/local/pgsql/bin/oid2name -d megafox_trial -U ... -P ...
-t article_property_tree
Oid of table article_property_tree from database "megafox_trial":
---------------------------------
VERY scary: more than one table with that name found!!

--
Best,

Frank.

Some additional info:

# select * from pg_class where relname = 'article_property_tree';
-[ RECORD 1 ]--
+-------------------------------------------------------------------
relname        | article_property_tree
relnamespace   | 2200
reltype        | 8349772
relowner       | 1000
relam          | 0
relfilenode    | 8349771
relpages       | 4
reltuples      | 299
reltoastrelid  | 8349775
reltoastidxid  | 0
relhasindex    | t
relisshared    | f
relkind        | r
relnatts       | 6
relchecks      | 1
reltriggers    | 3
relukeys       | 0
relfkeys       | 0
relrefs        | 0
relhasoids     | t
relhaspkey     | t
relhasrules    | f
relhassubclass | f
relacl         | {vugtf=a*r*w*d*R*x*t*/vugtf,=arwdRxt/vugtf,postgres=arwdRxt/
vugtf}
-[ RECORD 2 ]--
+-------------------------------------------------------------------
relname        | article_property_tree
relnamespace   | 2200
reltype        | 8354495
relowner       | 1000
relam          | 0
relfilenode    | 8354494
relpages       | 4
reltuples      | 299
reltoastrelid  | 8354498
reltoastidxid  | 0
relhasindex    | t
relisshared    | f
relkind        | r
relnatts       | 6
relchecks      | 1
reltriggers    | 5
relukeys       | 0
relfkeys       | 0
relrefs        | 0
relhasoids     | t
relhaspkey     | t
relhasrules    | f
relhassubclass | f
relacl         | {vugtf=a*r*w*d*R*x*t*/vugtf,=arwdRxt/vugtf,postgres=arwdRxt/
vugtf}

--
Best,

Frank.

Frank van Vugt <ftm.van.vugt@foxi.nl> writes:

This is consistent with the idea that pg_attribute_relid_attnum_index is
corrupted. I would suggest saving a copy of that file for postmortem
analysis and then trying to REINDEX pg_attribute.

Uhm...... looking for a file named 8349771 located in the directory base/<nr>
where nr corresponds to the proper db, using oid2name.....
We seem to have a small problem : there is no such file.....

I'm sorry, I meant to say save a copy of
pg_attribute_relid_attnum_index. The correct filename for it can be
found via
select relfilenode from pg_class where relname = 'pg_attribute_relid_attnum_index';

Oid of table article_property_tree from database "megafox_trial":
---------------------------------
VERY scary: more than one table with that name found!!

Do you have more than one such table (presumably in different schemas)?
If so this isn't scary at all. I don't believe oid2name has been fixed
to be schema-aware :-(

regards, tom lane

I'm sorry, I meant to say save a copy of
pg_attribute_relid_attnum_index. The correct filename for it can be
found via
select relfilenode from pg_class where relname =
'pg_attribute_relid_attnum_index';

Got it, made a backup of the entire database as well. Since the db wasn't
filled yet, both files are fairly small when bzipped (<0.5MB and < 5MB) and
both can be made available if you want.

Issuing 'reindex table pg_attribute' did not solve the problem, though....

Oid of table article_property_tree from database "megafox_trial":
---------------------------------
VERY scary: more than one table with that name found!!

Do you have more than one such table (presumably in different schemas)?
If so this isn't scary at all. I don't believe oid2name has been fixed
to be schema-aware :-(

Nope, all user-tables are in public at the moment.

Also:

megafox_trial=# select * from article_property_tree;
ERROR: catalog is missing 6 attribute(s) for relid 8349771

--
Best,

Frank.

Tom Lane wrote:

Frank van Vugt <ftm.van.vugt@foxi.nl> writes:

At one point, I arrived at the following situation:

psql:/home/data/megadump.sql:5169: WARNING: specified item offset is too
large
psql:/home/data/megadump.sql:5169: PANIC: failed to add item to the page for
"pg_attribute_relid_attnum_index"

Looks like a bug to me. Can you create a reproducible test case?

I have seen one occurence of that during a Slony test run on the log
table. I think it can be reproduced (not reliable though) with a high
frequency of insert, select and delete of growing keys with a very high
frequency of vacuums at the same time.

Concurrency seems to be an issue here.

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #