PANIC: failed to re-find parent key in "100924" for split pages 1606/1673

val <valiouk@yahoo.co.uk> writes:

I have a database that refuses to start due to the afformentioned error. I am running POstgreSQL 8.1.11 on a Debian Etch box.

Jan 5 10:36:29 db2 postgres[17111]: [11-1] PANIC: failed to re-find parent key in "100924" for split pages 1606/1673

Hmm ... I wonder if this is telling us that our patch here was
incomplete?
http://archives.postgresql.org/pgsql-committers/2006-11/msg00004.php

At the time we thought this failure could only occur during _bt_pagedel
but you have evidently got a case where a split is failing. It might
just be garden-variety index corruption, or it might be a real bug.

Is this database sufficiently small and non-proprietary that you could
send me a filesystem copy of it (a tarball of all of $PGDATA including
the WAL files)?

regards, tom lane

I have a database that refuses to start due to the

afformentioned error. I am running POstgreSQL 8.1.11 on a
Debian Etch box.

Jan 5 10:36:29 db2 postgres[17111]: [11-1] PANIC:

failed to re-find parent key in "100924" for split
pages 1606/1673

Is this database sufficiently small and non-proprietary
that you could
send me a filesystem copy of it (a tarball of all of
$PGDATA including
the WAL files)?

I solved my problem by reseting the next transaction ID with the pg_resetxlog utility.

Sorry I cannot send you the database since it is proprietary and is also quiet big, but if there is anything else I can do just let me know.

thanks,
val

On Mon, 2009-01-05 at 14:25 -0500, Tom Lane wrote:

val <valiouk@yahoo.co.uk> writes:

I have a database that refuses to start due to the afformentioned error. I am running POstgreSQL 8.1.11 on a Debian Etch box.

Jan 5 10:36:29 db2 postgres[17111]: [11-1] PANIC: failed to re-find parent key in "100924" for split pages 1606/1673

Hmm ... I wonder if this is telling us that our patch here was
incomplete?
http://archives.postgresql.org/pgsql-committers/2006-11/msg00004.php

At the time we thought this failure could only occur during _bt_pagedel
but you have evidently got a case where a split is failing. It might
just be garden-variety index corruption, or it might be a real bug.

Did you catch this had occurred during recovery?

Can we downgrade the error from PANIC to LOG please? One corrupt index
shouldn't prevent us from restarting the whole server. Plus, if we have
to use pg_resetxlog to get us out of trouble it isn't going to help much
with diagnosis. We can rebuild indexes once server is up.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs <simon@2ndQuadrant.com> writes:

On Mon, 2009-01-05 at 14:25 -0500, Tom Lane wrote:

Hmm ... I wonder if this is telling us that our patch here was
incomplete?
http://archives.postgresql.org/pgsql-committers/2006-11/msg00004.php

At the time we thought this failure could only occur during _bt_pagedel
but you have evidently got a case where a split is failing. It might
just be garden-variety index corruption, or it might be a real bug.

Did you catch this had occurred during recovery?

Yes, I did. Which is one of the reasons I think there might be a real
bug there, but without any evidence to look at it's hard to do much
about it now. (Also, our solution to the underlying problem is quite
different now than it was in 8.1, so I'm doubtful that the bug still
exists in current code even if it's real in 8.1.)

Can we downgrade the error from PANIC to LOG please?

No, that seems utterly unsafe to me. We'd have a corrupt index and
nothing to cause it to get repaired.

regards, tom lane

On Thu, 2009-01-08 at 13:40 -0500, Tom Lane wrote:

Can we downgrade the error from PANIC to LOG please?

No, that seems utterly unsafe to me. We'd have a corrupt index and
nothing to cause it to get repaired.

We do exactly this with GIN and GIST indexes currently.

I'd rather have a database that works, but has a corrupt index than one
that won't come up at all.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs <simon@2ndQuadrant.com> writes:

On Thu, 2009-01-08 at 13:40 -0500, Tom Lane wrote:

No, that seems utterly unsafe to me. We'd have a corrupt index and
nothing to cause it to get repaired.

We do exactly this with GIN and GIST indexes currently.

Which are not used in any system indexes.

I'd rather have a database that works, but has a corrupt index than one
that won't come up at all.

If the btree in question is a critical system index, your value of
"work" is going to be pretty damn small.

regards, tom lane

On Thu, 2009-01-08 at 14:19 -0500, Tom Lane wrote:

Simon Riggs <simon@2ndQuadrant.com> writes:

On Thu, 2009-01-08 at 13:40 -0500, Tom Lane wrote:

No, that seems utterly unsafe to me. We'd have a corrupt index and
nothing to cause it to get repaired.

We do exactly this with GIN and GIST indexes currently.

Which are not used in any system indexes.

I'd rather have a database that works, but has a corrupt index than one
that won't come up at all.

If the btree in question is a critical system index, your value of
"work" is going to be pretty damn small.

Those are good points.

So if its a system index we can throw a PANIC, else just LOG. Whilst a
corrupt index is annoying in the extreme, a total server outage is not
something we should allow. IMHO.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs <simon@2ndQuadrant.com> writes:

On Thu, 2009-01-08 at 14:19 -0500, Tom Lane wrote:

If the btree in question is a critical system index, your value of
"work" is going to be pretty damn small.

So if its a system index we can throw a PANIC, else just LOG. Whilst a
corrupt index is annoying in the extreme, a total server outage is not
something we should allow. IMHO.

I think an appropriate solution would be to institute some mechanism
that forces a reindex of the corrupted index at completion of recovery.
Merely fooling around with message severity levels doesn't fix anything
at all, it just opens the door to more trouble than you've already got.

Whether this is important enough to get done in the near future is
a different discussion...

regards, tom lane

On Thu, 2009-01-08 at 15:04 -0500, Tom Lane wrote:

Simon Riggs <simon@2ndQuadrant.com> writes:

On Thu, 2009-01-08 at 14:19 -0500, Tom Lane wrote:

If the btree in question is a critical system index, your value of
"work" is going to be pretty damn small.

So if its a system index we can throw a PANIC, else just LOG. Whilst a
corrupt index is annoying in the extreme, a total server outage is not
something we should allow. IMHO.

I think an appropriate solution would be to institute some mechanism
that forces a reindex of the corrupted index at completion of recovery.
Merely fooling around with message severity levels doesn't fix anything
at all, it just opens the door to more trouble than you've already got.

Well you know I agree on the longer term solution.

But with a down server, you just force people to do pg_resetxlog, which
loses both the corruption (probably) and real, useful data (likely) and
*then* they bring up the server. I don't see why we should force people
to take a manual action and lose data to bring up the server. It's not
like they'll just look at it and say how much of a shame it is it won't
start. They will be bringing up the server, somehow, or they get the
sack. IMHO. I'll say no more though; its not an argument.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs <simon@2ndQuadrant.com> writes:

But with a down server, you just force people to do pg_resetxlog, which
loses both the corruption (probably) and real, useful data (likely) and
*then* they bring up the server. I don't see why we should force people
to take a manual action and lose data to bring up the server.

That's all fine, but simply reducing the message level from PANIC to LOG
remains an utterly unacceptable "solution". What will happen is that
the server will start, the DBA will go back to sleep after ignoring
(most likely, never even reading) the log message, and the corruption
will get worse. The potential consequences of corruption in a pg_class
index, for example, are just horrid. Frankly I'd rather "rm -rf $PGDATA"
and force someone to go back to their last backup than let them continue
to run with a database that is known to be broken and the system didn't
do anything more to warn them than emit a LOG message someplace.

(No, I'm not seriously proposing that as a recovery technique. But it's
no more irresponsible than ignoring a corruption condition.)

regards, tom lane