Corrupted index file after restoring WAL on warm spare server

Started by Michael Nolanalmost 19 years ago4 messagesgeneral
Jump to latest
#1Michael Nolan
htfoot@gmail.com

Recently I went through the steps to refresh the database (8.3.4) on my
development server (running Fedora Core 5 Linux),
making a tarball of the live database, then restoring it on the development
server, and running all the archived WAL files.

Everything worked fine as far as I can tell, I don't see any errors in any
of my logs.

Howevever, when I go to access one table, the index appears to be corrupted
because the record I get for one query doesn't match the record key I give.

The search works correctly on the live server.

Reindexing that table on the warm spare system fixes the problem.

I redid setting up the warm spare server a second time to see if it was a
one-time anomaly, but I have the exact same condition again.

What could be causing this? If this is a bug, what can I do to help track
it down?
--
Mike Nolan
htfoot@gmail.com

#2Michael Nolan
htfoot@gmail.com
In reply to: Michael Nolan (#1)
Re: Corrupted index file after restoring WAL on warm spare server

Sorry, I meant 8.2.4 (darn typo virus)

Show quoted text

On 5/24/07, Michael Nolan <htfoot@gmail.com> wrote:

Recently I went through the steps to refresh the database (8.3.4) on my
development server (running Fedora Core 5 Linux),
making a tarball of the live database, then restoring it on the
development server, and running all the archived WAL files.

Everything worked fine as far as I can tell, I don't see any errors in any
of my logs.

Howevever, when I go to access one table, the index appears to be
corrupted because the record I get for one query doesn't match the record
key I give.

The search works correctly on the live server.

Reindexing that table on the warm spare system fixes the problem.

I redid setting up the warm spare server a second time to see if it was a
one-time anomaly, but I have the exact same condition again.

What could be causing this? If this is a bug, what can I do to help track
it down?
--
Mike Nolan
htfoot@gmail.com

#3Tom Lane
tgl@sss.pgh.pa.us
In reply to: Michael Nolan (#1)
Re: Corrupted index file after restoring WAL on warm spare server

"Michael Nolan" <htfoot@gmail.com> writes:

Howevever, when I go to access one table, the index appears to be corrupted
because the record I get for one query doesn't match the record key I give.

Reindexing that table on the warm spare system fixes the problem.

I redid setting up the warm spare server a second time to see if it was a
one-time anomaly, but I have the exact same condition again.

What could be causing this? If this is a bug, what can I do to help track
it down?

It sounds like you have a reproducible test case --- can you make it
available to someone else?

regards, tom lane

#4Michael Nolan
htfoot@gmail.com
In reply to: Tom Lane (#3)
Re: Corrupted index file after restoring WAL on warm spare server

On 5/24/07, Tom Lane <tgl@sss.pgh.pa.us> wrote:

It sounds like you have a reproducible test case --- can you make it
available to someone else?

Not very easily, Tom. The tarball files are around 35 GB (uncompressed)

and I assume they'd only work on a fairly similar system anyway.

I assume shell level access would be needed. I may be able to set up a way
to get to someone through the firewall, but there are some security issues
I'd have to clear with our Executive Director, since there's quite a bit of
confidential information in the database and elsewhere on the server.

Is this something where I could call someone and have them walk me through
checking things?

I tried the same thing on a 3rd server (similar hardware and same O/S), and
the query is working correctly there. I'm not sure what that tells me.
--
Mike Nolan