help with error "unexpected pageaddr"

"Scot Kreienkamp" <SKreien@la-z-boy.com> writes:

We have a PG 8.3.7 server that is doing WAL log shipping to 2 other
servers that are remote mirrors. This has been working well for almost
two years. Last night we did some massive data and structure changes to
one of our databases. Since then I get these errors on the two mirrors:

2010-09-15 08:35:05 EDT: LOG: restored log file
"0000000100000301000000D9" from archive

2010-09-15 08:35:27 EDT: LOG: restored log file
"0000000100000301000000DA" from archive

2010-09-15 08:35:40 EDT: LOG: restored log file
"0000000100000301000000DB" from archive

2010-09-15 08:35:40 EDT: LOG: unexpected pageaddr 301/47000000 in log
file 769, segment 219, offset 0

This appears to indicate that you archived the wrong contents of log
file 0000000100000301000000DB. If you don't still have the correct
contents on the master, I think the only way to recover is to take a
fresh base backup so you can make the slaves roll forward from a point
later than this log segment. There's no reason to suppose that there's
data corruption on the master, just bad data in the WAL archive.

You'd probably be well advised to look closely at your WAL archiving
script to see if it has any race conditions that might be triggered by
very fast generation of WAL.

Also, one additional question. I don't have a 00001.history file which
makes the PITRTools complain constantly. Is there any way to regenerate
this file?

Just ignore that, it's cosmetic (the file isn't supposed to exist).

regards, tom lane

"Scot Kreienkamp" <SKreien@la-z-boy.com> writes:

We have a PG 8.3.7 server that is doing WAL log shipping to 2 other
servers that are remote mirrors. This has been working well for

almost

two years. Last night we did some massive data and structure changes

to

one of our databases. Since then I get these errors on the two

mirrors:

2010-09-15 08:35:05 EDT: LOG: restored log file
"0000000100000301000000D9" from archive

2010-09-15 08:35:27 EDT: LOG: restored log file
"0000000100000301000000DA" from archive

2010-09-15 08:35:40 EDT: LOG: restored log file
"0000000100000301000000DB" from archive

2010-09-15 08:35:40 EDT: LOG: unexpected pageaddr 301/47000000 in log
file 769, segment 219, offset 0

This appears to indicate that you archived the wrong contents of log
file 0000000100000301000000DB. If you don't still have the correct
contents on the master, I think the only way to recover is to take a
fresh base backup so you can make the slaves roll forward from a point
later than this log segment. There's no reason to suppose that there's
data corruption on the master, just bad data in the WAL archive.

You'd probably be well advised to look closely at your WAL archiving
script to see if it has any race conditions that might be triggered by
very fast generation of WAL.

Also, one additional question. I don't have a 00001.history file

which

makes the PITRTools complain constantly. Is there any way to

regenerate

this file?

Just ignore that, it's cosmetic (the file isn't supposed to exist).

regards, tom lane

Tom,

I tried to take a new base backup about 45 minutes ago. The master has
rolled forward a number of WAL files since I last tried, but it still
fails.

LOG: restored log file "0000000100000301000000FE" from archive
LOG: restored log file "000000010000030200000000" from archive
LOG: restored log file "000000010000030200000001" from archive
LOG: restored log file "000000010000030200000002" from archive
LOG: restored log file "000000010000030200000003" from archive
LOG: unexpected pageaddr 301/50000000 in log file 770, segment 3,
offset 0
LOG: redo done at 302/2BCE828
LOG: last completed transaction was at log time 2010-09-15
15:07:01.040854-04
LOG: restored log file "000000010000030200000002" from archive
LOG: selected new timeline ID: 2

My entire WAL archiving script is 4 cp %p %f commands. It's so short I
don't even have a script, it's directly in the postgresql.conf archive
command.

"Scot Kreienkamp" <SKreien@la-z-boy.com> writes:

I tried to take a new base backup about 45 minutes ago. The master has
rolled forward a number of WAL files since I last tried, but it still
fails.

LOG: restored log file "0000000100000301000000FE" from archive
LOG: restored log file "000000010000030200000000" from archive
LOG: restored log file "000000010000030200000001" from archive
LOG: restored log file "000000010000030200000002" from archive
LOG: restored log file "000000010000030200000003" from archive
LOG: unexpected pageaddr 301/50000000 in log file 770, segment 3,
offset 0

Hmmm ... is it possible that your WAL archive contains log files
numbered higher than where your master is?

regards, tom lane

Shouldn't have, the only thing we did to the server was restart it and
run our database queries. Clearing out all the wal files from pg_xlog
along with a new base backup did fix it though.

Thanks for the help Tom!

Scot Kreienkamp
skreien@la-z-boy.com