Re: Strange database corruption with PostgreSQL 7.4.x o

On Wed, 2006-09-20 at 15:14 +0200, Matthias.Pitzl@izb.de wrote:

Hello Scott!

Thank you for the quick answer. I'll try to check our hardware which is a
Compaq DL380 G4 with a batteyr buffered write cache on our raid controller.
As the system is running stable at all i think it's not the cpu or memory.
At moment i tend more to a bad disk or SCSI controller but even with that i
don't get any message in my logs...
Any ideas how i could check the hardware?

Keep in mind, a single bad memory location is all it takes to cause data
corruption, so it could well be memory. CPU is less likely if the
machine is otherwise running stable.

The standard tool on x86 hardware is memtest86 www.memtest86.com

So, you'd have to schedule a maintenance window to run the test in since
you have to basically down the machine and run just memtest86. I think
a few live linux distros have it built in (FC has a memtest label in
some versions I think)

My first suspicion is always memory. We ordered a batch of memory from
a very off brand supplier, and over 75% tested bad. And it took >24
hours to find some of the bad memory.

good luck with your testing, let us know how it goes.

Hello Scott!

Thank you. Memtest86 i know. I think we will use this for testing our
hardware too.
Got some other nice information meanwhile from someone also running a DL380
server which had a defect backplane causing similar issues.
He also gave me the hint that there's a test suite CD by Compaq to run some
hardware diagnostic checks on our machine. I will try this out as soon as
possible.
I will inform you when i know more :)

-- Matthias

On Wed, 20 Sep 2006, Matthias.Pitzl@izb.de wrote:

Any ideas how i could check the hardware?

1. memtest86 or memtest86+ at least 8 hours

2. CPU Burn-in
http://users.bigpond.net.au/cpuburn/ at least 8 hours

3. badblocks -s -v -t random /dev/sd%
WARNING: this will destroy your data!

4. smartctl -a /dev/sd%
Does not have to work. Sometimes needs some hacking to make it work.

Dave "KernelSlacker" Jones wrote a very good article about hardware
problems:
http://people.redhat.com/davej/hardware-problems.txt

Regards
Tometzky
--
...although Eating Honey was a very good thing to do, there was a
moment just before you began to eat it which was better than when you
were...
Winnie the Pooh

Hello Tom!

Not yet, but i will try this one too. Anything special i should look for
when dumping out the bad pages?

-- Matthias

Hello all!

Ok, i found out some more informations. According to
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&
taskId=110&prodSeriesId=397634&prodTypeId=15351&prodSeriesId=397634&objectID
=PSD_EX050119_CW01 one of our four disks in the server has a firmware issue.
The problem are incomplete writes onto disk while on high I/O load...
We will check this one first. If it won't help, we will try the hardware
diagnostics and some other tests...
Meanwhile thank you all for your suggestions :)

-- Matthias

Matthias.Pitzl@izb.de writes:

Not yet, but i will try this one too. Anything special i should look for
when dumping out the bad pages?

"If we knew what it was we would learn, it wouldn't be research" ...

regards, tom lane

Hello everyone!

Small update on this issue:
Our server has four 146GB disks as pairwise RAID 1 and one of these is
affected by the bug mentioned in the HP support page.
As quick fix i moved our database to the the other raid device built of
unaffected disks. Till now i don't got any new database corruption, so i
think the one disk with the firmware bug is the cause of our problems. Since
only the database does a lot of I/O onto the disks, this will help us for
the next days till we can upgrade or replace the bugged disk.
Thank you all for your hints and suggestions!

-- Matthias

-----Original Message-----
From: pgsql-general-owner@postgresql.org
[mailto:pgsql-general-owner@postgresql.org] On Behalf Of
Matthias.Pitzl@izb.de
Sent: Wednesday, September 20, 2006 4:51 PM
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] Strange database corruption with
PostgreSQL 7.4.x o

Hello all!

Ok, i found out some more informations. According to

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&
taskId=110&prodSeriesId=397634&prodTypeId=15351&prodSeriesId=397634&objectID
=PSD_EX050119_CW01