[Fwd: PGBuildfarm member narwhal Branch HEAD Status changed from OK to InstallCheck failure]

Dave Page <dpage@postgresql.org> writes:

I've been seeing this failure intermittently on Narwhal HEAD, and once
on 8.1. Other branches have been OK, as have other animals running on
the same physical box. Narwhal-HEAD is run more often than any other
builds however.

Anyone have any idea what might be wrong? It seems unlikely to be a
hardware issue given that it's the exact same test failures each time.

Yeah, I'd been wondering about that too, but have no clue what's up.
It seems particularly odd that all the failures are in installcheck
not check.

If you want to poke at it, I'd suggest changing the ERROR to PANIC
(it's in bufmgr.c) to cause a core dump, run installchecks till you
get a panic, and then look around in the dump to see what you can find.
It'd be particularly interesting to see what the buffer actually
contains. Also you could look at the corresponding page of the disk
file (which in theory should be the same as the buffer contents,
since this error check is only made just after a read() ...)

regards, tom lane

Tom Lane wrote:

Dave Page <dpage@postgresql.org> writes:

I've been seeing this failure intermittently on Narwhal HEAD, and once
on 8.1. Other branches have been OK, as have other animals running on
the same physical box. Narwhal-HEAD is run more often than any other
builds however.

Anyone have any idea what might be wrong? It seems unlikely to be a
hardware issue given that it's the exact same test failures each time.

Yeah, I'd been wondering about that too, but have no clue what's up.
It seems particularly odd that all the failures are in installcheck
not check.

If you want to poke at it, I'd suggest changing the ERROR to PANIC
(it's in bufmgr.c) to cause a core dump, run installchecks till you
get a panic, and then look around in the dump to see what you can find.
It'd be particularly interesting to see what the buffer actually
contains. Also you could look at the corresponding page of the disk
file (which in theory should be the same as the buffer contents,
since this error check is only made just after a read() ...)

Hmm, I'll give it a go when I'm back in the office, but bear in mind
this is a Mingw build on which debugging is nigh-on impossible.

Regards, Dave.

Dave Page <dpage@postgresql.org> writes:

Tom Lane wrote:

If you want to poke at it, I'd suggest changing the ERROR to PANIC
(it's in bufmgr.c) to cause a core dump, run installchecks till you
get a panic, and then look around in the dump to see what you can find.
It'd be particularly interesting to see what the buffer actually
contains. Also you could look at the corresponding page of the disk
file (which in theory should be the same as the buffer contents,
since this error check is only made just after a read() ...)

Hmm, I'll give it a go when I'm back in the office, but bear in mind
this is a Mingw build on which debugging is nigh-on impossible.

I was afraid of that. Well, at least get a dump of page 104 in that
index so we can see what's on-disk.

regards, tom lane

Tom Lane wrote:

Dave Page <dpage@postgresql.org> writes:

Tom Lane wrote:

If you want to poke at it, I'd suggest changing the ERROR to PANIC
(it's in bufmgr.c) to cause a core dump, run installchecks till you
get a panic, and then look around in the dump to see what you can find.
It'd be particularly interesting to see what the buffer actually
contains. Also you could look at the corresponding page of the disk
file (which in theory should be the same as the buffer contents,
since this error check is only made just after a read() ...)

Hmm, I'll give it a go when I'm back in the office, but bear in mind
this is a Mingw build on which debugging is nigh-on impossible.

I was afraid of that. Well, at least get a dump of page 104 in that
index so we can see what's on-disk.

Sure - I'll have to try with 8.1/8.2 unless you have a pg_filedump
that'll work with -HEAD?

/D

Dave Page <dpage@postgresql.org> writes:

Tom Lane wrote:

I was afraid of that. Well, at least get a dump of page 104 in that
index so we can see what's on-disk.

Sure - I'll have to try with 8.1/8.2 unless you have a pg_filedump
that'll work with -HEAD?

No, I don't, but a plain hex/ascii dump is probably the best thing
anyway, since we know the page header is wrong. So use any old
version of pg_filedump with -d switch.

regards, tom lane

Hmm, I'll give it a go when I'm back in the office, but bear
in mind this is a Mingw build on which debugging is nigh-on
impossible.

I use the Snapshot
http://prdownloads.sf.net/mingw/gdb-6.3-2.exe?download from sf.net.
It has some issues, but it is definitely useable.

Andreas

Zeugswetter Andreas ADI SD wrote:

Hmm, I'll give it a go when I'm back in the office, but bear
in mind this is a Mingw build on which debugging is nigh-on
impossible.

I use the Snapshot
http://prdownloads.sf.net/mingw/gdb-6.3-2.exe?download from sf.net.
It has some issues, but it is definitely useable.

I'll give it a go - thanks.

Regards, Dave.

Dave Page wrote:

If you want to poke at it, I'd suggest changing the ERROR to PANIC
(it's in bufmgr.c) to cause a core dump, run installchecks till you
get a panic, and then look around in the dump to see what you can find.

Well, in typical fashion after 25+ runs this morning there's not a
failure in sight :-(. I'll keep trying this afternoon, but in case that
doesn't work, I've tweaked my buildfarm config to leave error trees in
place so maybe we can catch it that way (though that'll be without the
PANIC of course).

Regards, Dave

Dave Page <dpage@postgresql.org> writes:

I've been seeing this failure intermittently on Narwhal HEAD, and once
on 8.1. Other branches have been OK, as have other animals running on
the same physical box. Narwhal-HEAD is run more often than any other
builds however.

Oh, this is interesting:

http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=baiji&amp;dt=2007-04-26%2022:00:02

Different compiler, different OS, not quite the same block number (109,
whereas IIRC all the previous examples have complained of block 104).
Is this the same physical machine as narwhal?

regards, tom lane

Tom Lane wrote:

Dave Page <dpage@postgresql.org> writes:

I've been seeing this failure intermittently on Narwhal HEAD, and once
on 8.1. Other branches have been OK, as have other animals running on
the same physical box. Narwhal-HEAD is run more often than any other
builds however.

Oh, this is interesting:

http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=baiji&amp;dt=2007-04-26%2022:00:02

Different compiler, different OS, not quite the same block number (109,
whereas IIRC all the previous examples have complained of block 104).
Is this the same physical machine as narwhal?

Yes, it is. It's an FC6 box running VMWare server, with a Win 2k3r2 VM
and a Vista ultimate VM, both with mingw and msvc animals.

I'm still not convinced it's a hardware problem - aside from the fact
that it's the same error every time (although, I note in this case it
was in check, not installcheck), I would expect at least one of SMART,
FC6, VMware or 2k3/Vista to spot that there was a problem. I have also
recreated the virtual disks of both VMs since this started happening. I
wonder if we're hitting some odd bug in VMware.

Anyhoo, unfortunately Baiji wasn't set to keep error builds - I've
changed that now and will run it a few times again. I'll also run a
sector level check of Narwhal's virtual disk and see if that complains.

Regards, Dave.

Dave Page <dpage@postgresql.org> writes:

Tom Lane wrote:

Is this the same physical machine as narwhal?

Yes, it is. It's an FC6 box running VMWare server, with a Win 2k3r2 VM
and a Vista ultimate VM, both with mingw and msvc animals.

I'm still not convinced it's a hardware problem - aside from the fact
that it's the same error every time (although, I note in this case it
was in check, not installcheck), I would expect at least one of SMART,
FC6, VMware or 2k3/Vista to spot that there was a problem. I have also
recreated the virtual disks of both VMs since this started happening. I
wonder if we're hitting some odd bug in VMware.

I concur it's too regular to be a hardware issue. The VMware idea is
a bit plausible though. If that's it, we ought to see failures of this
ilk on all four animals sooner or later ...

regards, tom lane

Tom Lane wrote:

I concur it's too regular to be a hardware issue. The VMware idea is
a bit plausible though. If that's it, we ought to see failures of this
ilk on all four animals sooner or later ...

I've run full disk scans in both Windows VMs, and forced an fsck of the
host just to be on the safe side and nothing showed up. By chance it
seems that VMWare released an update just yesterday so I've upgraded
everything. Hopefully the problem will go away now, but I'm not holding
my breath!

If not, one other option would be to roll back a couple of versions of
VMware - an older version hosted Bandicoot for some time with no problems.

Regards, Dave