invalid page header
Hi All,
We are evaluating Postgresql as a db platform for one of our future
applications. Some tables in the database will contain more than
10.000.000 records, which as i understand it, should be no problem with
postgresql.
We have been trying to find the most effective/fastest way to manipulate
external data and insert them in the database. Our developper has
written some stored procedures in PL/PGSQL to do all this.
Everything works fine but we are constantly running into the same error.
After inserting about 2.000.000 records we get this error:
CETERROR: XX001: invalid page header in block 22182 of relation "dunn_main"
Some more error codes associated with this error:
CETCONTEXT: SQL statement "SELECT phone FROM dunn_main WHERE source_id
= $1 AND duns = $2 "
PL/pgSQL function "proc_dunn" line 29 at select into variables
CETLOCATION: ReadBuffer, bufmgr.c:257
CETSTATEMENT: SELECT proc_dunn ('Carriage Transfer','SL2
4BP','211121699','1','01','S','01753-648
900','5531',155,154,75176)
From what I could find about this error, I understand that it's
basically a data corruption in the table.
My main question is: why is this occuring?
We are running on a HP DL380G4 machine. That is running Novell OES sp2,
which is basically a Suse Linux Enterprise 9 setup + some Novell
extensions. I've compiled Postgresql 8.1.3 from source.
The test database runs on a single SCSI disk. No mirror, no Raid 5,
nothing. It is however connected to a HP Smartarray controller.
The server also runs other Postgres databases under another instance
without (until now) any problems. They are however not so big and don't
have a heavy insert/update load.
Does anybody have any clues or can tell me how to more exactly pinpoint
the problem? Hardware/bug in Postgresql?
Any advice would be more than welcome, because I'm afraid the developper
will demand to have an Oracle installation if I can't fix this (it was
also tested and ran without any issues.
Kind regard,
Jo De Haes.
"Jo De Haes" <jo.de_nospam_haes@indicator.be> wrote
CETERROR: XX001: invalid page header in block 22182 of relation
"dunn_main"
My main question is: why is this occuring?
I suppose there is no system error happens during the period (like lost
power). Can you attach the gdb at "b bufmgr.c:257" and print the content of
"(PageHeader) bufBlock"?
Regards,
Qingqing
"Qingqing Zhou" <zhouqq@cs.toronto.edu> writes:
"Jo De Haes" <jo.de_nospam_haes@indicator.be> wrote
CETERROR: XX001: invalid page header in block 22182 of relation
"dunn_main"
I suppose there is no system error happens during the period (like lost
power). Can you attach the gdb at "b bufmgr.c:257" and print the content of
"(PageHeader) bufBlock"?
Probably easier (especially if he's not using a debug-enabled build):
dump out the page from the disk file using pg_filedump, or even just
"dd | od -x".
regards, tom lane
Tom Lane wrote:
"Qingqing Zhou" <zhouqq@cs.toronto.edu> writes:
"Jo De Haes" <jo.de_nospam_haes@indicator.be> wrote
CETERROR: XX001: invalid page header in block 22182 of relation
"dunn_main"
I suppose there is no system error happens during the period (like lost
power). Can you attach the gdb at "b bufmgr.c:257" and print the content of
"(PageHeader) bufBlock"?Probably easier (especially if he's not using a debug-enabled build):
dump out the page from the disk file using pg_filedump, or even just
"dd | od -x".
Thanks for the 'easier' alternative. I tried to do this last friday,
but i was amazed when a count(*) on the table and a rerun of the query
that had caused the invalid page header error to occur just worked now.
I asked the developper to delete all imported data again an restart the
import. This import crashed again with the same error but this time on
another block.
2006-03-27 00:15:25.458 CESTERROR: XX001: invalid page header in block
48068 of relation "dunn_main"
2006-03-27 00:15:25.458 CESTCONTEXT: SQL statement "SELECT phone FROM
dunn_main WHERE source_id = $1 AND duns = $2 "
PL/pgSQL function "proc_dunn" line 29 at select into variables
2006-03-27 00:15:25.458 CESTLOCATION: ReadBuffer, bufmgr.c:257
2006-03-27 00:15:25.458 CESTSTATEMENT: SELECT proc_dunn ('J M
Darby','TA4 3BU','215517942','1','01','S',NULL,'0219',156,1
54,387166)
But again, when i do the 'SELECT proc_dunn ('J M Darby','TA4
3BU','215517942','1','01','S',NULL,'0219',156,1
54,387166)' statement now, it works without errors.
If I would like to dump block 48068 now with pg_dumpfile, how do i know
which file this block belongs to?
Jo.
Jo De Haes <jo.de_nospam_haes@indicator.be> writes:
I asked the developper to delete all imported data again an restart the
import. This import crashed again with the same error but this time on
another block.
2006-03-27 00:15:25.458 CESTERROR: XX001: invalid page header in block
48068 of relation "dunn_main"
2006-03-27 00:15:25.458 CESTCONTEXT: SQL statement "SELECT phone FROM
dunn_main WHERE source_id = $1 AND duns = $2 "
PL/pgSQL function "proc_dunn" line 29 at select into variables
2006-03-27 00:15:25.458 CESTLOCATION: ReadBuffer, bufmgr.c:257
2006-03-27 00:15:25.458 CESTSTATEMENT: SELECT proc_dunn ('J M
Darby','TA4 3BU','215517942','1','01','S',NULL,'0219',156,1
54,387166)
But again, when i do the 'SELECT proc_dunn ('J M Darby','TA4
3BU','215517942','1','01','S',NULL,'0219',156,1
54,387166)' statement now, it works without errors.
That is *really* strange. Are you certain that the function is
examining the same table you are? I'm wondering about multiple
similarly-named tables in different schemas, or something like that.
If I would like to dump block 48068 now with pg_dumpfile, how do i know
which file this block belongs to?
See
http://www.postgresql.org/docs/8.1/static/storage.html
and/or use contrib/oid2name.
regards, tom lane
Ok, So we reran everything and got the same error message again, now
i'm able to reproduce it.
2006-03-28 12:05:18.638 CESTERROR: XX001: invalid page header in block
39248 of relation "dunn_main"
2006-03-28 12:05:18.638 CESTLOCATION: ReadBuffer, bufmgr.c:257
2006-03-28 12:05:18.638 CESTSTATEMENT: VACUUM;
The output of the block in question is as follows:
* File: /backup/pgsql/data/base/16384/934173
* Options used: -R 39248 -f
*
* Dump created on: Tue Mar 28 13:43:09 2006
*******************************************************************
Block 39248 ********************************************************
<Header> -----
Block Offset: 0x132a0000 Offsets: Lower 536 (0x0218)
Block: Size 8192 Version 2 Upper 4796 (0x12bc)
LSN: logid 26 recoff 0xa9cad42c Special 8176 (0x1ff0)
Items: 129 Free Space: 4260
Length (including item array): 540
Error: Invalid header information.
0000: 1a000000 2cd4caa9 01000000 1802bc12 ....,...........
0010: f01f0220 d49f3800 b49f4000 949f4000 ... ..8...@...@.
0020: 749f4000 5c9f3000 449f3000 2c9f3000 t.@.\.0.D.0.,.0.
0030: 149f3000 fc9e3000 e49e3000 cc9e3000 ..0...0...0...0.
0040: b49e3000 9c9e3000 849e3000 6c9e3000 ..0...0...0.l.0.
0050: 549e3000 3c9e3000 249e3000 0c9e3000 T.0.<.0.$.0...0.
0060: f49d3000 dc9d3000 c49d3000 ac9d3000 ..0...0...0...0.
0070: 949d3000 7c9d3000 649d3000 4c9d3000 ..0.|.0.d.0.L.0.
0080: 289d4800 049d4800 e49c4000 c89c3800 (.H...H...@...8.
0090: a89c4000 909c3000 6c9c4800 549c3000 ..@...0.l.H.T.0.
00a0: ec923800 389c3800 1c9c3800 fc9b4000 ..8.8.8...8...@.
00b0: dc9b4000 bc9b4000 9c9b4000 7c9b4000 ..@...@...@.|.@.
00c0: 5c9b4000 3c9b4000 1c9b4000 009b3800 \.@.<.@...@...8.
00d0: e89a3000 d09a3000 b89a3000 a09a3000 ..0...0...0...0.
00e0: 849a3800 689a3800 509a3000 2c9a4800 ..8.h.8.P.0.,.H.
00f0: 0c9a4000 bc923000 d4923000 08933000 ..@...0...0...0.
0100: f4993000 dc993000 c4993000 ac993000 ..0...0...0...0.
0110: 94993000 7c993000 64993000 4c993000 ..0.|.0.d.0.L.0.
0120: 34993000 1c993000 04993000 ec983000 4.0...0...0...0.
0130: d4983000 bc983000 a4983000 8c983000 ..0...0...0...0.
0140: 74983000 5c983000 44983000 2c983000 t.0.\.0.D.0.,.0.
0150: 14983000 fc973000 e4973000 cc973000 ..0...0...0...0.
0160: b4973000 9c973000 84973000 6c973000 ..0...0...0.l.0.
0170: 54973000 3c973000 24973000 0c973000 T.0.<.0.$.0...0.
0180: f4963000 dc963000 c4963000 ac963000 ..0...0...0...0.
0190: 94963000 7c963000 64963000 4c963000 ..0.|.0.d.0.L.0.
01a0: 34963000 1c963000 04963000 ec953000 4.0...0...0...0.
01b0: d4953000 bc953000 a4953000 8c953000 ..0...0...0...0.
01c0: 74953000 5c953000 44953000 2c953000 t.0.\.0.D.0.,.0.
01d0: 14953000 fc943000 e4943000 cc943000 ..0...0...0...0.
01e0: b4943000 9c943000 80943800 64943800 ..0...0...8.d.8.
01f0: 48943800 2c943800 10943800 f4933800 H.8.,.8...8...8.
0200: cc935000 a8934800 80935000 58935000 ..P...H...P.X.P.
0210: 3c933800 20933800 20933800 <.8. .8. .8.
<Data> ------
Item 1 -- Length: 28 Offset: 8148 (0x1fd4) Flags: USED
1fd4: 00000f45 20001c40 13000000 62727574 ...E ..@....brut
1fe4: 6f2d7569 746b6572 696e6700 o-uitkering.
Item 2 -- Length: 32 Offset: 8116 (0x1fb4) Flags: USED
1fb4: 0000dd92 32002040 16000000 62727574 ....2. @....brut
1fc4: 6f2d6d61 616e6473 616c6172 69730000 o-maandsalaris..
Item 3 -- Length: 32 Offset: 8084 (0x1f94) Flags: USED
1f94: 00008e29 15002040 18000000 62727574 ...).. @....brut
1fa4: 6f6d6161 6e647361 6c617269 7373656e omaandsalarissen
Item 4 -- Length: 32 Offset: 8052 (0x1f74) Flags: USED
1f74: 0000af3a 32002040 18000000 62727574 ...:2. @....brut
1f84: 6f6d6161 6e647361 6c617269 7373656e omaandsalarissen
Item 5 -- Length: 24 Offset: 8028 (0x1f5c) Flags: USED
1f5c: 0000a70c 14001840 0e000000 62727574 .......@....brut
1f6c: 6f6d6172 67650000 omarge..
Item 6 -- Length: 24 Offset: 8004 (0x1f44) Flags: USED
1f44: 00003b0d 3d001840 0e000000 62727574 ..;.=..@....brut
1f54: 6f6d6172 67650000 omarge..
Item 7 -- Length: 24 Offset: 7980 (0x1f2c) Flags: USED
1f2c: 00007618 19001840 0e000000 62727574 ..v....@....brut
1f3c: 6f6d6172 67650000 omarge..
Item 8 -- Length: 24 Offset: 7956 (0x1f14) Flags: USED
1f14: 0000af19 25001840 0e000000 62727574 ....%..@....brut
1f24: 6f6d6172 67650000 omarge..
Item 9 -- Length: 24 Offset: 7932 (0x1efc) Flags: USED
1efc: 00007c1e 0e001840 0e000000 62727574 ..|....@....brut
1f0c: 6f6d6172 67650000 omarge..
Item 10 -- Length: 24 Offset: 7908 (0x1ee4) Flags: USED
1ee4: 00001420 11001840 0e000000 62727574 ... ...@....brut
1ef4: 6f6d6172 67650000 omarge..
Item 11 -- Length: 24 Offset: 7884 (0x1ecc) Flags: USED
1ecc: 00007528 2a001840 0e000000 62727574 ..u(*..@....brut
1edc: 6f6d6172 67650000 omarge..
Item 12 -- Length: 24 Offset: 7860 (0x1eb4) Flags: USED
1eb4: 0000c231 1f001840 0e000000 62727574 ...1...@....brut
1ec4: 6f6d6172 67650000 omarge..
Item 13 -- Length: 24 Offset: 7836 (0x1e9c) Flags: USED
1e9c: 00008240 21001840 0e000000 62727574 ...@!..@....brut
1eac: 6f6d6172 67650000 omarge..
Item 14 -- Length: 24 Offset: 7812 (0x1e84) Flags: USED
1e84: 0000f45e 39001840 0e000000 62727574 ...^9..@....brut
1e94: 6f6d6172 67650000 omarge..
Item 15 -- Length: 24 Offset: 7788 (0x1e6c) Flags: USED
1e6c: 00009362 24001840 0e000000 62727574 ...b$..@....brut
1e7c: 6f6d6172 67650000 omarge..
Item 16 -- Length: 24 Offset: 7764 (0x1e54) Flags: USED
1e54: 0000b666 3d001840 0e000000 62727574 ...f=..@....brut
1e64: 6f6d6172 67650000 omarge..
Item 17 -- Length: 24 Offset: 7740 (0x1e3c) Flags: USED
1e3c: 0000b76b 1b001840 0e000000 62727574 ...k...@....brut
1e4c: 6f6d6172 67650000 omarge..
Item 18 -- Length: 24 Offset: 7716 (0x1e24) Flags: USED
1e24: 00005d71 07001840 0e000000 62727574 ..]q...@....brut
1e34: 6f6d6172 67650000 omarge..
Item 19 -- Length: 24 Offset: 7692 (0x1e0c) Flags: USED
1e0c: 00009571 0b001840 0e000000 62727574 ...q...@....brut
1e1c: 6f6d6172 67650000 omarge..
Item 20 -- Length: 24 Offset: 7668 (0x1df4) Flags: USED
1df4: 0000ca89 20001840 0e000000 62727574 .... ..@....brut
1e04: 6f6d6172 67650000 omarge..
Item 21 -- Length: 24 Offset: 7644 (0x1ddc) Flags: USED
1ddc: 0000608b 0d001840 0e000000 62727574 ..`....@....brut
1dec: 6f6d6172 67650000 omarge..
Item 22 -- Length: 24 Offset: 7620 (0x1dc4) Flags: USED
1dc4: 0000f78c 1f001840 0e000000 62727574 .......@....brut
1dd4: 6f6d6172 67650000 omarge..
Item 23 -- Length: 24 Offset: 7596 (0x1dac) Flags: USED
1dac: 000065b0 16001840 0e000000 62727574 ..e....@....brut
1dbc: 6f6d6172 67650000 omarge..
Item 24 -- Length: 24 Offset: 7572 (0x1d94) Flags: USED
1d94: 00004ab1 22001840 0e000000 62727574 ..J."..@....brut
1da4: 6f6d6172 67650000 omarge..
Item 25 -- Length: 24 Offset: 7548 (0x1d7c) Flags: USED
1d7c: 00008d62 0f001840 0f000000 62727574 ...b...@....brut
1d8c: 6f6d6172 67653b00 omarge;.
Item 26 -- Length: 24 Offset: 7524 (0x1d64) Flags: USED
1d64: 00001620 22001840 0f000000 62727574 ... "..@....brut
1d74: 6f6d6172 67657300 omarges.
Item 27 -- Length: 24 Offset: 7500 (0x1d4c) Flags: USED
1d4c: 00001b69 1f001840 10000000 62727574 ...i...@....brut
1d5c: 6f6d696e 696d756d ominimum
Item 28 -- Length: 36 Offset: 7464 (0x1d28) Flags: USED
1d28: 00008d35 2c002440 1a000000 62727574 ...5,.$@....brut
1d38: 6f6d696e 696d756d 6a657567 646c6f6e ominimumjeugdlon
1d48: 656e0000 en..
Item 29 -- Length: 36 Offset: 7428 (0x1d04) Flags: USED
1d04: 00001e75 3b002440 1a000000 62727574 ...u;.$@....brut
1d14: 6f6d696e 696d756d 6a657567 646c6f6e ominimumjeugdlon
1d24: 656e0000 en..
Item 30 -- Length: 32 Offset: 7396 (0x1ce4) Flags: USED
1ce4: 00009974 39002040 16000000 62727574 ...t9. @....brut
1cf4: 6f2d6d69 6e696d75 6d6c6f6e 656e0000 o-minimumlonen..
Item 31 -- Length: 28 Offset: 7368 (0x1cc8) Flags: USED
1cc8: 0000181f 3f001c40 14000000 62727574 ....?..@....brut
1cd8: 6f6d696e 696d756d 6c6f6f6e ominimumloon
Item 32 -- Length: 32 Offset: 7336 (0x1ca8) Flags: USED
1ca8: 0000b220 1c002040 15000000 62727574 ... .. @....brut
1cb8: 6f2d6d69 6e696d75 6d6c6f6f 6e000000 o-minimumloon...
Item 33 -- Length: 24 Offset: 7312 (0x1c90) Flags: USED
1c90: 00002283 32001840 0f000000 62727574 ..".2..@....brut
1ca0: 6f2d6e65 74746f00 o-netto.
Item 34 -- Length: 36 Offset: 7276 (0x1c6c) Flags: USED
1c6c: 0000e39f 01002440 1b000000 62727574 ......$@....brut
1c7c: 6f2d6e65 74746f72 656b656e 6d616368 o-nettorekenmach
1c8c: 696e6500 ine.
Item 35 -- Length: 24 Offset: 7252 (0x1c54) Flags: USED
1c54: 0000a81c 3c001840 0f000000 62727574 ....<..@....brut
1c64: 6f2d6f6d 7a657400 o-omzet.
Item 36 -- Length: 28 Offset: 4844 (0x12ec) Flags: USED
12ec: 00001ecb 1a001c40 13000000 62727574 .......@....brut
12fc: 6f2d6f70 6272656e 67737400 o-opbrengst.
Item 37 -- Length: 28 Offset: 7224 (0x1c38) Flags: USED
1c38: 00004e1d 22001c40 13000000 62727574 ..N."..@....brut
1c48: 6f2d6f70 6272656e 67737400 o-opbrengst.
Item 38 -- Length: 28 Offset: 7196 (0x1c1c) Flags: USED
1c1c: 000025a3 22001c40 13000000 62727574 ..%."..@....brut
1c2c: 6f2d6f70 6272656e 67737400 o-opbrengst.
Item 39 -- Length: 32 Offset: 7164 (0x1bfc) Flags: USED
1bfc: 00003008 32002040 15000000 62727574 ..0.2. @....brut
1c0c: 6f2d6f70 6272656e 67737465 6e000000 o-opbrengsten...
Item 40 -- Length: 32 Offset: 7132 (0x1bdc) Flags: USED
1bdc: 0000ac1f 31002040 15000000 62727574 ....1. @....brut
1bec: 6f2d6f70 70657276 6c616b74 65000000 o-oppervlakte...
Item 41 -- Length: 32 Offset: 7100 (0x1bbc) Flags: USED
1bbc: 0000f225 2a002040 15000000 62727574 ...%*. @....brut
1bcc: 6f2d6f70 70657276 6c616b74 65000000 o-oppervlakte...
Item 42 -- Length: 32 Offset: 7068 (0x1b9c) Flags: USED
1b9c: 0000284f 3c002040 15000000 62727574 ..(O<. @....brut
1bac: 6f2d6f70 70657276 6c616b74 65000000 o-oppervlakte...
Item 43 -- Length: 32 Offset: 7036 (0x1b7c) Flags: USED
1b7c: 00002f55 19002040 15000000 62727574 ../U.. @....brut
1b8c: 6f2d6f70 70657276 6c616b74 65000000 o-oppervlakte...
Item 44 -- Length: 32 Offset: 7004 (0x1b5c) Flags: USED
1b5c: 00003155 17002040 15000000 62727574 ..1U.. @....brut
1b6c: 6f2d6f70 70657276 6c616b74 65000000 o-oppervlakte...
Item 45 -- Length: 32 Offset: 6972 (0x1b3c) Flags: USED
1b3c: 00006b75 09002040 15000000 62727574 ..ku.. @....brut
1b4c: 6f2d6f70 70657276 6c616b74 65000000 o-oppervlakte...
Item 46 -- Length: 32 Offset: 6940 (0x1b1c) Flags: USED
1b1c: 0000ac88 18002040 15000000 62727574 ...... @....brut
1b2c: 6f2d6f70 70657276 6c616b74 65000000 o-oppervlakte...
Item 47 -- Length: 28 Offset: 6912 (0x1b00) Flags: USED
1b00: 000038ba 02001c40 11000000 62727574 ..8....@....brut
1b10: 6f70656e 73696f65 6e000000 opensioen...
Item 48 -- Length: 24 Offset: 6888 (0x1ae8) Flags: USED
1ae8: 0000db0d 27001840 0f000000 62727574 ....'..@....brut
1af8: 6f707265 6d696500 opremie.
Item 49 -- Length: 24 Offset: 6864 (0x1ad0) Flags: USED
1ad0: 00002a18 10001840 0f000000 62727574 ..*....@....brut
1ae0: 6f707265 6d696500 opremie.
Item 50 -- Length: 24 Offset: 6840 (0x1ab8) Flags: USED
1ab8: 00006541 12001840 0f000000 62727574 ..eA...@....brut
1ac8: 6f707265 6d696500 opremie.
Item 51 -- Length: 24 Offset: 6816 (0x1aa0) Flags: USED
1aa0: 00007262 12001840 0e000000 62727574 ..rb...@....brut
1ab0: 6f707269 6a730000 oprijs..
Item 52 -- Length: 28 Offset: 6788 (0x1a84) Flags: USED
1a84: 0000c413 24001c40 12000000 62727574 ....$..@....brut
1a94: 6f72656e 64656d65 6e740000 orendement..
Item 53 -- Length: 28 Offset: 6760 (0x1a68) Flags: USED
1a68: 0000ea09 3c001c40 13000000 62727574 ....<..@....brut
1a78: 6f2d7265 6e64656d 656e7400 o-rendement.
Item 54 -- Length: 24 Offset: 6736 (0x1a50) Flags: USED
1a50: 0000d54c 28001840 0e000000 62727574 ...L(..@....brut
1a60: 6f72656e 74650000 orente..
Item 55 -- Length: 36 Offset: 6700 (0x1a2c) Flags: USED
1a2c: 000086c1 2b002440 19000000 62727574 ....+.$@....brut
1a3c: 6f2d7265 6e74652d 696e6b6f 6d737465 o-rente-inkomste
1a4c: 6e000000 n...
Item 56 -- Length: 32 Offset: 6668 (0x1a0c) Flags: USED
1a0c: 000086c1 23002040 18000000 62727574 ....#. @....brut
1a1c: 6f72656e 74652d69 6e6b6f6d 7374656e orente-inkomsten
Item 57 -- Length: 24 Offset: 4796 (0x12bc) Flags: USED
12bc: 00001fcc 13001840 10000000 62727574 .......@....brut
12cc: 6f73616c 61726973 osalaris
Item 58 -- Length: 24 Offset: 4820 (0x12d4) Flags: USED
12d4: 0000fbca 2b001840 10000000 62727574 ....+..@....brut
12e4: 6f73616c 61726973 osalaris
Item 59 -- Length: 24 Offset: 4872 (0x1308) Flags: USED
1308: 000008ca 1b001840 10000000 62727574 .......@....brut
1318: 6f73616c 61726973 osalaris
Item 60 -- Length: 24 Offset: 6644 (0x19f4) Flags: USED
19f4: 000035c3 27001840 10000000 62727574 ..5.'..@....brut
1a04: 6f73616c 61726973 osalaris
Item 61 -- Length: 24 Offset: 6620 (0x19dc) Flags: USED
19dc: 000069c7 25001840 10000000 62727574 ..i.%..@....brut
19ec: 6f73616c 61726973 osalaris
Item 62 -- Length: 24 Offset: 6596 (0x19c4) Flags: USED
19c4: 000042c5 31001840 10000000 62727574 ..B.1..@....brut
19d4: 6f73616c 61726973 osalaris
Item 63 -- Length: 24 Offset: 6572 (0x19ac) Flags: USED
19ac: 0000cbbe 24001840 10000000 62727574 ....$..@....brut
19bc: 6f73616c 61726973 osalaris
Item 64 -- Length: 24 Offset: 6548 (0x1994) Flags: USED
1994: 0000dec2 0e001840 10000000 62727574 .......@....brut
19a4: 6f73616c 61726973 osalaris
Item 65 -- Length: 24 Offset: 6524 (0x197c) Flags: USED
197c: 0000d8c0 18001840 10000000 62727574 .......@....brut
198c: 6f73616c 61726973 osalaris
Item 66 -- Length: 24 Offset: 6500 (0x1964) Flags: USED
1964: 00000dc2 05001840 10000000 62727574 .......@....brut
1974: 6f73616c 61726973 osalaris
Item 67 -- Length: 24 Offset: 6476 (0x194c) Flags: USED
194c: 00004380 1e001840 10000000 62727574 ..C....@....brut
195c: 6f73616c 61726973 osalaris
Item 68 -- Length: 24 Offset: 6452 (0x1934) Flags: USED
1934: 000027bb 17001840 10000000 62727574 ..'....@....brut
1944: 6f73616c 61726973 osalaris
Item 69 -- Length: 24 Offset: 6428 (0x191c) Flags: USED
191c: 00000806 0b001840 10000000 62727574 .......@....brut
192c: 6f73616c 61726973 osalaris
Item 70 -- Length: 24 Offset: 6404 (0x1904) Flags: USED
1904: 00009207 0f001840 10000000 62727574 .......@....brut
1914: 6f73616c 61726973 osalaris
Item 71 -- Length: 24 Offset: 6380 (0x18ec) Flags: USED
18ec: 00004809 29001840 10000000 62727574 ..H.)..@....brut
18fc: 6f73616c 61726973 osalaris
Item 72 -- Length: 24 Offset: 6356 (0x18d4) Flags: USED
18d4: 0000bb09 23001840 10000000 62727574 ....#..@....brut
18e4: 6f73616c 61726973 osalaris
Item 73 -- Length: 24 Offset: 6332 (0x18bc) Flags: USED
18bc: 0000710a 31001840 10000000 62727574 ..q.1..@....brut
18cc: 6f73616c 61726973 osalaris
Item 74 -- Length: 24 Offset: 6308 (0x18a4) Flags: USED
18a4: 0000e00d 35001840 10000000 62727574 ....5..@....brut
18b4: 6f73616c 61726973 osalaris
Item 75 -- Length: 24 Offset: 6284 (0x188c) Flags: USED
188c: 00006611 01001840 10000000 62727574 ..f....@....brut
189c: 6f73616c 61726973 osalaris
Item 76 -- Length: 24 Offset: 6260 (0x1874) Flags: USED
1874: 0000aa12 0d001840 10000000 62727574 .......@....brut
1884: 6f73616c 61726973 osalaris
Item 77 -- Length: 24 Offset: 6236 (0x185c) Flags: USED
185c: 00002d13 3d001840 10000000 62727574 ..-.=..@....brut
186c: 6f73616c 61726973 osalaris
Item 78 -- Length: 24 Offset: 6212 (0x1844) Flags: USED
1844: 00001316 39001840 10000000 62727574 ....9..@....brut
1854: 6f73616c 61726973 osalaris
Item 79 -- Length: 24 Offset: 6188 (0x182c) Flags: USED
182c: 00006018 08001840 10000000 62727574 ..`....@....brut
183c: 6f73616c 61726973 osalaris
Item 80 -- Length: 24 Offset: 6164 (0x1814) Flags: USED
1814: 0000811a 0c001840 10000000 62727574 .......@....brut
1824: 6f73616c 61726973 osalaris
Item 81 -- Length: 24 Offset: 6140 (0x17fc) Flags: USED
17fc: 0000ba1e 2f001840 10000000 62727574 ..../..@....brut
180c: 6f73616c 61726973 osalaris
Item 82 -- Length: 24 Offset: 6116 (0x17e4) Flags: USED
17e4: 00007120 1b001840 10000000 62727574 ..q ...@....brut
17f4: 6f73616c 61726973 osalaris
Item 83 -- Length: 24 Offset: 6092 (0x17cc) Flags: USED
17cc: 0000aa23 0b001840 10000000 62727574 ...#...@....brut
17dc: 6f73616c 61726973 osalaris
Item 84 -- Length: 24 Offset: 6068 (0x17b4) Flags: USED
17b4: 00002930 09001840 10000000 62727574 ..)0...@....brut
17c4: 6f73616c 61726973 osalaris
Item 85 -- Length: 24 Offset: 6044 (0x179c) Flags: USED
179c: 0000b531 32001840 10000000 62727574 ...12..@....brut
17ac: 6f73616c 61726973 osalaris
Item 86 -- Length: 24 Offset: 6020 (0x1784) Flags: USED
1784: 00000c32 05001840 10000000 62727574 ...2...@....brut
1794: 6f73616c 61726973 osalaris
Item 87 -- Length: 24 Offset: 5996 (0x176c) Flags: USED
176c: 00008732 29001840 10000000 62727574 ...2)..@....brut
177c: 6f73616c 61726973 osalaris
Item 88 -- Length: 24 Offset: 5972 (0x1754) Flags: USED
1754: 00009e32 15001840 10000000 62727574 ...2...@....brut
1764: 6f73616c 61726973 osalaris
Item 89 -- Length: 24 Offset: 5948 (0x173c) Flags: USED
173c: 0000cb32 2d001840 10000000 62727574 ...2-..@....brut
174c: 6f73616c 61726973 osalaris
Item 90 -- Length: 24 Offset: 5924 (0x1724) Flags: USED
1724: 0000e833 05001840 10000000 62727574 ...3...@....brut
1734: 6f73616c 61726973 osalaris
Item 91 -- Length: 24 Offset: 5900 (0x170c) Flags: USED
170c: 0000d034 20001840 10000000 62727574 ...4 ..@....brut
171c: 6f73616c 61726973 osalaris
Item 92 -- Length: 24 Offset: 5876 (0x16f4) Flags: USED
16f4: 0000a136 14001840 10000000 62727574 ...6...@....brut
1704: 6f73616c 61726973 osalaris
Item 93 -- Length: 24 Offset: 5852 (0x16dc) Flags: USED
16dc: 0000eb38 14001840 10000000 62727574 ...8...@....brut
16ec: 6f73616c 61726973 osalaris
Item 94 -- Length: 24 Offset: 5828 (0x16c4) Flags: USED
16c4: 00006c41 33001840 10000000 62727574 ..lA3..@....brut
16d4: 6f73616c 61726973 osalaris
Item 95 -- Length: 24 Offset: 5804 (0x16ac) Flags: USED
16ac: 00008d43 16001840 10000000 62727574 ...C...@....brut
16bc: 6f73616c 61726973 osalaris
Item 96 -- Length: 24 Offset: 5780 (0x1694) Flags: USED
1694: 0000e66a 19001840 10000000 62727574 ...j...@....brut
16a4: 6f73616c 61726973 osalaris
Item 97 -- Length: 24 Offset: 5756 (0x167c) Flags: USED
167c: 0000786c 06001840 10000000 62727574 ..xl...@....brut
168c: 6f73616c 61726973 osalaris
Item 98 -- Length: 24 Offset: 5732 (0x1664) Flags: USED
1664: 0000426d 18001840 10000000 62727574 ..Bm...@....brut
1674: 6f73616c 61726973 osalaris
Item 99 -- Length: 24 Offset: 5708 (0x164c) Flags: USED
164c: 00008b6e 1e001840 10000000 62727574 ...n...@....brut
165c: 6f73616c 61726973 osalaris
Item 100 -- Length: 24 Offset: 5684 (0x1634) Flags: USED
1634: 0000976e 11001840 10000000 62727574 ...n...@....brut
1644: 6f73616c 61726973 osalaris
Item 101 -- Length: 24 Offset: 5660 (0x161c) Flags: USED
161c: 0000086f 28001840 10000000 62727574 ...o(..@....brut
162c: 6f73616c 61726973 osalaris
Item 102 -- Length: 24 Offset: 5636 (0x1604) Flags: USED
1604: 0000c86f 22001840 10000000 62727574 ...o"..@....brut
1614: 6f73616c 61726973 osalaris
Item 103 -- Length: 24 Offset: 5612 (0x15ec) Flags: USED
15ec: 00004770 11001840 10000000 62727574 ..Gp...@....brut
15fc: 6f73616c 61726973 osalaris
Item 104 -- Length: 24 Offset: 5588 (0x15d4) Flags: USED
15d4: 00005270 3b001840 10000000 62727574 ..Rp;..@....brut
15e4: 6f73616c 61726973 osalaris
Item 105 -- Length: 24 Offset: 5564 (0x15bc) Flags: USED
15bc: 00007373 3e001840 10000000 62727574 ..ss>..@....brut
15cc: 6f73616c 61726973 osalaris
Item 106 -- Length: 24 Offset: 5540 (0x15a4) Flags: USED
15a4: 00008b7f 17001840 10000000 62727574 .......@....brut
15b4: 6f73616c 61726973 osalaris
Item 107 -- Length: 24 Offset: 5516 (0x158c) Flags: USED
158c: 00002283 01001840 10000000 62727574 .."....@....brut
159c: 6f73616c 61726973 osalaris
Item 108 -- Length: 24 Offset: 5492 (0x1574) Flags: USED
1574: 00002988 1f001840 10000000 62727574 ..)....@....brut
1584: 6f73616c 61726973 osalaris
Item 109 -- Length: 24 Offset: 5468 (0x155c) Flags: USED
155c: 0000a68b 13001840 10000000 62727574 .......@....brut
156c: 6f73616c 61726973 osalaris
Item 110 -- Length: 24 Offset: 5444 (0x1544) Flags: USED
1544: 0000c492 2b001840 10000000 62727574 ....+..@....brut
1554: 6f73616c 61726973 osalaris
Item 111 -- Length: 24 Offset: 5420 (0x152c) Flags: USED
152c: 00004a9f 20001840 10000000 62727574 ..J. ..@....brut
153c: 6f73616c 61726973 osalaris
Item 112 -- Length: 24 Offset: 5396 (0x1514) Flags: USED
1514: 000035a4 30001840 10000000 62727574 ..5.0..@....brut
1524: 6f73616c 61726973 osalaris
Item 113 -- Length: 24 Offset: 5372 (0x14fc) Flags: USED
14fc: 000041a4 24001840 10000000 62727574 ..A.$..@....brut
150c: 6f73616c 61726973 osalaris
Item 114 -- Length: 24 Offset: 5348 (0x14e4) Flags: USED
14e4: 000010ab 03001840 10000000 62727574 .......@....brut
14f4: 6f73616c 61726973 osalaris
Item 115 -- Length: 24 Offset: 5324 (0x14cc) Flags: USED
14cc: 00004db2 3e001840 10000000 62727574 ..M.>..@....brut
14dc: 6f73616c 61726973 osalaris
Item 116 -- Length: 24 Offset: 5300 (0x14b4) Flags: USED
14b4: 0000cab5 05001840 10000000 62727574 .......@....brut
14c4: 6f73616c 61726973 osalaris
Item 117 -- Length: 24 Offset: 5276 (0x149c) Flags: USED
149c: 000005b6 2e001840 10000000 62727574 .......@....brut
14ac: 6f73616c 61726973 osalaris
Item 118 -- Length: 28 Offset: 5248 (0x1480) Flags: USED
1480: 00006611 09001c40 11000000 62727574 ..f....@....brut
1490: 6f2d7361 6c617269 73000000 o-salaris...
Item 119 -- Length: 28 Offset: 5220 (0x1464) Flags: USED
1464: 0000901b 0b001c40 11000000 62727574 .......@....brut
1474: 6f2d7361 6c617269 73000000 o-salaris...
Item 120 -- Length: 28 Offset: 5192 (0x1448) Flags: USED
1448: 0000362e 1e001c40 11000000 62727574 ..6....@....brut
1458: 6f2d7361 6c617269 73000000 o-salaris...
Item 121 -- Length: 28 Offset: 5164 (0x142c) Flags: USED
142c: 0000532f 0a001c40 11000000 62727574 ..S/...@....brut
143c: 6f2d7361 6c617269 73000000 o-salaris...
Item 122 -- Length: 28 Offset: 5136 (0x1410) Flags: USED
1410: 0000be70 35001c40 11000000 62727574 ...p5..@....brut
1420: 6f2d7361 6c617269 73000000 o-salaris...
Item 123 -- Length: 28 Offset: 5108 (0x13f4) Flags: USED
13f4: 00002593 3e001c40 11000000 62727574 ..%.>..@....brut
1404: 6f2d7361 6c617269 73000000 o-salaris...
Item 124 -- Length: 40 Offset: 5068 (0x13cc) Flags: USED
13cc: 0000c14a 2c002840 1d000000 62727574 ...J,.(@....brut
13dc: 6f2d7361 6c617269 73766572 67656c69 o-salarisvergeli
13ec: 6a6b696e 67000000 jking...
Item 125 -- Length: 36 Offset: 5032 (0x13a8) Flags: USED
13a8: 00000f45 05002440 1a000000 62727574 ...E..$@....brut
13b8: 6f2d7363 68616465 76657267 6f656469 o-schadevergoedi
13c8: 6e670000 ng..
Item 126 -- Length: 40 Offset: 4992 (0x1380) Flags: USED
1380: 0000d029 12002840 1e000000 62727574 ...)..(@....brut
1390: 6f737469 6b73746f 6670726f 64756374 ostikstofproduct
13a0: 69656e6f 726d0000 ienorm..
Item 127 -- Length: 40 Offset: 4952 (0x1358) Flags: USED
1358: 0000d029 32002840 20000000 62727574 ...)2.(@ ...brut
1368: 6f737469 6b73746f 6670726f 64756374 ostikstofproduct
1378: 69656e6f 726d656e ienormen
Item 128 -- Length: 28 Offset: 4924 (0x133c) Flags: USED
133c: 000039ba 1b001c40 12000000 62727574 ..9....@....brut
134c: 6f756974 6b657269 6e670000 ouitkering..
Item 129 -- Length: 28 Offset: 4896 (0x1320) Flags: USED
1320: 0000382e 22001c40 13000000 62727574 ..8."..@....brut
1330: 6f2d7569 746b6572 696e6700 o-uitkering.
<Special Section> -----
BTree Index Section:
Flags: 0x0001 (LEAF)
Blocks: Previous (12811) Next (12769) Level (0)
1ff0: 0b320000 e1310000 00000000 01000000 .2...1..........
*** End of Requested Range Encountered. Last Block Read: 39248 ***
Tom Lane wrote:
Show quoted text
Jo De Haes <jo.de_nospam_haes@indicator.be> writes:
I asked the developper to delete all imported data again an restart the
import. This import crashed again with the same error but this time on
another block.2006-03-27 00:15:25.458 CESTERROR: XX001: invalid page header in block
48068 of relation "dunn_main"
2006-03-27 00:15:25.458 CESTCONTEXT: SQL statement "SELECT phone FROM
dunn_main WHERE source_id = $1 AND duns = $2 "
PL/pgSQL function "proc_dunn" line 29 at select into variables
2006-03-27 00:15:25.458 CESTLOCATION: ReadBuffer, bufmgr.c:257
2006-03-27 00:15:25.458 CESTSTATEMENT: SELECT proc_dunn ('J M
Darby','TA4 3BU','215517942','1','01','S',NULL,'0219',156,1
54,387166)But again, when i do the 'SELECT proc_dunn ('J M Darby','TA4
3BU','215517942','1','01','S',NULL,'0219',156,1
54,387166)' statement now, it works without errors.That is *really* strange. Are you certain that the function is
examining the same table you are? I'm wondering about multiple
similarly-named tables in different schemas, or something like that.If I would like to dump block 48068 now with pg_dumpfile, how do i know
which file this block belongs to?See
http://www.postgresql.org/docs/8.1/static/storage.html
and/or use contrib/oid2name.regards, tom lane
---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
OK. The saga continues, everything is a little bit more clear, but at
the same time a lot more confusing.
Today i wanted to reproduce the problem again. And guess what? A vacuum
of the database went thru without any problems.
I dump the block i was having problems with yesterday. It doesn't
report an invalid header anymore and it contains other data!!!
Turns out the data that was returned yesterday belongs to another database!
Some more detail about the setup. This server runs 2 instances of
postgresql. One production instance which is version 8.0.3. And
another testing instance installed in a different folder which runs
version 8.1.3 Am I wrong thinking this setup ought to work?
Both instances use completely seperated data folders.
So the first dump returned data that actually belongs to an 8.0.3
database (that runs fine). And today without _any_ intervention that
same block returns the correct data and the complete database is fine.
Where is the problem?
The fact that i'm running 2 different instances?
Cache on raid controller messing up?
Some strange voodoo?
Jo De Haes wrote:
Show quoted text
Ok, So we reran everything and got the same error message again, now
i'm able to reproduce it.
Tom Lane wrote:
Jo De Haes <jo.de_nospam_haes@indicator.be> writes:
I asked the developper to delete all imported data again an restart
the import. This import crashed again with the same error but this
time on another block.2006-03-27 00:15:25.458 CESTERROR: XX001: invalid page header in
block 48068 of relation "dunn_main"
2006-03-27 00:15:25.458 CESTCONTEXT: SQL statement "SELECT phone
FROM dunn_main WHERE source_id = $1 AND duns = $2 "
PL/pgSQL function "proc_dunn" line 29 at select into variables
2006-03-27 00:15:25.458 CESTLOCATION: ReadBuffer, bufmgr.c:257
2006-03-27 00:15:25.458 CESTSTATEMENT: SELECT proc_dunn ('J M
Darby','TA4 3BU','215517942','1','01','S',NULL,'0219',156,1
54,387166)But again, when i do the 'SELECT proc_dunn ('J M Darby','TA4
3BU','215517942','1','01','S',NULL,'0219',156,1
54,387166)' statement now, it works without errors.That is *really* strange. Are you certain that the function is
examining the same table you are? I'm wondering about multiple
similarly-named tables in different schemas, or something like that.If I would like to dump block 48068 now with pg_dumpfile, how do i
know which file this block belongs to?See
http://www.postgresql.org/docs/8.1/static/storage.html
and/or use contrib/oid2name.regards, tom lane
---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
Jo De Haes wrote:
OK. The saga continues, everything is a little bit more clear, but at
the same time a lot more confusing.Today i wanted to reproduce the problem again. And guess what? A
vacuum of the database went thru without any problems.I dump the block i was having problems with yesterday. It doesn't
report an invalid header anymore and it contains other data!!!
Inconsistant problems esp. with PostgreSQL are usually the result of
hardware failure.
Turns out the data that was returned yesterday belongs to another
database!Some more detail about the setup. This server runs 2 instances of
postgresql. One production instance which is version 8.0.3. And
another testing instance installed in a different folder which runs
version 8.1.3 Am I wrong thinking this setup ought to work?
No. Ihave done it before too. PostgreSQL instances running on
different ports or addresses are sufficiently isolated to prevent this
from being a problem.
Both instances use completely seperated data folders.
So the first dump returned data that actually belongs to an 8.0.3
database (that runs fine). And today without _any_ intervention that
same block returns the correct data and the complete database is fine.Where is the problem?
The fact that i'm running 2 different instances?
Cache on raid controller messing up?
Some strange voodoo?
I would see what sort of memory testing suite you can run on your system
first (memtestx86, for example) and go from there. It sounds to me like
some sort of a hardware issue. It *could* be bits flipped anywhere,
from the writehead on the disk to the main system memory or the CPU.
The likelihood that it is a random RAM error is reduced if you are using
ECC RAM. Otherwise it could be anything.
This being said, when I have seen bits flipped by the CPU usually you
get a lot of index issues and shared memory corruptions, so I would be
more inclined to think that this was RAM or RAID cache.
Best Wishes,
Chris Travers
Metatron Technology Consulting
Just a little followup on this problem.
We've moved the database to another server where it ran without problems.
HP just released new raid controller drivers for Suse and a firmware
update for the controller itself.
Until now the problem hasn't occurred anymore.
Thanks!
Jo.
Chris Travers wrote:
Show quoted text
Jo De Haes wrote:
OK. The saga continues, everything is a little bit more clear, but at
the same time a lot more confusing.Today i wanted to reproduce the problem again. And guess what? A
vacuum of the database went thru without any problems.I dump the block i was having problems with yesterday. It doesn't
report an invalid header anymore and it contains other data!!!Inconsistant problems esp. with PostgreSQL are usually the result of
hardware failure.Turns out the data that was returned yesterday belongs to another
database!Some more detail about the setup. This server runs 2 instances of
postgresql. One production instance which is version 8.0.3. And
another testing instance installed in a different folder which runs
version 8.1.3 Am I wrong thinking this setup ought to work?No. Ihave done it before too. PostgreSQL instances running on
different ports or addresses are sufficiently isolated to prevent this
from being a problem.Both instances use completely seperated data folders.
So the first dump returned data that actually belongs to an 8.0.3
database (that runs fine). And today without _any_ intervention that
same block returns the correct data and the complete database is fine.Where is the problem?
The fact that i'm running 2 different instances?
Cache on raid controller messing up?
Some strange voodoo?I would see what sort of memory testing suite you can run on your system
first (memtestx86, for example) and go from there. It sounds to me like
some sort of a hardware issue. It *could* be bits flipped anywhere,
from the writehead on the disk to the main system memory or the CPU.The likelihood that it is a random RAM error is reduced if you are using
ECC RAM. Otherwise it could be anything.This being said, when I have seen bits flipped by the CPU usually you
get a lot of index issues and shared memory corruptions, so I would be
more inclined to think that this was RAM or RAID cache.Best Wishes,
Chris Travers
Metatron Technology Consulting---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?