Block-level CRC checks

Started by Alvaro Herreraabout 18 years ago368 messageshackers
Jump to latest
#1Joshua D. Drake
jd@commandprompt.com
In reply to: Paul Schlie (#20)
Re: Block-level CRC checks

Paul Schlie wrote:

this can not be relied upon, a strategy potentially utilizing the suspect
data as if it were good likely needs to be adopted, accompanied somehow with
a persistent indication that the query results (or specific sub-results) are
themselves suspect, as it may often be a lesser evil than the alternative
(but not always). Or use a file system like ZFS, and let it do its thing,
and hope for the best.

Problem is, most people are running PostgreSQL on one of two operating
systems. Linux or Win32. (/me has been amazed at how many people are
running win32)

ZFS is not an option; generally speaking.

Joshua D. Drake

#2Alvaro Herrera
alvherre@commandprompt.com

A customer of ours has been having trouble with corrupted data for some
time. Of course, we've almost always blamed hardware (and we've seen
RAID controllers have their firmware upgraded, among other actions), but
the useful thing to know is when corruption has happened, and where.

So we've been tasked with adding CRCs to data files.

The idea is that these CRCs are going to be checked just after reading
files from disk, and calculated just before writing it. They are
just a protection against the storage layer going mad; they are not
intended to protect against faulty RAM, CPU or kernel.

This code would be run-time or compile-time configurable. I'm not
absolutely sure which yet; the problem with run-time is what to do if
the user restarts the server with the setting flipped. It would have
almost no impact on users who don't enable it.

The implementation I'm envisioning requires the use of a new relation
fork to store the per-block CRCs. Initially I'm aiming at a CRC32 sum
for each block. FlushBuffer would calculate the checksum and store it
in the CRC fork; ReadBuffer_common would read the page, calculate the
checksum, and compare it to the one stored in the CRC fork.

A buffer's io_in_progress lock protects the buffer's CRC. We read and
pin the CRC page before acquiring the lock, to avoid having two buffer
IO operations in flight.

I'd like to submit this for 8.4, but I want to ensure that -hackers at
large approve of this feature before starting serious coding.

Opinions?

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

#3Jonah H. Harris
jonah.harris@gmail.com
In reply to: Alvaro Herrera (#2)
Re: Block-level CRC checks

On Tue, Sep 30, 2008 at 2:02 PM, Alvaro Herrera
<alvherre@commandprompt.com> wrote:

A customer of ours has been having trouble with corrupted data for some
time. Of course, we've almost always blamed hardware (and we've seen
RAID controllers have their firmware upgraded, among other actions), but
the useful thing to know is when corruption has happened, and where.

Agreed.

So we've been tasked with adding CRCs to data files.

Awesome.

The idea is that these CRCs are going to be checked just after reading
files from disk, and calculated just before writing it. They are
just a protection against the storage layer going mad; they are not
intended to protect against faulty RAM, CPU or kernel.

This is the common case.

This code would be run-time or compile-time configurable. I'm not
absolutely sure which yet; the problem with run-time is what to do if
the user restarts the server with the setting flipped. It would have
almost no impact on users who don't enable it.

I've supported this forever!

The implementation I'm envisioning requires the use of a new relation
fork to store the per-block CRCs. Initially I'm aiming at a CRC32 sum
for each block. FlushBuffer would calculate the checksum and store it
in the CRC fork; ReadBuffer_common would read the page, calculate the
checksum, and compare it to the one stored in the CRC fork.

A buffer's io_in_progress lock protects the buffer's CRC. We read and
pin the CRC page before acquiring the lock, to avoid having two buffer
IO operations in flight.

If the CRC gets written before the block, how is recovery going to
handle it? I'm not too familiar with the new forks stuff, but
recovery will pull the old block, compare it against the checksum, and
consider the block invalid, correct?

I'd like to submit this for 8.4, but I want to ensure that -hackers at
large approve of this feature before starting serious coding.

IMHO, this is a functionality that should be enabled by default (as it
is on most other RDBMS). It would've prevented severe corruption in
the 20 or so databases I've had to fix, and other than making it
optional, I don't see the reasoning for a separate relation fork
rather than storing it directly on the block (as everyone else does).
Similarly, I think Greg Stark was playing with a patch for it
(http://archives.postgresql.org/pgsql-hackers/2007-02/msg01850.php).

--
Jonah H. Harris, Senior DBA
myYearbook.com

#4Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#2)
Re: Block-level CRC checks

Alvaro Herrera <alvherre@commandprompt.com> writes:

The implementation I'm envisioning requires the use of a new relation
fork to store the per-block CRCs.

That seems bizarre, and expensive, and if you lose one block of the CRC
fork you lose confidence in a LOT of data. Why not keep the CRCs in the
page headers?

A buffer's io_in_progress lock protects the buffer's CRC.

Unfortunately, it doesn't. See hint bits.

regards, tom lane

#5Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Alvaro Herrera (#2)
Re: Block-level CRC checks

Alvaro Herrera wrote:

A customer of ours has been having trouble with corrupted data for some
time. Of course, we've almost always blamed hardware (and we've seen
RAID controllers have their firmware upgraded, among other actions), but
the useful thing to know is when corruption has happened, and where.

So we've been tasked with adding CRCs to data files.

The idea is that these CRCs are going to be checked just after reading
files from disk, and calculated just before writing it. They are
just a protection against the storage layer going mad; they are not
intended to protect against faulty RAM, CPU or kernel.

This has been suggested before, and the usual objection is precisely
that it only protects from errors in the storage layer, giving a false
sense of security.

Doesn't some filesystems include a per-block CRC, which would achieve
the same thing? ZFS?

This code would be run-time or compile-time configurable. I'm not
absolutely sure which yet; the problem with run-time is what to do if
the user restarts the server with the setting flipped. It would have
almost no impact on users who don't enable it.

Yeah, seems like it would need to be compile-time or initdb-time
configurable.

The implementation I'm envisioning requires the use of a new relation
fork to store the per-block CRCs. Initially I'm aiming at a CRC32 sum
for each block. FlushBuffer would calculate the checksum and store it
in the CRC fork; ReadBuffer_common would read the page, calculate the
checksum, and compare it to the one stored in the CRC fork.

Surely it would be much simpler to just add a field to the page header.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

#6Joshua D. Drake
jd@commandprompt.com
In reply to: Jonah H. Harris (#3)
Re: Block-level CRC checks

On Tue, 30 Sep 2008 14:33:04 -0400
"Jonah H. Harris" <jonah.harris@gmail.com> wrote:

I'd like to submit this for 8.4, but I want to ensure that -hackers
at large approve of this feature before starting serious coding.

IMHO, this is a functionality that should be enabled by default (as it
is on most other RDBMS). It would've prevented severe corruption in

What other RDMS have it enabled by default?

Sincerely,

Joshua D. Drake

--
The PostgreSQL Company since 1997: http://www.commandprompt.com/
PostgreSQL Community Conference: http://www.postgresqlconference.org/
United States PostgreSQL Association: http://www.postgresql.us/

#7Jonah H. Harris
jonah.harris@gmail.com
In reply to: Joshua D. Drake (#6)
Re: Block-level CRC checks

On Tue, Sep 30, 2008 at 2:49 PM, Joshua Drake <jd@commandprompt.com> wrote:

On Tue, 30 Sep 2008 14:33:04 -0400
"Jonah H. Harris" <jonah.harris@gmail.com> wrote:

I'd like to submit this for 8.4, but I want to ensure that -hackers
at large approve of this feature before starting serious coding.

IMHO, this is a functionality that should be enabled by default (as it
is on most other RDBMS). It would've prevented severe corruption in

What other RDMS have it enabled by default?

Oracle and (I belive) SQL Server >= 2005

--
Jonah H. Harris, Senior DBA
myYearbook.com

#8Markus Wanner
markus@bluegap.ch
In reply to: Alvaro Herrera (#2)
Re: Block-level CRC checks

Hello Alvaro,

some random thoughts while reading your proposal follow...

Alvaro Herrera wrote:

So we've been tasked with adding CRCs to data files.

Disks get larger and relative reliability shrinks, it seems. So I agree
that this is a worthwhile thing to have. But shouldn't that be the job
of the filesystem? Think of ZFS or the upcoming BTRFS.

The idea is that these CRCs are going to be checked just after reading
files from disk, and calculated just before writing it. They are
just a protection against the storage layer going mad; they are not
intended to protect against faulty RAM, CPU or kernel.

That sounds reasonable if we do it from Postgres.

This code would be run-time or compile-time configurable. I'm not
absolutely sure which yet; the problem with run-time is what to do if
the user restarts the server with the setting flipped. It would have
almost no impact on users who don't enable it.

I'd say calculating a CRC is close enough to be considered "no impact".
A single core of a modern CPU easily reaches way above 200 MiB/s
throughput for CRC32 today. See [1]Crypto++ benchmarks: http://www.cryptopp.com/benchmarks.html.

Maybe consider Adler-32 which is 3-4x faster [2]Wikipedia about hash functions: http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes, also part of zlib and
AFAIK about equally safe for 8k blocks and above.

The implementation I'm envisioning requires the use of a new relation
fork to store the per-block CRCs. Initially I'm aiming at a CRC32 sum
for each block. FlushBuffer would calculate the checksum and store it
in the CRC fork; ReadBuffer_common would read the page, calculate the
checksum, and compare it to the one stored in the CRC fork.

Huh? Aren't CRCs normally stored as part of the block they are supposed
to protect? Or how do you expect to ensure the data from the CRC
relation fork is correct? How about crash safety (a data block written,
but not its CRC block or vice versa)?

Wouldn't that double the amount of seeking required for writes?

I'd like to submit this for 8.4, but I want to ensure that -hackers at
large approve of this feature before starting serious coding.

Very cool!

Regards

Markus Wanner

[1]: Crypto++ benchmarks: http://www.cryptopp.com/benchmarks.html
http://www.cryptopp.com/benchmarks.html

[2]: Wikipedia about hash functions: http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes
http://en.wikipedia.org/wiki/List_of_hash_functions#Computational_costs_of_CRCs_vs_Hashes

#9Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Alvaro Herrera (#2)
Re: Block-level CRC checks

Alvaro Herrera wrote:

Initially I'm aiming at a CRC32 sum
for each block. FlushBuffer would calculate the checksum and store it
in the CRC fork; ReadBuffer_common would read the page, calculate the
checksum, and compare it to the one stored in the CRC fork.

There's one fundamental problem with that, related to the way our hint
bits are written.

Currently, hint bit updates are not WAL-logged, and thus no full page
write is done when only hint bits are changed. Imagine what happens if
hint bits are updated on a page, but there's no other changes, and we
crash so that only one half of the new page version makes it to disk (=
torn page). The CRC would not match, even though the page is actually valid.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

#10Mark Woodward
pgsql@mohawksoft.com
In reply to: Alvaro Herrera (#2)
Re: Block-level CRC checks

A customer of ours has been having trouble with corrupted data for some
time. Of course, we've almost always blamed hardware (and we've seen
RAID controllers have their firmware upgraded, among other actions), but
the useful thing to know is when corruption has happened, and where.

That is an important statement, to know when it happens not necessarily to
be able to recover the block or where in the block it is corrupt. Is that
correct?

So we've been tasked with adding CRCs to data files.

CRC or checksum? If the objective is merely general "detection" there
should be some latitude in choosing the methodology for performance.

The idea is that these CRCs are going to be checked just after reading
files from disk, and calculated just before writing it. They are
just a protection against the storage layer going mad; they are not
intended to protect against faulty RAM, CPU or kernel.

It will actually find faults in all if it. If the CPU can't add and/or a
RAM location lost a bit, this will blow up just as easily as a bad block.
It may cause "false identification" of an error, but it will keep a bad
system from hiding.

This code would be run-time or compile-time configurable. I'm not
absolutely sure which yet; the problem with run-time is what to do if
the user restarts the server with the setting flipped. It would have
almost no impact on users who don't enable it.

CPU capacity on modern hardware within a small area of RAM is practically
infinite when compared to any sort of I/O.

The implementation I'm envisioning requires the use of a new relation
fork to store the per-block CRCs. Initially I'm aiming at a CRC32 sum
for each block. FlushBuffer would calculate the checksum and store it
in the CRC fork; ReadBuffer_common would read the page, calculate the
checksum, and compare it to the one stored in the CRC fork.

Hell, all that is needed is a long or a short checksum value in the block.
I mean, if you just want a sanity test, it doesn't take much. Using a
second relation creates confusion. If there is a CRC discrepancy between
two different blocks, who's wrong? You need a third "control" to know. If
the block knows its CRC or checksum and that is in error, the block is
bad.

A buffer's io_in_progress lock protects the buffer's CRC. We read and
pin the CRC page before acquiring the lock, to avoid having two buffer
IO operations in flight.

I'd like to submit this for 8.4, but I want to ensure that -hackers at
large approve of this feature before starting serious coding.

Opinions?

If its fast enough, its a good idea. It could be very helpful in
protecting users data.

Show quoted text

--
Alvaro Herrera
http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#11Greg Smith
gsmith@gregsmith.com
In reply to: Heikki Linnakangas (#5)
Re: Block-level CRC checks

On Tue, 30 Sep 2008, Heikki Linnakangas wrote:

Doesn't some filesystems include a per-block CRC, which would achieve the
same thing? ZFS?

Yes, there is a popular advoacy piece for ZFS with a high-level view of
why and how they implement that at
http://blogs.sun.com/bonwick/entry/zfs_end_to_end_data The guarantees are
stronger than what you can get if you just put a CRC in the block itself.
I'd never really thought too hard about putting this in the database
knowing that ZFS is available for environments where this is a concern,
but it certainly would be a nice addition.

The best analysis I've ever seen that makes a case for OS or higher level
disk checksums of some sort, by looking at the myriad ways that disks and
disk arrays fail in the real world, is in
http://www.usenix.org/event/fast08/tech/full_papers/bairavasundaram/bairavasundaram.pdf
(there is a shorter version that hits the high points of that at
http://www.usenix.org/publications/login/2008-06/openpdfs/bairavasundaram.pdf
)

One really interesting bit in there I'd never seen before is that they
find real data that supports the stand that enterprise drives are
significantly more reliable than consumer ones. While general failure
rates aren't that different, "SATA disks have an order of magnitude higher
probability of developing checksum mismatches than Fibre Channel disks. We
find that 0.66% of SATA disks develop at least one mismatch during the
first 17 months in the field, whereas only 0.06% of Fibre Channel disks
develop a mismatch during that time."

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

#12Bruce Momjian
bruce@momjian.us
In reply to: Alvaro Herrera (#2)
Re: Block-level CRC checks

Alvaro Herrera wrote:

A customer of ours has been having trouble with corrupted data for some
time. Of course, we've almost always blamed hardware (and we've seen
RAID controllers have their firmware upgraded, among other actions), but
the useful thing to know is when corruption has happened, and where.

So we've been tasked with adding CRCs to data files.

Maybe a stupid question, but what I/O subsystems corrupt data and fail
to report it?

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

#13Jeffrey Baker
jwbaker@gmail.com
In reply to: Bruce Momjian (#12)
Re: Block-level CRC checks

On Tue, Sep 30, 2008 at 1:41 PM, Bruce Momjian <bruce@momjian.us> wrote:

Alvaro Herrera wrote:

A customer of ours has been having trouble with corrupted data for some
time. Of course, we've almost always blamed hardware (and we've seen
RAID controllers have their firmware upgraded, among other actions), but
the useful thing to know is when corruption has happened, and where.

So we've been tasked with adding CRCs to data files.

Maybe a stupid question, but what I/O subsystems corrupt data and fail
to report it?

Practically all of them. Here is a good paper on various checksums, their
failure rates, and practical applications.

"Parity Lost and Parity Regained"
http://www.usenix.org/event/fast08/tech/full_papers/krioukov/krioukov_html/index.html

-jwb

#14Jim Nasby
decibel@decibel.org
In reply to: Mark Woodward (#10)
Re: Block-level CRC checks

On Sep 30, 2008, at 2:17 PM, pgsql@mohawksoft.com wrote:

A customer of ours has been having trouble with corrupted data for
some
time. Of course, we've almost always blamed hardware (and we've seen
RAID controllers have their firmware upgraded, among other
actions), but
the useful thing to know is when corruption has happened, and where.

That is an important statement, to know when it happens not
necessarily to
be able to recover the block or where in the block it is corrupt.
Is that
correct?

Oh, correcting the corruption would be AWESOME beyond belief! But at
this point I'd settle for just knowing it had happened.

So we've been tasked with adding CRCs to data files.

CRC or checksum? If the objective is merely general "detection" there
should be some latitude in choosing the methodology for performance.

See above. Perhaps the best win would be a case where you could
choose which method you wanted. We generally have extra CPU on the
servers, so we could afford to burn some cycles with more complex
algorithms.

The idea is that these CRCs are going to be checked just after
reading
files from disk, and calculated just before writing it. They are
just a protection against the storage layer going mad; they are not
intended to protect against faulty RAM, CPU or kernel.

It will actually find faults in all if it. If the CPU can't add and/
or a
RAM location lost a bit, this will blow up just as easily as a bad
block.
It may cause "false identification" of an error, but it will keep a
bad
system from hiding.

Well, very likely not, since the intention is to only compute the CRC
when we write the block out, at least for now. In the future I would
like to be able to detect when a CPU or memory goes bonkers and poops
on something, because that's actually happened to us as well.

The implementation I'm envisioning requires the use of a new relation
fork to store the per-block CRCs. Initially I'm aiming at a CRC32
sum
for each block. FlushBuffer would calculate the checksum and
store it
in the CRC fork; ReadBuffer_common would read the page, calculate the
checksum, and compare it to the one stored in the CRC fork.

Hell, all that is needed is a long or a short checksum value in the
block.
I mean, if you just want a sanity test, it doesn't take much. Using a
second relation creates confusion. If there is a CRC discrepancy
between
two different blocks, who's wrong? You need a third "control" to
know. If
the block knows its CRC or checksum and that is in error, the block is
bad.

I believe the idea was to make this as non-invasive as possible. And
it would be really nice if this could be enabled without a dump/
reload (maybe the upgrade stuff would make this possible?)
--
Decibel!, aka Jim C. Nasby, Database Architect decibel@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828

Attachments:

smime.p7sapplication/pkcs7-signature; name=smime.p7sDownload
#15Joshua D. Drake
jd@commandprompt.com
In reply to: Jeffrey Baker (#13)
Re: Block-level CRC checks

On Tue, 30 Sep 2008 13:48:52 -0700
"Jeffrey Baker" <jwbaker@gmail.com> wrote:

Practically all of them. Here is a good paper on various checksums,
their failure rates, and practical applications.

"Parity Lost and Parity Regained"
http://www.usenix.org/event/fast08/tech/full_papers/krioukov/krioukov_html/index.html

In a related article published in Login called "Data Corruption in the
storage stack: a closer look" they say:

During a 41-month period we observed more than 400,000 instances of
checksum mistmatches, 8% of which were discovered during RAID
reconstruction, creating the possibility of real data loss.

They also have a wonderful term they mention, "Silent Data corruptions".

Joshua D. Drake

[1]: Login June 2008

-jwb

--
The PostgreSQL Company since 1997: http://www.commandprompt.com/
PostgreSQL Community Conference: http://www.postgresqlconference.org/
United States PostgreSQL Association: http://www.postgresql.us/

#16Mark Woodward
pgsql@mohawksoft.com
In reply to: Jim Nasby (#14)
Re: Block-level CRC checks

I believe the idea was to make this as non-invasive as possible. And
it would be really nice if this could be enabled without a dump/
reload (maybe the upgrade stuff would make this possible?)
--

It's all about the probability of a duplicate check being generated. If
you use a 32 bit checksum, then you have a theoretical probability of 1 in
4 billion that a corrupt block will be missed (probably much lower
depending on your algorithm). If you use a short, then a 1 in 65 thousand
probability. If you use an 8 bit number, then 1 in 256.

Why am I going on? Well, if there are any spare bits in a block header,
they could be used for the check value.

#17Jim Nasby
decibel@decibel.org
In reply to: Heikki Linnakangas (#5)
Re: Block-level CRC checks

On Sep 30, 2008, at 1:48 PM, Heikki Linnakangas wrote:

This has been suggested before, and the usual objection is
precisely that it only protects from errors in the storage layer,
giving a false sense of security.

If you can come up with a mechanism for detecting non-storage errors
as well, I'm all ears. :)

In the meantime, you're way, way more likely to experience corruption
at the storage layer than anywhere else. We've had several corruption
events, only one of which was memory related... and we *know* it was
memory related because we actually got logs saying so. But with a SAN
environment there's a lot of moving parts, all waiting to screw up
your data:

filesystem
SAN device driver
SAN network
SAN BIOS
drive BIOS
drive

That's above things that could hose your data outside of storage:
kernel
CPU
memory
motherboard

Doesn't some filesystems include a per-block CRC, which would
achieve the same thing? ZFS?

Sure, some do. We're on linux and can't run ZFS. And I'll argue that
no linux FS is anywhere near as tested as ext3 is, which means that
going to some other FS that offers you CRC means you're now exposing
yourself to the possibility of issues with the FS itself. Not to
mention that changing filesystems on a large production system is
very painful.
--
Decibel!, aka Jim C. Nasby, Database Architect decibel@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828

Attachments:

smime.p7sapplication/pkcs7-signature; name=smime.p7sDownload
#18Greg Stark
greg.stark@enterprisedb.com
In reply to: Jim Nasby (#17)
Re: Block-level CRC checks

On 30 Sep 2008, at 10:17 PM, Decibel! <decibel@decibel.org> wrote:

On Sep 30, 2008, at 1:48 PM, Heikki Linnakangas wrote:

This has been suggested before, and the usual objection is
precisely that it only protects from errors in the storage layer,
giving a false sense of security.

If you can come up with a mechanism for detecting non-storage errors
as well, I'm all ears. :)

In the meantime, you're way, way more likely to experience
corruption at the storage layer than anywhere else.

Fwiw this hasn't been my experience. Bad memory is extremely common
and even the storage failures I've seen (excluding the drive crashes)
turned out to actually be caused by bad memory.

That said I've always been interested in doing this. The main use case
in my mind has actually been for data that's been restored from old
backups which have been lying round and floating between machines for
a while with many opportunities for bit errors to show up.

The main stumbling block I ran into was how to deal with turning the
option off and on. I wanted it to be possible to turn off the option
to have the database ignore any errors and to avoid the overhead.

But that means including an escape hatch value which is always
considered to be correct. But that dramatically reduces the
effectiveness of the scheme.

Another issue is it will make space available on each page smaller
making it harder to do in place upgrades.

If you can deal with those issues and carefully deal with the
contingencies so it's clear to people what to do when errra occur or
they want to turn the feature on or off then I'm all for it. That
despite my experience of memory errors being a lot more common than
undetected storage errors.

#19Andrew Chernow
ac@esilo.com
In reply to: Joshua D. Drake (#15)
Re: Block-level CRC checks

Joshua Drake wrote:

During a 41-month period we observed more than 400,000 instances of
checksum mistmatches, 8% of which were discovered during RAID
reconstruction, creating the possibility of real data loss.

They also have a wonderful term they mention, "Silent Data corruptions".

Exactely!

From my experience, the only assumption to be made about storage is that it can
and will fail ... frequently! It is unreliable (not to mention slooow) and
should not be trusted; regardless of the price tag or brand.

This could help detect:

- fs corruption
- vfs bug
- raid controller firmware bug
- bad disk sector
- power crash
- weird martian-like raid rebuilds

Although, this idea won't prevent anything. Everything would still sinisterly
fail on you. The difference is, no more silence.

--
Andrew Chernow
eSilo, LLC
every bit counts
http://www.esilo.com/

#20Paul Schlie
schlie@comcast.net
In reply to: Andrew Chernow (#19)
Re: Block-level CRC checks

If you are concerned with data integrity (not caused by bugs in the code
itself), you may be interested in utilizing ZFS; however, be aware that I
found and reported a bug in their implementation of the Fletcher checksum
algorithm they use by default to attempt to verify the integrity of the data
stored in their file system, and further aware that checksums/CRCs do not
enable the correction of errors in general, therefore be prepared to make
the decision of "what should be done in the event of a failure"; ZFS
effectively locks up in certain circumstances rather risk silently using
suspect data with some form of persistent indication that the result may be
corrupted. (strong CRC's and FEC's are relatively inexpensive to compute).

So in summary, my two cents: a properly implemented 32/64 bit Fletcher
checksum is likely adequate to detect most errors (and correct them if
presumed to be a result of a single flipped bit within 128KB or so, as such
a Fletcher checksum has a hamming distance of 3 within blocks of this size,
albeit fairly expensive to do so by trial and error; further presuming that
this can not be relied upon, a strategy potentially utilizing the suspect
data as if it were good likely needs to be adopted, accompanied somehow with
a persistent indication that the query results (or specific sub-results) are
themselves suspect, as it may often be a lesser evil than the alternative
(but not always). Or use a file system like ZFS, and let it do its thing,
and hope for the best.

#21Paul Schlie
schlie@comcast.net
In reply to: Paul Schlie (#20)
#22Laurenz Albe
laurenz.albe@wien.gv.at
In reply to: Jonah H. Harris (#7)
#23Zdenek Kotala
Zdenek.Kotala@Sun.COM
In reply to: Alvaro Herrera (#2)
#24Harald Armin Massa
haraldarminmassa@gmail.com
In reply to: Zdenek Kotala (#23)
#25Hannu Krosing
hannu@2ndQuadrant.com
In reply to: Mark Woodward (#16)
#26Tom Lane
tgl@sss.pgh.pa.us
In reply to: Hannu Krosing (#25)
#27Tom Lane
tgl@sss.pgh.pa.us
In reply to: Harald Armin Massa (#24)
#28Jonah H. Harris
jonah.harris@gmail.com
In reply to: Tom Lane (#27)
#29Mark Woodward
pgsql@mohawksoft.com
In reply to: Hannu Krosing (#25)
#30Mark Woodward
pgsql@mohawksoft.com
In reply to: Tom Lane (#26)
#31Brian Hurt
bhurt@janestcapital.com
In reply to: Paul Schlie (#21)
#32Brian Hurt
bhurt@janestcapital.com
In reply to: Brian Hurt (#31)
#33Tom Lane
tgl@sss.pgh.pa.us
In reply to: Mark Woodward (#30)
#34Paul Schlie
schlie@comcast.net
In reply to: Brian Hurt (#31)
#35Paul Schlie
schlie@comcast.net
In reply to: Brian Hurt (#32)
#36Jonah H. Harris
jonah.harris@gmail.com
In reply to: Tom Lane (#33)
#37Alvaro Herrera
alvherre@commandprompt.com
In reply to: Tom Lane (#4)
#38Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#37)
#39Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jonah H. Harris (#36)
#40Mark Woodward
pgsql@mohawksoft.com
In reply to: Tom Lane (#33)
#41Jonah H. Harris
jonah.harris@gmail.com
In reply to: Tom Lane (#39)
#42Tom Lane
tgl@sss.pgh.pa.us
In reply to: Paul Schlie (#34)
#43Alvaro Herrera
alvherre@commandprompt.com
In reply to: Tom Lane (#39)
#44Greg Stark
stark@enterprisedb.com
In reply to: Tom Lane (#39)
#45Jonah H. Harris
jonah.harris@gmail.com
In reply to: Alvaro Herrera (#43)
#46Paul Schlie
schlie@comcast.net
In reply to: Jonah H. Harris (#45)
#47Florian Weimer
fweimer@bfk.de
In reply to: Heikki Linnakangas (#9)
#48Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#43)
#49Csaba Nagy
nagy@ecircle-ag.com
In reply to: Greg Stark (#44)
#50Florian Weimer
fweimer@bfk.de
In reply to: Tom Lane (#39)
#51Aidan Van Dyk
aidan@highrise.ca
In reply to: Greg Stark (#44)
#52Paul Schlie
schlie@comcast.net
In reply to: Tom Lane (#42)
#53Mark Mielke
mark@mark.mielke.cc
In reply to: Aidan Van Dyk (#51)
#54Mark Mielke
mark@mark.mielke.cc
In reply to: Tom Lane (#42)
#55Greg Stark
stark@enterprisedb.com
In reply to: Aidan Van Dyk (#51)
#56Greg Stark
stark@enterprisedb.com
In reply to: Paul Schlie (#52)
#57Sam Mason
sam@samason.me.uk
In reply to: Alvaro Herrera (#43)
#58Kevin Grittner
Kevin.Grittner@wicourts.gov
In reply to: Tom Lane (#42)
#59Paul Schlie
schlie@comcast.net
In reply to: Kevin Grittner (#58)
#60Tom Lane
tgl@sss.pgh.pa.us
In reply to: Paul Schlie (#59)
#61Jonah H. Harris
jonah.harris@gmail.com
In reply to: Tom Lane (#60)
#62Tom Lane
tgl@sss.pgh.pa.us
In reply to: Aidan Van Dyk (#51)
#63Mark Woodward
pgsql@mohawksoft.com
In reply to: Tom Lane (#62)
#64Tom Lane
tgl@sss.pgh.pa.us
In reply to: Mark Woodward (#63)
#65Greg Stark
stark@enterprisedb.com
In reply to: Mark Woodward (#63)
#66Tom Lane
tgl@sss.pgh.pa.us
In reply to: Greg Stark (#65)
#67Aidan Van Dyk
aidan@highrise.ca
In reply to: Tom Lane (#66)
#68Jonah H. Harris
jonah.harris@gmail.com
In reply to: Aidan Van Dyk (#67)
#69Jonah H. Harris
jonah.harris@gmail.com
In reply to: Jonah H. Harris (#68)
#70Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Jonah H. Harris (#68)
#71Florian Weimer
fweimer@bfk.de
In reply to: Greg Stark (#56)
#72Hannu Krosing
hannu@2ndQuadrant.com
In reply to: Heikki Linnakangas (#70)
#73Brian Hurt
bhurt@janestcapital.com
In reply to: Jonah H. Harris (#68)
#74Jonah H. Harris
jonah.harris@gmail.com
In reply to: Brian Hurt (#73)
#75Brian Hurt
bhurt@janestcapital.com
In reply to: Jonah H. Harris (#74)
#76Greg Stark
stark@enterprisedb.com
In reply to: Jonah H. Harris (#74)
#77Jonah H. Harris
jonah.harris@gmail.com
In reply to: Brian Hurt (#75)
#78Jonah H. Harris
jonah.harris@gmail.com
In reply to: Greg Stark (#76)
#79Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Brian Hurt (#75)
#80Tom Lane
tgl@sss.pgh.pa.us
In reply to: Heikki Linnakangas (#79)
#81Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Tom Lane (#80)
#82Andrew Chernow
ac@esilo.com
In reply to: Jonah H. Harris (#69)
#83Jonah H. Harris
jonah.harris@gmail.com
In reply to: Andrew Chernow (#82)
#84Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Jonah H. Harris (#83)
#85Jonah H. Harris
jonah.harris@gmail.com
In reply to: Heikki Linnakangas (#84)
#86Tom Lane
tgl@sss.pgh.pa.us
In reply to: Heikki Linnakangas (#81)
#87Tom Lane
tgl@sss.pgh.pa.us
In reply to: Andrew Chernow (#82)
#88Jonah H. Harris
jonah.harris@gmail.com
In reply to: Tom Lane (#86)
#89Jonah H. Harris
jonah.harris@gmail.com
In reply to: Jonah H. Harris (#88)
#90Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jonah H. Harris (#89)
#91Robert Treat
xzilla@users.sourceforge.net
In reply to: Jim Nasby (#17)
#92Aidan Van Dyk
aidan@highrise.ca
In reply to: Tom Lane (#90)
#93Joshua D. Drake
jd@commandprompt.com
In reply to: Robert Treat (#91)
#94Jonah H. Harris
jonah.harris@gmail.com
In reply to: Aidan Van Dyk (#92)
#95Aidan Van Dyk
aidan@highrise.ca
In reply to: Jonah H. Harris (#94)
#96Jonah H. Harris
jonah.harris@gmail.com
In reply to: Aidan Van Dyk (#95)
#97Bruce Momjian
bruce@momjian.us
In reply to: Jonah H. Harris (#96)
#98Aidan Van Dyk
aidan@highrise.ca
In reply to: Bruce Momjian (#97)
#99Jonah H. Harris
jonah.harris@gmail.com
In reply to: Bruce Momjian (#97)
#100Greg Stark
greg.stark@enterprisedb.com
In reply to: Aidan Van Dyk (#95)
#101Bruce Momjian
bruce@momjian.us
In reply to: Jonah H. Harris (#99)
#102Jonah H. Harris
jonah.harris@gmail.com
In reply to: Greg Stark (#100)
#103Alvaro Herrera
alvherre@commandprompt.com
In reply to: Greg Stark (#100)
#104Aidan Van Dyk
aidan@highrise.ca
In reply to: Greg Stark (#100)
#105Greg Stark
stark@enterprisedb.com
In reply to: Aidan Van Dyk (#104)
#106Jonah H. Harris
jonah.harris@gmail.com
In reply to: Greg Stark (#105)
#107Jonah H. Harris
jonah.harris@gmail.com
In reply to: Alvaro Herrera (#103)
#108Aidan Van Dyk
aidan@highrise.ca
In reply to: Jonah H. Harris (#106)
#109Robert Treat
xzilla@users.sourceforge.net
In reply to: Tom Lane (#33)
#110Alvaro Herrera
alvherre@commandprompt.com
In reply to: Robert Treat (#109)
#111Alvaro Herrera
alvherre@commandprompt.com
In reply to: Jonah H. Harris (#107)
#112Aidan Van Dyk
aidan@highrise.ca
In reply to: Alvaro Herrera (#111)
#113Simon Riggs
simon@2ndQuadrant.com
In reply to: Alvaro Herrera (#111)
#114Dawid Kuroczko
qnex42@gmail.com
In reply to: Jonah H. Harris (#102)
#115Jim Nasby
decibel@decibel.org
In reply to: Sam Mason (#57)
#116Greg Stark
stark@enterprisedb.com
In reply to: Dawid Kuroczko (#114)
#117Jim Nasby
decibel@decibel.org
In reply to: Alvaro Herrera (#111)
#118Brian Hurt
bhurt@janestcapital.com
In reply to: Jim Nasby (#117)
#119Dawid Kuroczko
qnex42@gmail.com
In reply to: Brian Hurt (#118)
#120Bruce Momjian
bruce@momjian.us
In reply to: Brian Hurt (#118)
#121Aidan Van Dyk
aidan@highrise.ca
In reply to: Jim Nasby (#117)
#122Alvaro Herrera
alvherre@commandprompt.com
In reply to: Alvaro Herrera (#2)
#123Jonah H. Harris
jonah.harris@gmail.com
In reply to: Alvaro Herrera (#122)
#124Greg Stark
greg.stark@enterprisedb.com
In reply to: Jonah H. Harris (#123)
#125Jonah H. Harris
jonah.harris@gmail.com
In reply to: Greg Stark (#124)
#126Aidan Van Dyk
aidan@highrise.ca
In reply to: Greg Stark (#124)
#127Simon Riggs
simon@2ndQuadrant.com
In reply to: Alvaro Herrera (#122)
#128Alvaro Herrera
alvherre@commandprompt.com
In reply to: Simon Riggs (#127)
#129Simon Riggs
simon@2ndQuadrant.com
In reply to: Alvaro Herrera (#128)
#130Greg Stark
greg.stark@enterprisedb.com
In reply to: Alvaro Herrera (#128)
#131Markus Wanner
markus@bluegap.ch
In reply to: Alvaro Herrera (#122)
#132Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Markus Wanner (#131)
#133Paul Schlie
schlie@comcast.net
In reply to: Heikki Linnakangas (#132)
#134Alvaro Herrera
alvherre@commandprompt.com
In reply to: Simon Riggs (#127)
#135Alvaro Herrera
alvherre@commandprompt.com
In reply to: Alvaro Herrera (#134)
#136Alvaro Herrera
alvherre@commandprompt.com
In reply to: Alvaro Herrera (#134)
#137Alvaro Herrera
alvherre@commandprompt.com
In reply to: Alvaro Herrera (#134)
#138Alvaro Herrera
alvherre@commandprompt.com
In reply to: Alvaro Herrera (#136)
#139Greg Stark
stark@enterprisedb.com
In reply to: Alvaro Herrera (#138)
#140Zdenek Kotala
Zdenek.Kotala@Sun.COM
In reply to: Alvaro Herrera (#134)
#141Jonah H. Harris
jonah.harris@gmail.com
In reply to: Zdenek Kotala (#140)
#142Joshua D. Drake
jd@commandprompt.com
In reply to: Jonah H. Harris (#141)
#143Alvaro Herrera
alvherre@commandprompt.com
In reply to: Zdenek Kotala (#140)
#144Greg Stark
stark@enterprisedb.com
In reply to: Joshua D. Drake (#142)
#145Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jonah H. Harris (#141)
#146Jonah H. Harris
jonah.harris@gmail.com
In reply to: Tom Lane (#145)
#147Alvaro Herrera
alvherre@commandprompt.com
In reply to: Greg Stark (#139)
#148Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jonah H. Harris (#146)
#149Zdenek Kotala
Zdenek.Kotala@Sun.COM
In reply to: Jonah H. Harris (#141)
#150Jonah H. Harris
jonah.harris@gmail.com
In reply to: Tom Lane (#148)
#151Greg Stark
stark@enterprisedb.com
In reply to: Alvaro Herrera (#143)
#152Zdenek Kotala
Zdenek.Kotala@Sun.COM
In reply to: Alvaro Herrera (#143)
#153Tom Lane
tgl@sss.pgh.pa.us
In reply to: Zdenek Kotala (#152)
#154Zdenek Kotala
Zdenek.Kotala@Sun.COM
In reply to: Tom Lane (#153)
#155Alvaro Herrera
alvherre@commandprompt.com
In reply to: Greg Stark (#144)
#156Jonah H. Harris
jonah.harris@gmail.com
In reply to: Alvaro Herrera (#155)
#157Alvaro Herrera
alvherre@commandprompt.com
In reply to: Alvaro Herrera (#147)
In reply to: Greg Stark (#151)
#159Florian Weimer
fweimer@bfk.de
In reply to: Greg Stark (#100)
#160Alvaro Herrera
alvherre@commandprompt.com
In reply to: Alvaro Herrera (#157)
#161Paul Schlie
schlie@comcast.net
In reply to: Alvaro Herrera (#160)
In reply to: Alvaro Herrera (#122)
#163Tom Lane
tgl@sss.pgh.pa.us
In reply to: Martijn van Oosterhout (#162)
In reply to: Tom Lane (#163)
#165Greg Stark
greg.stark@enterprisedb.com
In reply to: Tom Lane (#163)
#166Alvaro Herrera
alvherre@commandprompt.com
In reply to: Greg Stark (#165)
#167Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#166)
#168Greg Stark
stark@enterprisedb.com
In reply to: Tom Lane (#167)
#169Alvaro Herrera
alvherre@commandprompt.com
In reply to: Greg Stark (#168)
#170Greg Stark
stark@enterprisedb.com
In reply to: Alvaro Herrera (#169)
In reply to: Greg Stark (#170)
#172Alvaro Herrera
alvherre@commandprompt.com
In reply to: Martijn van Oosterhout (#171)
#173Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#172)
#174Alvaro Herrera
alvherre@commandprompt.com
In reply to: Tom Lane (#173)
#175Alvaro Herrera
alvherre@commandprompt.com
In reply to: Alvaro Herrera (#174)
In reply to: Alvaro Herrera (#172)
#177Alvaro Herrera
alvherre@commandprompt.com
In reply to: Martijn van Oosterhout (#176)
#178Greg Stark
stark@enterprisedb.com
In reply to: Alvaro Herrera (#177)
#179Alvaro Herrera
alvherre@commandprompt.com
In reply to: Greg Stark (#178)
#180Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#179)
#181Aidan Van Dyk
aidan@highrise.ca
In reply to: Alvaro Herrera (#179)
#182Alvaro Herrera
alvherre@commandprompt.com
In reply to: Tom Lane (#180)
#183Alvaro Herrera
alvherre@commandprompt.com
In reply to: Aidan Van Dyk (#181)
#184Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#182)
#185Alvaro Herrera
alvherre@commandprompt.com
In reply to: Alvaro Herrera (#182)
#186Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Alvaro Herrera (#183)
#187Alvaro Herrera
alvherre@commandprompt.com
In reply to: Heikki Linnakangas (#186)
#188Aidan Van Dyk
aidan@highrise.ca
In reply to: Tom Lane (#184)
In reply to: Tom Lane (#180)
#190Tom Lane
tgl@sss.pgh.pa.us
In reply to: Martijn van Oosterhout (#189)
#191Alvaro Herrera
alvherre@commandprompt.com
In reply to: Tom Lane (#190)
In reply to: Alvaro Herrera (#191)
#193Tom Lane
tgl@sss.pgh.pa.us
In reply to: Martijn van Oosterhout (#192)
In reply to: Tom Lane (#193)
#195Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Martijn van Oosterhout (#194)
#196Greg Stark
greg.stark@enterprisedb.com
In reply to: Heikki Linnakangas (#195)
#197Aidan Van Dyk
aidan@highrise.ca
In reply to: Greg Stark (#196)
In reply to: Aidan Van Dyk (#197)
#199Aidan Van Dyk
aidan@highrise.ca
In reply to: Martijn van Oosterhout (#198)
#200Greg Stark
stark@enterprisedb.com
In reply to: Aidan Van Dyk (#197)
#201Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Greg Stark (#200)
#202Matthew T. O'Connor
matthew@zeut.net
In reply to: Aidan Van Dyk (#197)
#203Aidan Van Dyk
aidan@highrise.ca
In reply to: Matthew T. O'Connor (#202)
#204Paul Schlie
schlie@comcast.net
In reply to: Aidan Van Dyk (#203)
#205Greg Stark
stark@enterprisedb.com
In reply to: Paul Schlie (#204)
#206Paul Schlie
schlie@comcast.net
In reply to: Greg Stark (#205)
#207Alvaro Herrera
alvherre@commandprompt.com
In reply to: Aidan Van Dyk (#199)
#208Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Alvaro Herrera (#207)
#209Aidan Van Dyk
aidan@highrise.ca
In reply to: Alvaro Herrera (#207)
#210Tom Lane
tgl@sss.pgh.pa.us
In reply to: Heikki Linnakangas (#208)
#211Tom Lane
tgl@sss.pgh.pa.us
In reply to: Aidan Van Dyk (#209)
#212Aidan Van Dyk
aidan@highrise.ca
In reply to: Tom Lane (#211)
#213Joshua D. Drake
jd@commandprompt.com
In reply to: Aidan Van Dyk (#212)
#214Jaime Casanova
jcasanov@systemguards.com.ec
In reply to: Alvaro Herrera (#179)
#215Alvaro Herrera
alvherre@commandprompt.com
In reply to: Jaime Casanova (#214)
#216Josh Berkus
josh@agliodbs.com
In reply to: Alvaro Herrera (#215)
#217Jonah H. Harris
jonah.harris@gmail.com
In reply to: Josh Berkus (#216)
#218Alvaro Herrera
alvherre@commandprompt.com
In reply to: Jonah H. Harris (#217)
#219Jonah H. Harris
jonah.harris@gmail.com
In reply to: Alvaro Herrera (#218)
#220Bruce Momjian
bruce@momjian.us
In reply to: Jonah H. Harris (#219)
#221Jonah H. Harris
jonah.harris@gmail.com
In reply to: Bruce Momjian (#220)
#222Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jonah H. Harris (#221)
#223Alvaro Herrera
alvherre@commandprompt.com
In reply to: Jonah H. Harris (#221)
#224Jonah H. Harris
jonah.harris@gmail.com
In reply to: Tom Lane (#222)
#225Alvaro Herrera
alvherre@commandprompt.com
In reply to: Jonah H. Harris (#224)
#226Jonah H. Harris
jonah.harris@gmail.com
In reply to: Alvaro Herrera (#225)
#227Alvaro Herrera
alvherre@commandprompt.com
In reply to: Jonah H. Harris (#226)
#228Greg Stark
stark@enterprisedb.com
In reply to: Alvaro Herrera (#225)
#229Joshua D. Drake
jd@commandprompt.com
In reply to: Bruce Momjian (#220)
#230Alvaro Herrera
alvherre@commandprompt.com
In reply to: Joshua D. Drake (#229)
#231Alvaro Herrera
alvherre@commandprompt.com
In reply to: Greg Stark (#228)
#232Joshua D. Drake
jd@commandprompt.com
In reply to: Alvaro Herrera (#230)
#233Jonah H. Harris
jonah.harris@gmail.com
In reply to: Alvaro Herrera (#231)
#234Alvaro Herrera
alvherre@commandprompt.com
In reply to: Joshua D. Drake (#232)
#235Simon Riggs
simon@2ndQuadrant.com
In reply to: Alvaro Herrera (#122)
#236Joshua D. Drake
jd@commandprompt.com
In reply to: Simon Riggs (#235)
#237Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Simon Riggs (#235)
#238Simon Riggs
simon@2ndQuadrant.com
In reply to: Heikki Linnakangas (#237)
#239Aidan Van Dyk
aidan@highrise.ca
In reply to: Simon Riggs (#238)
#240Simon Riggs
simon@2ndQuadrant.com
In reply to: Aidan Van Dyk (#239)
#241Tom Lane
tgl@sss.pgh.pa.us
In reply to: Simon Riggs (#240)
#242Simon Riggs
simon@2ndQuadrant.com
In reply to: Tom Lane (#241)
#243Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Simon Riggs (#242)
#244Simon Riggs
simon@2ndQuadrant.com
In reply to: Heikki Linnakangas (#243)
#245Bruce Momjian
bruce@momjian.us
In reply to: Simon Riggs (#244)
#246Simon Riggs
simon@2ndQuadrant.com
In reply to: Bruce Momjian (#245)
#247Bruce Momjian
bruce@momjian.us
In reply to: Simon Riggs (#246)
#248Robert Haas
robertmhaas@gmail.com
In reply to: Heikki Linnakangas (#237)
#249Bruce Momjian
bruce@momjian.us
In reply to: Robert Haas (#248)
#250Simon Riggs
simon@2ndQuadrant.com
In reply to: Bruce Momjian (#247)
#251Bruce Momjian
bruce@momjian.us
In reply to: Simon Riggs (#250)
#252Bruce Momjian
bruce@momjian.us
In reply to: Bruce Momjian (#251)
#253Simon Riggs
simon@2ndQuadrant.com
In reply to: Bruce Momjian (#251)
#254Simon Riggs
simon@2ndQuadrant.com
In reply to: Bruce Momjian (#252)
#255Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Bruce Momjian (#247)
#256marcin mank
marcin.mank@gmail.com
In reply to: Heikki Linnakangas (#237)
#257Andres Freund
andres@anarazel.de
In reply to: marcin mank (#256)
#258Aidan Van Dyk
aidan@highrise.ca
In reply to: Andres Freund (#257)
#259Robert Haas
robertmhaas@gmail.com
In reply to: Heikki Linnakangas (#255)
#260Andres Freund
andres@anarazel.de
In reply to: Aidan Van Dyk (#258)
#261Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Robert Haas (#259)
#262Simon Riggs
simon@2ndQuadrant.com
In reply to: Heikki Linnakangas (#255)
#263Robert Haas
robertmhaas@gmail.com
In reply to: Heikki Linnakangas (#261)
#264Florian Weimer
fweimer@bfk.de
In reply to: Simon Riggs (#262)
#265Tom Lane
tgl@sss.pgh.pa.us
In reply to: Florian Weimer (#264)
#266Simon Riggs
simon@2ndQuadrant.com
In reply to: Heikki Linnakangas (#261)
#267Tom Lane
tgl@sss.pgh.pa.us
In reply to: Simon Riggs (#266)
#268Robert Haas
robertmhaas@gmail.com
In reply to: Simon Riggs (#266)
#269Joshua D. Drake
jd@commandprompt.com
In reply to: Bruce Momjian (#247)
#270Simon Riggs
simon@2ndQuadrant.com
In reply to: Tom Lane (#267)
#271Joshua D. Drake
jd@commandprompt.com
In reply to: Tom Lane (#267)
#272Bruce Momjian
bruce@momjian.us
In reply to: Simon Riggs (#262)
#273Robert Haas
robertmhaas@gmail.com
In reply to: Joshua D. Drake (#271)
#274Simon Riggs
simon@2ndQuadrant.com
In reply to: Bruce Momjian (#272)
#275Joshua D. Drake
jd@commandprompt.com
In reply to: Robert Haas (#273)
#276Tom Lane
tgl@sss.pgh.pa.us
In reply to: Bruce Momjian (#272)
#277Tom Lane
tgl@sss.pgh.pa.us
In reply to: Simon Riggs (#274)
#278Aidan Van Dyk
aidan@highrise.ca
In reply to: Tom Lane (#277)
#279Greg Stark
gsstark@mit.edu
In reply to: Tom Lane (#276)
#280Tom Lane
tgl@sss.pgh.pa.us
In reply to: Greg Stark (#279)
#281Josh Berkus
josh@agliodbs.com
In reply to: Tom Lane (#276)
#282Kevin Grittner
Kevin.Grittner@wicourts.gov
In reply to: Josh Berkus (#281)
#283Greg Stark
gsstark@mit.edu
In reply to: Josh Berkus (#281)
#284Robert Haas
robertmhaas@gmail.com
In reply to: Aidan Van Dyk (#278)
#285Greg Stark
gsstark@mit.edu
In reply to: Robert Haas (#284)
#286Greg Stark
gsstark@mit.edu
In reply to: Greg Stark (#285)
#287Robert Haas
robertmhaas@gmail.com
In reply to: Greg Stark (#285)
#288Tom Lane
tgl@sss.pgh.pa.us
In reply to: Robert Haas (#287)
#289Andrew Dunstan
andrew@dunslane.net
In reply to: Tom Lane (#288)
#290Bruce Momjian
bruce@momjian.us
In reply to: Andrew Dunstan (#289)
#291Bruce Momjian
bruce@momjian.us
In reply to: Tom Lane (#276)
#292Tom Lane
tgl@sss.pgh.pa.us
In reply to: Bruce Momjian (#291)
#293Richard Huxton
dev@archonet.com
In reply to: Bruce Momjian (#291)
#294Jim Nasby
decibel@decibel.org
In reply to: Tom Lane (#277)
#295Bruce Momjian
bruce@momjian.us
In reply to: Tom Lane (#292)
#296Greg Stark
gsstark@mit.edu
In reply to: Richard Huxton (#293)
#297Jim Nasby
decibel@decibel.org
In reply to: Kevin Grittner (#282)
#298Greg Stark
gsstark@mit.edu
In reply to: Jim Nasby (#294)
#299Bruce Momjian
bruce@momjian.us
In reply to: Greg Stark (#279)
#300Bruce Momjian
bruce@momjian.us
In reply to: Greg Stark (#298)
#301Richard Huxton
dev@archonet.com
In reply to: Greg Stark (#296)
#302Tom Lane
tgl@sss.pgh.pa.us
In reply to: Bruce Momjian (#299)
#303Bruce Momjian
bruce@momjian.us
In reply to: Tom Lane (#302)
#304Tom Lane
tgl@sss.pgh.pa.us
In reply to: Richard Huxton (#301)
#305Tom Lane
tgl@sss.pgh.pa.us
In reply to: Bruce Momjian (#303)
#306Greg Stark
gsstark@mit.edu
In reply to: Tom Lane (#302)
#307Jim Nasby
decibel@decibel.org
In reply to: Greg Stark (#296)
#308Tom Lane
tgl@sss.pgh.pa.us
In reply to: Greg Stark (#306)
#309Greg Stark
gsstark@mit.edu
In reply to: Tom Lane (#308)
#310Robert Haas
robertmhaas@gmail.com
In reply to: Greg Stark (#298)
#311Bruce Momjian
bruce@momjian.us
In reply to: Robert Haas (#310)
#312Robert Haas
robertmhaas@gmail.com
In reply to: Bruce Momjian (#311)
#313Bruce Momjian
bruce@momjian.us
In reply to: Robert Haas (#312)
#314Greg Smith
greg@2ndquadrant.com
In reply to: Robert Haas (#312)
#315Robert Haas
robertmhaas@gmail.com
In reply to: Bruce Momjian (#313)
#316Robert Haas
robertmhaas@gmail.com
In reply to: Greg Smith (#314)
#317Aidan Van Dyk
aidan@highrise.ca
In reply to: Greg Stark (#309)
#318Bruce Momjian
bruce@momjian.us
In reply to: Robert Haas (#315)
#319David Fetter
david@fetter.org
In reply to: Bruce Momjian (#313)
#320Dimitri Fontaine
dfontaine@hi-media.com
In reply to: Bruce Momjian (#313)
#321Greg Stark
gsstark@mit.edu
In reply to: Dimitri Fontaine (#320)
#322Bruce Momjian
bruce@momjian.us
In reply to: David Fetter (#319)
#323Dimitri Fontaine
dfontaine@hi-media.com
In reply to: Greg Stark (#321)
#324Robert Haas
robertmhaas@gmail.com
In reply to: Bruce Momjian (#318)
#325Simon Riggs
simon@2ndQuadrant.com
In reply to: Robert Haas (#324)
#326Robert Haas
robertmhaas@gmail.com
In reply to: Simon Riggs (#325)
#327Greg Smith
greg@2ndquadrant.com
In reply to: Robert Haas (#324)
#328Robert Haas
robertmhaas@gmail.com
In reply to: Greg Smith (#327)
#329Peter Eisentraut
peter_e@gmx.net
In reply to: Greg Stark (#283)
#330Peter Eisentraut
peter_e@gmx.net
In reply to: Tom Lane (#302)
#331Greg Smith
greg@2ndquadrant.com
In reply to: Robert Haas (#328)
#332Robert Haas
robertmhaas@gmail.com
In reply to: Greg Smith (#331)
#333Greg Smith
greg@2ndquadrant.com
In reply to: Robert Haas (#332)
#334Robert Haas
robertmhaas@gmail.com
In reply to: Greg Smith (#333)
#335Greg Stark
gsstark@mit.edu
In reply to: Robert Haas (#328)
#336Tom Lane
tgl@sss.pgh.pa.us
In reply to: Greg Stark (#335)
#337Robert Haas
robertmhaas@gmail.com
In reply to: Tom Lane (#336)
#338Jonah H. Harris
jonah.harris@gmail.com
In reply to: Joshua D. Drake (#275)
#339Jim Nasby
decibel@decibel.org
In reply to: Jonah H. Harris (#338)
#340Simon Riggs
simon@2ndQuadrant.com
In reply to: Jim Nasby (#339)
In reply to: Kevin Grittner (#282)
#342Greg Stark
gsstark@mit.edu
In reply to: Simon Riggs (#340)
#343Bruce Momjian
bruce@momjian.us
In reply to: Jim Nasby (#339)
#344Simon Riggs
simon@2ndQuadrant.com
In reply to: Bruce Momjian (#343)
#345Alvaro Herrera
alvherre@commandprompt.com
In reply to: Bruce Momjian (#343)
#346Bruce Momjian
bruce@momjian.us
In reply to: Simon Riggs (#344)
#347Simon Riggs
simon@2ndQuadrant.com
In reply to: Bruce Momjian (#346)
#348Simon Riggs
simon@2ndQuadrant.com
In reply to: Alvaro Herrera (#345)
#349Heikki Linnakangas
heikki.linnakangas@enterprisedb.com
In reply to: Simon Riggs (#348)
#350Greg Stark
gsstark@mit.edu
In reply to: Simon Riggs (#347)
#351Greg Stark
gsstark@mit.edu
In reply to: Greg Stark (#350)
#352Alvaro Herrera
alvherre@commandprompt.com
In reply to: Greg Stark (#351)
#353Alvaro Herrera
alvherre@commandprompt.com
In reply to: Heikki Linnakangas (#349)
#354Robert Haas
robertmhaas@gmail.com
In reply to: Alvaro Herrera (#353)
#355Tom Lane
tgl@sss.pgh.pa.us
In reply to: Simon Riggs (#344)
#356Tom Lane
tgl@sss.pgh.pa.us
In reply to: Greg Stark (#351)
#357Simon Riggs
simon@2ndQuadrant.com
In reply to: Tom Lane (#355)
#358Tom Lane
tgl@sss.pgh.pa.us
In reply to: Robert Haas (#354)
#359Simon Riggs
simon@2ndQuadrant.com
In reply to: Greg Stark (#350)
#360Bruce Momjian
bruce@momjian.us
In reply to: Robert Haas (#354)
#361Robert Haas
robertmhaas@gmail.com
In reply to: Bruce Momjian (#360)
#362Alvaro Herrera
alvherre@commandprompt.com
In reply to: Massa, Harald Armin (#341)
#363Bruce Momjian
bruce@momjian.us
In reply to: Robert Haas (#361)
#364Chuck McDevitt
cmcdevitt@greenplum.com
In reply to: Bruce Momjian (#363)
#365Simon Riggs
simon@2ndQuadrant.com
In reply to: Chuck McDevitt (#364)
In reply to: Alvaro Herrera (#362)
#367Greg Stark
stark@mit.edu
In reply to: Tom Lane (#356)
#368Greg Stark
gsstark@mit.edu
In reply to: Chuck McDevitt (#364)