Arguments Pro/Contra Software Raid
Hi,
I've just had some discussion with colleagues regarding the usage of
hardware or software raid 1/10 for our linux based database servers.
I myself can't see much reason to spend $500 on high end controller
cards for a simple Raid 1.
Any arguments pro or contra would be desirable.
From my experience and what I've read here:
+ Hardware Raids might be a bit easier to manage, if you never spend a
few hours to learn Software Raid Tools.
+ There are situations in which Software Raids are faster, as CPU power
has advanced dramatically in the last years and even high end controller
cards cannot keep up with that.
+ Using SATA drives is always a bit of risk, as some drives are lying
about whether they are caching or not.
+ Using hardware controllers, the array becomes locked to a particular
vendor. You can't switch controller vendors as the array meta
information is stored proprietary. In case the Raid is broken to a level
the controller can't recover automatically this might complicate manual
recovery by specialists.
+ Even battery backed controllers can't guarantee that data written to
the drives is consistent after a power outage, neither that the drive
does not corrupt something during the involuntary shutdown / power
irregularities. (This is theoretical as any server will be UPS backed)
--
Regards,
Hannes Dorbath
Hi Hannes,
Hannes Dorbath a �crit :
Hi,
I've just had some discussion with colleagues regarding the usage of
hardware or software raid 1/10 for our linux based database servers.I myself can't see much reason to spend $500 on high end controller
cards for a simple Raid 1.
Naa, you can find ATA &| SATA ctrlrs for about EUR30 !
Any arguments pro or contra would be desirable.
From my experience and what I've read here:
+ Hardware Raids might be a bit easier to manage, if you never spend a
few hours to learn Software Raid Tools.
I'd the same (mostly as you still have to punch a command line for
most of the controlers)
+ There are situations in which Software Raids are faster, as CPU power
has advanced dramatically in the last years and even high end controller
cards cannot keep up with that.
Definitely NOT, however if your server doen't have a heavy load, the
software overload can't be noticed (essentially cache managing and
syncing)
For bi-core CPUs, it might be true
+ Using SATA drives is always a bit of risk, as some drives are lying
about whether they are caching or not.
?? Do you intend to use your server without a UPS ??
+ Using hardware controllers, the array becomes locked to a particular
vendor. You can't switch controller vendors as the array meta
information is stored proprietary. In case the Raid is broken to a level
the controller can't recover automatically this might complicate manual
recovery by specialists.
?? Do you intend not to make backups ??
+ Even battery backed controllers can't guarantee that data written to
the drives is consistent after a power outage, neither that the drive
does not corrupt something during the involuntary shutdown / power
irregularities. (This is theoretical as any server will be UPS backed)
RAID's "laws":
1- RAID prevents you from loosing data on healthy disks, not from faulty
disks,
1b- So format and reformat your RAID disks (whatever SCSI, ATA, SATA)
several times, with destructive tests (see "-c -c" option from
the mke2fs man) - It will ensure that disks are safe, and also
make a kind of burn test (might turn to... days of formating!),
2- RAID doesn't prevent you from power suply brokeage or electricity
breakdown, so use a (LARGE) UPS,
2b- LARGE UPS because HDs are the components that have the higher power
consomption (a 700VA UPS gives me about 10-12 minutes on a machine
with a XP2200+, 1GB RAM and a 40GB HD, however this fall to......
less than 25 secondes with seven HDs ! all ATA),
2c- Use server box with redudancy power supplies,
3- As for any sensitive data, make regular backups or you'll be as
sitting duck.
Some hardware ctrlrs are able to avoid the loss of a disk if you turn
to have some faulty sectors (by relocating internally them); software
RAID doesn't as sectors *must* be @ the same (linear) addresses.
BUT a hardware controler is about EUR2000 and a (ATA/SATA) 500GB HD
is ~ EUR350.
That means you have to consider:
* The server disponibility (time to change a power supply if no
redudancies, time to exchange a not hotswap HD... In fact, how much
down time you can "afford"),
* The volume of the data (from which depends the size of the backup
device),
* The backup device you'll use (tape or other HDs),
* The load of the server (and the number of simultaneous users =>
Soft|Hard, ATA/SATA|SCSI...),
* The money you can spend in such a server
* And most important, the color of your boss' tie the day you'll
take the decision.
Hope it will help you
Jean-Yves
On 09.05.2006 12:10, Jean-Yves F. Barbier wrote:
Naa, you can find ATA &| SATA ctrlrs for about EUR30 !
Sure, just for my colleagues Raid Controller = IPC Vortex, which resides
in that price range.
For bi-core CPUs, it might be true
I've got that from pgsql.performance for multi-way opteron setups.
?? Do you intend to use your server without a UPS ??
Sure there will be an UPS. I'm just trying to nail down the differences
between soft- and hardware raid, regardless if they matter in the end :)
?? Do you intend not to make backups ??
Sure we do backups, this all is more hypothetical thinking..
Hope it will help you
It has, thanks.
--
Regards,
Hannes Dorbath
-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160
Hannes Dorbath wrote:
Hi,
I've just had some discussion with colleagues regarding the usage of
hardware or software raid 1/10 for our linux based database servers.I myself can't see much reason to spend $500 on high end controller
cards for a simple Raid 1.Any arguments pro or contra would be desirable.
One pro and one con off the top of my head.
Hotplug. Depending on your platform, SATA may or may not be hotpluggable
(I know AHCI mode is the only one promising some kind of a hotplug,
which means ICH6+ and Silicon Image controllers last I heard). SCSI
isn't hotpluggable without the use of special hotplug backplanes and
disks. You lose that in software RAID, which effectively means you need
to shut the box down and do maintenance. Hassle.
CPU. It's cheap. Much cheaper than your average hardware RAID card. For
the 5-10% overhead usually imposed by software RAID, you can throw in a
faster CPU and never even notice it. Most cases aren't CPU-bound
anyways, or at least, most cases are I/O bound for the better part. This
does raise the question of I/O bandwidth your standard SATA or SCSI
controller comes with, though. If you're careful about that and handle
hotplug sufficiently, you're probably never going to notice you're not
running on metal.
Kind regards,
- --
Grega Bremec
gregab at p0f dot net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)
iD8DBQFEYHRAfu4IwuB3+XoRA9jqAJ9sS3RBJZEurvwUXGKrFMRZfYy9pQCggGHh
tLAy/YtHwKvhd3ekVDGFtWE=
=vlyC
-----END PGP SIGNATURE-----
On May 9, 2006, at 2:16 AM, Hannes Dorbath wrote:
Hi,
I've just had some discussion with colleagues regarding the usage
of hardware or software raid 1/10 for our linux based database
servers.I myself can't see much reason to spend $500 on high end controller
cards for a simple Raid 1.Any arguments pro or contra would be desirable.
From my experience and what I've read here:
+ Hardware Raids might be a bit easier to manage, if you never
spend a few hours to learn Software Raid Tools.+ There are situations in which Software Raids are faster, as CPU
power has advanced dramatically in the last years and even high end
controller cards cannot keep up with that.+ Using SATA drives is always a bit of risk, as some drives are
lying about whether they are caching or not.
Don't buy those drives. That's unrelated to whether you use hardware
or software RAID.
+ Using hardware controllers, the array becomes locked to a
particular vendor. You can't switch controller vendors as the array
meta information is stored proprietary. In case the Raid is broken
to a level the controller can't recover automatically this might
complicate manual recovery by specialists.
Yes. Fortunately we're using the RAID for database work, rather than
file
storage, so we can use all the nice postgresql features for backing up
and replicating the data elsewhere, which avoids most of this issue.
+ Even battery backed controllers can't guarantee that data written
to the drives is consistent after a power outage, neither that the
drive does not corrupt something during the involuntary shutdown /
power irregularities. (This is theoretical as any server will be
UPS backed)
fsync of WAL log.
If you have a battery backed writeback cache then you can get the
reliability
of fsyncing the WAL for every transaction, and the performance of not
needing
to hit the disk for every transaction.
Also, if you're not doing that you'll need to dedicate a pair of
spindles to the
WAL log if you want to get good performance, so that there'll be no
seeking
on the WAL. With a writeback cache you can put the WAL on the same
spindles
as the database and not lose much, if anything, in the way of
performance.
If that saves you the cost of two additional spindles, and the space
on your
drive shelf for them, you've just paid for a reasonably proced RAID
controller.
Given those advantages... I can't imagine speccing a large system
that didn't
have a battery-backed write-back cache in it. My dev systems mostly use
software RAID, if they use RAID at all. But my production boxes all
use SATA
RAID (and I tell my customers to use controllers with BB cache,
whether it
be SCSI or SATA).
My usual workloads are write-heavy. If yours are read-heavy that will
move the sweet spot around significantly, and I can easily imagine that
for a read-heavy load software RAID might be a much better match.
Cheers,
Steve
On Tue, May 09, 2006 at 12:10:32 +0200,
"Jean-Yves F. Barbier" <7ukwn@free.fr> wrote:
Naa, you can find ATA &| SATA ctrlrs for about EUR30 !
But those are the ones that you would generally be better off not using.
Definitely NOT, however if your server doen't have a heavy load, the
software overload can't be noticed (essentially cache managing and
syncing)
It is fairly common for database machines to be IO, rather than CPU, bound
and so the CPU impact of software raid is low.
Some hardware ctrlrs are able to avoid the loss of a disk if you turn
to have some faulty sectors (by relocating internally them); software
RAID doesn't as sectors *must* be @ the same (linear) addresses.
That is not true. Software raid works just fine on drives that have internally
remapped sectors.
On Tue, 2006-05-09 at 04:16, Hannes Dorbath wrote:
Hi,
I've just had some discussion with colleagues regarding the usage of
hardware or software raid 1/10 for our linux based database servers.I myself can't see much reason to spend $500 on high end controller
cards for a simple Raid 1.Any arguments pro or contra would be desirable.
From my experience and what I've read here:
+ Hardware Raids might be a bit easier to manage, if you never spend a
few hours to learn Software Raid Tools.
Depends. Some hardware RAID cards aren't that easy to manage, and
sometimes, they won't let you do some things that software will. I've
run into situations where a RAID controller kicked out two perfectly
good drives from a RAID 5 and would NOT accept them back. All data
lost, and it would not be convinced to restart without formatting the
drives first. arg! With Linux kernel sw RAID, I've had a similar
problem pop up, and was able to make the RAID array take the drives
back. Of course, this means that software RAID relies on you not being
stupid, because it will let you do things that are dangerous / stupid.
I found the raidtools on linux to be well thought out and fairly easy to
use.
+ There are situations in which Software Raids are faster, as CPU power
has advanced dramatically in the last years and even high end controller
cards cannot keep up with that.
The only times I've found software RAID to be faster was against the
hybrid hardware / software type RAID cards (i.e. the cheapies) or OLDER
RAID cards, that have a 33 MHz coprocessor or such. Most modern RAID
controllers have coprocessors running at several hundred MHz or more,
and can compute parity and manage the array as fast as the attached I/O
can handle it.
The one thing a software RAID will never be able to match the hardware
RAID controller on is battery backed cache.
+ Using SATA drives is always a bit of risk, as some drives are lying
about whether they are caching or not.
This is true whether you are using hardware RAID or not. Turning off
drive caching seems to prevent the problem. However, with a RAID
controller, the caching can then be moved to the BBU cache, while with
software RAID no such option exists. Most SATA RAID controllers turn
off the drive cache automagically, like the escalades seem to do.
+ Using hardware controllers, the array becomes locked to a particular
vendor. You can't switch controller vendors as the array meta
information is stored proprietary. In case the Raid is broken to a level
the controller can't recover automatically this might complicate manual
recovery by specialists.
And not just a particular vendor, but likely a particular model and even
firmware revision. For this reason, and 24/7 server should have two
RAID controllers of the same brand running identical arrays, then have
them set up as a mirror across the controllers, assuming you have
controllers that can run cooperatively. This setup ensures that even if
one of your RAID controllers fails, you then have a fully operational
RAID array for as long as it takes to order and replace the bad
controller. And having a third as a spare in a cabinet somewhere is
cheap insurance as well.
+ Even battery backed controllers can't guarantee that data written to
the drives is consistent after a power outage, neither that the drive
does not corrupt something during the involuntary shutdown / power
irregularities. (This is theoretical as any server will be UPS backed)
This may be theoretically true, but all the battery backed cache units
I've used have brought the array up clean every time the power has been
lost to them. And a UPS is no insurance against loss of power.
Cascading power failures are not uncommon when things go wrong.
Now, here's my take on SW versus HW in general:
HW is the way to go for situations where a battery backed cache is
needed. Heavily written / updated databases are in this category.
Software RAID is a perfect match for databases with a low write to read
ratio, or where you won't be writing enough for the write performance to
be a big issue. Many data warehouses fall into this category. In this
case, a JBOD enclosure with a couple of dozen drives and software RAID
gives you plenty of storage for chicken feed. If the data is all
derived from outside sources, then you can turn on the write cache in
the drives and turn off fsync and it will be plenty fast, just not crash
safe.
Don't buy those drives. That's unrelated to whether you use hardware
or software RAID.
Sorry that is an extremely misleading statement. SATA RAID is perfectly
acceptable if you have a hardware raid controller with a battery backup
controller.
And dollar for dollar, SCSI will NOT be faster nor have the hard drive
capacity that you will get with SATA.
Sincerely,
Joshua D. Drake
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
On May 9, 2006, at 8:51 AM, Joshua D. Drake wrote:
("Using SATA drives is always a bit of risk, as some drives are lying
about whether they are caching or not.")
Don't buy those drives. That's unrelated to whether you use hardware
or software RAID.Sorry that is an extremely misleading statement. SATA RAID is
perfectly acceptable if you have a hardware raid controller with a
battery backup controller.
If the drive says it's hit the disk and it hasn't then the RAID
controller
will have flushed the data from its cache (or flagged it as correctly
written). At that point the only place the data is stored is in the non
battery backed cache on the drive itself. If something fails then you'll
have lost data.
You're not suggesting that a hardware RAID controller will protect
you against drives that lie about sync, are you?
And dollar for dollar, SCSI will NOT be faster nor have the hard
drive capacity that you will get with SATA.
Yup. That's why I use SATA RAID for all my databases.
Cheers,
Steve
On May 9, 2006, at 11:51 AM, Joshua D. Drake wrote:
Sorry that is an extremely misleading statement. SATA RAID is
perfectly acceptable if you have a hardware raid controller with a
battery backup controller.And dollar for dollar, SCSI will NOT be faster nor have the hard
drive capacity that you will get with SATA.
Does this hold true still under heavy concurrent-write loads? I'm
preparing yet another big DB server and if SATA is a better option,
I'm all (elephant) ears.
Attachments:
Vivek Khera <vivek@khera.org> writes:
On May 9, 2006, at 11:51 AM, Joshua D. Drake wrote:
And dollar for dollar, SCSI will NOT be faster nor have the hard
drive capacity that you will get with SATA.Does this hold true still under heavy concurrent-write loads? I'm
preparing yet another big DB server and if SATA is a better option,
I'm all (elephant) ears.
Correct me if I'm wrong, but I've never heard of a 15kRPM SATA drive.
-Doug
Vivek Khera wrote:
On May 9, 2006, at 11:51 AM, Joshua D. Drake wrote:
Sorry that is an extremely misleading statement. SATA RAID is
perfectly acceptable if you have a hardware raid controller with a
battery backup controller.And dollar for dollar, SCSI will NOT be faster nor have the hard drive
capacity that you will get with SATA.Does this hold true still under heavy concurrent-write loads? I'm
preparing yet another big DB server and if SATA is a better option, I'm
all (elephant) ears.
I didn't say better :). If you can afford, SCSI is the way to go.
However SATA with a good controller (I am fond of the LSI 150 series)
can provide some great performance.
I have not used, but have heard good things about Areca as well. Oh, and
make sure they are SATA-II drives.
Sincerely,
Joshua D. Drake
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
You're not suggesting that a hardware RAID controller will protect
you against drives that lie about sync, are you?
Of course not, but which drives lie about sync that are SATA? Or more
specifically SATA-II?
Sincerely,
Joshua D. Drake
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
On May 9, 2006, at 11:26 AM, Joshua D. Drake wrote:
You're not suggesting that a hardware RAID controller will protect
you against drives that lie about sync, are you?Of course not, but which drives lie about sync that are SATA? Or
more specifically SATA-II?
SATA-II, none that I'm aware of, but there's a long history of dodgy
behaviour designed to pump up benchmark results down in the
consumer drive space, and low end consumer space is where a
lot of SATA drives are. I wouldn't be surprised to see that beahviour
there still.
I was responding to the original posters assertion that drives lying
about sync were a reason not to buy SATA drives, by telling him
not to buy drives that lie about sync. You seem to have read this
as "don't buy SATA drives", which is not what I said and not what I
meant.
Cheers,
Steve
Douglas McNaught wrote:
Vivek Khera <vivek@khera.org> writes:
On May 9, 2006, at 11:51 AM, Joshua D. Drake wrote:
And dollar for dollar, SCSI will NOT be faster nor have the hard
drive capacity that you will get with SATA.Does this hold true still under heavy concurrent-write loads? I'm
preparing yet another big DB server and if SATA is a better option,
I'm all (elephant) ears.Correct me if I'm wrong, but I've never heard of a 15kRPM SATA drive.
Best I have seen is 10k but if I can put 4x the number of drives in the
array at the same cost... I don't need 15k.
Joshua D. Drake
-Doug
---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
On Tue, 2006-05-09 at 12:52, Steve Atkins wrote:
On May 9, 2006, at 8:51 AM, Joshua D. Drake wrote:
("Using SATA drives is always a bit of risk, as some drives are lying
about whether they are caching or not.")Don't buy those drives. That's unrelated to whether you use hardware
or software RAID.Sorry that is an extremely misleading statement. SATA RAID is
perfectly acceptable if you have a hardware raid controller with a
battery backup controller.If the drive says it's hit the disk and it hasn't then the RAID
controller
will have flushed the data from its cache (or flagged it as correctly
written). At that point the only place the data is stored is in the non
battery backed cache on the drive itself. If something fails then you'll
have lost data.You're not suggesting that a hardware RAID controller will protect
you against drives that lie about sync, are you?
Actually, in the case of the Escalades at least, the answer is yes.
Last year (maybe a bit more) someone was testing an IDE escalade
controller with drives that were known to lie, and it passed the power
plug pull test repeatedly. Apparently, the escalades tell the drives to
turn off their cache. While most all IDEs and a fair number of SATA
drives lie about cache fsyncing, they all seem to turn off the cache
when you ask.
And, since a hardware RAID controller with bbu cache has its own cache,
it's not like it really needs the one on the drives anyway.
Joshua D. Drake wrote:
Vivek Khera wrote:
On May 9, 2006, at 11:51 AM, Joshua D. Drake wrote:
Sorry that is an extremely misleading statement. SATA RAID is
perfectly acceptable if you have a hardware raid controller with a
battery backup controller.And dollar for dollar, SCSI will NOT be faster nor have the hard drive
capacity that you will get with SATA.Does this hold true still under heavy concurrent-write loads? I'm
preparing yet another big DB server and if SATA is a better option, I'm
all (elephant) ears.I didn't say better :). If you can afford, SCSI is the way to go.
However SATA with a good controller (I am fond of the LSI 150 series)
can provide some great performance.
Basically, you can get away with cheaper hardware, but it usually
doesn't have the reliability/performance of more expensive options.
You want an in-depth comparison of how a server disk drive is internally
better than a desktop drive:
http://www.seagate.com/content/docs/pdf/whitepaper/D2c_More_than_Interface_ATA_vs_SCSI_042003.pdf
--
Bruce Momjian http://candle.pha.pa.us
EnterpriseDB http://www.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Scott Marlowe wrote:
Actually, in the case of the Escalades at least, the answer is yes.
Last year (maybe a bit more) someone was testing an IDE escalade
controller with drives that were known to lie, and it passed the power
plug pull test repeatedly. Apparently, the escalades tell the drives to
turn off their cache. While most all IDEs and a fair number of SATA
drives lie about cache fsyncing, they all seem to turn off the cache
when you ask.And, since a hardware RAID controller with bbu cache has its own cache,
it's not like it really needs the one on the drives anyway.
You do if the controller thinks the data is already on the drives and
removes it from its cache.
--
Bruce Momjian http://candle.pha.pa.us
EnterpriseDB http://www.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
William Yu wrote:
We upgraded our disk system for our main data processing server earlier
this year. After pricing out all the components, basically we had the
choice of:LSI MegaRaid 320-2 w/ 1GB RAM+BBU + 8 15K 150GB SCSI
or
Areca 1124 w/ 1GB RAM+BBU + 24 7200RPM 250GB SATA
My mistake -- I keep doing calculations and they don't add up. So I
looked again on pricewatch and it turns out the actual comparison was
for 4 SCSI drives, not 8! ($600 for a 15K 145GB versus $90 for a 7200
250GB.) No wonder our decision seemed to much more decisive back then.
Import Notes
Reply to msg id not found: e3r8ip$11a$1@news.hub.org
On May 9, 2006, at 11:26 AM, Joshua D. Drake wrote:
Of course not, but which drives lie about sync that are SATA? Or
more specifically SATA-II?
I don't know the answer to this question, but have you seen this tool?
http://brad.livejournal.com/2116715.html
It attempts to experimentally determine if, with your operating
system version, controller, and hard disk, fsync() does as claimed.
Of course, experimentation can't prove the system is correct, but it
can sometimes prove the system is broken.
I say it's worth running on any new model of disk, any new
controller, or after the Linux kernel people rewrite everything (i.e.
on every point release).
I have to admit to hypocrisy, though...I'm running with systems that
other people ordered and installed, I doubt they were this thorough,
and I don't have identical hardware to run tests on. So no real way
to do this.
Regards,
Scott
--
Scott Lamb <http://www.slamb.org/>
