Test server strategies

My use of the word "bit" in the phrase "reading multiple drives for every bit of data" was a poor choice of words. I meant "bit" in the sense of "little piece" not as in one binary digit. If you substitute the word "byte" the sentence gives both the intended impression and is technically correct also.

John

I doubt if the TCP-C benchmarking question involves whether IBM or Oracle or any other participant would "give up a drop of performance" in a benchmark situation. My strong suspicion is that the benchmark rules require a RAID 5 disk configuration. Although I haven't been able to verify this conclusively from the tcp.org site, check out page 148 of this document: http://www.tpc.org/tpc_app/spec/TPC-App_V1.3.pdf
--my reading is that RAID 5 is a requirement of the benchmark. Historically, RAID 5 no doubt made sense, and now, when it's of much less practical use, they may just want to keep things consistent with older test results (back in the day of 2GB drives, you needed RAID 5 to get a big array).

But, just looking at raw performance, a simple measure of disk subsystem performance (using say IOMeter's File Server simulator) with RAID 1 vs. RAID 5 will definitely show that the disk subsystem performs substantially better with RAID 1 (particularly a fair number of queued IOs and for a high read/write ratio--and a library system is a fairly extreme case of reads to writes). Note that the raw performance of the disk subsystem is independent of how the OS interacts with the drives vs. how the DB does: there's no getting around the fact that a RAID 5 system will have higher seek times (even with good command queuing, positioning heads on 1 drive will beat positioning them on 3--although it won't be a simple 1 to 3 ratio, of course) and the XORs to recreate the actual data read from the RAID 5 array can't be free, no matter how efficient they are.

I'll be interested to see your test results, if you have time to do them (without using something like IOMeter, it may be moderately difficult to simulate an appropriate load at which the performance would really diverge--but more on that below), but the only reason for using RAID 5 over RAID 1 or RAID 0 + 1 is that it's more efficient from a cost-per-GB point of view than RAID 1 or 0+1. (The fact that you can have a hot spare drive configured for either setup means the fault tolerance is equivalent: either survives a single-drive failure.) This cost-per-GB fact was important in years past, perhaps, and for truly huge databases it might still be a consideration. But, for a library DB (all but the absolute biggest ones I've encountered in 17 years in the business are less than the size of one 73GB drive--most a LOT less), with current hard drive technology, the cost difference of more spindles (taking system cost as a whole into consideration) is insignificant. There's also the issue that RAID 5 performance varies tremendously from RAID controller to controller--but since RAID 1 is so much simpler, if you don't happen to have the best RAID card available, you've got a better chance of getting good performance from whatever you've got if you take the XORs and more complex head-positioning out of the mix by not using RAID 5.

I certainly wouldn't recommend anyone ever assume that they're getting the best performance from a DB server if they host the DB on a RAID 5 setup. We have a test server at one of our clients' sites, it has copies of the live DB on both RAID 5 and just individual disks (no RAID--which is slower for reads than RAID 1, but faster for writes). The individual disks are definitely faster (and not just a little faster--I mean at least 3 to 5 times faster). The RAID 5 performance is acceptable (especially at low load levels), but we would never use that arrangement for a production system. (We have the one copy on the RAID 5 array because there's extra space there in addition to the backups we keep on the array.) It is true that the RAID 5 performance of the Adaptec controller in that box isn't stellar.

I agree with the "try it yourself" philosophy too, and my actual experience with the same DB server hitting different DBs on different RAID arrangements has been crystal clear: RAID 1 or 0 + 1 gives substantially better performance. At low loads you might not be able to see the difference, but in the case of Evergreen's keyword searching, for instance, which is highly random in terms of disk accesses, my guess is that doing cache-cold searches for common terms would immediately show a perceptible (not just a measurable) difference.

My real-world experience, plus the fact that disk space (even employing the fastest available drives) is a relatively inexpensive part of a system, leads me to conclude that there's no reason to use RAID 5 to store DB data. And if I were spec'ing a system now, rather than a RAID 5 array for backups, I'd consider using mirrored big and inexpensive SATA drives for that too. For databases of the size typically encountered in the library world, RAID 5 has no practical benefits.

I'm sure Mike's right about RAID 5 vs. RAID 10 for particular controllers and benchmarks. I maintain that RAID 5 is essentially rendered obsolete by available disk capacities (when talking about the size of databases involved in the library world--as Mike suggests).

RE RAID 1+0 vs. 0+1, though, I want to suggest that the naming may not match the patterns as exactly as Mike implies. I've not encountered the type of combination of stripes and mirroring that Mike refers to in which a one drive failure kills the whole array, but I have encountered different terms for the same conceptual beast among the different vendors. And also different ways of setting up the combination of striping and mirroring (which can make the terminology all that much more confusing--especially if you have in mind that one way provides fault-tolerance and one does not).

Depending upon the controller manufacturer's terminology, RAID levels 0 + 1, 1 + 0, and 10 are somewhat interchangable. And, although the way the setup actually happens suggests that it should be called one thing or the other, the end result is the same--and is fault-tolerant. (3Ware and Adaptec, for instance, use opposite setup approaches, as I recall--befuddled me no end, initially because I was concerned about just the sort of issue Mike raised with regard to fault-tolerance). It turns out that the end result is the same: striped and mirrored with fault-tolerance of a single-disk failure.

With one of these vendor's cards, you have to set up your 4 drives (the minimum for either of these approaches) as RAID 0 (2 sets of 2 striped drives each) and then mirror the one striped pair onto the other striped pair: disks 0 & 1 are a striped pair; 2 & 3 are a striped pair; and then 2 & 3 are set up to mirror 0 & 1. Most logically, IMO, this might be referred to as 0 + 1: RAID 0 [striped] followed by the addition of RAID 1 [mirrored]. With the other manufacturer, you set up 2 mirrored pairs and then stripe across the 2 mirrored pairs: 0 & 1 are mirrors; 2 & 3 are mirrors; and 2 & 3 stripe 0 & 1. Now, for naming, I'd choose 1 + 0 for this. But, the thing is, even though the sequence is different between 3Ware and Adaptec, the terminology isn't consistent with what my brain thinks it should be: so they may call it the same thing even though the levels get applied in a different order. (Is the terminology a FIFO: first applied level listed first; or a FILO: last applied level first? And if you have that straight, doesn't
it seem that you'd attend to which order you apply the levels? Well, apparently not.)

The important point is that whatever your RAID controller manufacturer calls this RAID setup combining striping and mirroring (which is generally as good as you can get for a DB, in terms of general-purpose performance--the advantage of the striping being relatively minimal in most cases), you do not lose the whole array with a 1-drive failure. Each drive in such a setup contains a copy of the stripes for half the array. Failure of one drive means you have to rely only on the other half of the mirror for that set of stripes (until your hot spare comes on line--hot spares are generally worth the cost, I think). So, regardless of whether you apply the stripes and then mirror, or the other way around (which would possibly imply the different naming mentioned by Mike), you do have fault tolerance for single-drive losses.

I'm not disagreeing with Mike that there are non-fault-tolerant ways to set things up, but, in general, anything you can do via a RAID controller in terms of combining drives in any kind of configuration that includes a 1 (mirroring): 1 + 0; 0 + 1; or 10, will, based on my experience with 3 different manufacturer's controllers, give you the ability to survive a single disk failure--regardless of what they choose to call it and regardless of what order the setup happens in.

If it's a software RAID setup, I don't have enough experience to offer an opinion. Given Mike's comments, it sounds like some care would be in order in that case. But if you're configuring a server for a serious production system, you wouldn't want software RAID for anything except perhaps the OS and application installations anyway (RAID 1). And I would not recommend using any type of RAID 0 (striping) in a software RAID situation anyway: the only way it's any good is in hardware with mirroring. storagereview.com is an excellent site for many things disk related. Their discussion of RAID 0 is enlightening (they basically show that the received wisdom about striping is not correct).

Happy RAIDing.