To Raid or not to Raid? Anyone here running Raid5?

[Shield]

So, here's my dilemma. Right now my Sage box consists of the following disk subsystem:

1. Hardware Serial ATA Raid Controller - 8 channel internal LSI Logic 8308ELP with 128 MB/cache. Supports RAID 0,1,5,50. Only thing to note here is write back cache is disabled as I don't have the battery backup unit (WBC cannot be enabled unless the BBU exists).

2. 2 x 320 GB SATA drives in a RAID1 mirror. This stores the Operating System, the "Movie Times" directory, and a backup of some important files from other computers.

3. 4 X 300 GB Sata drives "Just a bunch of disks.
E Drive = Sage recording drive only
F Drive = DVD1 - Ripped DVD Movies #1
G Drive = DVD2 + Kid's Movies
H Drive = TV Shows (archive) and WWII Documentaries

I have 2 analog tuners + a dual tuner HD Homerun, and I do use Dirmon2 + Showanalyzer to skip commercials just after the show(s) end.


So, of course I'm now running out of disk space on two drives - the E: and the G: drives. Too many ripped kids movies and saved TV shows!

Tell me which option you'd go with:

1. Buy 2 750GB drives (The WD units for $159 each on Newegg) and replace two of my drives that are nearing full capacity. No redundancy here though.
Cost=$318

2. Buy 4 750 GB drives and have a 2.25 terabyte Raid5 array (before formatting) and get rid of the other drives as I copy data from them.
Cost =$636

3. Buy 5 750 GB drives and have 3.00 TB's Raid5
Cost=$795

4. Buy 3 1TB drives and have 2.00 TB in a Raid5 array.
Cost=750

5. Buy 4 1TB drives and have 3.00 TB in a Raid5 array.
Cost=$1000

Keep in mind my server/client is in the basement so noise and heat are not a problem.

My main concern is by building a huge array that it'll get bogged down while recording 2 shows in HD, watching HD playback and possibly recording 2 SD (and commercial skipping) all at the same time.

Anyone doing all of this with a huge hardware RAID 5 solution? I'd love to hear from you!

Thanks,
Shawn

[KarylFStein]

Well, I can't comment on performance yet because I haven't finished my new SageTV server, but I have been researching RAID. I'm currently leaning toward a RAID 10 (4 disks) for all data or 2 RAID 1 (2 disks each), one for TV and the other for data, (videos, pictures, music, etc.) My issue with RAID 5 (again, I haven't seen it in action) is that writes have a big hit, (at least on the ICH8R, which I plan to use--your controller may be better), compared to RAID 1 or RAID 10.

My main goal is to be able to take a disk failure and still operate. I also want an easy way to dump important data (pictures, etc.) to an external disk for storage off site. (For the backups, I'll just have a drive with the same capacity of my data disks in a removable tray and set it to sync when inserted. Every week or so I'll rotate that drive with another one I keep off site.)

I like the idea of one big disk pool so all space is available to whatever needs it. However, I *think* 2 RAID 1 arrays will give better performance and it makes upgrades less expensive, (e.g. if I'm running out of TV space, I just need to buy 2 new disks instead of upgrading all 4 in the RAID 10).

So, no recommendations or experience, sorry. I've just heard that unless you have a good controller, RAID 5 writes are slower than RAID 1.

[valnar]

Shawn
I run RAID 5 on my Sage Server (also in the basement) with four 500GB drives and will soon to go six drives total in the array. I am not familiar with your LSI Logic 8308ELP controller, but I can tell you from experience that building a RAID 5 array would be totally pointless unless you can enable write-back. Before buying any drives, get the BBU first.

I have two analog Hauppauge WinTV PVR-250 PCI cards and a HDHomeRun. I have no problem recording and watching on my C2D 6600 server. See my sig.

Robert

[pez]

I'm going with 4x 1TB WD GP drives as i live in AZ and it gets kinda hot here in the summer. I'm hopping to remove as much heat from the system as possible.

[Shield]

Originally Posted by valnar

Shawn
I run RAID 5 on my Sage Server (also in the basement) with four 500GB drives and will soon to go six drives total in the array. I am not familiar with your LSI Logic 8308ELP controller, but I can tell you from experience that building a RAID 5 array would be totally pointless unless you can enable write-back. Before buying any drives, get the BBU first.

I have two analog Hauppauge WinTV PVR-250 PCI cards and a HDHomeRun. I have no problem recording and watching on my C2D 6600 server. See my sig.

Robert

Well, I somewhat disagree. Sure the write performance will take somewhat of a hit, but I'm most concerned with read performance anyway. I say this because I think I'm just going to use the Raid5 for pictures, DVD / HD video storage and let SAGE write to a separate single disk drive or a couple of them. I record so much all the time anyway and I'm not concerned if a drive fails that contains recorded TV. Besides, why have Showanalyzer and the sage tuners pound on the array all the time? 600 GB is enough time for recorded tv.

Now, if I'd have to go back and re-rip all my DVD's...

That's the time waster I'm trying to avoid. I have the controller using a RAID1 mirror right now and the write performance is more than acceptable.

[valnar]

Originally Posted by Shield

Well, I somewhat disagree. Sure the write performance will take somewhat of a hit, but I'm most concerned with read performance anyway.

I'm sorry, but you speak as somebody who does not have much RAID experience. Nobody runs RAID 5 in write-through mode. It's totally counterproductive.

-Robert

[Shield]

Originally Posted by valnar

I'm sorry, but you speak as somebody who does not have much RAID experience. Nobody runs RAID 5 in write-through mode. It's totally counterproductive.

-Robert

Wow, really? I've worked on servers professionally for 12 years and built hundreds of SCSI arrays in various flavors - the box I'm typing this on has 10 147GB SCA 10k drives in a Raid 50 setup. That's a pretty offensive statement considering I just mentioned that this box isn't going to do much writing of data once the files are copied over. All the write back does is allow the disk to complete a full write from the controller's cache instead of system memory - sure for sequential small cluster writes it greatly improves performance and the BBU will protect the system in the event of a power loss. No one is saying here that write-through is faster are they? If data is critical I'd rather have write-through turned on that write-back without a BBU - all I was saying is my controller won't even allow me to force it even though the box is plugged into a UPS. I'm not going for the highest performance for the disk anyway - it's in no way ever a bottleneck except when copying over gigs of DVD images. After all, we're not talking about multiple user I/O writes anyway - if all the Sage box is serving up read requests, then big deal.
I have setup multiple tb San's that are in production for our Federal Gov't. I was running raid 5 with Micropolis drives back in the 1990's.

Oh, and the reason I didn't opt for the BBU initially? It was almost the same price as the controller, and I knew it'd be doing mostly reads. Why am I wasting my breath ?

[valnar]

Wow. Try to help a guy....

Anyone doing all of this with a huge hardware RAID 5 solution? I'd love to hear from you!

Perhaps you should re-read your own post.

This:

That's a pretty offensive statement considering I just mentioned that this box isn't going to do much writing of data once the files are copied over.

conflicts with this:

My main concern is by building a huge array that it'll get bogged down while recording 2 shows in HD, watching HD playback and possibly recording 2 SD (and commercial skipping) all at the same time.

Yes, for that you need write-back.

I assume you needed help since you posted a message asking for assistance. Why would somebody with all your RAID experience ask a simple RAID 5 question?

-Robert

[stanger89]

Originally Posted by Shield

So, here's my dilemma. Right now my Sage box consists of the following disk subsystem:

1. Hardware Serial ATA Raid Controller - 8 channel internal LSI Logic 8308ELP with 128 MB/cache. Supports RAID 0,1,5,50. Only thing to note here is write back cache is disabled as I don't have the battery backup unit (WBC cannot be enabled unless the BBU exists).

If you've got it on a UPS then there's really no reason to disable write cache.

Tell me which option you'd go with:

4. Buy 3 1TB drives and have 2.00 TB in a Raid5 array.
Cost=750

5. Buy 4 1TB drives and have 3.00 TB in a Raid5 array.
Cost=$1000

I'd go with TB drives in RAID-5, I'm tired of having to replace drives/arrays that I built with "small" drives when bigger ones were available. Sure it costs a bit more up front, but drives drop in price over time and this will extend the life of the array.

My main concern is by building a huge array that it'll get bogged down while recording 2 shows in HD, watching HD playback and possibly recording 2 SD (and commercial skipping) all at the same time.

Do you really keep that many recordings? I personally don't record to my RAID-5 array, it's just for 'static' data (DVDs primarilly).

That said, from what I've seen RAID-5 on a good controller is faster than a single drive, even for pseudo-random reads/writes.

[CollinR]

It really shouldn't matter much if the data is static DVDs, RAID5 for more efficent use of the drives, RAID1 for cheap and easy.
RAID10 though works best for the TV recordings but you say you don't care too much if they are lost. So really you can do anything but JBOD.

The only problem with RAID5 is it writes the parity across the array. With SageTV's dynamic control of the large highly compressed video data makes for fragmentation over time. That also hits on it's slightly slower write capacity.

If you are just putting DVDs out there and adding a new one occasionally it's unlikely you'll notice.

[sixdoubleo]

I also run RAID 5 on my server and echo what Stanger said....get the big drives. I built my RAID 5 at the tail end of 2004 with six 250GB drives ($129 each at the time) and am now looking at the hassle of replacing the whole thing.

Like others here, I don't record directly to the RAID 5...I record to a couple 500GB dedicated drives. The RAID 5 is used for library, DVD rips, music, etc. My RAID 5 is on a different server than the Sage box.

Also, and I'm sure you're probably addressing this already, I would avoid splitting the thing up into partitions and multiple drive letters. Just allocate the whole thing so you avoid fragmentation of your free space. Drive letters are going the way of powdered wigs. With NTFS permissions and DFS (assuming you run AD) you can virtualize your space and graft it all together to look however you want.

[Shield]

I just couldn't justify the extra $100 per drive for the cheapest 1TB drives, plus they were 5400 rpm. I bought 4 of the WD 7200 750's and a new case that will allow me to add 4 more when the time comes.

Oh, and I did change my post about writing Sage recordings to a non-raid disk before I was belittled about not having write-back enabled (to Valnar).

[Shield]

Originally Posted by stanger89

If you've got it on a UPS then there's really no reason to disable write cache.

I do have it on a UPS, but it cannot be enabled (it's disabled by LSIlogic if a BBU is not detected - it's the only raid card I've owned that I've seen do this)

Originally Posted by stanger89

I'd go with TB drives in RAID-5, I'm tired of having to replace drives/arrays that I built with "small" drives when bigger ones were available. Sure it costs a bit more up front, but drives drop in price over time and this will extend the life of the array.



Do you really keep that many recordings? I personally don't record to my RAID-5 array, it's just for 'static' data (DVDs primarilly).

That said, from what I've seen RAID-5 on a good controller is faster than a single drive, even for pseudo-random reads/writes.

Yes, that's been my experience also.

[stanger89]

Originally Posted by Shield

I do have it on a UPS, but it cannot be enabled (it's disabled by LSIlogic if a BBU is not detected - it's the only raid card I've owned that I've seen do this)

How nice of them. Not that it really makes much difference with writing large files, since you'll quickly fill that.

[Shield]

Well the good news is these BBU's for these cards used to be well over $200 - this is an expensive 8 port card - it's not flaky like some Highpoint controllers - but some guy was selling the BBU's for $55 on ebay, so I picked one up.

Every box in my house has either RAID 5 SCSI or SATA (the performance of the scsi is obviously night and day over the SATA once more than one file is being written) and that LSI 8308ELP is the only one that forces you to have a BBU for write-back. Hell, none of my other cards do (LSI PCI-X 320-2 and 150x4) so who knows.

[mikesm]

Well, which OS are you using for the server?

Windows software raid is notoriously unreliable, but a good hardware raid should have no problems doing what you are asking of it. All those applications are doing sequential access at rates that any decent array should be able to support just fine.

As an example, I currently store all media on a linux server running software RAID of about 6 TB of storage in one shared volume (8 500GB disks in RAID5, and 4 1 TB disks in RAID5). Showanalyzer, plus Sage server with 5 tuners ( 2 from the HDHomeRun, 1 from an R500-HD, 1 ATI 650 and 1 PVR 150) with the analog tuners recording at max quality, 2 Sage Clients driving HD displays and 1 MVP, and everything remoted over gigabit ethernet to the server. No sweat.

Of course, this is linux software RAID, and I can get over 200 Mbytes/sec out of the disks easy, so this is total overkill for even a bunch of HD streams.

I assume windows hardware raid can do this same sort of performance, and as folks said before, you really need a UPS on the system.

Thanks
Mike

[sixdoubleo]

Originally Posted by mikesm

As an example, I currently store all media on a linux server running software RAID of about 6 TB of storage in one shared volume (8 500GB disks in RAID5, and 4 1 TB disks in RAID5). Showanalyzer, plus Sage server with 5 tuners ( 2 from the HDHomeRun, 1 from an R500-HD, 1 ATI 650 and 1 PVR 150) with the analog tuners recording at max quality, 2 Sage Clients driving HD displays and 1 MVP, and everything remoted over gigabit ethernet to the server. No sweat.

Can you give me more specifics about your setup? Which distribution? What tool did you use to set it up? Will it easily resize if you add additional drives?

Also, how are you getting 12+ drive connections on one system? If it is an add-on SATA adapter, which add-on card are you using?

I'm considering a Linux SW RAID myself for backup and would be curious what you're using.

Thanks!

[mikesm]

Originally Posted by sixdoubleo

Can you give me more specifics about your setup? Which distribution? What tool did you use to set it up? Will it easily resize if you add additional drives?

Also, how are you getting 12+ drive connections on one system? If it is an add-on SATA adapter, which add-on card are you using?

I'm considering a Linux SW RAID myself for backup and would be curious what you're using.

Thanks!

You should look at my posts in this thread: http://forums.sagetv.com/forums/showthread.php?t=25709

Basically, I use 4 motherboard SATA ports, plus 2 SI 3132 SATA II ports hooked to two addonics 5:1 SATA port multipliers. Each port multiplier gives you 5 SATA ports for 1 host port. Modern linux kernels support several SATA controllers hooked to PMP's. For example, the 6 AHCI SATA ports found on many intel motherboards can support 5 disks each with port multipliers, with no performance hit. That's 30 drives on motherboard ports alone. (Actually, I think 2 of the six have some issues due to the way intel did it, but even 20 is a lot of drives).

My SI 3132 PCIE card cost me $20. I use an old DFI NF4 motherboard with an AMD 3800 X2 S939 CPU a 1GB of ram in it. Plenty fast for this.

Linux software RAID can expand RAID5 volumes online just fine. Do an man mdadm and look for the grow option. I also use LVM which lets combine physical volumes into logical volumes pretty easily. For example, I used to have 2 RAID5 arrays, a 5x320, and a 5x500. Both were combined into one logical volume which was exported via SAMBA, FTP, WebDav, etc...

Well, I had 3 new 500GB disks ready to add, but found a nice deal on 1 TB disks from best buy, and I didn't want to add another hotswap cage, so I added the 3 new 500GB disks to the existing 5x500 array, expanded the physical volume using LVM, and then did a pvremove on the old 5x320GB array. LVM then moved all the data stored on that 5x320GB array to the the new 8x500 array, all while the filesystem was completely live. I then removed the 5x320GB array from service, replaced the 320's with the 4 1TB disks I just got, created a new 4x1TB RAID5 array, and then added it to the LVM volume. Voila. 6 TB online, and I did all this without having to take the volume offline. It took longer since the filesystem was active, but I like my uptime.

I also use XFS which is well suited for media serving of large files. Performance is excellent. I generally see 40-50 Mbytes/sec in windows copies through SAMBA over Gigabit ethernet. Faster if I don't have to have windows machines involved.

It's so cool to see all this technology developed for large enterprise use by the open source community be applicable to my "home datacenter" as a friend of mine refers to his own system.

Hope this is helpful. Of course, I know linux fairly well, and while this stuff is pretty straightforward, if you have no unix experience, I would stay away from it.



Thanks,
mike

[sixdoubleo]

Originally Posted by mikesm

You should look at my posts in this thread: http://forums.sagetv.com/forums/showthread.php?t=25709

Basically, I use 4 motherboard SATA ports, plus 2 SI 3132 SATA II ports hooked to two addonics 5:1 SATA port multipliers. Each port multiplier gives you 5 SATA ports for 1 host port. Modern linux kernels support several SATA controllers hooked to PMP's. For example, the 6 AHCI SATA ports found on many intel motherboards can support 5 disks each with port multipliers, with no performance hit. That's 30 drives on motherboard ports alone. (Actually, I think 2 of the six have some issues due to the way intel did it, but even 20 is a lot of drives).

My SI 3132 PCIE card cost me $20. I use an old DFI NF4 motherboard with an AMD 3800 X2 S939 CPU a 1GB of ram in it. Plenty fast for this.

Linux software RAID can expand RAID5 volumes online just fine. Do an man mdadm and look for the grow option. I also use LVM which lets combine physical volumes into logical volumes pretty easily. For example, I used to have 2 RAID5 arrays, a 5x320, and a 5x500. Both were combined into one logical volume which was exported via SAMBA, FTP, WebDav, etc...

Well, I had 3 new 500GB disks ready to add, but found a nice deal on 1 TB disks from best buy, and I didn't want to add another hotswap cage, so I added the 3 new 500GB disks to the existing 5x500 array, expanded the physical volume using LVM, and then did a pvremove on the old 5x320GB array. LVM then moved all the data stored on that 5x320GB array to the the new 8x500 array, all while the filesystem was completely live. I then removed the 5x320GB array from service, replaced the 320's with the 4 1TB disks I just got, created a new 4x1TB RAID5 array, and then added it to the LVM volume. Voila. 6 TB online, and I did all this without having to take the volume offline. It took longer since the filesystem was active, but I like my uptime.

I also use XFS which is well suited for media serving of large files. Performance is excellent. I generally see 40-50 Mbytes/sec in windows copies through SAMBA over Gigabit ethernet. Faster if I don't have to have windows machines involved.

It's so cool to see all this technology developed for large enterprise use by the open source community be applicable to my "home datacenter" as a friend of mine refers to his own system.

Hope this is helpful. Of course, I know linux fairly well, and while this stuff is pretty straightforward, if you have no unix experience, I would stay away from it.

Thanks,
mike

Excellent!

I'm well-versed in Linux and Samba, but have never fooled around with its software RAID. Used LVM before though. Your setup looks very interesting.

Where did you mount the PMPs? Inside the CM Stacker case? (edit: nevermind....I googled them and came up with the eSATA ones. I see what you're using now).

I want one. Must. Resist. Urge. To. Rebuild. RAID. Server.

[mikesm]

Originally Posted by sixdoubleo

Excellent!

I'm well-versed in Linux and Samba, but have never fooled around with its software RAID. Used LVM before though. Your setup looks very interesting.

Where did you mount the PMPs? Inside the CM Stacker case? (edit: nevermind....I googled them and came up with the eSATA ones. I see what you're using now).

I want one. Must. Resist. Urge. To. Rebuild. RAID. Server.

I got the kind which has a base base to them, and I just mounted it with double sticky tape in the base. The challenge is keeping all the SATA cabling neat. I basically wiretied all of the SATA cables coming from the PMP into one bundle that only broke out at the SATA hotswap rack. I like the addonics 5in3 a lot, but Fry's has the AMS clone of it for $95, which is pretty good.

I like the hotswap racks a lot. A must if you like doing online disk reconfigs. :-)

If you'fine with Samba and LVM, RAID will be trivial for you. You just need to pick the right hardware, use a good chunk size (128K or 256K), and then tweak all the buffering (that's what my diskopt.sh script does). Like I said, performance is great.

If you need a good excuse to upgrade, get a cable R5000-HD. Full HD from premium movie channels burns disk like mad... :-)

thx
mike