to raid or not

[toony]

Ok I think I did a dumb thing here, my server is growing at about 1 TB every few months, things are getting full. I never emplemented raid.

Currently I have the os and sage on a .5TB drive, 3 x 1TB & 1x 1.5TB drives of media.

Can raid be implimented at this point or do I have to start fresh and which raid would be recomended? I was thinking Raid 5.

[fidget]

If you want to just convert your drives to RAID, it is too late. Some RAID boxes will allow you to extend an existing RAID (JBOD, RAID0 or RAID5) setup, so you could get a ReadyNAS or Thecus box (I have two Thecus 5100 boxes) and put two or three new drives in there and build the RAID from there. Copy your files over and then add a new drive to the mix, copy more files and add another drive, etc, etc. A JBOD configuration wouldn't need to rebuild from parity and can handle different sized disks.

There is a downside if the drives are too large (750+ GB). I remember reading an article on either Arstechnica or Slashdot (I can't find the link) that said with the larger drives, the amount of data being stored will increase the failure rate to the point where a RAID5 may not be able to be rebuilt after a failure. This was a factor of the error rate and the size (disk size * error rate > 95%). I have played with rebuilding my 5 drive RAID5 box (5 500GB drives) and it rebuilt fine. I haven't tried my 5 750GB drive box.

My solution has been to build a new NAS, copy files over to the new box, and go from there. Since I use my NASses to hold DVDs, when I built my 5 750 GB NAS, I copied all my DVDs over and reformatted the RAID5 box to be RAID0.

[stanger89]

Well you can't just convert your current drives into a RAID array. The creation of a RAID array is a destructive process. Also a RAID array needs to be made of all the same size drives (or the space of the larger drives goes unused).

If you wanted to go RAID at this point, you'd want to figure out what your current required capacity is, and probably how much you're going to want in the future. That should give you an idea of the capabilities you need in your RAID controller.

[alfi33]

How do you guys feel about RAID 0? I was thinking about getting a couple 1TB drives to add to my server. Is RAID 0 worthwhile for a recording drive? I don't care about redundancy. I'm currently having some trouble watching something while Comskip is running on my single 500GB drive (with 64k clusters). The drive just can't keep up. Do you think RAID 0 would help much?

[Striker:WG]

RAID0 is good for speed, but if you are having issues with your 500GB drive and comskip then there are other issues in your system.

Currently I have several hard drives in my system but only 1 is relevent here. My recorded tv drive is a 250GB IDE 16MB Buffer 7200RPM drive. This evening I was capturing 2x SD shows and 1x HD show simultaneously with two instances of commskip running to parse the SD shows while playing back the HD show and there was no issues.

The point is that if my 1 hard drive can keep up doing all that then you have other issues to troubleshoot as your 500GB drive should be more capable than my old clunker 250GB

-Striker-

[fidget]

Originally Posted by alfi33

How do you guys feel about RAID 0? I was thinking about getting a couple 1TB drives to add to my server. Is RAID 0 worthwhile for a recording drive? I don't care about redundancy. I'm currently having some trouble watching something while Comskip is running on my single 500GB drive (with 64k clusters). The drive just can't keep up. Do you think RAID 0 would help much?

The larger the cluster size the less fragmentation you will get. My NAS will allow me to format up to 1MB clusters on my RAID (0, 1, 5, 6 or JBOD) setup. I am currently using 256KB.

Two problems that I have had are: I have the NAS spin down after 30 minutes of inactivity and it can take about 30 seconds to spin back up (this wasn't much ofa problem, actually), and deleting files could take a while to do and it would cause problems with the Sage server (up to the point of getting dropouts in the recording). The second problem is the reason that I stopped using my NAS for recording live programming. I sent the log to Sage and they found that the delete call was taking a long time to complete. Basically, Windows was not letting Sage run until the delete had finished.

[stanger89]

Originally Posted by fidget

There is a downside if the drives are too large (750+ GB). I remember reading an article on either Arstechnica or Slashdot (I can't find the link) that said with the larger drives, the amount of data being stored will increase the failure rate to the point where a RAID5 may not be able to be rebuilt after a failure. This was a factor of the error rate and the size (disk size * error rate > 95%). I have played with rebuilding my 5 drive RAID5 box (5 500GB drives) and it rebuilt fine. I haven't tried my 5 750GB drive box.

It shouldn't have anything to do with the overall capacity, but the number of drives. I tried to run through the numbers one day, and as the number of drives added gets large, the probability of an array failure (not just a drive failure) can get rather large.

[alfi33]

Originally Posted by Striker:WG

The point is that if my 1 hard drive can keep up doing all that then you have other issues to troubleshoot as your 500GB drive should be more capable than my old clunker 250GB

Interesting. HDTach and HDTune seem to think the drive is running just as it should. I have no other problems at all....unless Comskip is running....and then the drive thrashing begins. I guess I should just try another drive. However, at what point or under what conditions would RAID 0 make a difference where I would want to use it in my server?

[bastafidli]

Not sure what OS you are using, but I believe you can convert your existing drives to Software Raid under Linux nondestructively. First to go from non raid to SW Raid 1. Then there is a way how to convert SW Raid 1 to SW Raid 5 if you needed. Have not done it myself but I was reading such instructions when I was planning my transition from HW raid under W2K3 to SW raid under CentOD.

Of course these days and with 1.5 TB disks you can find discussion that RAID 5 is not sufficient and that the chance of second disk failing is very high while reconstructing the array. Disks are cheap, I would go with RAID1. And these days I am actually switching (or adding for some volumes) mirroring across machines rather than RAID1 to avoid the whole machine dying.

Of course I always recommend to have good backup first :-)

[Skirge01]

Have you considered something like FlexRAID? You can certainly begin using that at any time, without migrating your data.

Small sidenote: What's Cat6e?

[Striker:WG]

Originally Posted by stanger89

It shouldn't have anything to do with the overall capacity, but the number of drives. I tried to run through the numbers one day, and as the number of drives added gets large, the probability of an array failure (not just a drive failure) can get rather large.

that's why larger arrays are usually RAID6. Dual parity drives allows for two disk failures before total data loss. So if one drive dies, you can begin rebuilding and still have a second drive fail during the rebuild without any data loss.

also, if you are really running a large array you should be allocating at least 1 if not 2 hot swap spare drives that the array can auto-rebuild with if an error occurs and a drive fails.

-Striker-

[Striker:WG]

Originally Posted by alfi33

Interesting. HDTach and HDTune seem to think the drive is running just as it should. I have no other problems at all....unless Comskip is running....and then the drive thrashing begins. I guess I should just try another drive. However, at what point or under what conditions would RAID 0 make a difference where I would want to use it in my server?

maybe the issue doesn't lie in the drive itself, but the rest of your system?

i'm running an Intel C2D Q6600 /w 4GB DD2 1066 RAM, what's your system core?

-Striker-

[toony]

Well now I think I a bit more confused.

I've been doing some reading, I'm not sure that JBOD is the way to go.
Flexi Raid seems interesting

My head is swimming and I don't think I can properly decide right now. I'm now questioning using XP Pro as my server. It work fine for my two extender setup. I will I am sure put one more, possibly two extenders in the future, long ways away and one client pc which will be a touch screen for my wife in the kitchen to view recipies and watch her cooking shows.

My main concern is speed and if I go to raid I don't want things to slow down. I can serve 2 movies at full 1080p DTS to both extenders at once without any issues and I don't want to lose that ability. At the same time I am sure it will suck if a disk fails and I lose some data.

Can I not just run some HDD monitoring software which will warn me of any issues creeping up. Perhaps it could creat a daily report for me and if there are any issues I can change out that drive.

I was not really thinking of building a NAS for more storage instead I was thinking of e-sata box (s) conected right to my server.

To much thinking going on here.

[stanger89]

Originally Posted by toony

Can I not just run some HDD monitoring software which will warn me of any issues creeping up. Perhaps it could creat a daily report for me and if there are any issues I can change out that drive.

You can, but that won't do you any good if a disk actually fails.

I was not really thinking of building a NAS for more storage instead I was thinking of e-sata box (s) conected right to my server.

The only comment I'll make on that is depending on size, an eSATA box is almost as much as a NAS.

How much space are you looking for and how much are you looking to spend (not counting drives)?

[bastafidli]

What you can do instead of RAID is use mirroring and keep a second copy of all your data/recording on what is basically read-only file server. This way even if your power supply or motherboard breaks on your primary server, the secondary server is still fine. The computers are cheap, especially since the second one would need to be just a headless file server and the expenses for disks are the same (or less) than you would pay for RAID since you move them to the other box.

[toony]

Originally Posted by stanger89

The only comment I'll make on that is depending on size, an eSATA box is almost as much as a NAS.

How much space are you looking for and how much are you looking to spend (not counting drives)?

I want to do this the right way, I have seen some e-sata boxes for under $1000

Right now I am up to 5 TB, I imagine I will be up to 7TB by summer.

I think I am getting way over my head here. I was looking at some of those Raid NAS towers and they werein the 3- 4 thousand range! That's crazy. Certainly I could build an NAS for allot cheaper? couldn't I?

The mirroring seems like an easy solution but not the cheapest, as everything needs to be doubled. And now that I found out about the large TB drives being more suseptable to failure has got me wondering how large a drive I should go.

At the very least for now I think I should install some HDD monitoring software.

[Skirge01]

I was in this boat not all that long ago. I wanted redundancy. I wanted lots of storage. I wanted uptime. I thought RAID was the answer. Then the cost factor set in. Between the RAID card(s) and port expansion cards, it ballooned to a huge amount. NAS turned out to be nearly as expensive. Then, you have the possible proprietary nature of these things.

Instead, I opted for a big case (the Lian-Li 343B), then added multiple Addonics 5SA drive cages, and the Addonics 5X1 Port Multiplier. Then, I added WHS as the OS and FlexRAID. WHS handles duplication for the critically important data I cannot afford to lose (tax returns, photos, etc.) and FlexRAID provides parity coverage for my media. Now, due to wanting to do this right and only once, I did buy some expensive parts and it has probably gone into what a NAS box would've cost. However, I can add just about an unlimited number of drives and nothing is proprietary, so if any part fails, I can replace it with something similar without fear of it not being compatible.

[S_M_E]

Originally Posted by toony

The mirroring seems like an easy solution but not the cheapest, as everything needs to be doubled. And now that I found out about the large TB drives being more suseptable to failure has got me wondering how large a drive I should go.

Personally, this is why I like WHS as a Sage server. It uses more drives on duplicated shares (like mirroring) but as long as you have more freespace than your single largest drive it won't matter if a large drive fails, you won't lose data and it's dynamically expandable. Large drives are here to stay and it makes more sense, to me, to use fewer large drives than dozens of smaller drives. I doubt it'll be very long before we see 5T and 10T drives and faster interfaces than SATAII.

[toony]

Well I think you guys have my mind made up. I am going look into port replicators and WHS. I just bought a new case with 8 HDD slots. Really it's a fancy server case with 6 full size case fans, it's meant for water cooling but the internal temp is pretty nice and cool with out it. Regardless.... I am going to start researching this.

Just curious if I want to make the switch to WHS and flexraid to use parity do I have to clear my drives off or can it set up the raid as long as it has one free drive?

[S_M_E]

I'm not sure about the flexraid setup (I don't use it) but when you add drives to the WHS pool it will format them. The pool is one of the great features of WHS. Skirge01 can explain how flexraid works better than I could...

Also, the port replicators will allow you to add a lot of drives but they share the bw of one port. You might be better off adding a cheap multi-port PCI/PCI-e card to add ports if performance is more important than space. You wouldn't need the RAID features of a card if you use WHS or flexraid instead.