aaaaaaaaaaaagh! Disk array corrupt.

[sleonard]

I just lost all of my recordings and saved video when my disk array got corrupted. I have created a new array and it is formatting right now. When I start Sage back up how is it going to handle an empty recording directory?

[Opus4]

SageTV will check for the existence of the recordings in its database (wiz.bin) by checking where it last noted they were. If the drive is accessible, it will see that they don't exist & will removed them from the list of recordings. If the drive isn't there (say you had them on drive E, but now it is drive D, and E is not there), then the files will be kept in the list, though you can't access them... at least that is what happened the last time I checked for non-existent files. If you wnat all the non-existent files to be removed from the database, make sure the orig drive is accessible. (And: did that paragraph make sense!?)

- Andy

[stanger89]

Any idea what happened?

[sleonard]

Thanks Opus, I was pretty sure that was going to happen but wanted to make sure before starting Sage up again. Sure enough, that's exactly what happened.

I started a defrag last night and when I woke up Windows was giving me delayed write errors and the defrag was still trying to finish the analyze phase. I never got to actually start the defrag as it never finished analyzing. I just cancelled and went to work. When I got home I couldn't watch any thing and was getting weird connection problems from the client. I rebooted the server and that's when the raid card told me that the array was corrupt. Took forever to format all 480 GB

Oh well, at least the disk is perfectly defragged now

[Opus4]

All hail RAID 5! All hail RAID 5! Keeper of our data; Guardian of its safety. Data redundancy is good. Raid 5 brings redundancy. All hail RAID 5!

Um... I watched MIB 2 the other day & you have to chant that real fast while confined to a small space.



<FLASH> Now back to reality.

- Andy

[sleonard]

I don't think that even raid5 would've help in this case. I think the defrag utility messed with the array on a low level. The adaptec card reported 3 of the 4 disks broken from the array and corrupted even though they are physically OK. Nevertheless, I would still like to have raid5. Maybe on a future file server for music, DVD, and video. I'm not sure recorded SDTV is worth it though so I'd probably keep the current 4 drive stripe for Sage recordings and manually move what I want to keep. Unless maybe Sage makes the jump to HD.

[mls]

Originally Posted by Opus4

<FLASH> Now back to reality.

There is such a thing? Been wonder what every happened to it since somewhere back in the middle of the Clinton administration.

sleonard - Really sorry to hear you lost all your files. Bummer.

[Opus4]

Well, I hope you realize my RAID 5 chant was a more of wishful thinking on my part than a dig at your lost data. I've had enough drives fail & lose enough recordings to know that is a real pain. (I'm hoping that was heat related in my old PC & that the new one won't have that problem with better cooling for the drives.) Since you think this was defrag related & not a drive failure... I sure hope you figure out what it was before you end up going through the same thing! At least when mine failed, it was only part of my recordings instead of _everything_.

- Andy

[mdmint]

Originally Posted by Opus4

All hail RAID 5! All hail RAID 5! Keeper of our data; Guardian of its safety. Data redundancy is good. Raid 5 brings redundancy. All hail RAID 5!

I only wish that were true. Unfortunately "data" isn't redundant with RAID5, only the drives themselves. The data (and parity) is striped across multiple drives yes, but that's for speed not data redundancy. RAID1 (mirroring) is data redundant. Yes you can loose one drive and data is maintained via on the fly via parity reconstruct but it isn't the same as redundant. Even though I'm running RAID5 with online spare so will immediately rebuild if an HD fails I know all too well that my PVR recordings are still at risk. If the RAID controller fails in the wrong way it's all scrambled and gone for good. RAID1 also susceptible to lost data because of failed controller. That is the purpose of backups, to guard against catostrophic data failure. Unfortunately I don't have a method in place for my 1.5TB Sage storage. Eventually may build another mungo storage system with a second RAID controller and either RAID10 it with current 1.5TB RAID5, would be cheaper (and faster) than a tape backup system capable of backing up this data amount. Can't afford it at the moment...

[Opus4]

Rats, I knew someone might pick apart my wording... OK, OK, I know that, but I just couldn't see chanting about "striping parity data across drives for added security, though it isn't failsafe", so some poetic license was taken. You have to remember that these beings have blind faith in their most current idol.

- Andy

[stanger89]

Not to be a smart @$$ or anything but RAID-5 does provide redundancy, all be it not a robust as RAID-1

Redundant - Duplication or repetition of elements in electronic equipment to provide alternative functional channels in case of failure.

I will agree with you that RAID-5 (or 1 for that matter) isn't equivalent to backup, I look at it as something to protect my investment, either time (Ripped DVD), or to provide additional security for things not practical to backup (RAID arrays have far exceeded any practical backup method).

[Cayars]

If you're thinking of building a secondary Raid array don't for backup purposes. Look at ebay and you'll find Dell Powervaults on there in the range of $1500-$5000. I picked up one last year for 2K that holds 36 tapes (auto changer) and has two internal tape drives. Each tape is 100GB without compression (200GB with)

The box itself will hold up to 6 internal tape drivers. It's SCSI and works with just about every backup software there is including what's built into the operating system. Once a tape is filled the auto changer can pull the tape and put a new one in so the backup limit for our purposes is pretty much unlimited. One of the cool things you can do with multiple drives is is run them duplexed or mirrowed and have two exact copies of every backup (just in case). If you have 3 or more tape drives you can also set them up as RAID drives which is pretty cool.

It's not your typical backup device but for something that sells new for close to 30K being able to pick one up like new for 2K is very nice especially when you have a terrabyte or two of online storage without a resonable way to back it up.

Carlo

[mdmint]

Originally Posted by stanger89

Not to be a smart @$$ or anything but RAID-5 does provide redundancy, all be it not a robust as RAID-1

I agree to agree AND disagree! It's a matter of semantics and who's definition.

• RAID-5. (Most commonly used) This type includes a rotating parity array, thus addressing the write limitation in RAID-4. All read and write operations can be overlapped. RAID-5 stores parity information but not redundant data (but parity information can be used to reconstruct data if enough of it remains).

In RAID5 the data itself is not redundant, not in more than one place, but data recovery or redundancy is achieved if a HD fails by reconstructing data via parity information. Though part of the data is actually in reality missing, being on the failed HD, it is indeed available so some definitions call it data redundant some do not.

[mdmint]

Yeah, I'm familiar with large backup multi drive library systems that can back up hundreds of TB nightly, some so big you even walk inside! But I'm leaning towards not going tape library but rather a second Escalade 9500S-8 RAID Ctrl and more 250GB HDs yielding even higher system availability. Would likely redo current 1.5TB 8HD RAID5 with online spare and go 1.75TB 15HD RAID10 (stripe of mirrors) with online spare, stripe of 7 pairs of mirrored 250GB SATA each of the two ctrls having one of the mirrored HDs of each set. This way even if one of the ctrls failed would still be functional as RAID0 stripe of broken mirrors. Than again this still wouldn't be a backup scenario with the added protection ability to store tapes in another location.
Gee, if funds were unlimited I'd just do both. But after 13+ years with HP formerly Compaq formerly DEC and dodging many many rounds I've recently become another statistic in the continuing Corporate America outsourcing downsizing saga and the future is as yet uncertain. And HP proudly proclaimed over $1Billion profit last quarter! (Upper management must have their large 5, 6 and even 7 figure bonuses don't you know)

Originally Posted by Cayars

If you're thinking of building a secondary Raid array don't for backup purposes. Look at ebay and you'll find Dell Powervaults on there in the range of $1500-$5000. I picked up one last year for 2K that holds 36 tapes (auto changer) and has two internal tape drives. Each tape is 100GB without compression (200GB with)

The box itself will hold up to 6 internal tape drivers. It's SCSI and works with just about every backup software there is including what's built into the operating system. Once a tape is filled the auto changer can pull the tape and put a new one in so the backup limit for our purposes is pretty much unlimited. One of the cool things you can do with multiple drives is is run them duplexed or mirrowed and have two exact copies of every backup (just in case). If you have 3 or more tape drives you can also set them up as RAID drives which is pretty cool.

It's not your typical backup device but for something that sells new for close to 30K being able to pick one up like new for 2K is very nice especially when you have a terrabyte or two of online storage without a resonable way to back it up.

Carlo

[stanger89]

Originally Posted by Cayars

Look at ebay and you'll find Dell Powervaults on there in the range of $1500-$5000.

Personally I'd rather have the extra 2-5TB that that could buy me

And mdmint, by that definition I agree

[bluenote]

Mdmint: although you're semantically correct, I think the important thing for readers of this thread, is that raid 5 DOES protect your data.

Yes, RAID has its own limitations, but I disagree strongly with your implication that mirroring is so much better for data protection purposes in a holistic view than raid 5. For RAID purposes, 5, IMHO, is far better than mirroring for the reason that you specify -- "if the raid card fails a certain way..." given. You can take the resources saved ie: N+1 disks for data protection rather than N X 2 .. and nowadays, using diagonal parity (sometimes called double parity) N+3 (parity, dparity, spare) you can withstand any 3 disk failures at a time and still be A-OK.

But the chances that mirroring is going to make the difference for you, vs. RAID 5 is in my opinion unlikely. Incidently, you'd be much better going raid 5, gaining your data protection, performance from striping & adding spindles, and take the money you've saved from not having to double your hard disks to put into a proper backup method (or to get out of the whole IDE RAID scene altogether).

You know that IDE drives are approximately HALF as reliable as SCSI (by MTBF). While I do have one box that uses IDE RAID at work, that's the only one we use IDE in, because the purpose of it IS disk backup, as one level of protection between RAID 5, online disk backup (IDE), tape backup (IBM 8307(?) library), and snapshot (hardware specific.. Netapp NAS) plus offsite live replicated volume.

FWIW

Cory

[Opus4]

Amazing what that MIB2-inspired joke has spawned...

But, I might have to go watch the first one, since I don't remember ever having seen that one.

- Andy

[teedublu]

The main problem with a redundant array (mirror or parity array type) is that it only protects against hardware failure -- and hopefully that hardware failure is quick n clean.

The problem is that most problems are data corruption problems, which can only be protected by a backup program -- so that you have time to say "don't backup my bad data" (overwriting my good backup with bad), which is what happens in a RAID situation.

The main drawback with a conventional daily backup is that if there is a crash, you may loose up to a days worth of stuff.

Certainly the ideal situation is a RAID system that is also backed up -- that's overkill I think for this stuff.

I think the best thing to do with those RAID disks is turn them back into volumes that are setup for a "backup buffer" Files are constantly being added as they're created on the source disk, and only deleted from the backup when the source has been deleted AND there is no space left (the oldest "source-was-deleted" files being cleared first). Yes, doing a complete restore may be more of a hassle -- old files need to be cleaned up -- but even that could be simplified with a "delete the been-deleted" batch file that could be automatically maintained by the backup routine.

I haven't found a backup program to do this -- haven't really looked too hard. But that's my next to-do. Might have to write my own or maybe a combo of the 2.

TW

[stanger89]

Originally Posted by Opus4

Amazing what that MIB2-inspired joke has spawned...

That's good, I didn't realize where that came from

Back on the RAID subject, a few random thoughts, basically my train of thought on the whole media server, RAID thing.

1) IMO, in general RAID has no place in a home PC. Firstly the general home PC doesn't have the storage requirements (>400GB or so) that many disks are necessary. Secondly, there's no performance advantage, despite the popularity of RAID-0 in gaming rigs, it's little more than a bragging right since while RAID-0 increases transfer rate, it also increases latency, and destop apps (including games) are bound more by latency than transfer rate. RAID-1 is also unneccessary because the general PC doesn't need the absolute 100% guaranteed uptime, and that's what RAID-1 is for, to guarantee that the PC doesn't go down if a drive fails. And RAID-5, well there's just no reason. For a general PC the best option is probably one or two big drives, and back up your unrecoverable data on CD or DVD. Or if you've got a lot of unrecoverable data then just use another one (or possibly two) drives for backup. Firewire/USB drives are great because you can backup, and then disconnect them easily and if your data is corrupted then you can recover.

2) There are special circumstances where RAID is a good idea. For example if you plan on doing video editing RAID-0 can be a great asset since you move a lot of data and are usually bound by disk transfer rate. RAID-1 is probably more for a corporate environment where downtime costs a great deal of money.

3) Now what we're talking about here isn't really a general PC, it's more of a media server, and that distinction is important. Why, because of the nature of the data stored and they type of use. A media server will typically contain recoverable data (ie DVD/CD rips), transatory data (like PVR files), and/or relatively unimportant data (stuff that if you lost, you could go on). A media server (we're talking at home here) is primarily an entertainment device, there to help make our hobby more enjoyable.

Along those lines, what are the primary criteria when building a media server? Well we have:
Capacity - we want to be able to store all our (fill in the blank)
Price - most of us don't have unlimited budgets, so "bang for the buck" is very important
Security - we don't want all our hard work to dissapear on us

So what options are there? Well there's:
JBOD - OK, but can be kind of messy with numerous drives, plus the controllers for them, plus if a drive dies, then you loose whatever's on it
RAID-0 - Definitely has the capacity and price (good $/GB) but actually reduces securty greatly
RAID-1 - This one has security nailed, but gets low marks for price/capacity (high $/GB) only 50% efficient
which leaves us with RAID-5 which gives us nearly the capacity of RAID-0, much of the security of RAID-1, and a good compromise on $/GB, making it great choice for a media server.

Of course I still back up (to DVD) the stuff that's really important (unrecoverable) like my digital photos.

[bluenote]

Stanger:

Agreed. My post above was taking the (big) assumption that you would want redundancy in the first place. Personally I would never consider putting RAID into my home PC either unless I had very specific requirements. And you're right about the bragging rights about RAID-1 .. I have a friend that this describes perfectly He still has to reinstall (and did just recently too ). I just couldn't let the assumption pass if someone might go out and make decisions based on what we were talking about here. I forgot to put in the most important part though -- that you're right in suggesting that there is very very few reasons where RAID is indicated in a home setup IMHO (in this case the H is humble even though re-reading my previous post I have to smile a bit)

Thanks for bringing things back to earth

Opus: I didn't realize this was all a lark either

Cory