Show "Hoarders", how much space did you use?

[IVB]

hey drvn, i have a hybrid hw/sw RAID card you can have for free. (Well, perhaps some speaker installation tips ). Cost me $300 a few years ago, can't tell you brand as the closet has halloween decorations for big shindig tonight. I couldn't get it to work without blowing out CPU, which is when I got the $700 RocketRaid.

[Fuzzy]

I've never seen a data center that did things the least expensive way. Does that mean they work any better? no. The off-the-shelf hardware solutions are simply easier to get going. With it you get professional consultants working for the manufacturer so you dont' have to think. These are things execs pay a lot of money for. Do they work any better than something like pfsense on equivalent hardware? no, not really.

Data centers and such do not usually incrementally upgrade. When the 4x2TB RAID array is getting full, they'll usually make a big capital expenditure and purchase a completely new 4x3TB array, transfer the data to the new one, and scrap the old one. seriously, this is how most american business operates. and no, it doesn't make sense. but it is easiest for that one particular person who was making that one particular decision, so that's what gets done. Look to data centers for their reliability and redundancy - not for their purchasing decisions.

[IVB]

Originally Posted by Fuzzy

Data centers and such do not usually incrementally upgrade. When the 4x2TB RAID array is getting full, they'll usually make a big capital expenditure and purchase a completely new 4x3TB array, transfer the data to the new one, and scrap the old one. seriously, this is how most american business operates. and no, it doesn't make sense. but it is easiest for that one particular person who was making that one particular decision, so that's what gets done. Look to data centers for their reliability and redundancy - not for their purchasing decisions.

Big companies have to pay for labor, plus can amortize/depreciate capital expenditure off taxes at a better rate than labor which is typically operating expense. Long story short, it might very well be cheaper for them to buy new stuff instead of maintaining older stuff.

And then, there's the rest of us, who work for nothing and have no tax writeoffs on HTPC playthings :-)

[Fuzzy]

Originally Posted by IVB

Big companies have to pay for labor, plus can amortize/depreciate capital expenditure off taxes at a better rate than labor which is typically operating expense. Long story short, it might very well be cheaper for them to buy new stuff instead of maintaining older stuff.

And then, there's the rest of us, who work for nothing and have no tax writeoffs on HTPC playthings :-)

I wasn't trying to make it sound like they are doing it wrong - just that how they do it is NOT how we should do it in this space.

[stanger89]

Originally Posted by drvnbysound

Sure, I understand your point (I'm not a data center)... but why are the data centers using a hardware based system when they could gain all of these advantages from being software based? I'm sure they've done the research.

It's all about the application, well that and level of support. But first, not everybody with a datacenter uses enterprise class hardware:

Backblaze uses consumer drives and regular HBAs rather than hardware RAID HBAs (and they list a few entities, including Netflix that are using similar systems):
http://blog.backblaze.com/2013/02/20...orage-pod-3-0/

https://signup.netflix.com/openconnect/hardware
"The hard drives are not hot swappable, as we wish to avoid the operational burden of field service. For lower power utilization and simpler sourcing we select commodity units from two vendors and use software to manage failure modes and avoid field replacement."

There's a good little discussion about applications down at the bottom of the Backblaze article (section "The Uses and Users of a Storage Pod").

Lots of enterprises, especially the ones you are likely referencing fall into the "Transactional storage" category, they have lots of people working simultaneously on the same, or near by data, and in the business world any time your employees spend waiting for data costs you money so you often want your storage to be fast. And by fast I mean supporting lots of IOPs across "random" data.

At home, especially for media, we fall into the "Bulk storage" category along with Backblaze. We want our data to be available all the time, but (for an entity like Backblaze) that data doesn't need to be available instantaneously, it can take a bit to prepare it since the data is rarely needed.

At home it's similar, there are only a few (one?) user, so while you need decent IOPs, that data is not very random, and you only need enough IOPs to support a couple video streams.

The other thing to consider is scale, at a very rough level you could think about it like this, the equivalent (in rough scale) to a single hard drive at home, is an entire storage array for an enterprise. Just think about that for a moment. If you're an enterprise, you don't think in terms of adding a hard drive to increase your capacity, you think about adding a whole storage array. With this mindset, whether you can add a hard disk to an array is irrelevant, you buy your arrays "complete".

Contrast this to home, where it's like you need maybe 8 TB today, and a year/months down the road you need a couple more TB. You don't want to buy a whole new storage array every time you need more storage at home, it's much simpler, cheaper, to just be able to add a new disk to your storage array.

Just as an example, my unRAID system is somewhere on the order of 8 drives today, but it started out with probably half that number. If I had gone the "enterprise" route, I would have purchased an 8-port RAID card, and either 8 drives initially (or less and relied on OCE). But either way I would have filled that up long before today since I would have built it on 2TB drives because that was the largest capacity available. And I'd be stuck, and out of space today, looking at either rebuilding the whole array from scratch or adding another array.

In contrast with unRAID, I just ordered a 4TB drive yesterday and I'll be installing it and expanding my array today. Much cheaper, much easier, much more in line with my requirements for storing my media.

Similarly, why do many enterprise networks utilize Cisco routers, firewalls, etc. instead of using pfSense? If they could save the money, I'm sure they would.

There's another factor not to be underestimated. Time is money for a business, so paying for robust support is often cheaper than doing it yourself. So Enterprises will go with Cisco etc in no small part due to the support those companies offer.

Vs home where often people value their money more than their time, and would rather set up, and administer a system themselves, than to pay someone to do it for them.

Yes, I will certainly have to draw a line with my own system and make the appropiate choices that meet my individual needs best (cost being a condsideratioin as well)... it's just a little harder to make those decisions when the cost difference is almost nothing. Can I afford a current model, Cisco enterprise-grade router? No. So what's the alternative? Do it in software, or ...

That's the only reason that I made the comment about the data centers and their use of hardware based RAID. The cost of going hardware vs. software is pretty negligable, so then it just comes down to what's better in terms of performance and my needs.

You can build an entire unRAID machine, standalone, for less than the cost of a good 8-port RAID controller, and it will be more expandable.

A good 8-port RAID controller is $500+. You can get a good, low power system for unRAID (excluding enclosure) for about $250 (Supermicro X7SPA+ram), that's 6 ports. Unraid Pro is another $120, so we're at about $400 or so for the equivalent hardware for an unRAID system to just a RAID card (and a RAID card still needs a system to run in, both still need and enclosure).

I ran an 8-drive RAID 5 array for a long time. I was sold on hardware RAID at the time. It worked pretty well though it was costly to set up. And it only worked well until I ran out of room, and then I was stuck with what to do. I ended up getting a 4-bay NAS, that offered a bit better expansion (maybe I should say expansion was possible) but the 4 bays was limiting.

I struggled for a long time on what to do when both were full, but ended up determining that for me, the flexibility and comparably low cost (see above) of unRAID vs either a retail NAS or a hardware RAID solution, made a lot more sense for storing my media at home.

I'm not trying to say that unRAID is the answer for you, but before judging your options based on what enterprises do, first make sure you take a look at what some enterprises that need lots of storage actually do (Netflix/Backblaze using commodity hardware with custom software solutions, not unlike unRAID/flexRAID in principal) and make sure to step back and think about how you're actually going to use it. Not just today, but how soon, how often you'll want to expand it as well, and how valuable ease of expansion would be.

[Skirge01]

Originally Posted by drvnbysound

Sure, I understand your point (I'm not a data center)... but why are the data centers using a hardware based system when they could gain all of these advantages from being software based? I'm sure they've done the research. Similarly, why do many enterprise networks utilize Cisco routers, firewalls, etc. instead of using pfSense? If they could save the money, I'm sure they would.

Everyone else has covered this pretty well, but want to specifically call out one thing I think needs to be detailed further: support. Data centers and businesses require 99% uptime. As others have said, time is money. That means that when something fails, you need to be able to get back up and running in hours. You simply cannot get that level of support from unRAID, tRAID, RocketRAID, pfSense, etc. That's where Cisco, HP, Dell, et all come in. To a business, getting a technician out to your facility at 3am merely 4 hours after the failure, with the exact hardware you need, is worth that level of support and the exorbitant costs associated with it. The company I work for replaces hardware as soon as the manufacturer EOLs it, even if it's still working fine and fits our needs. Why? The support ends shortly thereafter.

[Skirge01]

Originally Posted by Skirge01

I'm guessing you figured it out since you posted in the tRAID forums. But, for others following the thread, yes, I currently cannot record to the array. I think I tested both live streaming, as well as watching something after it recorded, but both had the same issue. I have another tRAID release to test, but haven't gotten around to it yet.

I finally found some time to test that tRAID release and submitted the results to the developer. Sadly, he's stated that SageTV is apparently "setting restrictions that no other process can access the file" and that I'd need to get SageTV to fix this. Obviously, that's out of the question.

I'm open to suggestions if anyone can think of a workaround for the file permissions issue, but I'm guessing there isn't one. I'm also curious if anyone can come up with a reason why SageTV would put such a restriction in place in the first place. It would appear that my only option is most likely going to be to record to one drive outside tRAID and copy the files over nightly. That's not exactly a big deal since I can automate it with a batch job, but it's still an annoying step to have to take.

I'm also kinda of surprised that this isn't an issue for unRAID.

[drvnbysound]

Originally Posted by Skirge01

I finally found some time to test that tRAID release and submitted the results to the developer. Sadly, he's stated that SageTV is apparently "setting restrictions that no other process can access the file" and that I'd need to get SageTV to fix this. Obviously, that's out of the question.

I'm open to suggestions if anyone can think of a workaround for the file permissions issue, but I'm guessing there isn't one. I'm also curious if anyone can come up with a reason why SageTV would put such a restriction in place in the first place. It would appear that my only option is most likely going to be to record to one drive outside tRAID and copy the files over nightly. That's not exactly a big deal since I can automate it with a batch job, but it's still an annoying step to have to take.

I'm also kinda of surprised that this isn't an issue for unRAID.

I'm definitely no expert and won't claim to be, but this makes some sense to me [unless I'm understanding the problem incorrectly]...

I'm assuming this is something that is specific to live recording to tRAID. My assumption for this would be so that no alternate program could access [particularly write to] the file while it's still being recorded to via STV. If unRAID doesn't have this problem, I'd have to assume that they are somehow just disregarding that information to 'lock' the file.

[Skirge01]

Originally Posted by drvnbysound

I'm definitely no expert and won't claim to be, but this makes some sense to me [unless I'm understanding the problem incorrectly]...

I'm assuming this is something that is specific to live recording to tRAID. My assumption for this would be so that no alternate program could access [particularly write to] the file while it's still being recorded to via STV. If unRAID doesn't have this problem, I'd have to assume that they are somehow just disregarding that information to 'lock' the file.

That makes sense. The developer said that he won't override that behavior since it could cause issues for other processes which need to do such a thing. This also makes sense.

It could be my lack of understanding, but something I find odd is that comskip is able to read the file while it's being recorded. I guess that does something similar to unRAID and just "ignores" that lock.

[drvnbysound]

Originally Posted by Skirge01

That makes sense. The developer said that he won't override that behavior since it could cause issues for other processes which need to do such a thing. This also makes sense.

It could be my lack of understanding, but something I find odd is that comskip is able to read the file while it's being recorded. I guess that does something similar to unRAID and just "ignores" that lock.

I honestly have no idea how it works at this level, but just thinking out-loud...

Maybe, but it's STV that's doing the recording and ComSkip is just a plugin of STV. So it may be that it is granted access because it's part of STV; that's like Microsoft Word running spell check while you are actively typing in the document.

[drvnbysound]

BTW: I started a dedicated thread to track my hardware, thoughts, plans, implementation, testing, etc here:

http://forums.sagetv.com/forums/showthread.php?t=60897

[stanger89]

Originally Posted by drvnbysound

I honestly have no idea how it works at this level, but just thinking out-loud...

Maybe, but it's STV that's doing the recording and ComSkip is just a plugin of STV. So it may be that it is granted access because it's part of STV; that's like Microsoft Word running spell check while you are actively typing in the document.

Comskip is a separate process, or in my case ShowAnalyzer is run on an entirely separate machine so there's no way for SageTV to grant access in that case.

Sounds to me like tRAID tries to open the file locked or for write itself (which doesn't make sense to me off hand but I'm not sure how it works).

It's quite easy for two applications to share a file, but they can't both open it for writing.

[drvnbysound]

Originally Posted by stanger89

Comskip is a separate process, or in my case ShowAnalyzer is run on an entirely separate machine so there's no way for SageTV to grant access in that case.

Sounds to me like tRAID tries to open the file locked or for write itself (which doesn't make sense to me off hand but I'm not sure how it works).

It's quite easy for two applications to share a file, but they can't both open it for writing.

Hmm, being a Windows application (tRAID)... is it 'opening' the file so that it can 'write it' to the storage array? [I don't expect you to actually know the answer]

[Skirge01]

I don't quite understand it myself, but I assume the developer of tRAID knows what his application needs to do and that there's no workaround he can currently think of which wouldn't have potential repercussions elsewhere.

The solution I have is pretty acceptable and it's already in place. I have a 750GB drive used by SageTV for recording (pretty much exclusively). Each morning at 9am (a slow time for recordings, but after nightly backups have run), I have a batch job which will copy the files over to tRAID, where I have SageTV's import directory. I've always had SageTV set to scan for new media every hour, so this should tidy things up after the moves. Through some testing yesterday, I found that when the recordings are moved first, SageTV triggers metadata lookups for the "missing" data which is actually still on the 750GB drive and waiting to be moved. However, if I move the comskip .edl files before the recordings, SJQv3 will needlessly retrigger a comskip process, possibly tying up the transfer of the recordings. So, my batch file looks like this:

robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.txt /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.log /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.csv /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.data /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.jpg /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.properties /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.mpg /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.ts /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.edl /mov /xo /r:20 /w:30

The /mov should be self explanatory. The /xo is so that only older files are overwritten, just in case there's some sort of conflict. The /r:20 is the number of retries in case the file is busy and the /w:30 is how many seconds to wait between retries.

Additionally, I have SJQv3 set to look for new media to comskip every 15 minutes. I have those set to not trigger any comskip processes between 9 and 11am, just in case a move is in progress. Finally, I currently have set SJQv3 not to look for comskip opportunities in my import directories.

This all may sound like a lot of work, but it was quickly obvious with some simple tests and took almost no time to set up. The hardest part was figuring out what program to use for moving the files and then the correct parameters to set. Robocopy is absolutely awesome. You can even have it generate a log file if you want.

[drvnbysound]

Can you explain the purpose of the nightly backups? What are you backing up, to where, and why?

[Fuzzy]

I agree this doesn't sound like it's really sage's fault. The file itself, while being written by sage, is 'locked', but not exclusive - meaning it can be read from (which is all he SHOULD need when reading it to calculate parity). This is evidenced by the many other processes people use to read the file while it is recording - from comskip, show analyzer, to sage clients over a network playing back live, etc. He, like usual, is pretty off base on this, and is looking at you as a minor part of his 'customer base' and doesn't feel like changing the way HIS process reads from the file to work around your use case.

[Fuzzy]

Originally Posted by Skirge01

I don't quite understand it myself, but I assume the developer of tRAID knows what his application needs to do and that there's no workaround he can currently think of which wouldn't have potential repercussions elsewhere.

The solution I have is pretty acceptable and it's already in place. I have a 750GB drive used by SageTV for recording (pretty much exclusively). Each morning at 9am (a slow time for recordings, but after nightly backups have run), I have a batch job which will copy the files over to tRAID, where I have SageTV's import directory. I've always had SageTV set to scan for new media every hour, so this should tidy things up after the moves. Through some testing yesterday, I found that when the recordings are moved first, SageTV triggers metadata lookups for the "missing" data which is actually still on the 750GB drive and waiting to be moved. However, if I move the comskip .edl files before the recordings, SJQv3 will needlessly retrigger a comskip process, possibly tying up the transfer of the recordings. So, my batch file looks like this:

robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.txt /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.log /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.csv /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.data /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.jpg /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.properties /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.mpg /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.ts /mov /xo /r:20 /w:30
robocopy \\w7-server\1_1 "\\w7-server\recordings (dvr)" *.edl /mov /xo /r:20 /w:30

The /mov should be self explanatory. The /xo is so that only older files are overwritten, just in case there's some sort of conflict. The /r:20 is the number of retries in case the file is busy and the /w:30 is how many seconds to wait between retries.

Additionally, I have SJQv3 set to look for new media to comskip every 15 minutes. I have those set to not trigger any comskip processes between 9 and 11am, just in case a move is in progress. Finally, I currently have set SJQv3 not to look for comskip opportunities in my import directories.

This all may sound like a lot of work, but it was quickly obvious with some simple tests and took almost no time to set up. The hardest part was figuring out what program to use for moving the files and then the correct parameters to set. Robocopy is absolutely awesome. You can even have it generate a log file if you want.

The fixes to get around the race conditions like this is to speed up the files moving out of, and into, sage's view. To do this, you need to first do a non-data move out of the sage folder (a move on the same drive does not move any data, just the pointer in the filesystem). Then move the files to a non-sage folder on the tRAID volume. Then move them them within the tRAID volume to the sage import folder.

assuming R: is your recording drive, and T: is your tRAID drive:

move R:\RECORDINGFOLDER\*.* R:\PreTransferZone
robocopy R:\PreTransferZone\*.* T:\PostTransferZone /mov /xo /r:20 /w:30
move T:\PostTransferZone\*.* T:\IMPORTFOLDER

[Fuzzy]

Actually went and looked at his response to your thread, and it looks like you are writing to the pool virtual drive - correct? I think what he is saying is that you can instead write directly to the data drives (this is how I had assumed you were doing it). In that case, you would only be writing to a single drive, and tRAID would be doing the parity off that data at it's leisure - possibly - I haven't investigated tRAID yet.

[Skirge01]

@Fuzzy: Thanks for the insight. (I assume that was you over on the tRAID forums, too.) I hadn't thought of the temp storage to alleviate the mobility of the files and that's a pretty good idea. If I write directly to the drives, I'm concerned about having to turn off caching on the pool, as well as the pool writing restrictions. It seems like there's a bit of a risk involved there.

[Skirge01]

Originally Posted by drvnbysound

Can you explain the purpose of the nightly backups? What are you backing up, to where, and why?

I'm backing up all of my computers, plus the SageTV client folders and the SageTV server folders, and the tRAID databases. I back them all up to the array (tRAID). As far as why, it's because I need backups in case something goes wrong.