Show "Hoarders", how much space did you use?

[drvnbysound]

Originally Posted by Skirge01

I'm backing up all of my computers, plus the SageTV client folders and the SageTV server folders, and the tRAID databases. I back them all up to the array (tRAID). As far as why, it's because I need backups in case something goes wrong.

Gotcha. I wasn't sure if it was something more.

This is part of why I want to change my configuration; I've never done any sort of backup of STV, other than a few times that I've captured the property and WIZ files before making changes. I also don't have any backup of my movie library - if that drive failed today, I would lose the ~150+ movies that are currently stored on it.

Moving STV over to a VM will allow me to easily capture backups (snapshots), and I feel pretty comfortable with the mitigation that RAID will offer in terms of my movie library. Personally, I don't care all that much about the DVR library; I'm not really deleting anything now, but I really wouldn't be all that upset if I lost the DVR data due to a drive failure - that's just not why I capture it, or how we use STV.

[Skirge01]

Originally Posted by drvnbysound

Gotcha. I wasn't sure if it was something more.

This is part of why I want to change my configuration; I've never done any sort of backup of STV, other than a few times that I've captured the property and WIZ files before making changes. I also don't have any backup of my movie library - if that drive failed today, I would lose the ~150+ movies that are currently stored on it.

Moving STV over to a VM will allow me to easily capture backups (snapshots), and I feel pretty comfortable with the mitigation that RAID will offer in terms of my movie library. Personally, I don't care all that much about the DVR library; I'm not really deleting anything now, but I really wouldn't be all that upset if I lost the DVR data due to a drive failure - that's just not why I capture it, or how we use STV.

Unless the latest version of ESXi has changed, when you have passthrough enabled, you cannot use snapshots.

Also, I just posted this over at the tRAID forums. I'm going to go back to RAID-F. With the fact that you can't record directly to the tRAID pool, it simply doesn't offer a SageTV user anything better than RAID-F does, so why not save $40? I happen to already have a RAID-F license, so it's a very easy decision for me. Now, to figure out how to actually revert.

[stanger89]

Originally Posted by Skirge01

The solution I have is pretty acceptable and it's already in place. I have a 750GB drive used by SageTV for recording (pretty much exclusively). Each morning at 9am (a slow time for recordings, but after nightly backups have run), I have a batch job which will copy the files over to tRAID, where I have SageTV's import directory.

Wild curiosity, can SJQ be setup to move recordings when they are finished?

[drvnbysound]

Originally Posted by Skirge01

Unless the latest version of ESXi has changed, when you have passthrough enabled, you cannot use snapshots.

Also, I just posted this over at the tRAID forums. I'm going to go back to RAID-F. With the fact that you can't record directly to the tRAID pool, it simply doesn't offer a SageTV user anything better than RAID-F does, so why not save $40? I happen to already have a RAID-F license, so it's a very easy decision for me. Now, to figure out how to actually revert.

I can't comment on the ability to do Snapshots with passthrough or not - I have never tried it before. However, I am planning to have a RAID 1 setup on the ESXi server for the primary datastore, so I'll have some backup there. This is another acceptable mitigation for me. I was mainly going to keep Snapshots for various upgrades (e.g. Java updates, etc) which have the possibility to break things, and would allow be to easily revert back.

[Skirge01]

Originally Posted by drvnbysound

I can't comment on the ability to do Snapshots with passthrough or not - I have never tried it before. However, I am planning to have a RAID 1 setup on the ESXi server for the primary datastore, so I'll have some backup there. This is another acceptable mitigation for me. I was mainly going to keep Snapshots for various upgrades (e.g. Java updates, etc) which have the possibility to break things, and would allow be to easily revert back.

You can use standard system restores for those types of things. Don't forget that your VMs function nearly identically to a physical computer, so the same rules generally apply.

[Fuzzy]

Originally Posted by Skirge01

@Fuzzy: Thanks for the insight. (I assume that was you over on the tRAID forums, too.) I hadn't thought of the temp storage to alleviate the mobility of the files and that's a pretty good idea. If I write directly to the drives, I'm concerned about having to turn off caching on the pool, as well as the pool writing restrictions. It seems like there's a bit of a risk involved there.

What actual benefit is the pool giving you?

[drvnbysound]

Originally Posted by Skirge01

You can use standard system restores for those types of things. Don't forget that your VMs function nearly identically to a physical computer, so the same rules generally apply.

Sure... just thinking about an additional layer of backup.

[Skirge01]

Originally Posted by Fuzzy

What actual benefit is the pool giving you?

Primarily, file tetris. I don't like micromanaging my server storage, having to look at how much storage is left on each drive and which one I can put a file onto. Previously, I've had times where I need to put a file or group of files on the server, but there wasn't enough space on a single drive, but there was enough free space in total. What this causes you to do is create 10 drives with the exact same folder layout (perhaps with numbers after them) and you've got no clue where something is actually stored.

[IVB]

Originally Posted by Skirge01

While I'm working through my tRAID recording issue, I'm currently doing exactly as Fuzzy described. I'm not even manually running the re-scan as it's set to happen automatically every so often and it works just fine. Every few days, I'm manually copying them over to tRAID and then deleting them from the recording drive. Honestly, on the off chance the recording issue can't be resolved in tRAID, I'll probably just set up a batch file to copy the recordings over every few days. Since the array is used for computer backups, music, photos, and everything else we want access to, it's much easier to have a single network location to go to than playing file tetris across 10 drives.

Got an HDHomerun Prime & Cablecard today, got it set up within minutes, added a ton of favorites so now I suddenly have to install the 4x4TB I got on Cyber Monday. After setting up tRAID and copying the data over, i'm going to yank the RocketRAID card but leave 1 1TB disk in the case for SageTV direct recording.

Are you still doing a manual copy or did you automate it? I assume I should set up a T:\RAIDTV, and then copy from (1TB) to T:RAIDTV every few weeks?

[Skirge01]

I'm doing a daily automatic copy (code below). I don't see a reason to wait a few weeks to copy things over. I had 2 failed drives I needed to RMA and just got my array back up again the other day, so I haven't gotten back to troubleshooting the problem with recording to the array, but will get back to that in the coming weeks.

robocopy \\W7-Server\1_1\SageRecs "\\W7-Server\Recordings (DVR)" *.txt /mov /zb /xo /r:140 /w:30 /v /fp /np /log+:c:\scripts\txtmovelog.txt
robocopy \\W7-Server\1_1\SageRecs "\\W7-Server\Recordings (DVR)" *.log /mov /zb /xo /r:140 /w:30 /v /fp /np /log+:c:\scripts\logmovelog.txt
robocopy \\W7-Server\1_1\SageRecs "\\W7-Server\Recordings (DVR)" *.csv /mov /zb /xo /r:140 /w:30 /v /fp /np /log+:c:\scripts\csvmovelog.txt
robocopy \\W7-Server\1_1\SageRecs "\\W7-Server\Recordings (DVR)" *.data /mov /zb /xo /r:140 /w:30 /v /fp /np /log+:c:\scripts\datamovelog.txt
robocopy \\W7-Server\1_1\SageRecs "\\W7-Server\Recordings (DVR)" *.jpg /mov /zb /xo /r:140 /w:30 /v /fp /np /log+:c:\scripts\jpgmovelog.txt
robocopy \\W7-Server\1_1\SageRecs "\\W7-Server\Recordings (DVR)" *.properties /mov /zb /xo /r:140 /w:30 /v /fp /np /log+:c:\scripts\propertiesmovelog.txt
robocopy \\W7-Server\1_1\SageRecs "\\W7-Server\Recordings (DVR)" *.edl /zb /xo /r:140 /w:30 /v /fp /np /log+:c:\scripts\edlmovelog.txt
robocopy \\W7-Server\1_1\SageRecs "\\W7-Server\Recordings (DVR)" *.mpg /mov /mt:1 /zb /xo /r:140 /w:30 /v /fp /np /log+:c:\scripts\mpgmovelog.txt
robocopy \\W7-Server\1_1\SageRecs "\\W7-Server\Recordings (DVR)" *.ts /mov /mt:1 /zb /xo /r:140 /w:30 /v /fp /np /log+:c:\scripts\tsmovelog.txt
del /Q \\W7-Server\1_1\SageRecs\*.edl
forfiles.exe /p Z:\SageRecs /m *.txt /d -7 /c "cmd /c del @file"
forfiles.exe /p Z:\SageRecs /m *.log /d -7 /c "cmd /c del @file"
forfiles.exe /p Z:\SageRecs /m *.csv /d -7 /c "cmd /c del @file"
forfiles.exe /p Z:\SageRecs /m *.data /d -7 /c "cmd /c del @file"
forfiles.exe /p Z:\SageRecs /m *.jpg /d -7 /c "cmd /c del @file"
forfiles.exe /p Z:\SageRecs /m *.properties /d -7 /c "cmd /c del @file"
forfiles.exe /p Z:\SageRecs /m *.edl /d -7 /c "cmd /c del @file"

I played with quite a few variations on the above, but this has been working for a few days now. I found theoretical or actual reasons for pretty much everything I chose (both the options and the specific sequence of events).

[IVB]

A)Many thanks, will dissect that.
B) HOLY CRAP I NEED MOAR DISKS!

Going back to Sage/cable, the wife/I set a bunch of random shows we thought would be cool to have. There's a lot of Seinfeld on TV, or other programs. I'm running damn near 50GB/day, not the 100GB/week I spec'ed. I might be pushing that 25-35TB marker sooner than I thought. (And Yes I know I don't have to record it all, but it doesn't hurt. Except the pocketbook for hard disks)

[VCRUser]

Now I don't feel so bad! I have 5tb, including an internal 1tb drive, where all the recording happens, then I convert any show that I don't expect to watch within a few days (i.e. most shows) to a much smaller file-size and store on one of the two 2 tb external drives. I currently have 140gb free on 1 external; 990 gb free on the other external, and 500gb free on the internal. I'm working on archiving some seasons of watched shows that I want to keep, and some unwatched shows that I may or may not want to watch some day. By archiving, I mean I copy the files to a separate external drive, along with the XML files containing their metadata, and also write them to DVDs.

There are also a few shows for which I watched a few episodes, then stopped. I need to decide whether to just dump the remainder or archive them for possible future watching. Because writing DVDs takes so long, I'm considering putting the "maybe I'll watch someday, but am not sure they're worth keeping" shows only on a external hd.

[wayner]

Have any of the hoarders here taken to using Backblaze and $50/year for unlimited cloud backups?

[Skirge01]

As good as that sounds (and I'm suspicious about their "unlimited" claim, especially for those in this thread--AT&T wireless, anyone?), how long would it take me to backup all my data and how long would it take to get it back? How likely would it be that I either hit my ISP's cap or they throttle me? Also, do I really want the NSA telling the MPAA and RIAA that I have all these movies/music and wind up with a settlement letter? Tinfoil hat? Maybe. But, I'll keep MY stuff right here in my house, thank you very much.

[panteragstk]

I've wondered the same. IIRC crash plan will ship you drives to back up the big stuff. I'm still thinking about using them as an off-site backup. Similar pricing to backblaze and they have an unraid plugin.

[wayner]

Originally Posted by Skirge01

As good as that sounds (and I'm suspicious about their "unlimited" claim, especially for those in this thread--AT&T wireless, anyone?), how long would it take me to backup all my data and how long would it take to get it back? How likely would it be that I either hit my ISP's cap or they throttle me? Also, do I really want the NSA telling the MPAA and RIAA that I have all these movies/music and wind up with a settlement letter? Tinfoil hat? Maybe. But, I'll keep MY stuff right here in my house, thank you very much.

To solve issue #1 you need Google Fiber as running full out at 1Gbps you could transfer 10.8TB per day. And they can Fedex you a hard drive to do the restore. Can't help you with the other problem!

[wayner]

Originally Posted by panteragstk

I've wondered the same. IIRC crash plan will ship you drives to back up the big stuff. I'm still thinking about using them as an off-site backup. Similar pricing to backblaze and they have an unraid plugin.

I know with Backblaze you have three restore optoins (1) download files online, (2) have them ship you a USB key and (3) have them ship you an external hard drive. Options 2 and 3 entail an additional charge.

[IVB]

Sigh. didn't make it a year on a 4x4TB RAID5 (12TB, although its really 11TB usable).

Adding more 4TB drives now...