How are you storing your ripped movies??

[Ericft]

Originally Posted by reggie14

I see from your signature that you already have a pretty big WHS server. Out of curiosity, why are you thinking about making a change?

Great comments....with regard to switching from WHS for my movies, i wanted as direct as possible access to videos...maybe the lags i was seeing were secondary to the HD100 that i was using but the inconsistency (not every bluray every time) let me to believe that it was an issue of drive read/network lag...i upgraded from onboard NIC to addon NIC, made sure everyting was cat 6 and both server and whs were on same switch...
despite that, i ended up with a rosewill 4 bay Esata enclosure that i think does software raid??? rsx-4 model. now the 4 1.5tb drives in that enclosure are alsmost full with bluray rips (raid 5 ) so i wanted to upgrade...
debating whether to just get a second 4 bay enclosure and plug into my second esata slot on the PCI card that came with the original enclosure or go ahead and convert the WHS box into an Unraid server...especially since WHS doesn't give me time machine backup capability natively, it was just a duplicated repository of saved docs, photos, software that i have accumulated for years...
that way, the movies are directly connected to the sagetv win 7 machine via esata with the least possible lag...and still have some raid 5 protection...

[reggie14]

It's just a guess, but maybe your drives under WHS were spinning down after idling for a while. That would cause some lag when trying to start up a movie, but it wouldn't happen every time. It doesn't seem like there ought to be any noticeable lag in moving videos to an external server/NAS. When you use UNC paths for an import directory, the extender gets the video directly from the server/NAS, not through the Sage server. Sure, the Sage server has to tell the extender where to go, but that really should be negligible.

Just so you know, if you don't like to have to wait a couple seconds when you try to watch a video stored on an unRAID server, you can tell unRAID to never spin down drives. There's probably a script to keep drives spun up during certain hours.

Like I said, I've been reasonably happy with unRAID, but I think its a pain to set up, and I don't like the fact that the developer disappears for extended periods of time, and doesn't always respond to support requests. It's definitely not a user-friendly piece of software, and some important functionality is missing from the core software (namely, UPS support, which is rather important in my opinion). But there is an active community of friendly unRAID users that develop plugins (like UPS support) and provide tech support in the forums (not unlike the Sage community).

[PiX64]

Reggie,

Try out unMENU which is an addon for unraid. its super simple to install, and it is leaps and bounds better than the stock ui you can do all kinds of stuff, including install additional packages right through the webpage one of which is UPS support...

not sure how long it has been since you used the customer support, but i just recently like past 1.5 months setup unraid and found it to be super easy to install and setup. There is certainly a little bit of al earning curve, but as far as how to get around, it was pretty straight forward.

u can install like airserver, webserver, ftp, so on and so forth!

http://code.google.com/p/unraid-unmenu/

~Pix64

[reggie14]

PiX64-
I sent Tom at Lime-Tech a few emails early this year. One before I purchased unRAID, and a couple after. I didn't get a response from him on any of those emails. I ended up getting what I needed from the forums, but I was not impressed with Lime-Tech's official support.

Yeah, I installed unMenu right away. It's practically a necessity. But it still bugs me that something like UPS support isn't built in. Considering the data corruption issues you could have with an unexpected shutdown I think you'd be crazy to not have it set up to automatically shutdown after XX minutes during a power failure.

Maybe I'm just getting grumpy, but I haven't been overly impressed with unRAID. It works well for my purposes, I just haven't been impressed. For example:

1. The UI: It's awful. I've been hoping that v5 would have a much improved UI, but I don't think that's going to happen.
2. Documentation is pretty weak. There's a very helpful and moderately complete FAQ, but last I checked many of the entries pointed to threads in the forums of someone working through the problem. In some cases, you have to read through a series of posts from someone working through a problem to figure out what to do.
3. I think drive management is pretty clunky. Why does the the array have to be taken offline while you pre-clear a drive you're installing? I know there's a script, but that should really be built-in.
4. While no other unRAID users seem to agree with me, I also think there should be an automated way to remove a drive and have the data copied over to the other drives. I know this is in WHS.
5. You have to be pretty careful about some poorly documented issues. One issue I ran into early was trying to figure what happens if you press the physical power button. Some documentation suggested that the server would shut down correctly, but an analysis by an unRAID user showed that that wasn't necessarily the case. There's a different script from the user community that does a better job, but even then its apparently a little hard to know for sure that it will work (it did for me).
6. Some stuff in unRAID is just sloppy. For instance, there is an FTP server turned on by default in unRAID using a default username/password from whatever FTP app is included. While the server is turned on, and it will accept logins, it doesn't quite work because of some issue with permissions. But I didn't even know it was on at all until I ran a Nessus scan on my home network shortly after setting up my unRAID server and it popped up as one of the issues.

That being said, once its up and going you can pretty much forget about it.

[PiX64]

yeah i hear ya.

the ui is a bit lackluster, and def could use some imporvements. some basic built in things would be nice as well.

I didnt' mean to bring up a sore subject :-) i just started using it and have been super happy with it. the whole "set it up and forget about it" is pretty awesome....Kinda like SageTV V7 has become! i love it!!

Hopefully in time i don't become "grumpy" about the subject :-)

[davephan]

Originally Posted by PiX64

don't forget in the PRO version you can use a cache drive which GREATLY speeds up system access... i dont' use it but have read if you need to do things like recording to the drives and so forth it is very solid!

Does the cache drive have to be as large as the largest data drive, like the parity drive? I wonder if anyone tried using a SSD for the cache drive. I don't know if that would make a difference from using a conventional cache hard drive.

Dave

[heffneil]

I just wanted to say I find this all very interesting. My 5.4 TB raid array is quickly getting outgrown and I don't know what to replace it with. I used to run a hosting operation and in the early days we used to build giant machines with 20 disks with 3ware raid cards and it was just a totally miserable experience! I am worried about going back and doing that. I was looking at external storage arrays and they are just SO amazingly expensive it is hard to justify for a video library that I have been amassing that will probably be totally available on Netflix shortly! Makes me want to stop ripping ASAP and just drink more beer

unraid sounds like its a royal pain in the butt! I wonder if anyone has network issues with this setup. If I understand it correctly you are writing to a host via the network from your sage server? I don't know the math but would network be the bottleneck or host based hard drives? I would think it was network if you asked me...

What controllers are you guys using with unraid and are you having any issues? I was looking at 6 port adapters from adaptec on ebay and they go for somewhere between $200 and $300 used..

It also sounds like if you are going to use this you absolutely need to use the cache drive but the question is: is the cache drive automatic in how it writes to the final destination? And why is that faster in this implementation? In our environment we could be recording 3 or 4 shows and watching 2 or 3 all at the same time...

Thanks!

[reggie14]

Originally Posted by davephan

Does the cache drive have to be as large as the largest data drive, like the parity drive? I wonder if anyone tried using a SSD for the cache drive. I don't know if that would make a difference from using a conventional cache hard drive.

The cache drive can be any size. Basically, the cache drive is just a temporary storage locations for writes. In practice, you probably want your cache drive to be large enough to temporarily hold all the files you might write to your unRAID array over the course of a day.

I suspect an SSD drive is overkill. Any modern hard drive should be able to do sequential writes pretty close to the maximum speed you'd get over gigabit ethernet, so I don't see a big benefit to using an SSD drive as a cache drive.

[reggie14]

Originally Posted by heffneil

unraid sounds like its a royal pain in the butt!

While I'm certainly not one to shower unRAID with praise, it's really not that bad.

Originally Posted by heffneil

I wonder if anyone has network issues with this setup. If I understand it correctly you are writing to a host via the network from your sage server? I don't know the math but would network be the bottleneck or host based hard drives? I would think it was network if you asked me...

I'm not sure I understand the question. unRAID is just software for a NAS server. It will expose one or more SMB shares on the network to read/write to. When doing writes to an unRAID server (from your Sage server or any other machine on your network), the hard drives and SATA controllers are the bottleneck, not the network. Like I said, without a cache drive you'll top out somewhere around 40 megabytes/sec for write operations. This is because for every block of data you write, unRAID needs to do 4 I/O operations, one read and one write on the data drive you're writing to, and one read and one write to write the correct value to the parity drive.

Partly because of that, you probably wouldn't want to write live TV recordings to the unRAID array. I don't think I would try to do that with a cache drive.

Originally Posted by heffneil

What controllers are you guys using with unraid and are you having any issues? I was looking at 6 port adapters from adaptec on ebay and they go
for somewhere between $200 and $300 used..

Right now I'm just using the SATA controller on my Gigabyte GA-MA785GT-UD3H motherboard. A lot of unRAID users with large setups seem to be using the Supermicro AOC-SASLP-MV8 add-on card. One of those lets you add 8 more drives.

Originally Posted by heffneil

It also sounds like if you are going to use this you absolutely need to use the cache drive but the question is: is the cache drive automatic in how it writes to the final destination? And why is that faster in this implementation? In our environment we could be recording 3 or 4 shows and watching 2 or 3 all at the same time...

It sounds like you want to write TV recordings directly to the unRAID server. I think that's probably a bad idea. It would probably be fine most of the time, but I still think its a bad idea. It seems like write speeds aren't terribly consistent, particularly early in a file transfer. A cache drive would probably help in that situation, but I still think its a bad idea. I think that's one of the only times a cache drive might be useful.

In any event, cache drives are nearly transparent to the user. I believe by default unRAID automatically moves things from the cache drive to the correct location in the protected array at 3:40AM. In the meantime, the file will be accessible off the cache drive. You don't particularly need to worry about whether its on the cache drive or whether its been moved- either way you access the file using the path of its final destination.

The cache drive would help if you're during simultaneous reads and writes in two ways. First, writes to the cache drive don't have all the overhead associated with having to write to the parity drive (meaning, a write to the cache drive is 1 I/O operation, while a write to the protected array is 4 I/O operations). Second, you would be writing recordings/DVDs to a different drive than what your clients are reading from.

[davephan]

Originally Posted by heffneil

unraid sounds like its a royal pain in the butt! I wonder if anyone has network issues with this setup. If I understand it correctly you are writing to a host via the network from your sage server? I don't know the math but would network be the bottleneck or host based hard drives? I would think it was network if you asked me...

What controllers are you guys using with unraid and are you having any issues? I was looking at 6 port adapters from adaptec on ebay and they go for somewhere between $200 and $300 used..Thanks!

I have only been using unRAID for two weeks now, so my unRAID experience is limited. I set up unRAID on a system board that has two built-in sata controllers. One for six drives, the other for four drives, plus two IDE drives can be used. So, a total of 12 drives could be setup, 1 cache drive and 11 data drives. I started with the 'Basic' 3 drive free version, then purchased the 'Pro' pre-configured USB flash drive pair (one for backup). I could add more drives with one new controller for 8 sata drives for about $350 - $450. Or maybe it would be cheaper to add two new controllers for 4 sata drives each.

The set up is time consuming, but not from the actual interaction time. For each drive, I first run the pre-clear script, which takes 28 to 30 hours for a 2 TB drive, then I add the drive to the array, then format the drive. So, it takes more than 1 day to add each drive. It takes several days to move the contents of a 2 TB RAID 1 drive pair from the SageTV computer to the unRAID computer. After the entire contents of the 2 TB RAID 1 drive pair is moved, I power down the SageTV computer and remove another pair of 2 TB drives so they can be added to the unRAID computer.

This process will take many weeks to move data and drives from the SageTV computer to the unRAID computer. I started with 10 sata drives and 2 IDE drives in the SageTV computer and three new 2 TB drives I bought for the unRAID computer. When I get done, I will have one IDE boot drive and one RAID drive pair in the SageTV computer for recordings. All the other drives will be in the unRAID computer. The recordings I want to keep will be moved to the unRAID computer, probably with SJQ.

So far, I haven't experienced any downsides to using unRAID, just the long setup time, but almost all of the time is kicking off processes, then doing something else for many hours. I haven't had any problems with HD playback from the unRAID computer. After my drives are moved to the unRAID computer, I should have about 20 TB of storage, which could be expanded to about 35 TB with the controller(s) and more drives. I don't think I'll need to expand past 20 TB for a long time.

Since unRAID runs on Linux, I might break down and buy the SageTV Linux license, since there isn't trialware for the SageTV Linux version, so I can try to get the whole SageTV system running on Linux.

Dave

[madpoet]

I was an original unRaid user from WAAAY back when 2004 or so I think. Say what you want, my original unRaid box stayed on and ticking for 6 years with 12 drives. I had 1 drive failure that was easy to fix. I'm upgrading it now to a souped up SATA version with 2TB drives, but at the heart of it unRaid is a great product.

Now, for that matter so is WHS. I'm running a WHS server as well and it works great. My problem with WHS is lack of real redundancy without cutting your available drive space in half. It's not a RAID solution, it's "I have 2 copies of everything in case something goes bad". That's not a horrible thing, it's just a reality about how the system works. Ultimately you have to decide what is more important to you.

[CollinR]

Seperate box using low powered miniITX mainboard running XP Pro, 3ware escalade 9500s hardware controller and a couple of RAIDs setup on it. RAID 5 for the DVD/BD and the music and RAID 1 for family photos and videos.

So far so good! It also has seperate DVD and BluRay burners so it can rip multiple sources at the same time if needed.

[heffneil]

Wait network is faster than host based drive? I don't think so.

[reggie14]

Originally Posted by heffneil

Wait network is faster than host based drive? I don't think so.

Gigabit ethernet is certainly faster than you'll ever be able to write to parity-protected unRAID array, which will top out around 40 MB/sec. So in the case of writes to an unRAID parity-protected array, the network is most definitely not the bottleneck- its the hard drives and SATA controllers.

It's not entirely clear what the bottleneck is in the case of writes to a cache drive. I believe the highest write speeds to a cache drive that anyone has reported is 73MB/sec. That's a bit less than what you ought to be able to get over gigabit ethernet, but also probably less than what you ought to be able to get with sequential writes to a modern hard drive. I'd guess the bottleneck is probably somewhere in the networking software stack.

Of course, in your case you seem to want high performance so you can write multiple streams to either the cache drive or the parity-protected array. Either way, writing multiple data streams is going to lower performance. That's not going to matter much from a networking perspective, but hard drive write speeds will take a decent hit. So, in that case I'm pretty sure the bottleneck would be the hard drive, not the network.

[heffneil]

That allure of cheap raid from unraid really is appealing but I just don't think it is for me. I rely too heavily on my library and I don't want another thing to fiddle with as of now I screw around with Sage too much just to enjoy some television. I found a card on newegg in a chassis someone suggested here that does raid 6 and 24 drives. 20 internal which is what this chassis does. I was also evaluating the price point of Promise's chassis raid unit. Also I was pricing out Dell's direct attached storage but they want $800 a drive for a 2 TB nearline SAS.

[briands]

Originally Posted by heffneil

That allure of cheap raid from unraid really is appealing but I just don't think it is for me. I rely too heavily on my library and I don't want another thing to fiddle with as of now I screw around with Sage too much just to enjoy some television. I found a card on newegg in a chassis someone suggested here that does raid 6 and 24 drives. 20 internal which is what this chassis does. I was also evaluating the price point of Promise's chassis raid unit. Also I was pricing out Dell's direct attached storage but they want $800 a drive for a 2 TB nearline SAS.

Take a look at external drive extender. Don't have experience with it, but 5 drive external box connecting with eSATA (card included) looks pretty sweet.

[heffneil]

I think I am going for broke. I just realized I don't need a ton of processing power unless I put SageTV on this box which isn't my intentions. I just want a bad fast storage machine that I can run shares off of. I thought about a cross over cable between my fat storage machine and the sage server so I can minimize the network traffic. I would assume if I had my storage on one machine and sage on another and an extender in another room I would double my network traffic because data would stream from the storage machine to the server to the extender. In this case I would stream directly to the server over a private network and then over the LAN with everything on it to the extender. It could be pretty elegant if it works?

[reggie14]

Originally Posted by heffneil

I thought about a cross over cable between my fat storage machine and the sage server so I can minimize the network traffic. I would assume if I had my storage on one machine and sage on another and an extender in another room I would double my network traffic because data would stream from the storage machine to the server to the extender.

Clients and extenders will stream video off the storage machine, not your Sage server (assuming you use UNC paths for your imported media locations, not mapped network drives). (Oops, apparently that's not right. Thanks paulbeers.)

Original post:
So, the only real case where there could be "double" network traffic is if you record TV to the storage machine and you're watching live TV, since there will be traffic going from SageServer->StorageServer, and StorageServer->extender. Even then it wouldn't really matter. The only machine that sees both data streams is the StorageServer, and one stream is upload and other download. So, assuming you're using full-duplex network gear it won't really matter.

I really think its overkill to mess around with a crossover cable. In that case clients would have to go through your Sage server to stream video, rather than just streaming it right off the storage device. I'm not even 100% sure it would work, since clients and extenders might try to stream directly off the storage device, but it would be accessible for them.

Besides, will you be using gigabit switches and NICs? If so, I'm really struggling to come up with a plausible scenario where the bottleneck is going to be the network. It's slightly different story if you want to use 100 megabit ethernet, but even then just putting a crossover cable between the storage device and the Sage server probably wouldn't help much. I'm not sure how (or even if) you could set this up, but I think if you really wanted to do this you would want to have 2 NICs in the storage device. You would want to set things up so that the SageServer communicates with the storage device over the crossover cable, and everything else on your network communicates with the storage device over its other NIC, which would be connected to your LAN. But again, that seems like overkill if you're already using gigabit ethernet, or a pretty clunky tweak if you're using 100mbit ethernet.

[paulbeers]

Originally Posted by reggie14

Clients and extenders will stream video off the storage machine, not your Sage server (assuming you use UNC paths for your imported media locations, not mapped network drives).
.

Actually Reggie, for the extenders, all data does go thru the server. For clients, they can access network shares directly, but not extenders. So network shares do in fact "double" the bandwidth in and out of the server to extenders (network share transfer to the server and back to the client).

However, even a full 1080i MPEG2 is only about 20mbps and a blu-ray is what maybe 50mpbs at the highest, so a gigabit network is plenty of bandwidth for 6+ extenders even from a network share to the extender.

[reggie14]

Originally Posted by paulbeers

Actually Reggie, for the extenders, all data does go thru the server. For clients, they can access network shares directly, but not extenders. So network shares do in fact "double" the bandwidth in and out of the server to extenders (network share transfer to the server and back to the client).

Thanks for the correction. I didn't realize it works differently for the extenders, although I guess that makes sense.

Though I still think a crossover cable is excessive. For imported media, there will be two data streams going at once, but one will be upload, the other download, so it shouldn't really matter. For live TV recordings I guess network traffic really is doubled (or quadrupled, depending on how you look at it), but I think its probably best to avoid using something like unRAID for TV recordings.