New Server - SageTV+unRAID under ESXi 4.1

[stanger89]

So, I've posted a number of questions and make a reasonable number of comments about this but I believe I've experimented enough so I'm finally starting. Not sure how long this will take, maybe today, maybe a week, maybe longer, we'll see. I'll start with some background.

A long time ago, can't remember how long actually and can't find the thread I thought I posted with pictures of the parts, I built my current SageTV server. It consists of an Athlon XP 1800+, a couple of recording drives, couple misc drives, and a 1.7TB 8x250GB RAID-5 array. A few drives got upgraded a couple times, namely the 2 recording drives were replaced with 1TB WD Blacks.

This server has worked, and really still does work well. But the reason I've been working on an upgrade is largely my storage situation. When I started looking my storage consisted of 8x250GB drives in my server, my 3-4 misc drives and another 4x1.5TBs in my ReadyNAS, and it was all full. I generally prefer hardware RAID-5 to some of the other options (like WHS's duplication/pool), I like NAS's, but they have both become very expensive. What I really liked was the concept of unRAID, but it's implementation, requiring it's own machine made a non-starter.

That was until ESXi 4, which is available free, and would allow me to run SageTV and unRAID on the same PC hardware. But the thing that finally pushed me over the edge was the release of ESXi 4.1. The idea was possible with 4.0, since it would let you passthrough PCI devices (tuners, USB controllers, etc), but not on my hardware so I kept pushing it off since I didn't want to buy new hardware. But 4.1 has added USB passthrough, and after a couple experiments I determined it worked on my hardware and with my USB devices (R5000, HD PVR, USB-UIRT).

So what follows will be basically the "log" of my setup of my ESXi 4.1 based SageTV+unRAID server.

[stanger89]

So, first things first, the hardware that's going into this thing:

The PC hardware is actually my old HTPC hardware:
AMD Athlon BE-2400 (45W TDP).
4GB memory
Gigabyte GA-MA78GM-S2H
Intel PWLA8391GTBLK PRO/1000 GT Desktop Network Adapter (the onboard GigE isn't recognized by ESXi)

For SageTV I'll be tossing in my two 1TB Caviar Blacks

For unRAID I'll be starting with 3 WD20EARS drives in an iStarUSA BPU-350 hot swap 5-in-3 cage.

I've got unRAID actually running in a VM right now. I sort of cheated, instead of using a USB key for unRAID I created a virtual disk for it. I'm using the free unRAID right now since I've only got 3 disks. I suspect I'll have to tackle either getting setting up passthrough with a USB Key or getting the registration stuff setup when I need to add a drive.

The Drives are direct passthrough, not virtual discs. Reading about ESXi this doesn't do much good for performance, but I like it because I can actually run the same discs in either a "real" or "virtual" unRAID. I created the virtual discs at the ESXi command line (SSH can be enabled in the security options now in the gui). I used the following command:

vmkfstools -a lsilogic -z /vmfs/devices/disks/<real disk> <virtual disk>

Then you just add the virtual disks to the unRAID VM, and then boot unRAID and configure it like normal.

My unRAID VM is actually doing a parity sync right now, performance seems quite good, it's doing about 58MB/sec on the sync.

[stanger89]

placeholder for SageTV config

[Fuzzy]

Looking forward to it. I've also looked into moving my storage to UnRaid, and ESXi would be nice, but like you, the R5000's held me back. Well, that, and I am currently using it as my main client as well. (If it would still pass through PCIe for hardware accelerated video, it would be great). Though, I'm so happy with my new i3 client in my bedroom, I may end up cloning that for the living room as well, and go headless on the server...

[Peter_h]

Really looking forward to this as well.

I've been doing the ESXi debate back in forth in my head for a couple of months now as well. I run a WHS and it's getting to a point where the price of redundancy and duplication is not worth it. I'm up around 11 TB w/ a third of that lost to duplication and not everything is backed up.

I've wanted to move to a linux based software raid for a long time now but i also need a windows server on the network for various reasons such as backing up the family computers, my movies, playon and SageTV Server (I don't trust myself to move the SageTV server over to linux...yet).

Moving to ESXi so that I can run both servers on one machine just makes more sense.

I'll let you work out all the bugs first before I make any moves.

[valnar]

I've had bad luck virtualizing anything that involves real time apps or multimedia. Just virtualizing Untangle on a ESXi 4.0 server (all by itself!) caused my ping times to go up and jitter became a problem. I had constant dropouts with VoIP calls. When I moved it to native hardware the problems went away.

Before you get too involved, just try this. Download this app and run it on a virtualized Windows box (where-ever you want to install Sage) and also your current Sage box.

[stanger89]

Originally Posted by valnar

I've had bad luck virtualizing anything that involves real time apps or multimedia. Just virtualizing Untangle on a ESXi 4.0 server (all by itself!) caused my ping times to go up and jitter became a problem. I had constant dropouts with VoIP calls. When I moved it to native hardware the problems went away.

Well the first step was to actually try this. I managed to record 2 HDHR channels, plus an R5000, plus an HDPVR recording already in this config. But yeah, definitely going to be watching closely before I do anything too drastic.

[stanger89]

I'll leave it go for a while, but actually the VM is looking to be a bit better latency wise. My current server hovers around 500us most of the time with peaks to maybe 1500us or so every once in a while. This is with no recordings going.

The VM, I'd say by eyeball averages about 250us but it a bit less stable (standard deviation is higher if you will). I'm not entirely sure what to think though. I had one >2000us spike on my VM shortly after I started DPClat, which is higher than the max of about 1700us on my current server. However I've not seen anything close to that since.

I just restarted both, and right now (running a couple minutes) the max on the VM is just under 1000us, and the current server is 1431.

Here's the interesting bit. As soon as my R5000 kicked off with a recording, my current server's latency basically doubled, averaging right at 1000us and with relatively frequent excursions into the 1000-1500us area. Say at least 2 within any given 30s window.

[valnar]

Originally Posted by stanger89

I'll leave it go for a while, but actually the VM is looking to be a bit better latency wise. My current server hovers around 500us most of the time with peaks to maybe 1500us or so every once in a while. This is with no recordings going.

Wow. That's surprising. Your original server is a little high.

My desktop averages between 3 and 80us. My Sage server idles around 180-220us, but it's a server board with a bunch of stuff going on. If any of my boxes idled higher than 500us, I'd consider that a problem and hunt down the offending device (driver).

The CPU time slicing will make the variation big, especially when other VM's are running and doing things. This will (may?) cause havoc on your recordings.

[Fuzzy]

stanger's older motherboard is mostly PCI based, not PCIe.. that would probably add to the latency on the USB and such.

[Bizarroterl]

I took a quick read through the Gigabyte GA-MA78GM-S2H specs and didn't see any references to virtualization technology. That may cause some issues with speed. ESXi is designed for MB/CPUs that have all the virtualization technology features.

That's not to say that stanger's setup won't work fine for his needs, it just means that if you're looking to build an ESXi box from scratch you may want to look into server based boards if speed is a major concern.

I'm running ESXi 4.1 set up on a Supermicro uATX server MB and an low power Intel 3100(IIRC) CPU. Currently it's running a Win2008 R2 DC and a Win 2008 exchange server. HDD is a Crucial SSD. The system is very responsive.

I also run an unraid NAS, but wasn't as ambitious as stanger is. My unraid is on a separate MB.

[stanger89]

Oh, the motherboard wasn't picked specifically for this, it was the best spare I've got laying around. That after say a couple months of light playing, seems to work fine.

If I were buying specifically for this, I'd probably get an Intel Core i5 and Q57 motherboard.

[ChaOConnor]

so hypetheically speaking if I did this, would putting sage on WHS be best or like and old XP license I have? I'm liking the idea of un-RAID, so i wouldn't be using WHS for much, just PC backups probably. Can the WHS see the unRaid Drives?

I'm not sure I have a real reason to do this... but I have to always be tinkering. ;-)

[stanger89]

It really depends on the functionality you want. Most likely you wouldn't want to use it's drive pool functionality as that would be redundant to the unRAID's purpose.

As far as if it could see unRAID shares, I don't know, they're standard Windows shares, so I'd think so, but I've never run WHS.

[ChaOConnor]

So I'm thinking then to not even use WHS. Just put my Sage Server on another OS. What are you going to put your Sage install on?

[stanger89]

An old copy of XP.

[ChaOConnor]

okay, i have an old copy of MCE, I could use that I guess, just disable all the old MC stuff. I also have an old Vista license, would the MCE be better for performance reasons?

[Fuzzy]

I don't think you'd gain anything using Vista (for a headless server). XP, especially if you lighten it, would be perfectly fine, and probably better virtualized.

[Gog]

I've been thinking about this recently. Ideally, I would like to be able to get the sage server to run on the linux setup of unraid. Maybe on a hidden directory of the cache drive but I don't think I'll ever get the free time to do so.

Your option seems to be simple enough, as these things go...

I was not sure what your strategy is with this though

Originally Posted by stanger89

For SageTV I'll be tossing in my two 1TB Caviar Blacks

If I was to do this, I'd use my oldest, smallest drive for the sage partition and use the cache drive directly for video recording. That way everything gets parity protection and unraid gains 2 Caviar TB.

Throughput shouldn't be affected much, you're basically going through the network loopback and saving parity calculation for off hours...

Gog

[stanger89]

By that I meant my recording drives. I'm not going to record to the unRAID.