New server - SageTV + WHS 2011 + Nexentastor (ZFS) under vmware ESXi 4.0

[teq]

I am currently experimenting with an (almost) new server mobo (Supermicro X8SIL-F with a Xeon processor and a LSI 9211-8i 8 port SAS adapter card) to find a successor for my trusty WHS v1/SageTV machine.

Originally I was just playing around with WHS 2011 on this mobo when I discovered that WHS 2011 - apart from no longer featuring DE (we all knew about that) - is limited to 8 GB RAM. Now I realize that this might be more than enough for a homeserver (and WHS v1 only included 4 GB), but I have 16GB in my mobo and this artificial limit (can you say 'crippleware'?) just made me angry. So I started to look for other options to make good use of the 16GB RAM and find a replacement for DE.

Inspired by Stanger89's interesting thread , I installed vmware ESXi 4.0. Unlike Stanger89, I didn't go for unRAID but wanted to try a ZFS based solution. Because I have no clue about Solaris, I originally tried FreeNAS which has a very nice UI, but unfortunately the current 8.n version doesn't work in vmware (at least not once I activated passthrough for the LSI SAS adapter). I finally discovered Nexentastor, which is a Solaris based storage product (free community edition as long as you store less than 18TB of data).

So far, I have been successful setting up a 'proof of concept', i.e., I have a running system with vmware, Nexentastor, WHS 2011 and SageTV installed. However, I am not sure whether this will be a viable 'productive solution' as there are many things that I haven't figured out yet (mostly to do with vmware - e.g., how can I configure the system for automatic start/shutdown, etc.). Nonetheless, I thought others might be interested in this approach, so here's roughly how I did it:

• Set up vmware ESXi 4.1
• Created a new vm; installed Nexentastor in this vm
• Set up passthrough for my LSI SAS card; added passthrough to the Nexenta vm
• In Nexenta web interface, configured ZFS RAIDZ1 pool
• In Nexenta web interface, configured an iSCSI target
• Created a new vm for WHS 2011; installed WHS 2011 in this vm
• In WHS 2011, set up an iSCSI initiator (connects to the Nexentastor ZFS iSCSI target)
• In WHS 2011, format the iSCSI target for NTFS and use it for the shared folders
• Install SageTV and other software as required

The biggest issue I had during my setup was that I'm using WD EARS drives, and ZFS doesn't play nicely with these drives. Fortunately there's a workaround for that (here and here), but it took me a long time to find it.

I'd be interested if anyone else has tried ZFS based solutions and what your experience has been so far.

Cheers,
teq

[Spartan]

Interesting concept. What tuners are you using with SageTV?

[mikejaner]

Originally Posted by teq

I'd be interested if anyone else has tried ZFS based solutions and what your experience has been so far.

Cheers,
teq

I just recently (1 month) set up an Openindiana box with Napp-It, which is a dead simple interface like Nexenta. I also just rebuilt my SageTV server so I am hammering out all kinks before I try to switch over to the network for recording. So far I love the ZFS setup. I am exporting via NFS and running VM's on it via two esx servers, while also exporting it via CIFS to my windows boxen for data files etc.... I was doing all this with an ubuntu server and a RAID6 LVM based setup, but after reading about ZFS and it's many advantages over other types of filesystems, I had to make the jump.

[drewg]

Originally Posted by teq

I'd be interested if anyone else has tried ZFS based solutions and what your experience has been so far.

I'm running SageTV on Ubuntu Linux with native ZFS. I have a raid-z made of of 3x 3TB Hitachi drives, with 32GB of L2-ARC on an SSD (other 32GB is my root disk). I described it more fully in the linux forum: http://forums.sagetv.com/forums/showthread.php?t=57024

There is another person running both ZFS and SageTV on FreeBSD. This is what I'd intended to do, but I just ran out of time. I even sponsored a project to get the Linux USB TV tuner drivers running on FreeBSD..

Drew

[sacrament055]

Originally Posted by teq

I am currently experimenting with an (almost) new server mobo (Supermicro X8SIL-F with a Xeon processor and a LSI 9211-8i 8 port SAS adapter card) to find a successor for my trusty WHS v1/SageTV machine.

Originally I was just playing around with WHS 2011 on this mobo when I discovered that WHS 2011 - apart from no longer featuring DE (we all knew about that) - is limited to 8 GB RAM. Now I realize that this might be more than enough for a homeserver (and WHS v1 only included 4 GB), but I have 16GB in my mobo and this artificial limit (can you say 'crippleware'?) just made me angry. So I started to look for other options to make good use of the 16GB RAM and find a replacement for DE.

Inspired by Stanger89's interesting thread , I installed vmware ESXi 4.0. Unlike Stanger89, I didn't go for unRAID but wanted to try a ZFS based solution. Because I have no clue about Solaris, I originally tried FreeNAS which has a very nice UI, but unfortunately the current 8.n version doesn't work in vmware (at least not once I activated passthrough for the LSI SAS adapter). I finally discovered Nexentastor, which is a Solaris based storage product (free community edition as long as you store less than 18TB of data).

So far, I have been successful setting up a 'proof of concept', i.e., I have a running system with vmware, Nexentastor, WHS 2011 and SageTV installed. However, I am not sure whether this will be a viable 'productive solution' as there are many things that I haven't figured out yet (mostly to do with vmware - e.g., how can I configure the system for automatic start/shutdown, etc.). Nonetheless, I thought others might be interested in this approach, so here's roughly how I did it:

• Set up vmware ESXi 4.0
• Created a new vm; installed Nexentastor in this vm
• Set up passthrough for my LSI SAS card; added passthrough to the Nexenta vm
• In Nexenta web interface, configured ZFS RAIDZ1 pool
• In Nexenta web interface, configured an iSCSI target
• Created a new vm for WHS 2011; installed WHS 2011 in this vm
• In WHS 2011, set up an iSCSI initiator (connects to the Nexentastor ZFS iSCSI target)
• In WHS 2011, format the iSCSI target for NTFS and use it for the shared folders
• Install SageTV and other software as required

The biggest issue I had during my setup was that I'm using WD EARS drives, and ZFS doesn't play nicely with these drives. Fortunately there's a workaround for that (here and here), but it took me a long time to find it.

I'd be interested if anyone else has tried ZFS based solutions and what your experience has been so far.

Cheers,
teq

Automatic startup/shutdown is under the Server (click on it, not one of your VM's) then click the Configuration tab and the left hand side (inside the "Software" menu you'll see "Virtual Machine Startup/Shutdown", Click on that, then click on Properties in the upper right hand corner. Tick the "Allow Virtual Machines to start and stop automatically with the system"
Now select your VM and use the "Move Up" button on the right until your VM is under the Automatic Startup heading.

Thats it.

[cenwesi]

ahhh... i set mine up last week as well. I really should have done this setup about 1yr ago. So far my VM is holding and Nexenta is just amazing. I tried them all and ended up on Nexenta. Still experimenting with my HDHomerun. I think i will need a new switch.

BTW, i am running Esxi 5 incase anyone is wondering.

[teq]

Originally Posted by sacrament055

Automatic startup/shutdown is under the Server (click on it, not one of your VM's) then click the Configuration tab and the left hand side (inside the "Software" menu you'll see "Virtual Machine Startup/Shutdown", Click on that, then click on Properties in the upper right hand corner. Tick the "Allow Virtual Machines to start and stop automatically with the system"
Now select your VM and use the "Move Up" button on the right until your VM is under the Automatic Startup heading.

Thats it.

Yes, I had already found that. However, there seems to be no 'clean' way to shutdown the ESXi itself (i.e., the server, not the virtual machines). For example, to shutdown the ESXi server at 12 AM every day. I guess this requirement isn't something you typically need in a datacenter...

Cheers,

teq

[gplasky]

Originally Posted by teq

Yes, I had already found that. However, there seems to be no 'clean' way to shutdown the ESXi itself (i.e., the server, not the virtual machines). For example, to shutdown the ESXi server at 12 AM every day. I guess this requirement isn't something you typically need in a datacenter...

Cheers,

teq

True. That's not a best practice and our datacenter ESXi hosts are only shutdown for ahrdware maintenance or updates, if needed.

Gerry

[teq]

Originally Posted by mikejaner

I just recently (1 month) set up an Openindiana box with Napp-It, which is a dead simple interface like Nexenta. I also just rebuilt my SageTV server so I am hammering out all kinks before I try to switch over to the network for recording. So far I love the ZFS setup. I am exporting via NFS and running VM's on it via two esx servers, while also exporting it via CIFS to my windows boxen for data files etc.... I was doing all this with an ubuntu server and a RAID6 LVM based setup, but after reading about ZFS and it's many advantages over other types of filesystems, I had to make the jump.

Interesting setup - Napp-It looks nice as well. Do your machines run 24x7, or how do you handle the startup/shutdown?

Cheers,

teq

[teq]

Originally Posted by Spartan

Interesting concept. What tuners are you using with SageTV?

Sorry for my late reply. I'm using a anysee E7 PS2 tuner on my production system, but I haven't tried running it on my new server yet...

Cheers,

teq

[m1abrams]

Is the performance of the iSCSI target drive good enough to use as a SageTV recording drive for multiple tuners say 4 HD tuners?

I assume that since the iSCSI target is on the same host your throughput becomes limited by the drive array and not by the virtual network.

Also what are you using for your datastore? Do you have any drive redundancy for it? Still researching my own setup for an ESXi server. Uptime is of high importance for me since I would be putting all my eggs in one basket and if want to avoid excessive downtime due to one drive failure.

[farscapesg1]

I'm currently in the process of finalizing my setup to move away from WHSv1 w/ SageTV to my virtual environment and thought I would share my experience so far...

Hardware

Primary Host
Supermicro X8ST3-F motherboard
Intel Xeon E5640 (quad-core + hyperthreading)
24GB RAM
(5) WDEADS20 drives - RAIDZ + spare
(4) Seagate Barracuda SE 750GB drives - striped mirror ZFS array
(4) 160GB 7200RPM 2.5" drives - striped mirror ZFS array
WD Scorpio Blue 160GB - ESXi install + SAN VM
Seagate 300 GB SATA - local temp datastore
Nvidia DualTV NTSC tuner (to be removed - passed to WHSv2 VM)
BR10i HBA card (device passed to SAN VM)
M1015 HBA card (device passed to SAN VM)
[Onboard] LSI 1068e HBA (device passed to SAN VM)
Intel quad-port gigabit NIC

Secondary Host
Supermicro X8SIL-F motherboard
Intel Xeon 3430 (quad-core, no hyperthreading)
8 GB RAM
160 GB SATA - ESXi install + local datastore
Intel quad-port gigabit NIC

Tertiary Host (ciurrently shut down)
Dell Optiplex 745
8 GB RAM
250 GB SATA - ESXi install + local datastore
Intel dual-port gigabit NIC

Temp iSCSI server (Freenas)
Dell Optiplex 755
4 GB RAM
250 GB SATA

I'm currently running/testing WHSv2 with SageTV and have had no problems with recordings from the Nvidia DualTV card and HDHomeRun (original) for recording sources. Granted, the testing has been pretty limited since it isn't in "production" in the house yet, but I've tried recording from all tuners and didn't notice any glitches in the recordings.

I'm running the following VMs in that environment also...
(2) Windows XP VMs for DVD ripping/conversion
OpenIndiana (v148) + Napp-it for the SAN
OpenIndiana (v151) + Napp-it (testing before migrating my ZFS arrays)
Untangle virtual router
vCenter (Server 2003)
Server 2008 AD server
Windows 7 test VM

Almost all of that is running on the primary host, except one XP VM and the Untangle VM. The secondary is mainly for HA in the event something fails ( which is actually sort of worthless with the SAN in an "all-in-one" setup ) and to help balance the VM load if necessary. I'm seriously considering taking the secondary host out and configuring it as a standalone ZFS SAN running OpenIndiana.

My HDHomeRun Prime should be sitting on my front porch when I get home (since UPS almost never knocks or rings the doorbell, so my wife probably won't know it is there). Will be connecting that up tonight to test out some cablecard goodness and to get things ready to move to the new setup during Thanksgiving.