ESXI build

[jdc]

I've been doing some research on limetech and have modeled this after some suggestions on there. However, was wondering if anyone has some recommendations for this build since my primary focus is for Sage, and also storage for family pics/vids.

After reading some of the other posts, it looks like maybe the SSD isn't the best option and maybe I should get a 7200rpm drive for sage (or more for any other production OSs) and RDM it as a boot drive into ESXI? I have a ton of 500GB 2.5" drives I can use to record to, and it seems like that is the best option?

Also, I am strictly OTA and DVDs but do record up to 4 shows and we watch everything from recordings, so I might have 4 recordings and 3 playing at a time at the most right now. More reading here has led me to believe that unRAID might bottleneck a bit with this max. Thoughts? I think I will record to dedicated drives and move the recordings over periodically. Can unRAID handle 3 playbacks at once?

Finally, this isn't a cheap build, but looking to get somewhere close to stable for awhile and just buy new components every 3 or so years, if necessary.

Components: Some of these items are out of stock at newegg but using the links for reference. I'm still shopping for the best prices.
CPU: Intel Xeon Quad-Core E3-1240V2 3.4GHz 5.0GT/s 8MB LGA1155 Processor, Retail BX80637E31240V2?
Motherboard: Supermicro DDR3 1600 LGA 1155 Server Motherboard X9SCM-IIF-O
RAM: Kingston Technology ValueRAM 32GB Kit (4 x 8GB) 1600MHz DDR3 ECC CL11 DIMM with TS Intel Desktop Memory KVR16E11K4/32I
Controller: IBM ServeRAID M1015 46M0831
SAS Expander: Intel RAID Twenty-four port Expander Card RES2SV240
Case: Norco RPC-4224 4U Server Case or SuperMicro SC846TQ-R900B (view thread)
PSU: Corsair Professional Series HX 750 Watt ATX/EPS Modular 80 PLUS Gold (HX750)
Datastore Drives: Samsung Electronics 840 Pro Series 256 GB MZ-7PD256BW and Seagate 4 TB 64MB Cache ST4000DM000
USB Flash: 8GB Kingston (spare parts)
Fans: Noctua NF-P12 Fan 3-Pack and 2 x Enermax Fan Uc-8Eb Enlobal
Parity Drive: Seagate Barracuda 2 TB ST2000DM001
Data Drives: 20 x Seagate Momentus 7200.4 ST9500423AS 500GB 7200 RPM (spare parts)
Cache Drive: Western Digital WD Black WD1002FAEX 1TB

Hypervisor: esxi 5.1
Guest OS: Win 7 (SageTV), Win 7 or Linux (Sab, sb, mysql, etc), unRAID,
Guest OS optional (testing or playing): Win 8, Mac OS Mountain Lion, Linux


Thanks for your help.
Jeff

[BobPhoenix]

I have a similar build for two of my SageTV servers.

Two ESXi servers:
ESXi: 5.0
VMs: 1 unRAID each, 1 SageTV (Win7x64) each, 1 WHS v1, various other VMs that are off most of the time.
CPU: E3-1230 V1 (Sandy Bridge - needed for AVerMedia duet)
MB: Tyan S5512GM2NR with 16GB memory each
Case: 2x Lian Li V2000 tower (will hold 12 HDD internal and 2 5x3 cages each)
Controllers: 2x IBM M1015s connected to 2x SAS expanders for unRAID VMs, 1 SASLP-MV8 for WHS v1 VM, 1 Highpoint 1742 for SageTV VM to 5 bay external encloser with port multiplier, 1 M1015 for other SageTV VM
HDDs: In process of migrating data but current specs: 13x 3TB (parity and data) unRAID - 5x 3TB SageTV on one, 1x 3TB (parity), 16x 2TB (data) unRAID - 6x 3TB SageTV on other, 4x 1TB laptop for WHS v1 in 4x1 cage.
Datastore: 300GB WD Raptor (had them) 7200rpm would have worked fine both have ~175GB free since I used Thin Provisioned boot drives for SageTV VMs and other miscellaneous VMs.
USB2.0 card: 1 for 2 HD-PVRs on one SageTV VM (passed through).
Tuners: (passed through to SageTV VMs) 1 Hauppauge HVR-2250 on each VM, 1 AVer Media Duet each VM, 1 additional Duet on one SageTV VM, 1 HDHR on other SageTV VM, maybe still 2x HVR-950Qs also.

2 other SageTV servers NOT virtualized one in Living Room as playback device.

NOTE: unRAID should have no problem serving up multiple files simultaneously at all even if they are from a single drive. Remember reading does not require parity access so full drive speed read access is possible. The only time you might have a problem is if you were writing to the array at the same time. If it becomes a problem then get a cache drive and use it for the shares you are writing to. Then unRAID will move it from the cache to the array at 3:40am when you are not likely to need high speed read access.

I had to stick with Sandy Bridge CPU because the AVerMedia Duet tuners are not recognized when an Ivy Bridge CPU is installed. I even had to downgrade my bios back to Sandy Bridge only compatible bios to get the Duets to be recognized. Don't think that has changed for SuperMicro or Tyan. I have an Ivy Bridge Intel DH77EB MB and Ivy Bridge CPU that I can only use HVR-2250s on currently. Supposedly Intel put out a new bios that will make my Duets work but I haven't had time to upgrade the bios yet to test. If I didn't need the Duet for Multi path issues I probably would have gone Ivy Bridge on my ESXi builds.

[jdc]

Thanks BobPhoenix. I saw in another thread you followed Johnm's build on lime. Are you running esxi and unRAID from a thumb drives?

[PosterBoy]

What kind of tuners are you using? My biggest pain with ESXi (I used 4.0, maybe 5.1 is better) was getting the pass through stuff to play nicely. I got it all working in the end stabley, but the limits imposed by the pass through made it less than ideal. On issue I had is that you cant do a snapshot backup on a running VM if you pass through hardware other than USB stuff (maybe this is fixed?). In the end I got rid of ESXi and just run server 2008 r2. I have a synology NAS for my storage needs. The are quite hackable and run linux, so installing any linux apps I need is pretty easy. As such I really didnt need any other VMs, so I just ripped it up and installed server 2008 r2 with the plan to run any VMs I need on that.

That was my experience, but yours will almost certainly be different. Nice setup you are looking at

[BobPhoenix]

Originally Posted by jdc

Thanks BobPhoenix. I saw in another thread you followed Johnm's build on lime. Are you running esxi and unRAID from a thumb drives?

Yes. My Tyan S5512GM2NR MBs have two internal USB ports so that makes it easy. If I was still using my X9SCM-F MB I would get something like this.

[jdc]

Originally Posted by PosterBoy

What kind of tuners are you using? My biggest pain with ESXi (I used 4.0, maybe 5.1 is better) was getting the pass through stuff to play nicely.

That was my experience, but yours will almost certainly be different. Nice setup you are looking at

I'm strictly OTA and am using two dual HDHRs, so I don't have the passthrough issue. I might consider going with a HDHR-Prime with a cablecard if I ever go back to comcast. I think my only concern is how long I will be using Sage, but I have 3 HD300s and 3 HD200s (1 spare), so I figure I'm set for at least another year or three. Either way, I think esxi will fit my needs long term.

Originally Posted by BobPhoenix

Yes. My Tyan S5512GM2NR MBs have two internal USB ports so that makes it easy. If I was still using my X9SCM-F MB I would get something like this.

Thanks. I was reading that esxi 5.1 passthrough was broken. Any idea if it is fixed in 5.1+? I actually rebuilt my AMD setup with esxi 5.1 (johnm's thread), but the passthrough was not available only to read about it recently here when I was researching Sage esxi setups.

Also, would you go with an a few smaller SSDs (60GB or so) for my 'production' drives and RDM them or stick with some decent 7200rpm drives?

[BobPhoenix]

Originally Posted by jdc

Thanks. I was reading that esxi 5.1 passthrough was broken. Any idea if it is fixed in 5.1+? I actually rebuilt my AMD setup with esxi 5.1 (johnm's thread), but the passthrough was not available only to read about it recently here when I was researching Sage esxi setups.

Also, would you go with an a few smaller SSDs (60GB or so) for my 'production' drives and RDM them or stick with some decent 7200rpm drives?

I'm on 5.0 so no problems with passthrough but that doesn't play well with Ivy Bridge. It works but the MB chipset doesn't display correctly. As far as I know the only part of pass through that is broken in 5.1 is USB devices and PCI port passthrough - PCIe port passthrough works fine. Also from what I remember one of the updates to 5.1 fixed PCI port passthrough but not USB device passthrough - but that may have been fixed now as well. If you can't get USB device passthrough to work on 5.1+ then I would just get a USB card and pass it through instead. SSD and ESXi - I just don't trust that Trim works in Windows on a RDM'd SSD so I swapped mine out in favor of my Raptors. What I though about trying however is a small (10-20GB - maybe less) virtual HDD to boot from (drive c) but installing Windows to D drive that was on a pass through controller that was SSD drive. I just never got around to trying it that way. I am using virtual "Thin Provisioned" drives for my boot drives on all but my WHSv1 install - that is a 320GB laptop drive RDM'd.

[jdc]

So I received everything for the build. I can ask on the vmware or lime forums, but I'm not sure how it all works.

I'm planning on using a reverse breakout cable to connect 4 drives from the motherboard. Then I'll connect the remaining 20 drives using the m1015 and RES2SV240.

In vmware, I'm assuming I should be able to select one of the raptors to RDM. Do you know if this is true? Otherwise, I would need some sort of tray adapter to direct connect the SATA port from the drive to the motherboard. Any thoughts?

[BobPhoenix]

Originally Posted by jdc

So I received everything for the build. I can ask on the vmware or lime forums, but I'm not sure how it all works.

I'm planning on using a reverse breakout cable to connect 4 drives from the motherboard. Then I'll connect the remaining 20 drives using the m1015 and RES2SV240.

In vmware, I'm assuming I should be able to select one of the raptors to RDM. Do you know if this is true? Otherwise, I would need some sort of tray adapter to direct connect the SATA port from the drive to the motherboard. Any thoughts?

I'm assuming you are planning on mounting the RES2SV240 to an empty slot cover on your Norco 4224? So you should have 3 available - 2 easily one a little tight to the X9SCM-iiF MB potentially. So here is how I would mount everything if I was doing it - expect yours would be different. I would use two of these and mount two laptop drives directly to MB one a datastore drive - you have to have one - and the other for my WHSv1 boot drive. Then use the remaining 4 MB connectors like you are suggesting with a reverse breakout cable connecting MB ports with a 4224 backplane and RDM those drives to whichever guest VM I needed to. In my case I might forgo the reverse breakout completely and connect all 4224 backplanes to the RES2SV240 and M1015 (just one cable to RES2SV240 works fine that is how I currently have mine) so that I could max out unRAID. All my other VMs would have virtual drives on the datastore laptop drive except for WHSv1 boot drive - but that is me.

Note - you should be able to RDM any drive. The type (of SATA) drive does not matter. So a Raptor should RDM just as well as the laptop drive I currently have RDM'd for WHSv1. I've only used the -r parameter to VMKFSTOOLS (spelling my be off - going from memory) but I hear the -z parameter gives the OS more control over the drive. So I may do that in the future if I have to RDM a drive.