ZFS rocks for SageTV!

[drewg]

So, I just installed a new combination home workstation / sagetv server with 16GB RAM, 3x Hitiachi "green" 3TB drives. My old server had 4GB, 3x 750GB WD "green" drives.

The big problem that I had was that the stupid linux page caching algorithms always decided last nights ~20-30Gb of recordings were more important than my mailer, my web browser, my X11 server, and everything else. So in the morning, it would take ~90 seconds to page in everything I cared about. I tried running with no swap, I tried setting vm.swappiness to every value imaginable, but no joy. Running without swap would occasionally lead to the OOM killer nuking my web browser if I had to start up a VM during the workday, and none of the swappiness settings had any impact.

Back in the old days, DEC OSF/1 (later Tru64) had a tunable (ubc.maxpercent) that would limit the amount of RAM that could be used as a disk cache. You could set this to something like 25, and never have to worry about the kind of problem I was having.

Enter ZFS... On my new system, I'm using native ZFS (from http://zfsonlinux.org/ via the PPA on that page) for my /home and for video / recording directories. With ZFS, you can do cool stuff like:

1) Limit the cache size
2) Disallow file and/or metadata caching on a per fs basis
3) Use an SSD to make a huge cache.

In case anybody is interested, I did the following non-default things:

1) limited ARC (zfs page cache) to 4GB via this line in /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=4294967296 zfs_vdev_cache_size=536870912

2) disallowed the video files from being stored in the page cache via:

zfs set primarycache=metadata tank/video
zfs set secondarycache=metadata tank/video

3) configured a small 32GB SSD as an L2ARC (secondary page cache):

My config looks like:

# zpool status
pool: tank
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
cache
sda3 ONLINE 0 0 0

errors: No known data errors



# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 2.03T 3.30T 7.08G /tank
tank/home 185G 4.11T 185G /home
tank/video 1.02T 3.30T 1.02T /video



After recording 2 NFL games and a movie this afternoon (well more than the 16GB RAM in the box), the machine is still as snappy as ever, and has over 6GB free (~5GB used by ZFS, ~5GB used by everything else)

Drew

[Fuzzy]

You wanna hear something that might be depressing to you.... I've been recording 4-5 programs continuously on a windows server to NTFS drives, with 3GB RAM, for many years, and have never had a problem with using a web browser at the same time...

[drewg]

Originally Posted by Fuzzy

You wanna hear something that might be depressing to you.... I've been recording 4-5 programs continuously on a windows server to NTFS drives, with 3GB RAM, for many years, and have never had a problem with using a web browser at the same time...

Yeah, the linux VM system sucks horribly when under any sort of memory pressure (like 30 tabs of chrome). My goal was to switch to FreeBSD, but I just didn't have the time.

Assuming ZFS didn't work, I was planning to switch SageTV to a VM with raw access to drives.

Drew

[stuckless]

I've been running a couple of years on linux, and I can't say that I've any memory issues. I've only ever recorded 2 HD shows at the same time.

I tried zfs a few years and I had huge performance issues, so I now use linux raid and it's been pretty good... although I miss the features of zfs If you are saying that zfs is really fast, I might switch back.

Something you could have tried as well, is tuning the "swapiness" kernel variable so that the system doesn't swap very often. One of the things with windows (and linux to a certain extent) is that is really wants to move stuff out into swap even if there is lots of ram available. I haven't set this on my server but on my laptop, it made a huge difference when I was running running multiple vmware sessions, eclipse, lotus notes, browsers, etc, and only had 3 gigs of memory.

[PiX64]

Originally Posted by Fuzzy

You wanna hear something that might be depressing to you.... I've been recording 4-5 programs continuously on a windows server to NTFS drives, with 3GB RAM, for many years, and have never had a problem with using a web browser at the same time...

+1.

3+ years on windows xp 32 bit sp3 with 4 GB of ram 3.5 gb usable of course...never had a problem like that before. I reguarly record 3+ hd shows without a hitch.

[DevNull]

I've recently added a HDHR Prime which updated me to 5 HD tuners. Thus far I've been able to record HD from all of them (the Prime uses MythTV as a backend because I'm having issues getting it to work properly in Sage) simultaneously while watching 1 HD feed on a HD200.

This is a system with a Athlon 64 X2 4200+ CPU (a whopping 2.2GHz) and 2.5G of RAM running CentOS 6 (64-bit). The OS runs in a ext3 partition, but my video storage uses XFS. I currently have swap disabled, and "free" is telling me I have 1.3GB cached.

Admittedly, I typically don't run X (run only in mode 3), but it does run Sage, MythTV, Apache, Samba, and MySQL.

[drewg]

Originally Posted by stuckless

I've been running a couple of years on linux, and I can't say that I've any memory issues. I've only ever recorded 2 HD shows at the same time.

I tried zfs a few years and I had huge performance issues, so I now use linux raid and it's been pretty good... although I miss the features of zfs If you are saying that zfs is really fast, I might switch back.

Did you try native ZFS, or ZFS under fuse? Or a FreeBSD or Solaris?

With 3 days under my belt, native ZFS seems blindingly fast so far. For raidz-1 with a tiny SSD l2-arc, I'm seeing 210MB/sec reads and 160MB/sec writes.

Originally Posted by stuckless

Something you could have tried as well, is tuning the "swapiness" kernel variable so that the system doesn't swap very often. One of the things with windows (and linux to a certain extent) is that is really wants to move stuff out into swap even if there is lots of ram available. I haven't set this on my server but on my laptop, it made a huge difference when I was running running multiple vmware sessions, eclipse, lotus notes, browsers, etc, and only had 3 gigs of memory.

Yes, I tried that. The kernel just insisted that the 25GB I'd recorded and/or watched was much more important than my thunderbird, chrome, etc.

Back when I had to hack sagetv with an LD_PRELOAD to fix some dvb issues, I was thinking to try to add an LD_PRELOAD of open() to make it so that an open of any .mpg would get the O_DIRECT flag OR'rd in.. But that would have meant a native comskip (didn't feel up to hacking wine)..

Drew

[stuckless]

It was zfs under fuse... there wasn't a native zfs for unbuntu at the time that I tried it.

[drewg]

Originally Posted by stuckless

It was zfs under fuse... there wasn't a native zfs for unbuntu at the time that I tried it.

Yes, I tried that the last time I built a system in 2008, and the performance was indeed ... lacking. Here is bonnie output from the native ZFS.

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
nol2arc  20480 130721 99.2 150756  9.7 84240 12.3 109947 76.4 183958 18.8 140.0  1.2
l2arc    20480 129911 99.2 149446  9.7 87720  9.8 140813 94.6 211060 11.1 596.2  3.0

This is with 3 HITACHI Deskstar 0S03230 3TB 5400rpm drives in raidz-1. The l2-arc is on a 32GB partition of a Crucial M4 CT064M4SSD2 SSD.

[mikejaner]

This is great news! I was told by a buddie that ZFS was only availible by going through FUSE. After looking at the site, I was a little thrown off by seeing that the "0.5.2 stable release does not yet support a mountable filesystem". What and how do I even process that? Are you using, 0.6.0-rc5 release candidate?

[drewg]

Originally Posted by mikejaner

This is great news! I was told by a buddie that ZFS was only availible by going through FUSE. After looking at the site, I was a little thrown off by seeing that the "0.5.2 stable release does not yet support a mountable filesystem". What and how do I even process that? Are you using, 0.6.0-rc5 release candidate?

Yes... It had been in RC so long that I finally gave up waiting & took the plunge.

I'm running it on the Ubuntu 11.10 "oneiric" beta. I'd really wanted to use CentOS 6, as I don't want to have to re-install the machine for a while. But the ZFS package for CentOS does not get automatically rebuilt after a kernel update, so I was worried that I'd run into issues with a kernel update. And Ubuntu 10.04 does not support TRIM for SSD, so I was boxed into running something 'new' .. so I chose the latest short-term Ubuntu.


Drew

[mikejaner]

Originally Posted by drewg

Yes... It had been in RC so long that I finally gave up waiting & took the plunge.

I'm running it on the Ubuntu 11.10 "oneiric" beta. I'd really wanted to use CentOS 6, as I don't want to have to re-install the machine for a while. But the ZFS package for CentOS does not get automatically rebuilt after a kernel update, so I was worried that I'd run into issues with a kernel update. And Ubuntu 10.04 does not support TRIM for SSD, so I was boxed into running something 'new' .. so I chose the latest short-term Ubuntu.


Drew

Ahh, I have been messing around with it on OpenIndiana with napp-it, but this would let me integrate it into my current Ubuntu server.

[zoop]

Originally Posted by DevNull

I've recently added a HDHR Prime which updated me to 5 HD tuners. Thus far I've been able to record HD from all of them (the Prime uses MythTV as a backend because I'm having issues getting it to work properly in Sage) simultaneously while watching 1 HD feed on a HD200.

Not to hijack a thread completely, but would you mind expanding on your experience with the HDHR Prime? I am interested in this device but if it doesn't work right with Sage then I am probably going to skip it... Given how nicely my old HDHR works I have been looking forward to it's release.

thx

/jer

[kbyrd]

Originally Posted by zoop

Not to hijack a thread completely, but would you mind expanding on your experience with the HDHR Prime?

+1 to this. I was under the impression HDHR Prime (that's the CableCard version, right?) only worked with Windows Media Center.

[zoop]

Originally Posted by kbyrd

+1 to this. I was under the impression HDHR Prime (that's the CableCard version, right?) only worked with Windows Media Center.

HDHR Prime works fine with non-WMC clients (MythTV for example) but only for copy-freely marked content. It looks like folks have it working fine with SageDCT on Windows, but I was wondering what the Linux version does with it....

thx

/jer

[DevNull]

Originally Posted by zoop

HDHR Prime works fine with non-WMC clients (MythTV for example) but only for copy-freely marked content. It looks like folks have it working fine with SageDCT on Windows, but I was wondering what the Linux version does with it....

thx

/jer

That what I am hoping to try and resolve. The directions for SageDCT seem to be pretty basic for Windows, but I'm having trouble getting it to function properly using a Linux server. I don't have any previous experience with SageDCT, so it could easily be a config issue on my end.

[najames]

Originally Posted by drewg

With 3 days under my belt, native ZFS seems blindingly fast so far. For raidz-1 with a tiny SSD l2-arc, I'm seeing 210MB/sec reads and 160MB/sec writes.
Drew

Although I've looked at unRAID and FreeNAS briefly, I'm currently testing Lubuntu 11.04 32bit running of a 16GB USB drive, Sempron 140 unlocked, Asus M4A785-M, 2x1GB RAM, Intel PCI-e NIC (2W), 300W Seasonic "Bronze" PSU. I created a 6x2TB mdadm soft RAID5 with XFS. It idles on the desktop at 64W, when drives sleep it drops to 43W, during post it briefly hits ~130W. Voltage settings are heavily tweaked though. This distro used less power usage than some others. Default "Screensaver" used 7W, ouch, it's gone now. Power testing is with KillAWatt, measured "at the wall".

I installed FreeNAS 8.x, opened up Samba to everyone and gave it a quick test, no problems but I wanted to look at some other distros. I also used unRAID for a while with 3 drives in the "free version" again no real problems but I felt limited to what I could do with it, ditto for FreeNAS.

I was going to use Mint LDXE but they were forcing me to use either Firefox or Epiphany (forces "Mint search credits" I suppose), no restrictions with Lubuntu and it installs Chrome by default, which I wanted. I have a nice Conkey panel set up that shows everything I need at a glance, from power to drive space. I'm not 100% this is my final system, but it has worked well now for a couple weeks after I weeded out some initial glitches. LXDE versions are fairly stripped and I only had to delete about 5-10 unwanted apps for a lean/clean setup, only using 144MB of RAM now on the desktop with Conky running. Uptime has been 6 days now, but it's not seen any use other than testing, and converting one ts movie to a MKV, until I get my new SageTV box working between recordings for max WAF, can't miss NFL games either for HAF. First SageTV on Vista test failed because I moved my Vista hard drive to a new PC using sysprep. It looked ok but old IDs were still there, not new ones created, SageTV 7 on Vista was using local, couldn't map to the Linux RAID, tuners were wonky, etc. I did a reinstall from within Vista, reactivated, and it seems happy now.

Using 3GB file size to exceed any caches with Iozone

iozone -s 3G /media/long RAID ID string/Documents/iozone.dat
read speed is 457.1MB/s, reread is 473.0MB/s
write speed is 195.3MB/s, rewrite is 195.8MB/s

[drewg]

Originally Posted by najames

Although I've looked at unRAID and FreeNAS briefly, I'm currently testing Lubuntu 11.04 32bit running of a 16GB USB drive, Sempron 140 unlocked, Asus M4A785-M, 2x1GB RAM, Intel PCI-e NIC (2W), 300W Seasonic "Bronze" PSU. I created a 6x2TB mdadm soft RAID5 with XFS. It idles on the desktop at 64W, when drives sleep it drops to 43W, during post it briefly hits ~130W. Voltage settings are heavily tweaked though. This distro used less power usage than some others. Default "Screensaver" used 7W, ouch, it's gone now. Power testing is with KillAWatt, measured "at the wall".

This box idled at 53w until I added my 3 tuners & USB sound dongle, which pushed it up to about 70w.

Originally Posted by najames

iozone -s 3G /media/long RAID ID string/Documents/iozone.dat
read speed is 457.1MB/s, reread is 473.0MB/s
write speed is 195.3MB/s, rewrite is 195.8MB/s

That's about what I'd expect, given 6 drives.

FWIW, in my machine with a 17GB file (16GB RAM), I see this from IOzone:
read speed is 1,169MB/s, re-read is 1195MB,s
write speed is 506MB/s, re-write 117MB/s.

Interesting affects from the SSD cache

Drew

[najames]

Sidetrack, my harder configuration of Linux works, but the "easy Vista 64 server" is kicking my butt and it doesn't feel good.

No recordings were scheduled so I stayed up late last night to get SageTV 7 working, (HDHomerun + HDPVR & USB-UIRT). About 1am it all worked great from the HD200, scheduled recordings and went to bed. This morning the extender said there were no cotton pickin servers. Grrrr. I came in to check it and it was stuck on a black screen saying something like win.loader failed. There was over 21,000 lines of crap in system logs, reinstalling from scratch again now. Maybe time to call Bill Gates tomorrow.

[zoop]

Originally Posted by DevNull

That what I am hoping to try and resolve. The directions for SageDCT seem to be pretty basic for Windows, but I'm having trouble getting it to function properly using a Linux server. I don't have any previous experience with SageDCT, so it could easily be a config issue on my end.

FYI SageDCT does work as advertised with SageTV Linux, no whammies encountered so far. Utilizing a UNC path for DCT to save via samba on the Linux side.

Andy was very helpful while i tried to sort out the DCT installation on Windows. Probably the most confusing bit is that you have to have Windows Firewall _on_ for the RTP stream to be processed properly. This is the case both in XP and Win7 (DCT works fine for me in both), although in XP at least i needed to disable the firewall in order for the DCT setup utility to "scan" for the HDHR Prime initially. YMMV.

/jer