SageTV performance tweaks: What do you do to gain extra SageTV performance?

[mkanet]

Presuming people have sufficient CPU/Physical RAM on their system to begin with; I am curious what fellow sageTV users do to gain extra performance out of their relatively extensive sageTV setups. I've included a few tweaks which helped me the most:

1- High-end large capacity SSD:
For me, the single biggest performance enhancement by far was getting an high end 256GB SSD HDD (divided into 2 partitions/logical drives). The one I use is the Corsair P256. However, Crucial and some OCZ brand SSDs are great as well.

EDIT: Based on feedback from a couple of SSD sageTV users (thank you for correcting me on this), I should make it clear that this tweak does NOT directly affect general sageTV performance on a basic sageTV setup with abundant RAM/CPU. However, it does increase performance significantly in certain circumstances (which my family use alot) such as retrieving large uncached fanart (or other large uncached files hosted on the SSD) and functions which require certain driver/OS/3rd party driver dependencies for sageTV. This tweak is better suited for servers which do more than just serve sageTV clients; such as Squeezebox services, Virtual Clone Drive, R5000 drivers (version 3.0c), webservers, etc all of which are utilized by my sageTV setup.

My SSD set me back $750 for a 256GB disk, but completely worth it. There are smaller capacity discs that have great performance and lower prices as well. It's similar to having everything on your hard disks kept in RAM at all times; except with the big advantage that files/temp files written are non-volatile (meaning you wont lose anything if you lose power to the PC). Since there are no movable parts (unlike traditional hard drives), there's no significant heat generated and no noise. Imagine a single, tiny device comfortably outperforming several WD Raptor high RPM HDD's raided together without any noise or heat.

The idea is that you can read files into RAM so quickly, its like having the files already in system RAM/RAM cache. Since there are no movable parts in the SSD drive, you can read/write to files at the blink of an eye.

This is how I have my SSD partitioned and used:
Drive C: (OS and Apps; including java/sageTV program directories)
Drive Z: (System temp files and swapfile (set to not grow or shrink at 12GB), SageTV Wiz.bin, sageTV fanart, etc)

I still have an external RAID for keeping my ripped bluray movie collection and TV recordings; since it's not practical to use SSD's for this; as SSD's these days aren't much larger than 256GBs. Some people with deep pockets go as far as to raid-0'ing multiple SSDs with a great HDD controller.. to give you an idea how much power that is.. in some cases, it can even exceed SATA-3 specs!

2- limited_carny_init=true properties setting:
(For faster sageTV startup) I just found out about this tweak not to long ago. This tweak seems to be ideal for people with large/older databases. This will speed up SageTV's startup process. The speed increase will be less for smaller/newer TV history databases.(wiz.bin). For me, it made a significant difference. You can read about it in more detail here. As far as I know, the only thing it adversely affects is intelligent recordings.

3- JVMMaxHeapSizeMB:
Many people have known about this tweak for a while. However, I thought I'd put it here just in case some people might have missed this. In several cases with more elaborate setups, the default value for this settings is way too low; which can cause severe performance issues and even instability. Here's a thread which tells you more about this setting and how to change it: http://forums.sagetv.com/forums/showthread.php?t=46741

4- Disable software based firewalls:
You can safely do this if you already have a dedicated hardware Internet router (which inherently offers firewall protections via NAT). There's no point in having two firewalls; especially if you have a good Internet Router with SPI, etc).

-MKANET

[Fuzzy]

I guess the first thing I'd do is ask what you are using to gauge 'SageTV Performance'.

[mkanet]

Significant responsiveness, less lag in accessing/processing/displaying data that affect sageTV server, clients, etc.. which don't require resource performance tools to notice a difference.

Originally Posted by Fuzzy

I guess the first thing I'd do is ask what you are using to gauge 'SageTV Performance'.

[Spectrum]

Originally Posted by mkanet

Since there are no movable parts (unlike traditional hard drives), there's no heat generated and no noise.

SSD's may not generate much heat, but they do generate some. Mechanical motion is not required to generate heat. Your CPU doesn't have any moving parts but you wouldn't run it without a heatsink right?

One of the big reasons that cpu design has shifted from high clock speed to more cores is heat. The Prescott CPUs got up to 115w TDP; that is a lot of heat coming off a tiny chip. The only way to keep making the transistors faster would have been to provide more power to the gates, which generates more heat, which requires stronger cooling systems and lowers chip reliability. Hence the shift to multi-core CPUs. The reason Intel/AMD can provide modern chips that still run in the 3 GHz range without consuming more power is due to shrinking the die. Chips made with a 45nm process can provide the same speed with less heat than chips made with a 90nm process. But I digress! I don't want this to become a conversation on ASIC design

I'm glad you posted the carny limit setting, I hadn't changed that since I upgraded to V7.

[GKusnick]

Originally Posted by mkanet

Significant responsiveness, less lag in accessing/processing/displaying data that affect sageTV server, clients, etc.. which don't require resource performance tools to notice a difference.

But your initial presumption was for systems with adequate physical RAM. If you have enough RAM to hold the entire working set (including Wiz.bin and the code that accesses it), then disk speed shouldn't make a dramatic difference. If it does, that's a sign you were doing excessive paging in the first place, and the usual solution to that is more RAM (or trim your working set by packaging code and data more efficiently).

I tend to agree with Fuzzy here: before throwing expensive hardware at a performance issue, it helps to have a clear idea of exactly what you're trying to fix and where the bad performance is coming from.

[mkanet]

Thanks for your reply. Hopefully, there will be other people who might find other performance tweaks they can share.

Actually, the heat generated from current generation, high-end consumer based SSDs' on board processors is trivial; hence why I worded it the way I did. It was in respect to heat generated from mechanical HDD.

It's very difficult to explain the value of an SSD in respect to "real life" system/app performance gains and other benefits compared to traditional HDD without seeing a system that didnt have it before, then upgrading to one. I dont really bother with posting IOPS, Sequential read/write per second, etc comparison charts. There are plenty of website that prove that. I just look at what would be noticeable even to a non-technical person such as my wife.

But, bottom-line, seeing is believing. Hairs can be split with disadvantages/advantages of SSDs; but bottom line, you get one REALLY big "real world" performance boost in most apps and OS subsystems... just ask anyone who upgraded to a very good SSD from a high end HDD.

Thanks again,
Michael

Originally Posted by Spectrum

SSD's may not generate much heat, but they do generate some. Mechanical motion is not required to generate heat. Your CPU doesn't have any moving parts but you wouldn't run it without a heatsink right?

One of the big reasons that cpu design has shifted from high clock speed to more cores is heat. The Prescott CPUs got up to 115w TDP; that is a lot of heat coming off a tiny chip. The only way to keep making the transistors faster would have been to provide more power to the gates, which generates more heat, which requires stronger cooling systems and lowers chip reliability. Hence the shift to multi-core CPUs. The reason Intel/AMD can provide modern chips that still run in the 3 GHz range without consuming more power is due to shrinking the die. Chips made with a 45nm process can provide the same speed with less heat than chips made with a 90nm process. But I digress! I don't want this to become a conversation on ASIC design

I'm glad you posted the carny limit setting, I hadn't changed that since I upgraded to V7.

[mkanet]

I'm sorry, but I have to disagree to some degree. There are files, registry modifications, etc that can't be kept in RAM for too long that require frequent writes/reads; and first-time reads to files that may not be cached/pre-fetched (such as large fanart, certain pagefile transactions, driver related transactions such as tuners, temp files, real-time virus protection written to disk, etc). All these hidden transaction can make a significant difference. So, while having a lot of RAM is good for overall system performance, it doesnt mean it will take the place of a very fast non-volatile storage device if setup correctly. I'm speaking from first-hand experience on a system that has adequate available free physical RAM.

Originally Posted by GKusnick

But your initial presumption was for systems with adequate physical RAM. If you have enough RAM to hold the entire working set (including Wiz.bin and the code that accesses it), then disk speed shouldn't make a dramatic difference. If it does, that's a sign you were doing excessive paging in the first place, and the usual solution to that is more RAM (or trim your working set by packaging code and data more efficiently).

I tend to agree with Fuzzy here: before throwing expensive hardware at a performance issue, it helps to have a clear idea of exactly what you're trying to fix and where the bad performance is coming from.

[davephan]

I occasionally have the SageTV 'spinning wheel' for extended time periods. Did you have the 'spinning wheel' or other lagging before the change to the SSD?

I think I read somewhere that the Windows swap files should be set to zero if you have a SSD, so the SSD is not prematurely worn out.

If the SSD does make a significant difference, maybe a 120 gig SSD would be enough, run as one partition. I don't know why you would need to set it up with two partitions, and why you need more than 120 gigs for the OS/programs drive.

Dave

[mkanet]

Hi Dave, as a matter of fact, you just reminded me, yes... I used to get short spinning wheels once in a while with my old HDD (which was very fast). But, the spinning wheel can be caused by various different things; especially when the spinning wheel gets stuck. I haven't seen a spinning wheel in over a half a year on any of my clients. I honestly can't prove if replacing my old HDD with the SSD stopped the spinning wheel; but, I know I have never gotten the spinning wheel with the SSD except for one instance when I had a problem with a plugin; and, never could reproduce it again.

I dont think people realize how fast a high end SSD really is. I can load windows in 10-12 seconds (thats including over 20 USB device drivers being initialized/loaded and other PCI peripherals. Before it used to take me close to 2 minutes or longer. Apps (even with prefetching) would take a little bit of time to load. Now, they just pop right up as if they were minimized and already in RAM. The best way to describe SSD performance is like loading everything into RAM with Disable Paging Executive & Large System Cache windows system features (available via registry settings). I tried that a long time ago; which was VERY fast with lots of RAM, but unfortunately weird things started happening using certain applications and after my machine crashed, it hosed everything. With an SSD, although it's not as fast as real DRAM, its as close as you can get to having everything in RAM without risk.

Disabling the Windows page file or setting the size to 0 is a very bad idea. There is page file activity for certain system processes that are hard coded in Windows; even if you have a gazillion GB's of RAM. Read about the details here. The trick is to set a pagefile that it doesnt grow or shrink; reducing fragmentation; and find the optimal size for it based on how much physical RAM you have and various apps that might require to use the pagefile.

The purpose of having two separate partitions for the SSD is to keep fragmentation to a minimum on the critical OS/APP partition and have another smaller partition for temp files (reimaging the temp partition when needed). Fragmentation can cause a performance degradation especially on an SSD.

As long as you have enough room for apps, temp files, fanart, etc (with some breathing room) that would be great.

-Michael

Originally Posted by davephan

I occasionally have the SageTV 'spinning wheel' for extended time periods. Did you have the 'spinning wheel' or other lagging before the change to the SSD?

I think I read somewhere that the Windows swap files should be set to zero if you have a SSD, so the SSD is not prematurely worn out.

If the SSD does make a significant difference, maybe a 120 gig SSD would be enough, run as one partition. I don't know why you would need to set it up with two partitions, and why you need more than 120 gigs for the OS/programs drive.

Dave

[Fuzzy]

If its mostly navigation speed that is your concern, then really the best thing you can do is ditch the extenders for pc clients. Keeping a local copy of your fanart on each client, even stored on a old crappy hdd will still be considerably more responsive than an extender. I've never felt a need to even look for performance tweaks, simply because I use pc clients, and have never noticed any sort of lag. Probably the reason for my response above.

[QueOnda]

Originally Posted by Fuzzy

If its mostly navigation speed that is your concern, then really the best thing you can do is ditch the extenders for pc clients. Keeping a local copy of your fanart on each client, even stored on a old crappy hdd will still be considerably more responsive than an extender. I've never felt a need to even look for performance tweaks, simply because I use pc clients, and have never noticed any sort of lag. Probably the reason for my response above.

Off topic but I wonder if usb sticks could ever be used on extenders (hd300, hd200) to keep fanart locally.

[Fuzzy]

I'm not sure it would really make it much faster. Most the slowdown is in the rendering of the ui, not the fetching of the fanart. This is especially true for v7, where the pc client uses full dx acelleration, making the gap even wider.

[Motofreak75]

the only tweak i did, was to install sagetv on a Linux box. Hands down the best thing I have down. Using the same hardware I have tried windows 2k3,2k8 and w7 and XP for over 6 years, Then for the past year and bit my Linux setup is solid. The only time I have to restart sage now is if has plugin update. 4 HD extenders & MVP and one off site extender & there is no lag except on my MVP unit (old POS) My linux box is using just a few sata drives no raid and its smooth and lag free.

I just wish Sage offered a free trial for linux users so that people can test out their hardware to see if everything works.

[mkanet]

Thanks Fuzzy. Actually, I do use a PC based client. I'm on one now. It happens to be my server as well. However, I also have an extender (which we use most).

Actually, we are very happy with the performance on the HD200 extender (not using wireless networking). It was the SSD upgrade that dramatically helped overall performance on all my clients; including extender. I use full 1920x 1080 Fanart backgrounds. Before the HDD, there was a noticeable lag that just made the extender feel a little "clunky" when doing different specific things; not just graphics intensive menus; but stuff related to the apps/system interacting with device drivers:

-Mounting/dismounting virtual blu-ray discs (ISO format) on the fly
-Playing radio stations on the extender (which has to initialize/start my hardware based loopback TV Tuner) for Slimplayer to work.
-Changing channels on my R5000 boxes and streaming (this has dramatically improved). I'm not sure how, but it went from taking sometimes taking over 2 seconds down to less than a second.

All I did was swap out my old HDD with the SSD and put the same image I had back on it. Immediately, I noticed incredible performance differences in the above mentioned items. For example, ripped ISOs mounted and streamed just as if they were MKVs. I could start playing any radio station in Slimplayer on my extender immediately... before it would take a second or so for the loopback channel to start recording/streaming. I could flip through graphics intensive lists using full res fanart almost as fast as being on the server itself (like I am now).

It would be interesting to see if anyone else here has tried using an SSD to speed up their HTPC.

Originally Posted by Fuzzy

I'm not sure it would really make it much faster. Most the slowdown is in the rendering of the ui, not the fetching of the fanart. This is especially true for v7, where the pc client uses full dx acelleration, making the gap even wider.

[Fuzzy]

Originally Posted by mkanet

Thanks Fuzzy. Actually, I do use a PC based client. I'm on one now. It happens to be my server as well. However, I also have an extender (which we use most).

Actually, we are very happy with the performance on the HD200 extender (not using wireless networking). It was the SSD upgrade that dramatically helped overall performance on all my clients; including extender. I use full 1920x 1080 Fanart backgrounds. Before the HDD, there was a noticeable lag that just made the extender feel a little "clunky" when doing different specific things; not just graphics intensive menus; but stuff related to the apps/system interacting with device drivers:

-Mounting/dismounting virtual blu-ray discs (ISO format) on the fly
-Playing radio stations on the extender (which has to initialize/start my hardware based loopback TV Tuner) for Slimplayer to work.
-Changing channels on my R5000 boxes and streaming (this has dramatically improved). I'm not sure how, but it went from taking sometimes taking over 2 seconds down to less than a second.

All I did was swap out my old HDD with the SSD and put the same image I had back on it. Immediately, I noticed incredible performance differences in the above mentioned items. For example, ripped ISOs mounted and streamed just as if they were MKVs. I could start playing any radio station in Slimplayer on my extender immediately... before it would take a second or so for the loopback channel to start recording/streaming. I could flip through graphics intensive lists using full res fanart almost as fast as being on the server itself (like I am now).

It would be interesting to see if anyone else here has tried using an SSD to speed up their HTPC.

Those things, I agree WOULD be affected by an SSD, as those are primarily windows OS based operations, and NOT sagetv server functions. my point was that the SageTV server is not really going to be sped up in any significant way by swapping out to an SSD, as any interface slowdowns are going to be more of a bottleneck than SAGETV's drive access.

[mkanet]

Yeah, these tweaks wont make much difference for stock sageTV without fanart. It's everything else on SageTV that it affects; which is why I put this thread in the customizations sub-forum instead of sageTV 7 software forum.

It's important to note that the SSD doesn't just affect the OS (such as, drivers, services, Windows system related functions), but significantly affects subsystems on the same machine as well. Such as: 2 web servers (IIS 7/Jetty), including Java (mainly cache-related) and Squeezebox Server. I know this doesnt count as SageTV really, but I can "popup" SageTV Webserver grid-view Program Guide with all 999 channels and respective logos and other images on any PC/Laptop/Netbook on my LAN as if the webpage was stored locally on the device. The list goes on and on with various circumstances that the SSD makes a noticeable difference on; which may not be necessarily sageTV related.

Originally Posted by Fuzzy

Those things, I agree WOULD be affected by an SSD, as those are primarily windows OS based operations, and NOT sagetv server functions. my point was that the SageTV server is not really going to be sped up in any significant way by swapping out to an SSD, as any interface slowdowns are going to be more of a bottleneck than SAGETV's drive access.

[mayamaniac]

mkanet, what do you consider as a high-end SSD, Sata III or HSDL? In your situation, maybe you need a 256GB drive, but for most SageTV servers, it seems a smaller 64GB SSD is probably sufficient, as Dave mentioned. And I think you are right about the benefits of SSD, especially running under Windows. The way Windows work, it will always require virtual memory/pagefile. And the best way to speed that up is to speed up the drive. As for SSD fragmentation, couldn't you just schedule it to defragment every day? I wish SSD drives gets a lot cheaper soon, I would love to get one for my OS drive.

And thanks for the "limited_carny_init=true" mention, I haven't set that since upgrading to v7. I think both this setting and the Java heap size should've been added to the SageTV UI options instead of hidden in the sage.properties file and registry.

[davephan]

Originally Posted by mkanet

Presuming people have sufficient CPU/Physical RAM on their system to begin with; I am curious what fellow sageTV users do to gain extra performance out of their relatively extensive sageTV setups. I've included a few tweaks which helped me the most:

1- High-end large capacity SSD:
For me, the single biggest performance enhancement by far was getting an ultra high end 256GB SSD HDD (divided into 2 partitions/logical drives). The one I use is the Corsair P256. However, Crucial and some OCZ brand SSDs are great as well.

It set me back $750 for a 256GB disk, but completely worth it. It's similar to having everything on your hard disks kept in RAM at all times; except with the big advantage that files/temp files written are non-volatile (meaning you wont lose anything if you lose power to the PC). Since there are no movable parts (unlike traditional hard drives), there's no heat generated and no noise. Imagine a single, tiny device comfortably outperforming several WD Raptor high RPM HDD's raided together without any noise or heat.

-MKANET

I have been thinking about switching from XP to Windows 7 64-bit for Linux with a SSD for the OS. Hopefully, the move will get rid of the annoying occasional spinning or stuck wheels, which are very annoying. I don't know if the spinning or stuck wheels is a related to the 16 TB of video library files on the mapped drives to the unRAID server. Maybe having thousands of video files slows down SageTV.

Some of the more expensive SSDs do have higher performance. What are the read/write specs for your SSD? I'm still not sure why it couldn't be done with a 120 gig SSD. Is your SSD using a sanforce controller? I head that SSDs that use sandforce controllers are better. I've also heard that using SLC are better than MLC. SLC SSDs are pretty uncommon though. Could you use a product like Perfect Disk to periodically defrag your SSD? I've also heard that SSDs can slow down over time. Maybe that is a result of the fragmentation.

I've also heard from several sources that you need to set your operating system disk caching to zero if you use a SSD, since the disk caching will prematurely wear out the SSD. I've also head that defragging the SSD will also prematurely wear out the SSD.

Originally Posted by Motofreak75

the only tweak i did, was to install sagetv on a Linux box. Hands down the best thing I have down. Using the same hardware I have tried windows 2k3,2k8 and w7 and XP for over 6 years, Then for the past year and bit my Linux setup is solid. The only time I have to restart sage now is if has plugin update. 4 HD extenders & MVP and one off site extender & there is no lag except on my MVP unit (old POS) My linux box is using just a few sata drives no raid and its smooth and lag free.

I just wish Sage offered a free trial for linux users so that people can test out their hardware to see if everything works.

I'm thinking about moving to Linux. If a free trial was available, I would have probably switched already. Maybe the there could be a free trial for only currently licensed SageTV Windows users. Linux might be a way to get rid of the occasional spinning or stuck wheels. Sometimes I turn off and on my HD-200 boxes when the wheel gets stuck or looks like it will be spinning for minutes. The spinning or stuck wheel is the most annoying part of SageTV.

I think I could get everything to work with Linux, except for PlayOn / Netflix streaming. I think I would need to run PlayOn with a separate Windows computer. I don't think I would want to add the complexity of setting up a Windows VM on the Linux SageTV computer.

If I moved to Linux, I would have to pay several hundred dollars for the Acronis Linux imaging software or switch to a cheaper imaging product that works with Linux. What do you use to image your Linux computer? If you don't have an image, the recovery is too painful.

I'm hesitant to move to Linux without separate hardware for the Linux build since it might take many days or weeks to get everything working with Linux. That would be a lot of SageTV downtime, which I want to avoid.

Dave

[razrsharpe]

Originally Posted by QueOnda

Off topic but I wonder if usb sticks could ever be used on extenders (hd300, hd200) to keep fanart locally.

Originally Posted by Fuzzy

I'm not sure it would really make it much faster. Most the slowdown is in the rendering of the ui, not the fetching of the fanart. This is especially true for v7, where the pc client uses full dx acelleration, making the gap even wider.

Plucky asked sage about that awhile and iirc they said they didn't plan on adding anything like that, mainly because it wouldn't do anything to the speed in fetching the fanart...

[razrsharpe]

Originally Posted by davephan

Could you use a product like Perfect Disk to periodically defrag your SSD? I've also heard that SSDs can slow down over time. Maybe that is a result of the fragmentation.

Defragmenting SSD's is not recommended. In fact it is considered to be a major no-no. SSD's slow down over time because of excessive erase/write operations. This is why defragmenting a SSD is major NO NO. Win7 and most modern SSD's have "trim" support that does garbage collection on the disk and this (more then anything else) keeps your SSD in tip top shape.

Trim
Wear Leveling
Defrag discussion