Live TV pictures freezes and speeds up to catch up!

[robogeek]

With the Total memory usage reaching the Max heap size setting, it means that at some point SageTV is maxing out the JVM memory. Once it maxes out, Java should automatically kick in the garbage collection routine to free up JVM memory for the running threads in the JVM that need more memory. This garbage collection will likely cause the video to pause until it finishes, which usually lasts around 15 to 45 seconds. Not sure why it does that, or why only the video pauses and not the audio too, but that's the experience I've had.

If your pauses are being caused by Java garbage collecion, there's not much that can be done about it. You can try pushing the JVM Heap Size up closer to 1200 or 1300, but that will only work if your SageTV machine has enough RAM. If you push the heap size higher and SageTV fails to start, you've gone to high for the amount of RAM you have available. In that case, just keep lowering the heap size until SageTV starts up again. If you run SageTV Client on the server machine, it's possible that you may increase the heap size so much that the SageTV Service will start, but the SageTV Client may then fail to start. In that case, you will again have to lower the heap size until both the service and client start without issues.

Just a couple of other thoughts... if you don't have a ton of new media coming into the server in the SageTV import directories, you can increase the time between media scans to limit the possibility that a large MP3 collection, BMT, or Phoenix is jumping the memory usage during media scans. The setting can be found in SageTV->Setup->Detailed Setup->General->Automatic Library Rescan Frequency. I set mine to 10080 minutes (one scan per week). If I add anything new that I want to watch or listen to right away, I just manually rescan at SageTV->Setup->Scan Imported Media. The only other thing that comes to mind is a property in the SageTV properties file that can be set to force regular Java garbage collection. The property is allow_periodic_explicit_gc and I think it defaults to false. I would think that would be the desirable setting since it will allow Java to do garbage collection only when needed instead of doing it regularly and possibly causing the pausing more often, but I could be wrong. Check that property in your Sage.properties file. If it's set to false, maybe try setting it to true for a couple of days to see what happens. Or if it's set to true, try setting it back to false...just be sure to stop the SageTV Service before editing the Sage.properties file.

[heffneil]

Sound is beat too! It is like it stops. I think you are on to something with the garbage collection. I have a huge library lots and lots of movies. I will look at the time frame I reindex and the other suggestions. You responses are very detailed and helpful. Thank you very much!

Neil

[wayner]

I have this problem too and I am pretty sure when I have it that it is a network issue. This happens most likely when watching OTA HD files which are the highest bitrate files that I would have.

In terms of running out of JVM - I have this happen all the time as I have 6 extenders and running 4 or more at once really stresses out your JVM and it also appears that putting extenders in standby doesn't free up aloof the JVM.

[heffneil]

I don't think my problem is drive related because last night when watching a video off my unraid server it did the same thing. Especially when I attempted to get the heap details.

I just bumped it up to 1241 MB and the details are showing

856 1241 1241 - I think diamond is absolutely killing me.

[wayner]

There are other threads here regarding JVM and others have reported bad behavior when setting the JVM size above 1024 so you probably shouldn't go bigger than that.

[heffneil]

I haven't done too much research but my machine has 8 gigs. I don't know what other people's machines have. I have to say I really don't like Java and only wish Sage would have compiled in C before going belly up.

[patja]

Any suggestions for how to monitor or log the times when JVM garbage collection happens? Is there a windows event or perfmon counter that captures (or can be configured to capture) these events?

[wayner]

There is a JVM monitor plugin that can send you notices when JVM usage is getting high or causing an error. It might also be good to do an occasional "therapeutic" restart of the sagetv service and/or your SageTV server.

If you are familiar with SJQv4 then here is some code that will determine if there are any clients attached and if there are any upcoming recordings: http://code.google.com/p/sagetv-addo....groovy?r=1535. You could then just run a batch file to restart the service and/or reboot your machine - the batch command "shutdown /r" from a command prompt will cause Windows to reboot.

You could always reboot at a specific time every day without doing this test but then you run the risk of screwing up a recording.

[wayner]

Originally Posted by heffneil

I have to say I really don't like Java and only wish Sage would have compiled in C before going belly up.

Wasn't the advantage of Java the fact that you could be platform independent? And Sage did run on Windows, Linux and Mac (at least before v7).

But apparently another shortcoming is that it used the 32 bit version of Java that limited the amount of memory that Java could use. If they used 64 bit Java then we wouldn't have had this issue.

[heffneil]

I just updated java today to 6.0 32 and my server wouldn't start. I am glad you said something about the 32 bit (crap) version. Turns out that the update broke it and on top of it I was installing the wrong older version 64 bit. What a darn shame

[andyvr4]

Originally Posted by heffneil

I just updated java today to 6.0 32 and my server wouldn't start. I am glad you said something about the 32 bit (crap) version. Turns out that the update broke it and on top of it I was installing the wrong older version 64 bit. What a darn shame

java 6 update 32 should work fine with sagetv. I've been using it now for a couple months without any issues.

[heffneil]

I can't get it to work. went back to 30 and it started back up just fine for me...

[patja]

I am back to thinking that at least in my case it is a network problem. Could be some other problem that as a downstream consequence causes a network problem, but network is definitely in the mix. I never get close to maxing out the JVM heap memory allocation.

I've been running a number of CPU and network perfmon counters, tracking when the "pause then fast catchup" glitch happens and then examining the perfmon reports. I am consistently seeing that the counters for Bytes Received/Sec and Bytes Sent/Sec on my NIC drop to zero for the duration of the glitch. Outside of the glitch they are never zero. After recovering from the glitch they peak a bit and then go back to a steady state/normal level.

I've got an Intel Gigabit PCIe card which I thought was best in class. My next step is to try to figure out if the glitch is due to some external event or problem caused by some other device on the LAN vs. being internal to the Sage server. I'm going to run the perfmon counters on other machines on my LAN that don't have anything to do with Sage. If there is some rogue device causing trouble on the wire (I have probably a dozen devices on my LAN chatting away at all times) this should find it. I suspect it is internal to the Sage machine, maybe a problem with the NIC, maybe a PCI bus problem?

[robogeek]

Here is a tool that might be able to tell you if it's a device driver behaving badly:

http://www.thesycon.de/deu/latency_check.shtml

Unfortunately it won't be able to tell you which specific device is causing high latency, but you can disable some likely culprits in the Windows Device Manager until you find the offending driver(s). If it turns out to be a bad driver, you'll have to check the device or motherboard manufacturer web site for updated drivers or, if you already have the latest drivers, you may need to go back to an earlier driver version...sometimes the newest drivers aren't the best drivers.

[patja]

I just wanted to report back here that I believe (knock on wood) that I have 100% eliminated this particular glitch by migrating my Sage server to an esxi virtual machine. I still suspect the problem was the NIC, but after months of swapping NICs around, looking at perfmon captures, and tweaking different driver settings got nowhere, I decided to try the dramatic change of moving to the VM.

The Colossus is working fine with DirecTV HDMI using PCI passthrough, and for OTA I am using a HDHomeRun HDHR-3.

The esxi server is built on a Supermicro X9SCA-F motherboard. It is a napp-it all-in-one implementation, which I've been very happy with running half a dozen VMs. Maybe it's the hardware, maybe it is the clean installation. I'm just happy to have that particular problem behind me.

It is not all perfect. The Colossus still has its issues where it produces bad recordings a few times a month and continues to do so until the server is rebooted, and I still do a nightly reboot, and I'm still not sure if the HDHomeRun is a keeper...I haven't been super happy with its performance.

[sacrament055]

The part of this issue that makes no sense to me is why live TV would be different than recorded TV. If it's a network of Java issue it's using the same network for previously recorded shows and the same java version etc as well.

I have found that if I watch a few seconds (maybe 10) behind live it appears to help/eliminate the issue.

[Clift]

What worked for me was to change out my SATA cables.
I was looking everywhere for a solution to this problem. It had gotten really really bad over the course of a couple to three months.
I started suspecting hardware issues such as hard drive or fragmentation.
The thing is, though, it was happening on both drives that I have. So thought it was the motherboard or HDD controller. I found a thread on HDD guru talking about how infrequently Hard Drive controllers actually fail vs hard drives themselves. Someone on that thread suggested changing out the SATA cables as a troubleshooting step for apparently "bad" hard drives that had no bad sector or errors that Windows or HDD utilities could find.

In my case I imagine the sustained data rate was enough to expose an issue that was masked by the random r/w of everyday usage.

I hope this ends up helping someone else.

[sacrament055]

Suppose that is worth a shot because I've gone to ssd to test with the same result so IO shouldn't be my issue.