Ideas for getting notified when Sage dies

[wayner]

I have had the odd issue with SageTV dying while the SageTV docker is still alive - for example see this thread.

So I want to be notified when SageTV goes down. I have written a python script that attempts to hit the SageTV web server. When the response code is not 200 or there is a timeout or other issue I then have an email sent to me. This is set to run every 5 minutes in a cron job under the root unRAID account.

Is this the best way to do this or is there a better way to do this?

One issue with this script is that it won't work if the unRAID server itself goes down.

[Fuzzy]

Originally Posted by wayner

One issue with this script is that it won't work if the unRAID server itself goes down.

It would if it was run on something besides the unRAID server. That's really the only way to do a watchdog script like this, is to have it completely seperate from the potential failure points.

[wayner]

That's a good point. I might run it on both the server and another computer as this would allow me to get more information. For example it would be helpful to know the following info:

Is the unRAID server up?
Is the SageTV docker process alive?
Is the SageTV process alive?

When it comes to the docker process and, arguably the SageTV process, you would need to run them from the unRAID server or run a remote SSH command.

[KarylFStein]

Originally Posted by wayner

That's a good point. I might run it on both the server and another computer as this would allow me to get more information. For example it would be helpful to know the following info:

Is the unRAID server up?
Is the SageTV docker process alive?
Is the SageTV process alive?

When it comes to the docker process and, arguably the SageTV process, you would need to run them from the unRAID server or run a remote SSH command.

Probably more than you're wanting to do, but I've used Nagios for a long, long time, (since before it was called Nagios--forget the original name). I haven't had time to see how much integration there may be with unRAID, though.

[KeithAbbott]

Before I undertake the work to make the switch to an unRAID/SageTV docker system, do I need to be worried about the stability of this setup? Is it really unstable enough that it requires monitoring from a separate computer? Your post has me questioning whether unRaid/SageTV is a good direction to head in.

[stuckless]

Originally Posted by KeithAbbott

Before I undertake the work to make the switch to an unRAID/SageTV docker system, do I need to be worried about the stability of this setup? Is it really unstable enough that it requires monitoring from a separate computer? Your post has me questioning whether unRaid/SageTV is a good direction to head in.

In my setup on linux in 7-8 years... I can count on one hand the number of times my server has locked up and needed a manual reboot... and you can bet that every one of those times I was away on business

The uptime on the server isn't in the years, but I usually get 2-3 months of uptime and then I need to reboot to because I've taken some updates, etc.

Overall I can't say the stability is bad, but i don't run Windows, so I also can't compare the setup with Windows.

[wayner]

My SageTV V7 Win7 system is very unstable and I have a similar notification system set up for that. But the reason for that instability is the insufficiency of 1GB of JVM heap size.

My unRAID system has been up for a month and I have had two crashes. It looks like the latest one was related to the YouTube plugin.

I just want to be pro-active and stay on top of any potential issues.

[EnterNoEscape]

I have not had any noteworthy issues running the SageTV Docker container on unRAID. I also purely use network-based encoders with it through OpenDCT which might make a difference. I can't speak for long uptimes due to the reboots needed for OS updates, but I have not had the container suddenly stop on me.

[wayner]

I have not had the docker stop on me either but twice I have had the SageTV java process die within the docker. So the docker is still active but it isn't doing very much. The logs will show something like:

./startsagecore: line 48: 100 Aborted java -Djava.awt.headless=$HEADLESS $JAVAMEM -XX:+UseAdaptiveSizePolicy -XX:MaxGCPauseMillis=25 -XX:GCTimeRatio=24 $(if [[ $EUID -eq 0 ]]; then echo '-XX:ThreadPriorityPolicy=1'; fi) $JAVAOPTS -cp Sage.jar:JARs/lucene-core-3.6.0.jar:/:$(echo JARs/*.jar | sed 's/ */:/g') sage.Sage 0 0 x "sagetv Sage.properties" 0>&-

I have seen this happen twice now in the roughly five weeks since I have been running SageTV on unRAID. The most recent issue was described here: [ReProcessHook@895f038] 9000 times per second!: (I guess a mod moved it to a different forum). Tmiranda is pretty sure that this was a Youtube plugin issue.

Would there be any way of having the SageTV server process automatically restart itself if it dies?

[KarylFStein]

Originally Posted by wayner

I have not had the docker stop on me either but twice I have had the SageTV java process die within the docker. So the docker is still active but it isn't doing very much. The logs will show something like:



I have seen this happen twice now in the roughly five weeks since I have been running SageTV on unRAID. The most recent issue was described here: [ReProcessHook@895f038] 9000 times per second!: (I guess a mod moved it to a different forum). Tmiranda is pretty sure that this was a Youtube plugin issue.

Would there be any way of having the SageTV server process automatically restart itself if it dies?

From the output you gave it looks like SageTV is being started within a script and is not being exec'd, so yes, it should be possible to catch that type of failure and restart the service.

[KarylFStein]

Originally Posted by KarylFStein

From the output you gave it looks like SageTV is being started within a script and is not being exec'd, so yes, it should be possible to catch that type of failure and restart the service.

I looked at the script in Git and it looks to already have a restart ability. If the file restart or restartsvc exists, it will restart, (but also remove the file). I am not sure if something else is supposed to create that file e.g. some timer so the service will restart if say it's been running OK for at least an hour, but if rapid-fire crashing it won't keep trying. If not it would be fairly simple to add logic like that and perhaps even trigger an alert of some sort if it can't resolve things.

[EnterNoEscape]

I believe it's supposed to do that already. If it's not as in your case, it must have frozen in a manner that didn't cause the JVM to actually stop. In that case, then it's likely some code spinning out of control in such a manner that the JVM appears to have crashed because it is no longer responding.

[wayner]

Should we mention this to stuckless and see if he can implement it?

[KarylFStein]

Originally Posted by KarylFStein

I looked at the script in Git and it looks to already have a restart ability. If the file restart or restartsvc exists, it will restart, (but also remove the file). I am not sure if something else is supposed to create that file e.g. some timer so the service will restart if say it's been running OK for at least an hour, but if rapid-fire crashing it won't keep trying. If not it would be fairly simple to add logic like that and perhaps even trigger an alert of some sort if it can't resolve things.

FYI, as far as I can tell it looks like the restart file is created during a planned restart of the server, (e.g. after updating a plugin that asks if you want to restart the server).

[Fuzzy]

Originally Posted by KarylFStein

FYI, as far as I can tell it looks like the restart file is created during a planned restart of the server, (e.g. after updating a plugin that asks if you want to restart the server).

That is correct. It essentially writes the restart file, and shuts down the JVM, returning control to the script, which then sees the restart file, and relaunches the jvm.

The issue with it blanket restarting when the jvm exits, is we'd never be able to cleanly shut down the server. Currently, when you tell the docker container to close, it sends a kill signal to all process, including sage. This triggers sage to do a clean shutdown (stopping recordings, stopping services, final writes on the databse files, etc), and then it closes the JVM. If what is suggested here is implemented and it autorestarts when closing, it may simply relaunch as soon as it exits, and eventually, after a timeout, the force kill will stop it uncleanly.

A change could certainly be made to the sage core that will write a 'running' file, that gets deleted before sage intentionally closes down, and if that file is present, the script will restart assuming it failed. This might provide a location to detect a 'sage died' action and send off a notification of some sort perhaps as well. Feel free to write up an issue for this capability on github.