Plugin: Sage Job Queue (SJQ)

[madas]

I rebooted and can run debug once...and only once..after it hangs

Sun 3/22 17:38:10.611 (org.mortbay.log) <DEBUG> REQUEST /sjq/SJQ on org.mortbay.jetty.HttpConnection@163bb7b
Sun 3/22 17:38:10.611 (org.mortbay.log) <DEBUG> sessionManager=org.mortbay.jetty.servlet.HashSessionManager@105f21c
Sun 3/22 17:38:10.611 (org.mortbay.log) <DEBUG> session=null
Sun 3/22 17:38:10.611 (org.mortbay.log) <DEBUG> servlet=SJQServlet
Sun 3/22 17:38:10.611 (org.mortbay.log) <DEBUG> chain=null
Sun 3/22 17:38:10.611 (org.mortbay.log) <DEBUG> servlet holder=SJQServlet
Sun 3/22 17:38:10.611 (org.mortbay.log) <DEBUG> loaded class java.lang.StackTraceElement
Sun 3/22 17:38:10.612 (org.mortbay.log) <DEBUG> loaded class java.lang.StackTraceElement from null
Sun 3/22 17:38:10.612 Sun Mar 22 17:38:10 EDT 2009: SJQ Application Dump
	QueueLoader thread is alive...
		java.lang.Thread.sleep(Native Method)
		com.google.code.sagetvaddons.sjq.server.QueueLoader.run(QueueLoader.java:94)
		java.lang.Thread.run(Unknown Source)
	ClientMonitor thread is alive...
		java.lang.Thread.sleep(Native Method)
		com.google.code.sagetvaddons.sjq.server.ClientMonitor.run(ClientMonitor.java:35)
		java.lang.Thread.run(Unknown Source)
	InteralTaskClient thread is alive...
		com.google.code.sagetvaddons.sjq.server.TaskQueue.pop(TaskQueue.java:120)
		com.google.code.sagetvaddons.sjq.server.InternalTaskClient.run(InternalTaskClient.java:39)
		java.lang.Thread.run(Unknown Source)
	Most recent LogCleaner thread is dead (this is normal behaviour)!
	Most recent FileCleaner thread is dead (this is normal behaviour)!

Sun 3/22 17:38:10.612 (org.mortbay.log) <DEBUG> RESPONSE /sjq/SJQ  200
Sun 3/22 17:38:10.613 (org.mortbay.log) <DEBUG> EOF
Sun 3/22 17:38:10.614 (org.mortbay.log) <DEBUG> REQUEST /sjq/SJQ on org.mortbay.jetty.HttpConnection@347cce
Sun 3/22 17:38:10.614 (org.mortbay.log) <DEBUG> sessionManager=org.mortbay.jetty.servlet.HashSessionManager@105f21c
Sun 3/22 17:38:10.614 (org.mortbay.log) <DEBUG> session=null
Sun 3/22 17:38:10.614 (org.mortbay.log) <DEBUG> servlet=SJQServlet
Sun 3/22 17:38:10.614 (org.mortbay.log) <DEBUG> chain=null
Sun 3/22 17:38:10.614 (org.mortbay.log) <DEBUG> servlet holder=SJQServlet
Sun 3/22 17:38:10.615 Sun Mar 22 17:38:10 EDT 2009: SJQ Application Dump
	QueueLoader thread is alive...
		java.lang.Thread.sleep(Native Method)
		com.google.code.sagetvaddons.sjq.server.QueueLoader.run(QueueLoader.java:94)
		java.lang.Thread.run(Unknown Source)
	ClientMonitor thread is alive...
		java.lang.Thread.sleep(Native Method)
		com.google.code.sagetvaddons.sjq.server.ClientMonitor.run(ClientMonitor.java:35)
		java.lang.Thread.run(Unknown Source)
	InteralTaskClient thread is alive...
		com.google.code.sagetvaddons.sjq.server.TaskQueue.pop(TaskQueue.java:120)
		com.google.code.sagetvaddons.sjq.server.InternalTaskClient.run(InternalTaskClient.java:39)
		java.lang.Thread.run(Unknown Source)
	Most recent LogCleaner thread is dead (this is normal behaviour)!
	Most recent FileCleaner thread is dead (this is normal behaviour)!

Sun 3/22 17:38:10.615 (org.mortbay.log) <DEBUG> RESPONSE /sjq/SJQ  200
Sun 3/22 17:38:10.616 (org.mortbay.log) <DEBUG> REQUEST /sjq/SJQ on org.mortbay.jetty.HttpConnection@7ce842
Sun 3/22 17:38:10.616 (org.mortbay.log) <DEBUG> sessionManager=org.mortbay.jetty.servlet.HashSessionManager@105f21c
Sun 3/22 17:38:10.616 (org.mortbay.log) <DEBUG> session=null
Sun 3/22 17:38:10.616 (org.mortbay.log) <DEBUG> servlet=SJQServlet
Sun 3/22 17:38:10.616 (org.mortbay.log) <DEBUG> chain=null
Sun 3/22 17:38:10.616 (org.mortbay.log) <DEBUG> servlet holder=SJQServlet

[Slugger]

Originally Posted by madas

Not looking good.

I cannot even get the DEBUG command to work, just comes back blank (after about a minute of waiting)

On 383?

[Slugger]

With the help of madas, I was able to track down the problem he was having and have fixed it in snapshot 386. Two different request threads were deadlocking trying to acquire each others' locks on the database resources. Problem has been solved in 386.

Between this deadlock issue and the thread synchronization issues fixed a few snapshots back, I think everything should be rather stable now. Please post a reply if you are having any problems with 386.

I'm going to wait a couple days to give those running 386 a chance to report back any issues (though at least two of us have been running 386 for some time now and have had no issues), but assuming none come up I'll package up 386 as the official 2.1.3 build. Then it's on to improving the performance of the db code (more on that when I have something ready to test).

[Wheemer]

With snapshot 386 I am unable to get the main page to load at all. It just says the server is taking too long to respond.

[Slugger]

Originally Posted by Wheemer

With snapshot 386 I am unable to get the main page to load at all. It just says the server is taking too long to respond.

Can you load the debug=TRUE URL and get the output from the SageTV debug logs? When SJQ isn't loading go to:

http://jetty_server/sjq/SJQ?debug=TRUE

Should get a response saying that the logs were dumped, then go to your SageTV debug logs and get the SJQ app dump for me.

[Wheemer]

I tried that and it still won't load. So I have deleted my database and I'm trying again.

Will report back.

[Wheemer]

I see this error in the log:

Mon 3/23 20:33:27.978 ERROR updating locator system of:java.net.SocketException: Software caused connection abort: socket write error

Not sure if its related

[Peter_h]

I upgraded to snapshot 404 and it seems to be stable. It's been going for 3 hours now w/o issue.

[Slugger]

Originally Posted by Peter_h

I upgraded to snapshot 404 and it seems to be stable. It's been going for 3 hours now w/o issue.

Builds 395+ include the rewrite of the thread/synchronization code. Most of the problems I found have been there for quite some time (dating back to v2.0.0.154 and older), but they didn't seem to manifest themselves until official v2.1.2, which isn't all that uncommon when dealing with multi-threaded code. Most of the problems were just carelessness on my part, which resulted in numerous race condition situations. Things had to happen at just the right time in order for the deadlock to occur. I still can't get SJQ v2.1.2.343 to crash/hang/die on my system, but with the help of a few users who got me debug logs, JVM dumps, etc. I was able to find the problem spots, which lead to a complete review of the code, which lead to build 395.

Here's a quick overview of what these builds contain. I'm now recommending that everyone upgrade to 395 or higher. I'll mark one of these recent snapshots as official v2.1.3 after a couple more days of testing myself along with feedback from everyone else who's testing.

Snapshots 395, 397, 398, 400

• First drop with rewrite of threaded code; complete review of code done, sync/deadlock issues resolved; many performance improvements (395)
• A few touch ups with the servlet code; upgrade JDBC driver to latest available while I'm at it (397)
• Ensure super.finalize() is called from local finalize() methods (398)
• Various logging enhancements (properly flush in memory log messages to db on Jetty shutdown) (400)
• Anyone running less than 395 should move to at least 395 or higher with 400 being preferred; these four snapshots contain all the multi-threading fixes

Snapshot 403+

• Timestamp each individual log message since there can be more than one Logger instance now; sort log messages by new timestamp field when sorting them for display (403)
• Security fix: Only accept commands from clients who have authenticated themselves [issue 60] (404)
• When purging task logs, purge all task logs even those orphaned by deletion of task entries from the complete and failed tables [issue 59] (405)

So I'd recommend all users move to at least 395, but moving all the way to 405 should be fine now. Again, however, please keep me updated on your results (good and bad). Please be sure to mention which version you're running.

[tomfisk]

The Last Run and Next Run is not being displayed under the QueueLoader Tab. Otherwise it's been running great.

Oooppsss! I take that back...it just showed up.

[Slugger]

Originally Posted by tomfisk

The Last Run and Next Run is not being displayed under the QueueLoader Tab. Otherwise it's been running great.

Oooppsss! I take that back...it just showed up.

Yeah, the fix for issue 60 broke this - sort of. The QueueLoader menu tries to load the schedule times as soon as you load the GUI, but the initial attempt will fail now because you have not authenticated. The first time that menu tries to update the times after you've authenticated then it will properly populate the menu. I'll add a fix to immediately reload that menu after authentication succeeds.

[95TBird]

Hi:

I can't seem to get the file cleaner to work.
I'm running SJQ v2.1.3.406 & client 2.1.2.331
& the file cleaner settings are
Delete files with an extension matching this regex: edl
When no corresponding file has an extension matching this regex: mpg|ts
Never delete files matching this regex: *.logo.txt

But whenever a recording is deleted in Sage, the .edl file never gets deleted.

I've been over & over all the settings & I can log in to SJQ with no problems, but the .edl files just sit there .

I'd appreciate any help !

-Bruce

[Slugger]

Originally Posted by 95TBird

Delete files with an extension matching this regex: edl
When no corresponding file has an extension matching this regex: mpg|ts
Never delete files matching this regex: *.logo.txt

But whenever a recording is deleted in Sage, the .edl file never gets deleted.

First, the FileCleaner thread runs periodically, so edl files won't be deleted as soon as the mpg or ts file is deleted. The FileCleaner runs every time the QueueLoader runs.

Next, I'm sure it's just the formatting of the copy/paste, but make sure there are no spaces in either of the first two boxes, especially before and after the "|" is the second box.

Finally, remove the contents of the third box. It's an invalid regular expression, which is probably why nothing is deleting and it's also not needed since you're not trying to delete txt files anyway.

(10 mins later after a quick scan of the javadocs for java.util.regex...) So it turns out Java's regex package throws an unchecked exception when a regex compile fails (due to an invalid regular expression). That seems awfully wrong to me, but anyway this is definitely the cause of your problems. Issue 61 opened for the fix, but in the meantime simply remove the contents of the third box, you don't need it anyway. But if you decide you want to start deleting txt files, then ensure that the third box contains a valid regex and to protect logo files generated during comskip runs, you'll want to use: ".*\.logo\.txt$" (no quotes).

[Peter_h]

It's been going for almost 24 hours and no hang.

Looks like this is all fixed.

Thanks so much for this plugin, slugger. You definately work hard and do a great job.

[95TBird]

Slugger:

It was in fact the entry in the third box.
I deleted it & now the edl files are being deleted appropriately!!

Thanks for a fantastic utility !!

-Bruce

[Wheemer]

snapshot 410 doesn't load up the webpage for me again...

[Slugger]

Originally Posted by Wheemer

snapshot 410 doesn't load up the webpage for me again...

I'm going to need a dump of your JVM to see what's going on. Grab me on MSN when you have time to do this.

Anyone else with results from 410?

[tomfisk]

Originally Posted by Slugger

I'm going to need a dump of your JVM to see what's going on. Grab me on MSN when you have time to do this.

Anyone else with results from 410?

Working great for me...no issues.

[nyplayer]

Originally Posted by Wheemer

snapshot 410 doesn't load up the webpage for me again...

I had the same problem I had to clear the Cache in the browser and delete the tempfiles.... then everything loaded ok.

[nyplayer]

Slugger,

I create mpg.properties and mpg.xml files for all my recordings. I would like to cleanup these files. I have it setup as below.... but it deletes the properties files even if an mpg avi etc... exists?

Delete files with an extension matching this regex
vprj|edl|my|mpg.properties|avi.properties|mkv.properties|mp4.properties

When no corresponding file has an extension matching this regex:
mpg|avi|mkv|mp4