Plugin: Sage Job Queue (SJQ)

[nyplayer]

Slugger,

Is there anything you can do with illegal characters such as ":" etc... " * / : < > ? \ |

see below As you can see the ":" completely destroys the path names.

Law & Order: Criminal Intent

Episode: The Posthumous Collection

:TRANSCODEOPTS "Copy-MP4 Move Play" "false" "\\\\GFORCENAS\\sagetvnas\\tv\\%t%"

translates to this.

destfile=\\\\GFORCENAS\\sagetvnas\\tv\\Law & Order\: Criminal Intent



example below. This fails

transcoder/jobs/70/clip_duration=0
transcoder/jobs/70/clip_start=0
transcoder/jobs/70/destfile=\\\\GFORCENAS\\sagetvnas\\tv\\Law & Order\: Criminal Intent
transcoder/jobs/70/formatname=Copy-MP4 Move Play
transcoder/jobs/70/processor=sagetv
transcoder/jobs/70/replace=true
transcoder/jobs/70/sourcefile=N\:\\sageuncn\\LawOrderCriminalIntent-ThePosthumousCollection-3211232-0.mp4
transcoder/jobs/70/status=0





No Illegal Characters. Runs fine

transcoder/jobs/66/clip_duration=0
transcoder/jobs/66/clip_start=0
transcoder/jobs/66/destfile=\\\\GFORCENAS\\sagetvnas\\tv\\The Outer Limits
transcoder/jobs/66/formatname=Copy-MP4 Move Play
transcoder/jobs/66/processor=sagetv
transcoder/jobs/66/replace=true
transcoder/jobs/66/sourcefile=M\:\\sageuncm\\TheOuterLimits-TheBorderland-3085427-0.mp4
transcoder/jobs/66/status=0

[Skirge01]

Is there a way to mark all recorded movies _UNWATCHED? I want to use this in order to ensure that Intelligent Recordings isn't bypassing a movie I may be interested in simply because someone else in the household may have marked it as watched.

[Slugger]

Shouldn't be too hard (untested):

if [Category == "Movie" && IsWatched == true] {
   _UNWATCHED
}

[Slugger]

Originally Posted by Slugger

Shouldn't be too hard (untested):

if [Category == "Movie" && IsWatched == true] {
   _UNWATCHED
}

I think maybe I'm seeing the catch to this question now. Are you wanting to mark all previously watched movies as unwatched (even those that have been deleted from disk)? If so, then SJQ can't do that. SJQ only processes media files on disk. The rule above will only unmark movies that are still on the hard drive, but have been marked as watched.

[Skirge01]

Originally Posted by Slugger

I think maybe I'm seeing the catch to this question now. Are you wanting to mark all previously watched movies as unwatched (even those that have been deleted from disk)? If so, then SJQ can't do that. SJQ only processes media files on disk. The rule above will only unmark movies that are still on the hard drive, but have been marked as watched.

Hehe... you got me! My guess is that other members of the household are finishing up movies and selecting "Delete & Set Watched" and totally screwing things up. I was hoping there was a way to "go back in time" and fix those. As it is, I'm trying to browse through the historical movie data at my leisure and trying to "unwatch" everything I find. As you can imagine, this is a lost cause. Oh well, it was worth a shot.

Thanks, Slugger!

[jreichen]

Nielm's web server has a "Watched Shows" menu item that will find everything in the database that's marked as watched. It's a set of presets for the search page. Once the results are displayed, you can refine the search to narrow down to what you're looking for.

[rsagetv99]

Or you could create a favorite based on the Movies category, go into the favorite, view airings, and I think there is an option to set all watched/unwatched. Pretty sure it will include previous airings, even if you have deleted it.

[rsagetv99]

Slugger, just an FYI. I had the high CPU usage experienced by a couple other users earlier in this thread. I have been pretty hard on the system lately, stopping, restarting, rebooting, re-queing jobs, etc and my sqlite db had grown to almost 30MB. I went ahead and cleaned up the db with the below commands, and everything is OK for now. If I have the problem again once everything settles down, I will send you a copy of the db.

DELETE FROM log WHERE mediaid <> 0;
VACUUM;

Here's what was in the sage log, you can see it is complaining that the DB is locked.

Fri 6/19 12:54:34.465 (org.mortbay.log) <DEBUG> REQUEST /sjq/SJQ on org.mortbay.jetty.HttpConnection@1cc3b51
Fri 6/19 12:54:34.465 (org.mortbay.log) <DEBUG> sessionManager=org.mortbay.jetty.servlet.HashSessionManager@1ed0790
Fri 6/19 12:54:34.465 (org.mortbay.log) <DEBUG> session=null
Fri 6/19 12:54:34.465 (org.mortbay.log) <DEBUG> servlet=SJQServlet
Fri 6/19 12:54:34.465 (org.mortbay.log) <DEBUG> chain=null
Fri 6/19 12:54:34.465 (org.mortbay.log) <DEBUG> servlet holder=SJQServlet
Fri 6/19 12:54:35.894 (org.mortbay.log) <ERROR> /sjq/SJQ
Fri 6/19 12:54:35.894 java.lang.RuntimeException: java.sql.SQLException: database is locked
Fri 6/19 12:54:35.901 	at com.google.code.sagetvaddons.sjq.server.DataStore.dumpLogs(DataStore.java:734)
Fri 6/19 12:54:35.901 	at com.google.code.sagetvaddons.sjq.server.Logger.flush(Logger.java:114)
Fri 6/19 12:54:35.901 	at com.google.code.sagetvaddons.sjq.server.SJQServlet.doPost(SJQServlet.java:492)
Fri 6/19 12:54:35.901 	at com.google.code.sagetvaddons.sjq.server.SJQServlet.doGet(SJQServlet.java:92)
Fri 6/19 12:54:35.901 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
Fri 6/19 12:54:35.902 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
Fri 6/19 12:54:35.902 	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
Fri 6/19 12:54:35.902 	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:362)
Fri 6/19 12:54:35.902 	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
Fri 6/19 12:54:35.902 	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
Fri 6/19 12:54:35.902 	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:729)
Fri 6/19 12:54:35.903 	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
Fri 6/19 12:54:35.903 	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206)
Fri 6/19 12:54:35.903 	at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
Fri 6/19 12:54:35.903 	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
Fri 6/19 12:54:35.903 	at org.mortbay.jetty.Server.handle(Server.java:320)
Fri 6/19 12:54:35.903 	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
Fri 6/19 12:54:35.903 	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:829)
Fri 6/19 12:54:35.904 	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
Fri 6/19 12:54:35.904 	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
Fri 6/19 12:54:35.904 	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
Fri 6/19 12:54:35.904 	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
Fri 6/19 12:54:35.904 	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)
Fri 6/19 12:54:35.904 Caused by: java.sql.SQLException: database is locked
Fri 6/19 12:54:35.905 	at org.sqlite.DB.throwex(DB.java:288)
Fri 6/19 12:54:35.905 	at org.sqlite.DB.exec(DB.java:68)
Fri 6/19 12:54:35.905 	at org.sqlite.Conn.commit(Conn.java:172)
Fri 6/19 12:54:35.905 	at com.google.code.sagetvaddons.sjq.server.DataStore.dumpLogs(DataStore.java:724)
Fri 6/19 12:54:35.905 	... 22 more
Fri 6/19 12:54:35.905 (org.mortbay.log) <DEBUG> GET /sjq/SJQ?proto=2&cmd=showQ&sid=-260287375 HTTP/1.1
Host: XXXX.NNNN
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Content-Type: text/plain; charset=utf-8
Referer: http://XXXX:NNNN/sjq/529E702F7A146C36CFB52788388C5E7F.cache.html


Fri 6/19 12:54:35.906 (org.mortbay.log) <DEBUG> Response already committed for handling java.lang.RuntimeException: java.sql.SQLException: database is locked
Fri 6/19 12:54:35.906 (org.mortbay.log) <DEBUG> RESPONSE /sjq/SJQ  200

[Slugger]

What version of SJQ are you running? The snapshot I created last night, 483, contains the fix that will auto vacuum the db file once a week. I'd try upgrading to that version. I'd still like to know which version you're running that's causing this issue (which I thought was fixed in 2.1.4).

[Skirge01]

Thanks to everyone for their suggestions to my little dilemma. I'll check them all out and see which one works best for me.

Slugger, do you mention anywhere on the forums when you update SJQ? I never even thought to look for newer versions, thinking that I'd eventually stumble across a post mentioning the update.

[rsagetv99]

Originally Posted by Slugger

What version of SJQ are you running? The snapshot I created last night, 483, contains the fix that will auto vacuum the db file once a week. I'd try upgrading to that version. I'd still like to know which version you're running that's causing this issue (which I thought was fixed in 2.1.4).

I'm running 466. I tried just vacuuming, but it did not shrink, so I think I had too many logs. That's why I ran the other command.

Slugger, will SJQ let me use Media Variables in the :TRANSCODEOPTS parameters? Like this?

:TRANSCODEOPTS "Copy-%e% Move Play" "false" "%d%/%t%.%e%"

[Slugger]

Originally Posted by Skirge01

Thanks to everyone for their suggestions to my little dilemma. I'll check them all out and see which one works best for me.

Slugger, do you mention anywhere on the forums when you update SJQ? I never even thought to look for newer versions, thinking that I'd eventually stumble across a post mentioning the update.

I usually will post a message here on an official release. I'll only post a msg about a snapshot build if it fixes something major that people need to know about or if it includes a new feature that I'm just dying to get people to try. Otherwise, you just have to monitor the project site (either manually or subscribe to the RSS feeds).

[Skirge01]

@Slugger: Just wanted to post to specifically say THANK YOU for this plugin. It absolutely ROCKS! Just using it for commercial skipping alone gives you so much more control.

[Slugger]

Originally Posted by rsagetv99

I'm running 466. I tried just vacuuming, but it did not shrink, so I think I had too many logs. That's why I ran the other command.

Slugger, will SJQ let me use Media Variables in the :TRANSCODEOPTS parameters? Like this?

:TRANSCODEOPTS "Copy-%e% Move Play" "false" "%d%/%t%.%e%"

It will expand media vars on the file name, not on the transcode profile name. If you want the latter, open a defect and I can probably add that quickly sometime when I return home next week.

[rsagetv99]

Hi slugger, guess who?

I ran out of drive space on my system drive, but even after reclaiming 2gb, I keep getting the following error with rules that were working before. The error refers to the variable FileExists, but when I remove that from all of the rules, it then complains about Filename. Once I remove Filename, I don't get the errors, but of course then my rules don't work right. Please help.

2009/06/20 23:48:43.031	Application exception:
java.lang.NullPointerException
	at com.google.code.sagetvaddons.sjq.server.MediaVariables.getVars(MediaVariables.java:72)
	at com.google.code.sagetvaddons.sjq.server.MediaVariables.expand(MediaVariables.java:39)
	at com.google.code.sagetvaddons.sjq.server.MediaVariables.expand(MediaVariables.java:62)
	at com.google.code.sagetvaddons.sjq.server.FieldFileExists.run(FieldFileExists.java:37)
	at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.evalTest(RulesParser.java:104)
	at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.<init>(RulesParser.java:57)
	at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.<init>(RulesParser.java:69)
	at com.google.code.sagetvaddons.sjq.server.RulesParser.loadQ(RulesParser.java:145)
	at com.google.code.sagetvaddons.sjq.server.RulesParser.<init>(RulesParser.java:119)
	at com.google.code.sagetvaddons.sjq.server.QueueLoader.run(QueueLoader.java:68)
	at java.lang.Thread.run(Unknown Source)

[Slugger]

Can you open an issue ticket for this, please? I took a quick look at the code from here and what it looks like is that your wiz.bin might be out of sync/corrupted due to the out of space errors you were having. For this NPE to happen, basically Sage is giving SJQ a media object that it still thinks exists on the hard drive, but the actual file is gone so when I query Sage for the list of files from the hard drive, Sage is returning null (which should never happen) and that is causing the errors you see.

I won't be able to sit down and work on a fix for this until probably after Canada Day (I'm moving out of city this week), but in the mean time, can you turn on debug logging and reproduce the error? When you do, the server logs should at least tell us which media object is causing the NPE and perhaps the wiz.bin can be corrected to avoid the error.

I obviously should still guard against the NPE, so please do open the ticket for this and I'll tackle it as soon as I can. Post the log details and we should at least be able to get the error stopped for now.

[rsagetv99]

Well I turned on the debugger, but it hasn't been much help. I am still getting the same error, but only with the Filename and FileExists tests. When I ran the debugger against specific recordings, all of the variables expanded without any problem, and after the debug that recording would show up in the active task list.

So where would I see the name of the file that is giving me a problem in the SJQ server log?

[Slugger]

Go to the settings and enable debug logging, then run a full QueueLoader run (from the QueueLoader menu). That should produce the output I'm looking for. You should see messages in the log like ("file" == NULL) or something like that.

[rsagetv99]

Originally Posted by Slugger

Go to the settings and enable debug logging, then run a full QueueLoader run (from the QueueLoader menu). That should produce the output I'm looking for. You should see messages in the log like ("file" == NULL) or something like that.

No, it doesn't.

I put this simple rule:

if [IsTV == true && IsWatched == false && IsActivelyRecording == false && IsScheduledRecording == true && Filename =$ ".mpg" && FileExists != "%d%/%p%.edl"]
{
   COM2SKIP
}

And after I run the queue loader all I get is:

2009/06/22 11:38:23.634	QueueLoader awakened: sleep interrupted
2009/06/22 11:38:23.743	Using MediaMask: 'T'
2009/06/22 11:38:23.759	Application exception:
java.lang.NullPointerException
	at com.google.code.sagetvaddons.sjq.server.MediaVariables.getVars(MediaVariables.java:72)
	at com.google.code.sagetvaddons.sjq.server.MediaVariables.expand(MediaVariables.java:39)
	at com.google.code.sagetvaddons.sjq.server.MediaVariables.expand(MediaVariables.java:62)
	at com.google.code.sagetvaddons.sjq.server.FieldFilename.run(FieldFilename.java:33)
	at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.evalTest(RulesParser.java:104)
	at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.<init>(RulesParser.java:57)
	at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.<init>(RulesParser.java:69)
	at com.google.code.sagetvaddons.sjq.server.RulesParser.loadQ(RulesParser.java:145)
	at com.google.code.sagetvaddons.sjq.server.RulesParser.<init>(RulesParser.java:119)
	at com.google.code.sagetvaddons.sjq.server.QueueLoader.run(QueueLoader.java:68)
	at java.lang.Thread.run(Unknown Source)

2009/06/22 11:38:23.899	LogCleaner instance started
2009/06/22 11:38:23.899	FileCleaner instance started
2009/06/22 11:38:23.946	LogCleaner instance finished [47ms]
2009/06/22 11:38:25.774	FileCleaner instance finished [1875ms]

[Slugger]

Yeah, seems the thread is crashing before the log entries are flushed to the db then. I know what's causing the problem, I just don't know which file is out of sync in your wiz.bin. As I said above, I won't have time to sit down and fix this bug probably until late next week (after Canada Day [Jul 1]) as I'm getting ready to move to another city so unfortunately you're going to have to remove the tests causing problems or perhaps try to force a rebuild of your wiz.bin (if that's possible)?

The underlying problem, from what my quick review has discovered, is that Sage is returning MediaFile objects, but those objects have no associated File objects, which I don't expect, and so this is why the null pointer error is slipping through.