Plugin: Sage Job Queue (SJQ)

[tomfisk]

Sounds pretty reasonable to me Slugger. While I'm not a "power" SJQ user certainly having access to a broader range of information typically isn't a bad thing. About the only downside I can see is having to reprogram some rules when upgrading from v2 to v3.

[Beefcake550]

this is a generic question, but I'm not suer of the answer.

My situation is that I use SJQ to run compskip and generate edl files. When I delete the show, the edl file still exists and I end up with a bunch of unused edl files. Is there a way to have SJQ clean these up? I also just thought this might be an optino is SageMC (which I am using) and I'll look into that.

I am thinking SJQ can't because it just scans all media files..

[Slugger]

Originally Posted by Beefcake550

this is a generic question, but I'm not suer of the answer.

My situation is that I use SJQ to run compskip and generate edl files. When I delete the show, the edl file still exists and I end up with a bunch of unused edl files. Is there a way to have SJQ clean these up? I also just thought this might be an optino is SageMC (which I am using) and I'll look into that.

I am thinking SJQ can't because it just scans all media files..

SJQ can do this... have a look at the file cleaner settings (both in the GUI, but especially in the User's Guide as it'll explain how to set it up).

[nyplayer]

Originally Posted by Beefcake550

this is a generic question, but I'm not suer of the answer.

My situation is that I use SJQ to run compskip and generate edl files. When I delete the show, the edl file still exists and I end up with a bunch of unused edl files. Is there a way to have SJQ clean these up? I also just thought this might be an optino is SageMC (which I am using) and I'll look into that.

I am thinking SJQ can't because it just scans all media files..

SJQ already does that bring up SJQ in your browser and look under tools file cleaner.

[Beefcake550]

ah, I forgot about this section. I recently had to reinstall SageTV and had to reconfigure SJQ and noticed these files building up. So, since I had setup SJQ many months ago when it first came out, I forgot to re-setup this file cleaning.

Thanks for reminding me.

On that note, is there a way to backup/save/restore the client and server rulesets?

[nyplayer]

Originally Posted by Beefcake550

ah, I forgot about this section. I recently had to reinstall SageTV and had to reconfigure SJQ and noticed these files building up. So, since I had setup SJQ many months ago when it first came out, I forgot to re-setup this file cleaning.

Thanks for reminding me.

On that note, is there a way to backup/save/restore the client and server rulesets?

The way I back them up is simply to go into the rules right click select all copy and paste into notepad and save somewhere. That way I can just paste them back in If I need to.

[Slugger]

Originally Posted by Beefcake550

ah, I forgot about this section. I recently had to reinstall SageTV and had to reconfigure SJQ and noticed these files building up. So, since I had setup SJQ many months ago when it first came out, I forgot to re-setup this file cleaning.

Thanks for reminding me.

On that note, is there a way to backup/save/restore the client and server rulesets?

Backup the sjq.sqlite file in the SageTV root installation directory. That database contains all those settings. If you need to reinstall then simply restore that db file and everything should be restored.

[Addict]

EDIT: I found the problem. I had cut/pasted from the PDF, which had # comments in it...after removing those, the system runs fine. Might cause issues for others, so might be worth changing the PDF for the time being. Cheers!

Hey there...been trying to find a solution to this in this (quite long) thread, but am not having any luck.

Everything seems to be configured correctly, I've run through the PDF document step-by-step and have a server and Windows-Service based client installed on the same machine. I see the client in the Clients screen, but have the below error log from there. Same error occurs every 5 minutes or so, and have yet to have it kick-off the task.

I have 6 items in my queue (Waiting). I have one task on the client (ComSkip, per instructions in PDF).

Thoughts? THANKS!!

HTTP ERROR 500
2009/10/11 17:47:01.600	Problem accessing /sjq/SJQ. Reason:
2009/10/11 17:47:01.741	    Unexpected '#' on line 1
Caused by:java.lang.RuntimeException: Unexpected '#' on line 1
2009/10/11 17:47:01.913		at com.google.code.sagetvaddons.sjq.server.ClientParser.<init>(ClientParser.java:192)
2009/10/11 17:47:02.053		at com.google.code.sagetvaddons.sjq.server.SJQServlet.doPost(SJQServlet.java:166)
2009/10/11 17:47:02.194		at com.google.code.sagetvaddons.sjq.server.SJQServlet.doGet(SJQServlet.java:92)
2009/10/11 17:47:02.335		at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
2009/10/11 17:47:02.475		at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
2009/10/11 17:47:02.616		at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
2009/10/11 17:47:02.772		at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389)
2009/10/11 17:47:02.928		at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
2009/10/11 17:47:03.100		at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
2009/10/11 17:47:03.335		at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
2009/10/11 17:47:03.600		at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
2009/10/11 17:47:03.866		at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
2009/10/11 17:47:04.038		at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
2009/10/11 17:47:04.210		at org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:230)
2009/10/11 17:47:04.366		at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
2009/10/11 17:47:04.522		at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
2009/10/11 17:47:04.694		at org.mortbay.jetty.Server.handle(Server.java:326)
2009/10/11 17:47:04.850		at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
2009/10/11 17:47:05.038		at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
2009/10/11 17:47:05.194		at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
2009/10/11 17:47:05.335		at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
2009/10/11 17:47:05.491		at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
2009/10/11 17:47:05.631		at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
2009/10/11 17:47:05.803		at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
2009/10/11 17:47:06.006

[Slugger]

Please paste in your client config here. Be sure to surround it in code tags in your reply for easier reading.

[razrsharpe]

I'm not a current SJQ user but have been following your work on you google page. The time and date availability rules is the last thing stopping me from switching from dirmon2 to SJQ... I do have a question about SJQ that loosely relates to availability.

Does SJQ have the notion of resource allocation... aka you can run multiple threads of comskip but only one video conversion thread? like in dirmon2 each "job" consumes x amount of resource out of a total y. Dirmon2 will then allocate jobs until the sum of all x is the maximum that does not exceed y.

not sure if this notion of resource allocation already... if so sorry.. maybe i should rtfm ... if not though it would be a welcome addition to SJQ

[Slugger]

Originally Posted by razrsharpe

I'm not a current SJQ user but have been following your work on you google page. The time and date availability rules is the last thing stopping me from switching from dirmon2 to SJQ... I do have a question about SJQ that loosely relates to availability.

If you're following this ticket on the project site then I guess you got the email saying that I started work on this today (actually working on it right now)? If not, well, I am.

Here's what I've done:

Add two options to the client config "OFFDAY" and "OFFHOUR". You can specify a range of days (of the week, 1-7) or hours of the day (0-23) and if the current day or hour matches any of the listed ranges then the client is disabled and will not receive new tasks to run (any tasks that are running are not killed).

Does SJQ have the notion of resource allocation... aka you can run multiple threads of comskip but only one video conversion thread? like in dirmon2 each "job" consumes x amount of resource out of a total y. Dirmon2 will then allocate jobs until the sum of all x is the maximum that does not exceed y.

not sure if this notion of resource allocation already... if so sorry.. maybe i should rtfm ... if not though it would be a welcome addition to SJQ

Yes, if I'm understanding how dirmon2 works. Basically in SJQ you can configure a client to run a max of X tasks simultaneously (aka max threads). Then you can also tell the client to run no more than Y instances of any specific task.

So you can say something like, "run up to 5 tasks simultaneously on this client, but no more than 3 comskips, 1 transcode and unlimited fanart downloads." Of course, "unlimited" is unlimited up to 5 so you'll never have more than 5 task threads running at once and never any more than 3 comskip threads and never any more than 1 transcode thread. Hopefully that makes sense?

So now I just have to get the bugs out of the interpreter for the off hours/off day settings.

[loonsailor]

Originally Posted by razrsharpe

I'm not a current SJQ user but have been following your work on you google page. The time and date availability rules is the last thing stopping me from switching from dirmon2 to SJQ... I do have a question about SJQ that loosely relates to availability.

Does SJQ have the notion of resource allocation... aka you can run multiple threads of comskip but only one video conversion thread? like in dirmon2 each "job" consumes x amount of resource out of a total y. Dirmon2 will then allocate jobs until the sum of all x is the maximum that does not exceed y.

not sure if this notion of resource allocation already... if so sorry.. maybe i should rtfm ... if not though it would be a welcome addition to SJQ

I'm not an sjq expert, but it's working quite well for me. You can limit the number of jobs that can be run simultaneously. I have a quad core, so I set it to 2, since Sage uses 1, and that always leaves one for an ssh or something. I don't know if it has the job weighting concept (maybe it does and I haven't learned about it) but, as a practical matter, I don't think there's a compelling reason for it. For me, comskip runs .mpg files from my HDHRs and .ts files from my hdpvr. The .mpgs are very fast - <2 minutes for a one hour show. The .ts files are much, much slower, 1-2 hours for a one hour show. If I could put a much larger weighting on the .ts files, I suppose all the .mpgs would run first (yes?), which would take < 4 minutes, because I can only record four at a time max, and then I might spend an hour or two processing the 1-2 .ts files recorded at the same time. Without weighting, the order is more random - the .ts files might go first - but the total time spent is the same. Since the .mpgs are so much faster, in either scenario 99% of the time will still be spent processing the .ts files. Unless I've got two .ts files to process (made be two hdpvr's recording simultaneously) in either scenario, one core will be busy for a long time doing a .ts if I've got one and the other will quickly crank out the .mpgs. The only disadvantage of no weighting, I guess, is that if I have two .ts recorded simultaneously, and I were to watch the .mpg very soon after the recording it is less likely to be comskipped without a weighting factor. That just doesn't seem like a big deal to me.

[razrsharpe]

Originally Posted by Slugger

If you're following this ticket on the project site then I guess you got the email saying that I started work on this today (actually working on it right now)? If not, well, I am.

Here's what I've done:

Add two options to the client config "OFFDAY" and "OFFHOUR". You can specify a range of days (of the week, 1-7) or hours of the day (0-23) and if the current day or hour matches any of the listed ranges then the client is disabled and will not receive new tasks to run (any tasks that are running are not killed).

yup got the email .. sounds exactly like what i'm looking for... what would happen if you were to specify 2000 - 0800? also is the OFFDAY/OFFHOUR on a per task or per client basis? For example I would want transcoding to be off from 1600 to 2359, but comskipping to be on all the time. If its on a per client basis can i have two virtual clients that are the same physical machine?.. then i can assign only specific tasks to the specific client... with the blackout rules...

Yes, if I'm understanding how dirmon2 works. Basically in SJQ you can configure a client to run a max of X tasks simultaneously (aka max threads). Then you can also tell the client to run no more than Y instances of any specific task.

So you can say something like, "run up to 5 tasks simultaneously on this client, but no more than 3 comskips, 1 transcode and unlimited fanart downloads." Of course, "unlimited" is unlimited up to 5 so you'll never have more than 5 task threads running at once and never any more than 3 comskip threads and never any more than 1 transcode thread. Hopefully that makes sense?

So now I just have to get the bugs out of the interpreter for the off hours/off day settings.

also sounds like it will do what i want thanks for explaining it again ... one question though... say i want it to be 3 comskips OR 1 transcode ... ie if its transcoding i don't want it do anything else processor intensive (comskipping)... its probably not really a big deal with the blackout rules but figured id ask

[Slugger]

Originally Posted by razrsharpe

yup got the email .. sounds exactly like what i'm looking for... what would happen if you were to specify 2000 - 0800? also is the OFFDAY/OFFHOUR on a per task or per client basis? For example I would want transcoding to be off from 1600 to 2359, but comskipping to be on all the time. If its on a per client basis can i have two virtual clients that are the same physical machine?.. then i can assign only specific tasks to the specific client... with the blackout rules...

Blackouts are client based, not task based. You can't say 2000-0800, but you can say:

20-23,0-8

Which would be the same thing. Blackouts are by hour only (as currently written). No, you can't have virtual machines. Machines are identified by IP address so you can't. Except on the localhost, you actually can have two separate clients running. One that connects to 127.0.0.1 and one that connects via the LAN IP (192.x.x.x or whatever). In that case only, you can actually have two separate clients on the same physical machine (this wasn't done on purpose, but happens to work in this case).

also sounds like it will do what i want thanks for explaining it again ... one question though... say i want it to be 3 comskips OR 1 transcode ... ie if its transcoding i don't want it do anything else processor intensive (comskipping)... its probably not really a big deal with the blackout rules but figured id ask

Can't be done. You set a max threads for the client then max values for each task. You can't say 3 x OR 1 y.

I might look into moving the client blackouts to the task level, but that's a big maybe b/c it really adds a whole new level of complications that I really don't want to write, test, nor maintain. The client level blackouts are actually quite easy to add, but I'll take a look (seems unlikely though).

[Slugger]

Ok, so I did a little reading of the source code and adding task based blackouts wasn't as big a deal as I thought it would be so I added it. Now clients can be blacked out and tasks can be blacked out.

This will all be added to the new SJQ v3 user guide, but as a preview (and a place for me to scribble some notes to remind me), here's what you can do with the clients and tasks now:

Ruleset

# We don't want to transcode Criminal Minds episodes between 8pm and 2am daily (for whatever reason)
if [IsObjMediaFile == true && $.GetShowTitle{} == "Criminal Minds"] {
   :OFFHOUR "20-23,0-2" # Blackout this task from 8:00pm-11:59pm and from midnight to 2:59am
   MYTRANSCODE
}

Client config

# The OFFHOUR and OFFDAY settings are OR'd together (i.e. if either one matches then the blackout test is considered true
:OFFHOUR "19-23" # This client is blacked out daily from 7pm-11:59pm
:OFFDAY "2-5" # This client is also blacked out all day Mon-Fri, inclusive
MYTRANSCODE {
   "/path/to/transcode/exe \"%c%\""
}

[razrsharpe]

Originally Posted by Slugger

Ok, so I did a little reading of the source code and adding task based blackouts wasn't as big a deal as I thought it would be so I added it. Now clients can be blacked out and tasks can be blacked out.

wowow thank you, i look forward to trying out v3

[Slugger]

Originally Posted by razrsharpe

wowow thank you, i look forward to trying out v3

[Pasted from the issue ticket...]

Some things to consider:

Let's say you don't want to run transcode tasks between 8pm and 11pm because that's
when you're watching tv so you black out the transcode task in your ruleset...

Well, if the server does a scan at 7:58pm it'll add the task to the queue (because
it's not between 8pm-11pm yet). A client might not check for work again until 8:01pm
and it'll get that task and run it during the "blackout period". Even if you blacked
out the client from 8pm-11pm the server might scan at 7:58 and add the task and the
client might check for work at 7:59 and get the task to run which means it'll be
running that task during the blackout period.

Task blackouts are used to prevent tasks from entering the queue. Once in the queue,
tasks will be assigned to clients that aren't blacked out and aren't running at capacity.

There are so many edge cases that simply can't be contained nor am I going to try to
contain them all. In other words, the blackout settings are a best effort kind of
thing, but it's almost a guarantee that your blackout rules won't be honoured and
it's because of scenarios like the one I just described.

[razrsharpe]

Originally Posted by Slugger

Let's say you don't want to run transcode tasks between 8pm and 11pm because that's
when you're watching tv so you black out the transcode task in your ruleset...

Well, if the server does a scan at 7:58pm it'll add the task to the queue (because
it's not between 8pm-11pm yet). A client might not check for work again until 8:01pm
and it'll get that task and run it during the "blackout period". Even if you blacked
out the client from 8pm-11pm the server might scan at 7:58 and add the task and the
client might check for work at 7:59 and get the task to run which means it'll be
running that task during the blackout period.

understood once tasks are in the queue they will be executed and completed... dirmon2 actually operates in a similar manner... so obviously the blackout times that are chosen need some smartness from the user (me)... ie transcode tasks normally take about 3.5 hours to complete for a 1 hour program on my q6600... therefore (using the example above) i would set the blackout time starting at 4pm for transcode tasks... that way if something is entered into the queue "around" 4pm it will still complete and not interfere with when i am watching/recording the majority of my tv.

[mguebert]

I have been able to get SJQ running fine, and have a ruleset that is setting up jobs to run. I am on SJQ ver 2.1.5.542.

I have tried to install the standalone client with 127.0.0.1, \\server, and localhost and I get the following output from the command line

SJQ Task Client v2.1.5.556
Using 'C:/sjqc' as location for configuration file...
Exception in thread "main" java.lang.RuntimeException: java.io.FileNotFoundExcep
tion: http://localhost:80/sjq/SJQ?proto=2&cmd=writeLog
at com.google.code.sagetvaddons.sjqc.TaskClient.log(Unknown Source)
at com.google.code.sagetvaddons.sjqc.TaskClient.log(Unknown Source)
at com.google.code.sagetvaddons.sjqc.TaskClient.pop(Unknown Source)
at com.google.code.sagetvaddons.sjqc.TaskClient.run(Unknown Source)
at com.google.code.sagetvaddons.sjqc.TaskClient.main(Unknown Source)
Caused by: java.io.FileNotFoundException: http://localhost:80/sjq/SJQ?proto=2&cm
d=writeLog
at sun.net.http://www.protocol.http.HttpURLConn...Stream(Unknown So
urce)
... 5 more


I have also tried the service and get the same results, here is the output from the wrapper.log

STATUS | wrapper | 2009/10/18 17:13:14 |
STATUS | wrapper | 2009/10/18 17:13:14 | Launching a JVM...
INFO | jvm 1 | 2009/10/18 17:13:14 | WrapperManager: Initializing...
INFO | jvm 1 | 2009/10/18 17:13:15 | SJQ Task Client v2.1.5.556
INFO | jvm 1 | 2009/10/18 17:13:15 | Using 'C:/sjqc' as location for configuration file...
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp:
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: Encountered an error running main:
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: java.lang.RuntimeException: java.io.FileNotFoundException: http://127.0.0.1:80/sjq/SJQ?proto=2&cmd=writeLog
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: at com.google.code.sagetvaddons.sjqc.TaskClient.log(Unknown Source)
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: at com.google.code.sagetvaddons.sjqc.TaskClient.log(Unknown Source)
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: at com.google.code.sagetvaddons.sjqc.TaskClient.pop(Unknown Source)
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: at com.google.code.sagetvaddons.sjqc.TaskClient.run(Unknown Source)
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: at com.google.code.sagetvaddons.sjqc.TaskClient.main(Unknown Source)
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: at java.lang.reflect.Method.invoke(Unknown Source)
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:236)
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: at java.lang.Thread.run(Unknown Source)
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: Caused by: java.io.FileNotFoundException: http://127.0.0.1:80/sjq/SJQ?proto=2&cmd=writeLog
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: at sun.net.http://www.protocol.http.HttpURLConn...Stream(Unknown Source)
INFO | jvm 1 | 2009/10/18 17:13:15 | WrapperSimpleApp: ... 11 more
STATUS | wrapper | 2009/10/18 17:13:17 | <-- Wrapper Stopped

I have Java ver 6 update 15 installed


Can anyone help me get the client working?

[btrcp2000]

Despite the presence of these in my ini:

output_default=0
output_edl=1

comskip will not output edl files when triggered by SJQ. It will do it if I manually drag and drop files.

Here is my sjq settings file if it helps:
:MAXPROCS 3
COMSKIP {
:MAX 3
:CPU "NORMAL"
:RETURNCODE 1
"C:/Program Files/SageTV/SageTV/comskip/comskip.exe \"%c%\""
}