Plugin: Sage Job Queue (SJQ)

[Slugger]

Originally Posted by perfessor101

ps -- I still have the sjq.sqllite file that was confusing the parser

Can you please send me the sjq.sqlite file? I want to take a peek inside it and see if I can't find and squish the parser bug.

I will investigate the split recording processing after I shovel the foot of snow out of my driveway. I've been staring at the driveway since 9am and now it's nearly 2:30 and the snow didn't manage to melt itself.

[Slugger]

Server snapshot 186 contains the fix for incorrect media variable expansion on all file segments after the first one (*-1.*, *-2.*, etc.). perfessor101 (and anyone else willing to help out), can you please test the fix and report back on its effectiveness?

[edbmdave]

Happy to test it.

Can tell me how to accomplish this?

Originally Posted by Slugger:
"With SJQ you can even use external transcoding programs/scripts as well as the SageTV internal transcoder as needed."

How do you do that within SJQ? and continue to have sage recognize the file? How do you get it to run an external transcoding?

I would love to have Handbrake run all my transcoding... and still keep all the files in Sage.

[Slugger]

Originally Posted by edbmdave

Happy to test it.

Can tell me how to accomplish this?

Originally Posted by Slugger:
"With SJQ you can even use external transcoding programs/scripts as well as the SageTV internal transcoder as needed."

How do you do that within SJQ? and continue to have sage recognize the file? How do you get it to run an external transcoding?

I would love to have Handbrake run all my transcoding... and still keep all the files in Sage.

You simply create rules that run external programs against your media files.

Load the queue:

if [IsTV == true && IsActivelyRecording == false && IsWatched == false] {
   EXTTRANS
}

Configure client(s) to run those queued jobs:

EXTTRANS {
   "C:/handbrake/handbrake.exe \"%c%\""
}

Other SJQ users are more experienced with running external transcode jobs via SJQ and can confirm this, but as long as the transcoded file contains the encoded airing id in the file name, SageTV should be able to pick it up and make the replacement. Personally, I don't transcode my content or on the very, very rare occasion I have then I've always used the internal SageTV transcoder, which handles all the reregistration of the files automatically.

[scott2020]

Is it possible to use SJQ to run compression jobs off of the main server? I have my server auto compressing my selected series recordings, but it takes a long time on my older server. It would be great to be able to tell SJQ to run the compress job on another client like we do for comskip.
Scott

[CollinR]

There is still an issue of deleted files gumming up the works.

How do you force a client to stop processing a file? I have a client tring to comskip a deleted show, it will not process anything else until it's completed and it'll never complete as it doesn't exist. Stopping/starting the service both server/client don't seem to remedy it's listed as active and started and assigned.

Is there something I can put in my task so it will check to ensure the media is there before running it?

It seems my config the time it takes to catch up from something like this will force this to happen again a couple times in the next few days unless I do something. Or maybe something in the rules so it a started task takes more then X amount of time just mark it failed and go on.

The client log shows:

Date: Sat Jan 24 02:31:13 CST 2009

HTTP ERROR: 500
INTERNAL_SERVER_ERROR
RequestURI=/sjq/SJQ

Caused by:
java.lang.NullPointerException
	at com.google.code.sagetvaddons.sjq.server.ClientParser$ClientTask.toJSONObject(Unknown Source)
	at com.google.code.sagetvaddons.sjq.server.SJQServlet.doPost(Unknown Source)
	at com.google.code.sagetvaddons.sjq.server.SJQServlet.doGet(Unknown Source)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:362)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:729)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206)
	at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:320)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:829)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)

[Beefcake550]

Please help, I think I'm 9/10 of the way there but I don't know what I'm doing...

I've got SJQ installed (v2.1) and sjqc. It looks like it is doing something becuase I can see the web pages and edit server/client rules. I'm looking to be able to setup comskip to be run on the TV shows I've already recorded in SageTV.

Here's my rules:
//Comskip on MPEG
if [IsTV == "true" && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.edl" && IsActivelyRecording == "false"] {
COMSKIPMPEG
}

Client:
// Running Comskip on MPEG files (much quicker than TS files)
COMSKIPMPEG {
:CPU "NORMAL" # optional; one of NORMAL, HIGH, LOW
:MAX 1 # Run no more than one instance of this task at a time on this client
"C:/Comskip/comskip.exe -n \"%c%\"" # The %c% is expanded to the full path of the file to be processed
}

I haven't done anything else but this.

I'm not seeing anything happen. I don't know how to tell whats happening or what I'm doing wrong. I've read about some log files, but I don't see anything anywhere.

Can someone please help me out?

-B

[CollinR]

Beefcake, copy paste this.

Server Rules:

//ComSkipSD
if [IsTV == "true" && IsHD == "false" && FileExists != "%d%\\%p%.edl" && Filename =$ ".mpg" && IsActivelyRecording == "false" && ChannelNumber != "0|1|1400" && IsScheduledRecording == "true"]
{
COMSKIPSD
}
//CompressSD
if [IsTV == "true" && IsHD == "false" && FileExists == "%d%\\%p%.edl" && Filename =$ ".mpg" && IsArchived == "true" && TranscodeQueueSize <= "5" && TimeOfDay <= "02:00"]
{
:TRANSCODEOPTS "MPEG4-High Quality Deinterlaced AVI" "false"
_TRANSCODE
}
if [IsTV == "true" && IsHD == "false" && FileExists == "%d%\\%p%.edl" && Filename =$ ".mpg" && MediaAge >= "30D" && TranscodeQueueSize <= "5" && TimeOfDay <= "02:00"]
{
:TRANSCODEOPTS "MPEG4-High Quality Deinterlaced AVI" "false"
_TRANSCODE
}

Client Tasks:

//Client options
:MAXPROCS 1
// TASK definition
COMSKIPSD {  
:CPU LOW
:MAX 1    
"C:\\Comskip\\comskip.exe -n \"%c%\""
}

That will do your comskip as well as compress any SD mpgs that are either older then 30 days or have been marked as archived to xvid HQ.

Have fun.

EDIT:
The problem was the slashes in your PATH\\TO/comskip.exe

[Slugger]

Originally Posted by CollinR

There is still an issue of deleted files gumming up the works.

How do you force a client to stop processing a file? I have a client tring to comskip a deleted show, it will not process anything else until it's completed and it'll never complete as it doesn't exist. Stopping/starting the service both server/client don't seem to remedy it's listed as active and started and assigned.

Is there something I can put in my task so it will check to ensure the media is there before running it?

It seems my config the time it takes to catch up from something like this will force this to happen again a couple times in the next few days unless I do something. Or maybe something in the rules so it a started task takes more then X amount of time just mark it failed and go on.

Which server version (as reported on the bottom right corner of the GUI) are you running? This issue was fixed in v2.1.0. Starting in v2.1.0 the server checks that the media exists just before it assigns the task to a client. Now there is a slight chance that the media is deleted during the second or two after the server checks and before the client starts the assigned task, but that should almost be a one in a million shot. If you're running v2.0.0 then upgrade to v2.1.0 (or a snapshot) and let me know if you're still having problems.

[Beefcake550]

CollinR:
Thanks for the quickest response I've ever seen!

Unfortunately, I still don't see anythign happening. I think the first step is to see jobs getting added to the queue, right? How can I see that? I've tried clicking on "data->active" on the webpage, but there is nothing listed.

Thanks.

[Slugger]

Originally Posted by Beefcake550

CollinR:
Thanks for the quickest response I've ever seen!

Unfortunately, I still don't see anythign happening. I think the first step is to see jobs getting added to the queue, right? How can I see that? I've tried clicking on "data->active" on the webpage, but there is nothing listed.

Thanks.

After you update the server rules you won't see anything added to the queue until the QueueLoader thread runs again (which could be up to six hours if using the default settings). To force a manual run, click on the time of the next scheduled run in the lower right corner, wait until the last run time stamp updates, then check the active queue again (if already loaded then give it 10-15 seconds to update).

[Beefcake550]

ok, did that ...no luck.. .

Let me double check my thoughts.....Do I have to have a client connected in order for the "active queue" to get populated? I would think no, since there is a column labeled "assigned client". Is this true?

Also, when installing jetty, there was something about a default user/pass of sage/frey I never had to enter that anywhere....what was that for? Any clues?

-B

[Slugger]

Originally Posted by Beefcake550

ok, did that ...no luck.. .

Let me double check my thoughts.....Do I have to have a client connected in order for the "active queue" to get populated? I would think no, since there is a column labeled "assigned client". Is this true?

Also, when installing jetty, there was something about a default user/pass of sage/frey I never had to enter that anywhere....what was that for? Any clues?

-B

You do not need a client setup to have the queue populated, but without a client setup then the queue will, of course, never empty out. But we need to get your queue filled up first.

SJQ does not use the Jetty authentication, it uses its own. If you install other apps in Jetty they may or may not use the Jetty authentication, which is what that id/pwd is for.

Ok, next for you is to go to the settings tab, enable debugging, then manually trigger another run of the QueueLoader thread. Depending on the size of your media library, this is going to dump a massive amount of data to the logs, but it's also going to tell us why nothing is getting into the queue. After the QueueLoader runs go to the server logs and study the output, if you're not sure what you're reading, paste it in here (be sure to surround it in a "code" block in this thread for easier reading). After you run it once, be sure to turn off debug output or else you're going to swell your SJQ database to a crazy size rather quickly.

[Beefcake550]

ok, sorry to be such a newb, but I dont know how to enable debug logging. I foudn the settings tab, but there is just an empty list and I don't see how to add anything.

[CollinR]

Originally Posted by Slugger

If you're running v2.0.0 then upgrade to v2.1.0 (or a snapshot) and let me know if you're still having problems.

That was it , thanks man!

[Beefcake550]

ok, sorry to be such a newb at this. I go the setting tabs and I just see an empty list and no apparent way to add an item. How do I turn on debug logging?

also, where would the log file be placed?

-B

[Slugger]

Originally Posted by Beefcake550

ok, sorry to be such a newb, but I dont know how to enable debug logging. I foudn the settings tab, but there is just an empty list and I don't see how to add anything.

You should be seeing an options grid as in the screen shot attached. Bring up the settings submenu, click settings, then flip the debug switch. Be sure to flip it back after you run the QueueLoader once. (As I write this, I realize I should maybe rename one of those buttons.)

[Beefcake550]

oh.. very interesting...I'm not seeing that. I am using IE7 in vista......I wonder what's wrong.

[Beefcake550]

snap....I was using 127.0.0.1 as the ip address and for some reason when I switch it to my servers 192.x.x.x IP, it works. Go figure...

Thanks all...I'll post back if I have more problems.

[Wheemer]

How can I make multiple copies of comskip run on each client?

I also would like to have the captions processed at the same time as the commercials...

An example of how that might be done would be great.

This plugin rocks... can have any of the spare computers I have on my network do some of the processing work.

Also how do I get it to fire off the commands as soon as it starts recording?