Plugin: Sage Job Queue (SJQ)

[Slugger]

I will create the JS debug build, but it won't be tonight. Going to have to do some reading up on generating debug JS code from GWT compiler so it'll be an evening task when I find a free one this week (may Mon, probably Wed).

[Slugger]

The GWT debug builds are available at the project download area. Be sure to grab a build with "gwtdbg" in the file name. Once you install that build then reproduce the problems you were having and then paste back the JavaScript error. The error message (file and line number) should point me to something I can trace.

[ghostlobster]

Wow! SJQ looks incredibly powerful! Unfortunately, I have absolutely NO idea what anything in the Wiki is talking about, nor do I honestly want to sped a lot of time rummaging through it and learning. I know, I know...if you don't want to learn it, then don't use it!
Basically, all I'm looking to do is use Comskip against h.264 recordings. I'm already testing SAV1 right now, and just donated to Comskip and got access to the donator's version. I want to do some side by side comparisons of their H.264 capabilities.
So...when I checked the Comskip forum for some instructions on setting it up with SageTV, they recommend using SJQ and point to this thread. In looking through this thread and then going through the Wiki, I can not find anything that remotely resembles instructions for setting up SJQ to launch comskip against recorded TV files? I have SJQ set up, I can access it within the Jetty server without a problem. I just have absolutely no clue what to do next. Has anyone written up a document for setting up SJQ to launch comskip?
Thanks!

[Slugger]

Originally Posted by ghostlobster

Wow! SJQ looks incredibly powerful! Unfortunately, I have absolutely NO idea what anything in the Wiki is talking about, nor do I honestly want to sped a lot of time rummaging through it and learning. I know, I know...if you don't want to learn it, then don't use it!
Basically, all I'm looking to do is use Comskip against h.264 recordings. I'm already testing SAV1 right now, and just donated to Comskip and got access to the donator's version. I want to do some side by side comparisons of their H.264 capabilities.
So...when I checked the Comskip forum for some instructions on setting it up with SageTV, they recommend using SJQ and point to this thread. In looking through this thread and then going through the Wiki, I can not find anything that remotely resembles instructions for setting up SJQ to launch comskip against recorded TV files? I have SJQ set up, I can access it within the Jetty server without a problem. I just have absolutely no clue what to do next. Has anyone written up a document for setting up SJQ to launch comskip?
Thanks!

The first example in the user guide does a complete step by step on how to setup comskip to process all recordings. A little extra effort and you can have it only process H.264 recordings.

http://docs.google.com/Doc?id=dz87gf...s_All_Schedule

[ghostlobster]

Thanks. I've already gone through that and all of my recordings are showing as "Waiting"
How does SJQ know what the COMSKIP job is and where the comskip.exe file is located and what parameters to run against it?
I tried going into the Clients tabe to enter:
COMSKIP {
"C:/comskip/comskip.exe \"%c%\""
}
but that field is not editable.
Also, when I go into Tools/Edit settings, the Save button is always blanked out. I've confirmed that I have selected something for each variable, entered 5 minutes as the Maxiumum time QueueLoader will sleep between runs (in minutes) and 30 as Amount of time the QueueLoader will delay before each run (in seconds) so there are values in those fields.
Obviously I cannot run in debug mode, as I can not Save the settings here.

Sorry for being dense.

[Slugger]

Originally Posted by ghostlobster

Thanks. I've already gone through that and all of my recordings are showing as "Waiting"
How does SJQ know what the COMSKIP job is and where the comskip.exe file is located and what parameters to run against it?
I tried going into the Clients tabe to enter:
COMSKIP {
"C:/comskip/comskip.exe \"%c%\""
}
but that field is not editable.

Did you install a task client? Is it running? Is it connected to the SJQ server? If it is, then clicking on the client menu should list the connected client then you can click on that client then edit it, which is where you would enter the stuff above.

Also, when I go into Tools/Edit settings, the Save button is always blanked out. I've confirmed that I have selected something for each variable, entered 5 minutes as the Maxiumum time QueueLoader will sleep between runs (in minutes) and 30 as Amount of time the QueueLoader will delay before each run (in seconds) so there are values in those fields.
Obviously I cannot run in debug mode, as I can not Save the settings here.

You're not the first person to report this problem. What version are you running? Unfortunately, I'm just not able to reproduce this problem and so I have no way to track it down at the moment. Could you try installing the GWT debug build (sjq-gwtdbg-2.1.5.389.zip) and then reproducing this and getting me the JavaScript error?

[ghostlobster]

Originally Posted by Slugger

Did you install a task client? Is it running? Is it connected to the SJQ server? If it is, then clicking on the client menu should list the connected client then you can click on that client then edit it, which is where you would enter the stuff above.

I don't believe that I've installed a task client. However, I may be incorrect in my interpretation of the term. I'm guessing that a Task Client is a secondary PC that is utilized for performing specific tasks, such as analyzing recordings for commercial skipping, offloading these processes from the main SageTV rig. If that is the case, then no, I have not. All I've done is put it in place on my WHS box. The reason I even looked at the clients page was because that was the only section of the document that had any reference to comskip.exe.
Just curious, are the docs implying that all I need to do is put SJQ in place on the server, then place
if [IsTV == true && IsScheduledRecording == true] {
:PRIORITY 10 # Optional, default priority is zero
COMSKIP
}
in the ruleset editor, and it'll just work? No additional configuration required, no pointer to comskip.exe, nothing? If so, then I'm way over complicating things in my mind!

Originally Posted by Slugger

You're not the first person to report this problem. What version are you running? Unfortunately, I'm just not able to reproduce this problem and so I have no way to track it down at the moment. Could you try installing the GWT debug build (sjq-gwtdbg-2.1.5.389.zip) and then reproducing this and getting me the JavaScript error?

About shows this:
Author: Derek Battams <derek AT battams DOT ca>
Project Site: http://sagetv-addons.googlecode.com
GWT: 1.5.3 | GWT Home Page
GWT Incubator: 1.5.3 | GWT Incubator Home Page
GWTSRWC: 0.1.0.28 | GWTSRWC Home Page

[Slugger]

Originally Posted by ghostlobster

I don't believe that I've installed a task client. However, I may be incorrect in my interpretation of the term. I'm guessing that a Task Client is a secondary PC that is utilized for performing specific tasks, such as analyzing recordings for commercial skipping, offloading these processes from the main SageTV rig. If that is the case, then no, I have not. All I've done is put it in place on my WHS box. The reason I even looked at the clients page was because that was the only section of the document that had any reference to comskip.exe.
Just curious, are the docs implying that all I need to do is put SJQ in place on the server, then place
if [IsTV == true && IsScheduledRecording == true] {
:PRIORITY 10 # Optional, default priority is zero
COMSKIP
}
in the ruleset editor, and it'll just work? No additional configuration required, no pointer to comskip.exe, nothing? If so, then I'm way over complicating things in my mind!

You must also install the task client. You can install it on the same system as the SJQ server, but it must be installed somewhere. From the last paragraph of example 1 in the user guide:

And that's it! You've setup the SJQ server and have configured it to queue up COMSKIP tasks for all recordings that have been scheduled. You can continue on reading the remaining examples for more advanced rulesets or you can move onto chapter three, which deals with installing and configuring a task client to actually perform the tasks you've just queued up. As it stands, those tasks will remain in the queue and stuck in the waiting state until you configure a task client that is capable of running COMSKIP tasks. Chapter three continues on with this example and goes through the details of configuring the task client to run the tasks you've just queued up in this example.

About shows this:
Author: Derek Battams <derek AT battams DOT ca>
Project Site: http://sagetv-addons.googlecode.com
GWT: 1.5.3 | GWT Home Page
GWT Incubator: 1.5.3 | GWT Incubator Home Page
GWTSRWC: 0.1.0.28 | GWTSRWC Home Page

The SJQ version is in the title bar of this popup window (i.e. SJQ v2.1.5.xxx), it's the xxx number that really means something to me (but convention is to refer to your version by the entire version number string).

[ghostlobster]

It's 466.
OK, I've assigned a task client to the comskip task. Rather than use my very underpowered WHS box for this, I'm using my main Vista rig, which was also handling the Show Analyzer processing. I've disabled the SA services so that they don't step on one another, I've put the donators version of Comskip into a c:\comskip directory on the vista box and used the syntax in the example to point to it.
Low and behold, my recordings are now being assigned to the remote client and changing from waiting to running.
I understand that as I'm running this on Vista, I cannot use the service method of running the task client. That's OK, as I'll just put a shortcut to the batch file in my startup group and make certain that a user is always logged in.

I'm guessing that any issues stumbled onto from here would be Comkip related.

Now, at some point within the next day or so I'll put your debug version in place and see what's going on with the settings page. Is there anything in particular I need to know about applying that beforehand?

[Slugger]

Originally Posted by ghostlobster

Now, at some point within the next day or so I'll put your debug version in place and see what's going on with the settings page. Is there anything in particular I need to know about applying that beforehand?

Nothing special, just check the Help>About menu after upgrading and ensure you're running 489 (or which ever version number is associated with the debug build, I believe it is 489). Then reproduce the problem with the settings page and then report back the JavaScript error(s) you are seeing.

Also, if you can enable debug logging in Jetty and paste in the logs from the request to the server, I'd really like to know if the request is throwing a 500 error back to the web browser.

[ghostlobster]

Will do in a bit.
Right now, though, something weird is happening. All of the recordings are showing up under Failed Tasks.
I've checked the RecordedTV folder on the server and the .txt, .edl and .log files are being created just fine. In looking at the .edl file, the final lines of one of the shows are:
107851 Frames Processed
####################################################
Time at end of run:
Tue Aug 11 17:49:16 2009
####################################################
In looking at the job log within SJQ's interface, I see the following:

2009/08/11 17:01:42.189 Tue Aug 11 17:01:43 EDT 2009: Starting task with priority 5/1/10
2009/08/11 17:01:42.283 Executing command line: C:/comskip/comskip.exe "y:\DivineDesign-CatchalltoSharedBedroom-4105566-0.ts"
2009/08/11 17:18:59.923 Comskip 0.80.023, made using:
2009/08/11 17:18:59.923 mpeg2dec-0.4.0 - by Michel Lespinasse <walken@zoy.org> and Aaron Holtzman
2009/08/11 17:18:59.923 0:00:00 - 46 frames in 1.00 sec (46.00 fps), 1.00 sec (46.00 fps), 0%
~~~~a bunch of similar lines~~~~~
2009/08/11 17:49:15.205 0:00:00 - 107840 frames in 2701.50 sec (39.92 fps), 4.26 sec (3.76 fps), 99%
2009/08/11 17:49:15.205
2009/08/11 17:49:15.205 107851 frames decoded in 2701.70 seconds (39.92 fps)
2009/08/11 17:49:15.205 [h264 @ 03990A80]error while decoding MB 18 0, bytestream (td)
2009/08/11 17:49:15.205 [h264 @ 03990A80]error while decoding MB 58 1, bytestream (td)
~~~~A bunch of similar lines~~~~

Now, even though the .txt, .log and .edl files are present, the show timeline does not display the presence of any commercial skip data (as it did when using Show Analyzer) I'm not certain if this is a SJQ issue or a Comskip issue. I'm leaning towards Comskip as the Comskip log shows 2 lines that repeat throughout just containing the following:
Audio error
Audio error

Any ideas?

[Slugger]

comskip returns 1 on success instead of the expected value of zero and so this is why SJQ thinks each run is failing. See the RETURNCODE option in the user guide for more info on how to fix that.

The other errors are comskip problems that are independent of SJQ.

[ghostlobster]

Originally Posted by Slugger

comskip returns 1 on success instead of the expected value of zero and so this is why SJQ thinks each run is failing. See the RETURNCODE option in the user guide for more info on how to fix that.

The other errors are comskip problems that are independent of SJQ.

Again, my apologies, but that doc was obviously written by developers, for developers. I'm not a developer. Thgere are 3 occurrences of the term RETURNCODE in the doc.
1 - In the TOC
2 - In the Data Views section, under Viewing Completed/Failed tasks.
3 - Under Client Configuration Syntax. whic I'm guessing is the correct section for my needs.
I don't see anything that would indicate the proper syntax and location for inserting the proper command/argument for this. Would I go into the Client, Configure the client and then chang this:
COMSKIP {
"C:/comskip/comskip.exe \"%c%\""
}
to this:
COMSKIP {
:RETURNCODE 1
"C:/comskip/comskip.exe \"%c%\""
}

?

[Slugger]

You got it.

I realize the doc is a little technical as I, the author of the software, wrote it and he who documents his own work, though better than none, is definitely doing so from a skewed point of view, I accept that. However, searching blindly for the keyword and reading a little section out of context definitely isn't the way the documentation is expected to work. For that section on RETURNCODE to be useful to someone I'd expect that they've also read up on how to apply option speicifiers (which was a few sections above). Even better, take the time to read how to completely configure a task client by reading, not skimming, chapter three. I simply suggested reading up on RETURNCODE based on the assumption that you had already worked through example 1 on the client side and server side.

[ghostlobster]

My post definitely does not read as I intended it, and I'm sorry about that. Just another case of RTFM

OK, I made the change and will wait and see what happens from here.

By the way, it's an enormous doc...much more than most bother to assemble and can be a bit intimidating. If I ever get the hang of SJQ and get a better understanding of the terminology, usage examples and capabilities, I might be able to put together an DEU (dumb end user) oriented idiot's guide to SJQ, citing case and step based examples from the concepts. I'm going to need to go and get some more beer before that though.

Actually, an overall commercial-skip doc might not be a bad idea at this point. Enough has changed with the advent of h.264 and all of the options we now have as opposed to a few years ago. Something outlining some basic best practices when using SJQ/Comskip, SAV1, and DTb...hmmm...perhaps I'll grab a bottle of gin instead of beer before going that deep.

[ghostlobster]

OK, after tinkering around with this and getting to know it, I have to say...Wow, what a powerful add-on this is! It's a bit intimdating at first, but once you get your brain around the methodology, it just plain rocks.
Nice work on it. I have not had a hiccup with it since I got over all of the end-user issues. Thanks again!

[nyplayer]

Slugger,

Is there a way for SJQ to determine when a recording has more than 1 part.... broken into say 2 files etc...?

[Slugger]

Yes. It actually already knows about multi-file objects and is suppose to run the task on each file part. Is it not doing that? Or do you want a test that tells you how many file parts are in the MediaFile object?

[nyplayer]

I would like a test if its more than 1 Part if possible. SageTV seems to have trouble recognizing multi part transcoded files using an external transcoder.

[Slugger]

Originally Posted by nyplayer

I would like a test if its more than 1 Part if possible. SageTV seems to have trouble recognizing multi part transcode files using an external transcoder.

Can you open a ticket for this? Won't happen today, probably some morning early next week before I start work. Really a simply/quick addition, just my day/weekend is full (big year end ball tournament starts tonight ).