Plugin: Sage Job Queue (SJQ)

[Grasshopper]

I have Sage Job Queue installed and working (I think). I have Comskip also installed and can manually run it by dragging a recorded show file onto it; I haven't tuned the commercial skipping yet, but it appears to work okay. But I'm at a loss when it comes to using Sage Job Queue to automate the comskip process. Can someone post an example of a ruleset (am I even talking about the right thing?) for a very simple comskip call without a lot of complex conditions? I have a very simple Sage server setup, running on a WHS box. I don't need to process on another pc, just the server is fine.

[scott2020]

Originally Posted by scott2020

Is it possible to use SJQ to run compression jobs off of the main server? I have my server auto compressing my selected series recordings, but it takes a long time on my older server. It would be great to be able to tell SJQ to run the compress job on another client like we do for comskip.
Scott

I'm sorry, I am a total moron! The post right above my question had a starting point to run a transcode job. Perhaps looking at the command lines on other programs will give some insight as to keeping the episode information and such.

After the compress/transcode job is done, I assume comskip will need to be run again? The jobs I did run on the server wiped out the comskip results obviously since the file is much different.
Scott

[Slugger]

Originally Posted by Wheemer

How can I make multiple copies of comskip run on each client?

Configure the client to run multiple tasks simultaneously. The setup is described in the wiki doc.

I also would like to have the captions processed at the same time as the commercials...

An example of how that might be done would be great.

No clue on this one; I assume it's a command line option to comskip or sa? If it is a command line option then simply pass that option when calling comskip or sa. Someone else may have an answer on this one.

This plugin rocks... can have any of the spare computers I have on my network do some of the processing work.

Also how do I get it to fire off the commands as soon as it starts recording?

Ok, this one's not in the wiki... will add a new doc shortly. On the settings grid, change the RunType setting from 'AfterRec' to 'DuringRec'. When set to 'DuringRec' the QueueLoader will run 10 seconds after the start of each recording, which will queue up the jobs, but it may be up to 5 more minutes before a client comes along and starts the queued job.

[Slugger]

Originally Posted by Grasshopper

I have Sage Job Queue installed and working (I think). I have Comskip also installed and can manually run it by dragging a recorded show file onto it; I haven't tuned the commercial skipping yet, but it appears to work okay. But I'm at a loss when it comes to using Sage Job Queue to automate the comskip process. Can someone post an example of a ruleset (am I even talking about the right thing?) for a very simple comskip call without a lot of complex conditions? I have a very simple Sage server setup, running on a WHS box. I don't need to process on another pc, just the server is fine.

The wiki doc has some simple examples that should help you on your way. (I just added another example showing a typical rule for setting up comskip/showanalyzer.)

[Slugger]

Originally Posted by scott2020

I'm sorry, I am a total moron! The post right above my question had a starting point to run a transcode job. Perhaps looking at the command lines on other programs will give some insight as to keeping the episode information and such.

After the compress/transcode job is done, I assume comskip will need to be run again? The jobs I did run on the server wiped out the comskip results obviously since the file is much different.
Scott

I'm not 100% sure, but as long as the new file keeps the airing id in the name of the file, Sage should be able to make the connection, but, again, don't quote me on that.

If you need to rerun comskip, then you can just add that command line to the same task configuration (a task can run more than one command).

[Wheemer]

I think I figured it out, does this look ok?

CLIENT:
// TASK definition
:MAXPROCS 5
COMSKIP{
:MAX 2
"C:\\comskip.bat \"%c%\""
}
CC{
:MAX 3
"C:\\cc.bat \"%c%\""
}

SERVER:
//
if [IsTV == "true" && FileExists != "%d%\\%p%.edl"]
{
COMSKIP}
if [IsTV == "true" && FileExists != "%d%\\%p%.smi"]
{
CC}

[Slugger]

Originally Posted by Wheemer

I think I figured it out, does this look ok?

CLIENT:
// TASK definition
:MAXPROCS 5
COMSKIP{
:MAX 2
"C:\\comskip.bat \"%c%\""
}
CC{
:MAX 3
"C:\\cc.bat \"%c%\""
}

SERVER:
//
if [IsTV == "true" && FileExists != "%d%\\%p%.edl"]
{
COMSKIP}
if [IsTV == "true" && FileExists != "%d%\\%p%.smi"]
{
CC}

That looks pretty good. As defined, the client will run up to 5 tasks simultaneously, no more than 3 CC tasks and no more than 2 COMSKIP tasks simultaneously.

[scott2020]

Originally Posted by Slugger

I'm not 100% sure, but as long as the new file keeps the airing id in the name of the file, Sage should be able to make the connection, but, again, don't quote me on that.

If you need to rerun comskip, then you can just add that command line to the same task configuration (a task can run more than one command).

Great, thanks. It looks like if the filename is the same with the airing ID Sage can figure it out, even with the file type and ext changing.

Now my problem is I can't get Handbrake to convert my MPEG files recorded by Sage. Right off the bat I get audio time sync errors and it drops audio frames and so on. The result is OK video and choppy audio. I'll have to keep working on that. My other DVD rips and MPEG files from Tivo work, but the Sage recordings don't.

}
CC{
:MAX 3
"C:\\cc.bat \"%c%\""
}

@Wheemer,
What does your CC.BAT do?

[Wheemer]

Hey, here's the contents:

cc.bat

c:
cd C:\Program Files\CCExtractor
start /b /w /belownormal ccextractorwin.exe -sami -utf -s --stream 15 "%~f1"


comskip.bat

c:
cd C:\Program Files\ComSkip
start /b /w /belownormal comskip.exe --ini="C:\Program Files\ComSkip\comskip.ini" -n "%~f1"
set comskiperror=%ERRORLEVEL%
if %comskiperror% EQU 1 exit 0
if %comskiperror% NEQ 1 exit 1

[Grasshopper]

Okay, forgive me for being dense. Here's my ruleset, shamelessly copied from the wiki:

if [IsTV == "true" && IsActivelyRecording == "false" && IsWatched == "false" && FileExists != "%d%/%p%.edl" && Filename =$ ".mpg"]
{
:PRIORITY 10
COMSKIP
}

This has resulted in an active job queue consisting of several hundred tasks, all of which are Waiting and a task ID of "COMSKIP". This is as I want it, at least for now.

But nothing's processing, all the tasks are Waiting. Must I designate the server as a client in some way? Must I write a comskip.bat file?

Desperately seeking a clue.

[Slugger]

Originally Posted by Grasshopper

Okay, forgive me for being dense. Here's my ruleset, shamelessly copied from the wiki:

if [IsTV == "true" && IsActivelyRecording == "false" && IsWatched == "false" && FileExists != "%d%/%p%.edl" && Filename =$ ".mpg"]
{
:PRIORITY 10
COMSKIP
}

This has resulted in an active job queue consisting of several hundred tasks, all of which are Waiting and a task ID of "COMSKIP". This is as I want it, at least for now.

But nothing's processing, all the tasks are Waiting. Must I designate the server as a client in some way? Must I write a comskip.bat file?

Desperately seeking a clue.

Yes, you must install the task client package. The server does not act as a client (for external tasks). To clear the built up queue of tasks, you must install one or more instances of the task client and configure the task client(s) to complete the type of tasks queued up by the server.

[scott2020]

Originally Posted by Wheemer

Hey, here's the contents:

cc.bat

c:
cd C:\Program Files\CCExtractor
start /b /w /belownormal ccextractorwin.exe -sami -utf -s --stream 15 "%~f1"


comskip.bat

c:
cd C:\Program Files\ComSkip
start /b /w /belownormal comskip.exe --ini="C:\Program Files\ComSkip\comskip.ini" -n "%~f1"
set comskiperror=%ERRORLEVEL%
if %comskiperror% EQU 1 exit 0
if %comskiperror% NEQ 1 exit 1

Thanks for posting that. I want to write a batch file that will process the original mpg file into an AVI. I have done this on my test machine using mencoder, because Handbrake I couldn't figure out. mencoder will convert the Sage mpg files with no problem but Handbrake kept complaining about time sync and needing to go back in time, etc. I can process a 1/2 hour show in 15 minutes or so, vs. taking several hours on my P4 2.8 machine using SageMC Video Tools. I guess what I need to do is process the original file to a new one, delete the original file, and then comskip the new file. I haven't tried to comskip an AVI file and I don't think that will work. I'd hate to lose my comskip on compressed shows so we'll see!
Scott

[nyplayer]

scott2020,

I encode to XVID using mencoder and I use the edl from the original mpg and that works fine. I do not change the framerate when I encode to avi I use the original FPS. Maybe you can post your mencoder command line.

[scott2020]

Originally Posted by nyplayer

scott2020,

I encode to XVID using mencoder and I use the edl from the original mpg and that works fine. I do not change the framerate when I encode to avi I use the original FPS. Maybe you can post your mencoder command line.

I think I may have a bad test... I don't see the edl files from the original at all so I think I screwed up along the way. I am encoding another one right now and I'll see how that works. The command line I use I took mostly from the profile settings in SageMC Video Tools settings. The %1's are the filename passed on the batch file command line.

C:\Mencoder\mencoder.exe %1 -oac mp3lame -ovc xvid -xvidencopts bitrate=1500:threads=4:autoaspect:trellis:chroma_me:chroma_opt:hq_ac:quant_type=h263 -o %1.avi

[scott2020]

I just finished up an episode of Friends and left the original edl files and such in place, but playing back in SageMC shows no "empty" spots on the timeline for the commercial skip.

I'm sorry, this is probably getting off topic at this point. Maybe we can PM or start another one at some point.

Scott

[edbmdave]

These are working for me. I convert to mp4 (Ipod)

Server rules:
// Comskip MPEG scan
if [IsTV == "true" && IsActivelyRecording == "false" && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.edl"]
{
:PRIORITY 10
COMSKIP
}


// Transcode
if [IsTV == "true" && IsActivelyRecording == "false" && FileExists == "%d%\\%p%.mpg" && FileExists == "%d%\\%p%.edl" && FileExists != "%d%\\%p%.AUTOCOMPRESS"]
{
:PRIORITY 5
CONVERT
}


Client Configuration:

:MAXPROCS 2
:MAPDIR "D:\\Media\\Videos" "\\\\server\\video01"
:MAPDIR "E:\\Media\\Videos" "\\\\server\\video02"

COMSKIP {
:CPU LOW
:MAX 1
"C:\\ComskipMonitor\\comskip.bat" "%c%"
}

CONVERT {
:CPU LOW
:MAX 1
"C:\\handbrake\\handbrake.bat" "%d%\\%p%.mpg" "%d%\\%p%.mp4"
}

Batch files:

Handbrake.BAT:
cd c:\handbrake
c:
start /b /w handbrakecli.exe -v -Z "iPhone & iPod Touch" -I -d -i %~f1 -o %~f2
set HBError=%ERRORLEVEL%
if %HBError% EQU 0 DEL /Q /F %~f1

CompSkip.BAT:
cd c:\ComskipMonitor\comskip80_007
c:
start /b /w comskip.exe -n "%~f1"
set comskiperror=%ERRORLEVEL%
if %comskiperror% EQU 1 exit 0
if %comskiperror% NEQ 1 exit 1

[scott2020]

Thanks for those batch files. The version of Handbrake I am using is 0.9.3 I believe, whatever the latest one is. It won't convert with those command lines without making horrible choppy audio. It gives loads of time sync errors and whatnot. I might try the 0.9.2 version and see if that works.

Encoding: task 1 of 1, 1.59 % (35.17 fps, avg 61.36 fps, ETA 00h14m25s)[18:00:06
] sync: audio 0 time went backwards 0 ms, dropped 4 frames (next 2749198, curren
t 2749198)
[18:00:06] sync: adding 96 ms of silence to audio 0  start 2769358, next 2760718

Signal 2 received, terminating - do it again in case it gets stuck
[18:00:06] sync: audio 0 time went backwards 0 ms, dropped 4 frames (next 277223
8, current 2772238)
[18:00:06] sync: adding 96 ms of silence to audio 0  start 2792398, next 2783758

[18:00:06] sync: audio 0 time went backwards 0 ms, dropped 3 frames (next 279527
8, current 2795278)
[18:00:06] sync: adding 96 ms of silence to audio 0  start 2812558, next 2803918

[18:00:06] sync: audio 0 time went backwards 0 ms, dropped 4 frames (next 281543
8, current 2815438)

[nyplayer]

Slugger,

In the server settings is the RunDelay in seconds?

[Slugger]

Originally Posted by nyplayer

Slugger,

In the server settings is the RunDelay in seconds?

MaxDelay is in minutes. As a data point, running the QueueLoader more than once every 60 seconds tends to bring out perf issues with SageTV. If you're going to run it on an interval, I really recommend setting it no lower than 2 minutes (I believe there's even a safe guard in place to prevent it going lower than 2 mins).

[nyplayer]

Slugger,

It's not the maximum delay it's RunDelay... I have the server to scan During record. I would like to delay the scan for at least 5 minutes after start of record.