Plugin: Sage Job Queue (SJQ)

[Slugger]

Originally Posted by Peter_h

Also, I had a job stuck in queue yesterday.

It was a comskip job and when i checked the logs it was at 53%. An hour later it was still at 53%.

I restarted the computer to clear sage and SJQ and checked on the file and it was finished processing but the job was still in SJQ as processing.

I cleared it and all was well.

Where would i start to troubleshoot this? I don't think this is on the comskip side but I could be wrong.

Define 'stuck' in the queue? The job was in the queue, stuck in waiting state and never assigned to a client? The job was in the queue, stuck in started state after being assigned to a client? The job was in the queue, stuck in running state and never left the queue after being assigned to a client? The job was stuck in the queue, put in started state, but no client was assigned to it? Something else?

If it was assigned to a client, marked as running state and then never left that state then you need to determine if the comskip process had actually completed and terminated (i.e. no longer shows up in the task manager). If that's confirmed then the problem may lie in the client not sending the update back to the server. And the update would be missed if the server was down or the client crashed or a bug in my code, etc. Which logs were you checking? The log output shown in SJQ or an external log file from comskip? If the former, then we don't know much because if it was an SJQ problem then the task client might have stopped sending logs to the server. The point here, is that this particular report is going to be very hard to track down and 1) confirm, 2) squash without very specific instructions on how to consistently reproduce the problem. First we need to define 'stuck in the queue' and then you need to determine if comskip actually finished and properly terminated itself.

Finally, you shouldn't need to reboot your entire system to restart SJQ. Simply touch the context file (see 'Restart' section), which will force a restart of SJQ and that will reset the task queue (though it still may not be sync'ed up if one were to restart SJQ while a task client were still running a task). In other words, kill all task clients before restarting the server then once the server is restarted properly then restart all task clients.

[Peter_h]

The job was in the queue, assigned to a client and was running.

I checked on the job by clicking on the client -> log and it showed what looked like the output of the comskip job.

This was at 53%. An hour later, the job was still in the "running state" and still at 53%. However, comskip was no longer running in Task Manager and when i checked the file, commercials were processed all the way through.

So, the comskip definately finished but SJQ never received anything past 53%? Once I get home I'll check the logfile for comskip to see if it did finish.

[Wheemer]

I am trying to extract usable smi files for captions. I have compskip outputting it and it usually does fine. However there are times when it outputs an unusable smi that is 1 kb.

I'm looking for a way to have these files that have 1 kb smi to be reprocessed using ccextractor.

But I also want to ensure that it only tries once, since i it fails as well there will still be the 1 kb smi, and I don't want a loop.

I'm really not sure if this is possible, please help if you can.

[Slugger]

Originally Posted by Wheemer

I am trying to extract usable smi files for captions. I have compskip outputting it and it usually does fine. However there are times when it outputs an unusable smi that is 1 kb.

I'm looking for a way to have these files that have 1 kb smi to be reprocessed using ccextractor.

But I also want to ensure that it only tries once, since i it fails as well there will still be the 1 kb smi, and I don't want a loop.

I'm really not sure if this is possible, please help if you can.

There is no way to test the size of a file not registered with SageTV (in wiz.bin) and perform an action based on that file size. The MediaSize test in SJQ is only capable of testing the size of files registered with SageTV (in the wiz.bin). SJQ can test for the existence of an external file so if you can somehow create a file to signify that you want to rerun ccextractor then you could trigger a rerun that way, but trying to determine that based on the size of some external file is not possible with SJQ.

[scott2020]

Is it possible to do a NOT test on the TimeOfDay test? Basically I want my COMSKIP job NOT to run between 7pm and 10pm.

TimeOf Day !< "19:00" || TimeOfDay !> "22:00"

Maybe I am thinking backwards!

Scott

[Slugger]

Originally Posted by scott2020

Is it possible to do a NOT test on the TimeOfDay test? Basically I want my COMSKIP job NOT to run between 7pm and 10pm.

TimeOf Day !< "19:00" || TimeOfDay !> "22:00"

Maybe I am thinking backwards!

Scott

TimeOfDay < "19:00" || TimeOfDay > "22:00"

This reads as, "the time of day must be before 7pm or after 10pm". In other words, the time of day can't be between 7pm and 10pm.

[theringnebula]

Hi,

Thank you for this great tool. I am having a minor issue with it that doesn't seem to hurt anything but is causing unnecessary processing and I want to make sure I'm not missing something. I am running the latest 2.1 release version of the server and the client service on the same machine. Here is my code:

Server Rules:

// Comskip scan
if [IsTV == true && IsActivelyRecording == false && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.edl" && ChannelName !^ "WGBH" && ChannelName !^ "WENH" && ChannelName !^ "WGBX" && ChannelName !^ "HBO" && ChannelName !^ "STRZ" && ChannelName !^ "ENCR"]
{
COMSKIP
}

//Comskip TS scan
if [IsTV == true && IsActivelyRecording == false && FileExists == "%d%\\%p%.ts" && FileExists != "%d%\\%p%.edl" && ChannelName !^ "WGBH" && ChannelName !^ "WENH" && ChannelName !^ "WGBX" && ChannelName !^ "HBO" && ChannelName !^ "STRZ" && ChannelName !^ "ENCR"]
{
COMSKIPTS
}

Client Rules:

//Client Options
:MAXPROCS 2

//Commercial Skip
COMSKIP {
:MAXRUNTIMERATIO 1.0
"C:\\Comskip\\comskip.bat \"%c%\""
}

//Commercial Skip for HD PVR
COMSKIPTS {
:MAX 1
:MAXRUNTIMERATIO 2.0
"C:\\Comskip\\comskip.bat \"%c%\""
}

The problem I have is that when watching live TV and the program changes SJQ is adding that recording to the queue and processing comskip on it before sage has a chance to delete the temporary recording. Is there a way to avoid this that I am missing?

Thanks!

[Slugger]

Originally Posted by theringnebula

The problem I have is that when watching live TV and the program changes SJQ is adding that recording to the queue and processing comskip on it before sage has a chance to delete the temporary recording. Is there a way to avoid this that I am missing?

Thanks!

Use the IsScheduledRecording test (i.e. IsScheduledRecording == true). That test will only match recordings that you specifically scheduled to record (either through a manual recording or a favourite recording).

[Slugger]

Originally Posted by theringnebula

Client Rules:

//Client Options
:MAXPROCS 2

//Commercial Skip
COMSKIP {
:MAXRUNTIMERATIO 1.0
"C:\\Comskip\\comskip.bat \"%c%\""
}

//Commercial Skip for HD PVR
COMSKIPTS {
:MAX 1
:MAXRUNTIMERATIO 2.0
"C:\\Comskip\\comskip.bat \"%c%\""
}

Replace MAXRUNTIMERATIO with MAXRUNRATIO, the names of those optoins were incorrectly documented at first, but have since been corrected in the wiki. As written, those options would simply be ignored right now meaning your tasks are not setting a timeout timer.

[perfessor101]

Hello,
I checked my logs and it would seem they go back a few weeks. I stayed with the default setting of clearing logs weekly, but I'm not sure it looks like it's still keeping at least some of the information.

I've attached a jpg of a log page and a jpg of the settings page ...

please don't judge the shows I record

I am currently running SJQv2.1.1.236

Thanks for your time,
Bobby

[Slugger]

Originally Posted by perfessor101

Hello,
I checked my logs and it would seem they go back a few weeks. I stayed with the default setting of clearing logs weekly, but I'm not sure it looks like it's still keeping at least some of the information.

I've attached a jpg of a log page and a jpg of the settings page ...

please don't judge the shows I record

I am currently running SJQv2.1.1.236

Thanks for your time,
Bobby

When you click on one of those entries to view the program output for that task it should be empty. That's the data that gets purged, the task history entry in the db itself is never purged (it's kept there to ensure that the same job never runs more than once). If you really want to remove the task history (and risk a successfully completed task running again) then right click in the grid and select 'remove all', though I don't think you really want to do that.

[Slugger]

SJQ v2.1.1 is now available. This is a bug fix release that addresses various issues with the server and task client packages. All users are encouraged to upgrade to this version. See the project home page for details of the changes included in this release.

[mgleed]

I'm personally looking forward to making use of the RETURNCODE option.

[mgleed]

What's the correct way to upgrade the service version of the client?

[ccourtney]

I have successfully upgraded the service version by stopping the service. Overwriting the files from the new distribution and then starting the service again. I suppose one could be more cautious and unregister before overwriting and register after to be on the safe side as well.

[Slugger]

Originally Posted by ccourtney

I have successfully upgraded the service version by stopping the service. Overwriting the files from the new distribution and then starting the service again. I suppose one could be more cautious and unregister before overwriting and register after to be on the safe side as well.

This is how I've always done it as well. Just make sure to stop the service before overwriting anything then you should be good to go. I haven't had a need to unregister the service - just stop, overwrite, restart. The task client installation wiki doc has been updated with a paragraph including this information.

[bnh]

i apologize for not reading the entire thread. What is the primary reason that you are using this plugin?

I'm considering using it to offload comskip and transcoding processing to two other clients in the house. is that the idea...to keep the sage server near idle? I have noticed when both the sagetranscoder and comskip are running, HD playback on the clients are "skippy", since the server nears 100% CPU usage. this plugin may alleviate that, but i like the ability to monitor transcode status within SageMC. Can i still observe that with this plugin? what are the advantages?

just thinking outloud...

[Slugger]

Originally Posted by bnh

i apologize for not reading the entire thread. What is the primary reason that you are using this plugin?

I'm considering using it to offload comskip and transcoding processing to two other clients in the house. is that the idea...to keep the sage server near idle? I have noticed when both the sagetranscoder and comskip are running, HD playback on the clients are "skippy", since the server nears 100% CPU usage. this plugin may alleviate that, but i like the ability to monitor transcode status within SageMC. Can i still observe that with this plugin? what are the advantages?

just thinking outloud...

The original purpose of the plugin - and still the main reason I use it - is to selectively process certain recordings while ignoring others. Basically I don't want any live tv recordings to be comskipped, I don't want recordings from channels known not to have commercials to be comskipped (HBO, PPV, etc.) since it's just a waste of time (and CPU). When I first wrote this plugin my Sage server was a little underpowered so I couldn't afford to be wasting CPU cycles running comskip on recordings I knew didn't have commercials - now I have a quad core CPU and wanted a way to make use of it, which is how v2.x came about. v2.x introduced parallel task execution and the natural addition to parallel task execution was distributed task execution. So now my quad core is setup to run multiple tasks simultaneously, I also have some Linux hardware on my LAN that I can offload some work to if I wanted to.

You can offload comskip jobs to other hardware for sure, but if you use the SageTV transcoder then those tasks can't be distributed to other hardware since SJQ just registers jobs with the Sage core and the Sage core runs all transcoding jobs on the server. But with SJQ you can choose when jobs are sent to the SageTV transcoder. For example, if you have problems transcoding while watching tv then you can tell SJQ not to send jobs to the transcoder if a client is connected or if an extender is watching tv or you tell SJQ not to send jobs to the transcoder if it's between 8pm and 11pm or you can tell SJQ not to send a recording to the transcoder until it's at least 30 days old. Or you can combine all those rules, etc. The possibilities are almost endless, and that's where the power of this plugin comes from.

SageTV transcode jobs triggered by SJQ would still be viewable in SageMC since SJQ just registers the jobs in the SageTV transcode queue.

[bnh]

Originally Posted by Slugger

you can tell SJQ not to send jobs to the transcoder if a client is connected or if an extender is watching tv

THAT is compelling. thanks!

[perfessor101]

Hello, Slugger
you were right ... my logs are getting deleted as expected.

I have a lot of recordings from previous SageTV and previous SJQ (or Dirmon) installs. (1,265 recordings)

I got used to using marker files to tag the levels of processing that way when I have to start over with a new install nothing needs reprocessing.

However, I'm guessing all the media files that have never been processed by SJQ 2.1 are getting run through every rule check each time the Queue Loader runs ... and with the number of media files I currently have it seems to be confusing something between SageTV, Jetty and SJQ.

I was curious is there a way of marking all tasks for all media files done?
(if there was a command I could type ...)

Another possibility I have been thinking of is changing the rules and the tasks to a set of "blank" rules and "blank" batch files so that SJQ could "process all files"

I've up, down and re-graded Jetty (1.4, 1.5) and tried sjq-2.1.1.186 and sjq-2.1.1.236 and I'm thinking maybe it's time to switch to linux

Thanks for your time,
Bobby