Plugin: Sage Job Queue (SJQ)

Slugger · #1 05-02-2007, 07:36 PM

[This overview is pasted from my online document. Please see the online doc for full details including system requirements and installation. You can download the plugin from here.]

Overview
Sage Job Queue (SJQ) is a loadable class plugin for the SageTV PVR system. With SJQ, you can selectively process your media files based on the contents of the file as stored in the SageTV database. For example, you can setup a ruleset that runs your commercial flagging programs on user scheduled programs only (i.e. don't commercial flag live tv recordings or IR based recordings). You can also specify rules that ignore commercial flagging for recordings from specific channels (say premium channels that don't have commercials). You can also ignore recordings of sports programming (based on program title), etc.

The example above was my motivation for this plugin. I wanted a way to ensure that live tv programs weren't being scanned, I also wanted PPV programs and other channels ignored as well.

Changelog

Code:

v0.0.1 -> v0.1.0
================
 
* Added new field type: Time
   + Format is: HH:MM:SS, 24 hour format
   + Valid operators: >, <, =
* Added new time field for comparisons:
   + TimeOfDay: Compares the current time of day with the given string; use for restricting time that SJQ will do specific jobs
* Added new boolean fields for comparison:
   + IsClientConnected: Returns true if there is at least one client connected to the SageTV server
   + IsViewingMedia: Returns true if media is being viewed on the Sage instance from which SJQ is being run (see online docs for further details)
* Added more details to SJQ database for completed jobs
   + For each completed action SJQ now records the exe that was run, the file(s) the action was run against and the time the action completed
   + varchar field 'exe' added to recordings table
   + New table 'files' created; records full path of the file and the time the action finished for a given media file id
* Added new table to SJQ database: settings
   + Holds various internal settings, including the schema version of the SJQ database, which is used for upgrading purposes

mayamaniac · #2 05-08-2007, 07:59 PM

of course, obvious request is to make a UI for this plugin so it is configurable within the SageTV menus. This has good potential to be handy for a lot of comskip users.

Unrelated but just curious, what comskip version are you using?

Slugger · #3 05-09-2007, 06:15 PM

It's unlikely I'd ever create an STVi for this plugin. I personally think it's best configured from the sjq.rules text file. If anything, I may create a servlet to add to the web server plugin to allow configuration from there, but that would be some time down the road. I further hesitate to create an STVi because I don't want to support both the default STV and SageMC because I only use SageMC.

I've already hit this problem (multiple STV support) with my SRE plugin. I originally wrote the STVi for the default STV then a week later I made the switch to SageMC. Seeing as how these plugins (SRE and SJQ) don't need to tie into the STV at all to function, I think my time's better spent creating servlets for the web server plugin. The servlet approach interests me even more because I need not maintain multiple versions for each STV.

As to which comskip app I use, I'm currently using ShowAnalyzer.

deria · #4 05-14-2007, 02:44 PM

This is very impressive. Up until now, I've been using the (also impressive) Cmmerical Skip Monitor.

After reading the documentation, it LOOKS like I should be able to make a rule that only runs between 11:00 pm and 9:00 am, by doing something like:

# RuleSet: Do commercial skip processing between 11:00 pm and 9:00 am, on scheduled TV recordings only,
# with names that end in .mpg, that are not currently recording. Also, omit recordings from the
# following channels: 211, 212

= IsTV: true # Only run SA against recorded TV
= IsScheduledRecording: true # Ignore live tv and IR recordings
$ Filename: .mpg # Only run SA against MPEG2 recordings
= IsActivelyRecording: false # Wait until program is finished before scanning
!% ChannelNumber: 211|212 # Ignore recordings from PPV channels and Movie Network; notice use of ! operator
> TimeOfDay: 23:00:00 # Time must be after 23:00.
< TimeOfDay: 23:59:59 # Time must be before 23:59:59.
COMMFLAG "C:\Program Files\comskip\comskip.exe"
= IsTV: true # Only run SA against recorded TV
= IsScheduledRecording: true # Ignore live tv and IR recordings
$ Filename: .mpg # Only run SA against MPEG2 recordings
= IsActivelyRecording: false # Wait until program is finished before scanning
!% ChannelNumber: 211|212 # Ignore recordings from PPV channels and Movie Network; notice use of ! operator
< TimeOfDay: 09:00:00 # Time must be before 9:00.
> TimeOfDay: 00:00:00 # Time must be after midnight.
COMMFLAG "C:\Program Files\comskip\comskip.exe"

I included two rules because you don't seem to have a "between" operator. At first, I was going to have a > and a < in the same rule, but I suspect that wouldn't work becasue at 23:50 the time would only satisfy one of the conditions (which would make the whole rule fail).

Does what I've planned here make sense?

Anyway, nice idea and nice implementation. I'm planning on giving this one a try this afternoon.

EDIT: forum killed my formatting. oh well, its still relatively legible

EDIT #2: I had a logic error in my ruleset. Each rule would need to have a < and > time, otherwise the commercials could get marked twice. Not the end of the world, but why make the machien work harder then it has to?

Slugger · #5 05-14-2007, 04:20 PM

Your rule sets look good based on what you say you want them to do. Obviously, you'll know better once you try to run it. The key is to look in the generated sjq.log file if things aren't happening the way you expected. I think you'll find the log does a pretty good job of telling you why an action did or did not execute. Just remember that everything the log tells you is 0-indexed so if the log says an action didn't execute because rule 0.3 returned false then look at the 4th test of your first rule.

As to the TimeOfDay issue, I'll definitely look at addressing that in the next version. As you pointed out, running an action between 11pm and 9am requires two different rules. However, if you modified it to run between midnight and 9am then you could do it all in one rule.

deria · #6 05-14-2007, 05:02 PM

Quote:

Originally Posted by Slugger

Your rule sets look good based on what you say you want them to do. Obviously, you'll know better once you try to run it. The key is to look in the generated sjq.log file if things aren't happening the way you expected. I think you'll find the log does a pretty good job of telling you why an action did or did not execute. Just remember that everything the log tells you is 0-indexed so if the log says an action didn't execute because rule 0.3 returned false then look at the 4th test of your first rule.

As to the TimeOfDay issue, I'll definitely look at addressing that in the next version. As you pointed out, running an action between 11pm and 9am requires two different rules. However, if you modified it to run between midnight and 9am then you could do it all in one rule.

I spent a little bit of time working on this, and so far it seems to be working (well, I see one instance of comskip running -- its not using any cpu time apparently, so I might not have set comskip up right, which is entirely possible since its a brand new installation of it). I do have some things I should pass along though:

1. In the installation instructions, its not clear what you mean by adding the class as a loadable class. I figured out that you meant that it should be added to the load_at_startup_runnable_classes line after some poking around though.

2. Under Vista, if you open the log in notepad while SJQ is doing its thing, it seems to crash (ie: the sage UI vanished on me). At first i thought it had just run into a bad media file or something, but it didn't crash if I left the log file alone during subsequent passes.

3. I don't think that the file i/o is doing a flush when writing to the log. I see comskip running on my system, but in the log I see no indication that it launched it or what file its actually processing. The log ends in mid-word. Would be nice if it listed the command it was executing and flushed the log file prior to doing it so that I could see what it was actually on.

4. Is there a way to control the frequency of when SJQ does its scan? It seems to be doing it every 4 minutes, but to be honest I really don't need it to fire that often (since it scans every single item of media on the system). If there was a way to set it to a frequency of my choosing, it would be nice.

deria · #7 05-14-2007, 05:29 PM

Heya, one more quick question.

Is there anything special that this module does to monitor the process that it laucnhes or throttle its cpu? Each time it launches comskip, comkip works for a minute or two and then goes to 0 percent CPU usage and stays there (seemingly forever). I manually ran it from the command line though (on the same file) and it seems to work fine. Any thoughts?

I suppose it could be a security thing in Vista, but the command prompt I used to run the program didn't throw any security warnings.

Slugger · #8 05-14-2007, 05:56 PM

Quote:

Originally Posted by deria

I spent a little bit of time working on this, and so far it seems to be working (well, I see one instance of comskip running -- its not using any cpu time apparently, so I might not have set comskip up right, which is entirely possible since its a brand new installation of it). I do have some things I should pass along though:

1. In the installation instructions, its not clear what you mean by adding the class as a loadable class. I figured out that you meant that it should be added to the load_at_startup_runnable_classes line after some poking around though.

2. Under Vista, if you open the log in notepad while SJQ is doing its thing, it seems to crash (ie: the sage UI vanished on me). At first i thought it had just run into a bad media file or something, but it didn't crash if I left the log file alone during subsequent passes.

3. I don't think that the file i/o is doing a flush when writing to the log. I see comskip running on my system, but in the log I see no indication that it launched it or what file its actually processing. The log ends in mid-word. Would be nice if it listed the command it was executing and flushed the log file prior to doing it so that I could see what it was actually on.

4. Is there a way to control the frequency of when SJQ does its scan? It seems to be doing it every 4 minutes, but to be honest I really don't need it to fire that often (since it scans every single item of media on the system). If there was a way to set it to a frequency of my choosing, it would be nice.

I'll work on cleaning up the install docs, thanks for the info on that.
I'm running on XP and I haven't seen this issue. I have kept the log file open while SJQ has been running, though I don't use Notepad.
I have noticed this and will try to clean it up.
In the sage.properties file look for slugger/SJQRunFreq and you can modify it as you like. It is the time between runs in milliseconds. You must stop Sage before editing the properties file otherwise the changes will be ignored. My planned configuration servlet will address this issue along with allowing a GUI like environment to create/edit the rules file. But don't hold your breath on the servlet, as it's probably going to be awhile before I get around to it.

Quote:

Originally Posted by deria

Heya, one more quick question.

Is there anything special that this module does to monitor the process that it laucnhes or throttle its cpu? Each time it launches comskip, comkip works for a minute or two and then goes to 0 percent CPU usage and stays there (seemingly forever). I manually ran it from the command line though (on the same file) and it seems to work fine. Any thoughts?

I suppose it could be a security thing in Vista, but the command prompt I used to run the program didn't throw any security warnings.

This might be an issue with the configuration of comskip. The module does nothing to control CPU usage. All the code does is create a Process via a call to Runtime.getRuntime().exec() and sleeps until the process finishes. The actual Process object is not restricted in anyway (by my module - I can't speak for the SageTV core, which may or may not throttle child threads, though I don't see any such behaviour when calling ShowAnalyzer via SJQ - I don't use comskip). One thing to consider is that the current working directory for any process triggered by SJQ is the base install dir of SageTV. Does comskip assume the cwd is its own base dir? Probably something like that would be the likely cause.

deria · #9 05-14-2007, 06:47 PM

Quote:

Originally Posted by Slugger

This might be an issue with the configuration of comskip. The module does nothing to control CPU usage. All the code does is create a Process via a call to Runtime.getRuntime().exec() and sleeps until the process finishes. The actual Process object is not restricted in anyway (by my module - I can't speak for the SageTV core, which may or may not throttle child threads, though I don't see any such behaviour when calling ShowAnalyzer via SJQ - I don't use comskip). One thing to consider is that the current working directory for any process triggered by SJQ is the base install dir of SageTV. Does comskip assume the cwd is its own base dir? Probably something like that would be the likely cause.

Its certainly strange. That comskip.exe process never finished (just stayed at 0%). The comskip log showed that it had started running, which file it was processing, and that it had started scanning and marking the content -- it just got stopped somewhere along the line. I think that the only way that I can narrow it down further might be to try using comskip monitor again (which I haven't done under Vista) and see if that works. If it does, the problem would have to be either Sage or the plugin. If it doesn't work, the problem is likely vista (and then I have a big problem because not being able to run ComSkip would suck alot). I have a license for SA, so I could also give that a try. I'll post my results.

toricred · #10 05-19-2007, 08:15 PM

I was thinking of using this instead of dirmon. Is there a way to set it up to clean up the old edl files after the mpg files are gone?

Slugger · #11 05-20-2007, 10:13 AM

Quote:

Originally Posted by toricred

I was thinking of using this instead of dirmon. Is there a way to set it up to clean up the old edl files after the mpg files are gone?

Not at this time, however what I've chosen to do, while I decide if such functionality should be in SJQ, is to run dirmon2 with no jobs defined. When you do that it still cleans up the files as defined in the cleanup section of dirmon2.

Slugger · #12 05-20-2007, 10:15 AM

I've just added v0.1.1 of SJQ to the downloads section. Changelog below. See first post in thread for links to file, docs, etc.

Code:

v0.1.0 -> v0.1.1
================

* Flush logs before creating process for action line
* Deprecated slugger/SJQRunFreq server property; it's no longer used
* Created slugger/SJQMaxSleep server property
	+ Maximum amount of time between runs of SJQ (see below)
	+ Default values is 21600000 (6 hours)
* Created slugger/SJQSleepType server property; takes one of following values:
	+ AfterRec: SJQ sleeps until two minutes following the end of the next scheduled recording or a maximum of SJQMaxSleep,
	  whichever comes first
        + StartRec: SJQ sleeps until two minutes following the start of the next scheduled recording or a maximum of SJQMaxSleep, whichever comes first
	+ Interval: SJQ simply runs every SJQMaxSleep milliseconds, 21600000 (6 hours) by default
* Expanded on installation documentation
* Properly documented all server properties used in module
* Added range operator to Date and Time fields; allows comparison of values in a given range; see examples

Slugger · #13 09-11-2007, 12:35 PM

SJQ v1.0.0 is now available.

Changelog

Code:

v0.1.1 -> v1.0.0
================

* Bug  7: Add ability to run internal tasks as actions in SJQ
* Bug  8: Update SQLite JDBC driver; move to pure Java driver
* Bug  9: Add support for interpolated strings in action lines and tests
* Bug 10: Add GKusnick's Studio Tools jar to packaged zip
* Bug 11: Add new boolean test: FileExists
* Bug 12: Schema upgrades not done in transaction
* Bug 15: 'files' table does not store job type

NOTE: Bug 16 has been deferred to the next release

NOTE: Please read upgrade instructions if upgrading from a previous
version of SJQ.  See online document link from download page.
You've been warned.

Slugger · #14 10-19-2007, 08:01 PM

SJQ v1.0.1 is now available.

Slugger · #15 04-03-2008, 06:51 PM

v1.1.0 is now available. A special thanks to cnovak for helping me test the RC builds and iron out the bugs!

v1.1.0 introduces a web configuration servlet compatible with the web server plugin. The servlet allows you to configure all of SJQ's options in real time via the web without the need to stop/restart the SageTV service. Also in this release is the web based rule editor. You no longer need to learn the ruleset syntax to setup processing rules in SJQ. Many other bug fixes and enhancements (new tests to test your media against) are also included. The online documentation has many more details.

Changelog:

Code:

v1.0.1 -> v1.1.0
================

* Bug 34: Add configuration servlet compatible with web server plugin
* Bug 43: Upgrade Studio Tools jar to latest version
* Bug 44: Handle case where ruleset file does not exist at plugin startup
* Bug 45: New test: Age; allows testing of a media object's age
* Bug 47: Move all logging to database
* Bug 48: Use complete, absolute path of file names in 'Filename' test
* Bug 49: Add link to docs from online rule editor
* Bug 54: Log arguments used in transcode action lines
* Bug 55: Update online docs
* Bug 56: Add ability to run SJQ on demand (wake it up early and run it from servlet)
* Bug 59: Fix issue where some processes triggered could hang if they exhausted the OS output buffer
* Bug 61: Store plugin settings in data store instead of using Sage properties
* Bug 63: New test: TranscodeQueueSize
* Bug 64: Log all stdout and stderr output from external commands to the database
* Bug 65: Fix synchronization issue in Logger class
* Bug 67: New test: IsSomethingRecording - returns true if Sage is currently recording anything
* Bug 68: New test: FreeSpacePercentage - allows checking the amount of free space available on the disks used for recording
* Bug 71: New test: ShowID - allows testing against the unique external show id

nyplayer · #16 04-04-2008, 06:24 PM

Slugger,

Maybe you can help me by providing a sample.

Currently I run videoredo quickfix on all my files. I then run showanalyzer against the fixed files.

1. I run Videoredo Qfix it creates a file with the extension qfix to let me know it is completed.

2. I then would like to run Showanalyzer against the mpg files the are qfixed only... creating EDL and VPRJ files.

Also I do not want videredo to run if a qfix file exists.

Slugger · #17 04-04-2008, 06:54 PM

Quote:

Originally Posted by nyplayer

Slugger,

Maybe you can help me by providing a sample.

Currently I run videoredo quickfix on all my files. I then run showanalyzer against the fixed files.

1. I run Videoredo Qfix it creates a file with the extension qfix to let me know it is completed.

2. I then would like to run Showanalyzer against the mpg files the are qfixed only... creating EDL and VPRJ files.

Also I do not want videredo to run if a qfix file exists.

I'm not sure how familiar you are with SJQ, but I'll provide a ruleset file that should do what you want, but I also sent you a PM with details on how to grab me on MSN as it might be easier to walk through an example together.

Code:

# Run Qfix
= IsTV: true
= IsActivelyRecording: false
!= FileExists: %d%\%b%.qfix
	QFIX "C:\Program Files\path\to\qfix.exe" %f%

# Run SA for files that have a qfix file
= FileExists: %d%\%b%.qfix
!= FileExists: %d%\%b%.edl   # Use this test to prevent SA running for recordings that already have an edl
= IsActivelyRecording: false # Don't run SA against actively recording files
	COMMSKIP "C:\Program Files\path\to\ShowAnalyzer.exe" %f%

This ruleset file can be made with little difficulty using the web based ruleset editor. You don't need to know all this syntax when using the web. Let me know if you have any questions (may not see them until tomorrow now).

NOTE: Copy/paste of the ruleset file may not work. The action lines must start with a tab character - not sure what this BB will do to tab characters.

nyplayer · #18 04-05-2008, 07:54 AM

Slugger,

Thanks for your help everything works as advertised. What you can do with your program is unlimited.

One thing I noticed the jobs run in Sequence on how they are in the rules file. Maybe you can add on server appl a way to re-organize them... maybe the ability to move them up and down on the Rulset Editor tab.

Slugger · #19 04-23-2008, 09:09 PM

v1.2.0 is now available.

This version is a reissue of v1.1.0 (no new functionality). But v1.2.0 introduces a Windows installer that completely automates the installation and setup of the plugin, including installation of the servlet into the web server plugin. With the installer all you need to do is run it and restart SageTV and then SJQ will be fully functional. No more having to modify properties files and web server config files - the installer does it all!

The installer will only run on Windows XP or below. I do not have access to a Vista system to create a Vista installer. Vista and Linux users will need to continue to use the manual installation process as outlined in the docs.

nyplayer · #20 05-02-2008, 02:49 PM

Slugger,

I seem to have run into a problem is there a limit on the amount of data a rules can contain?

[Fri May 02 13:41:21 PDT 2008] /sage/SJQ: init
[Fri May 02 13:43:10 PDT 2008] WARNING: POSTed data from Form content truncated to 10000 bytes
[Fri May 02 13:43:10 PDT 2008] unexpected problem running servlet net.sourceforge.sagetv.sjq.SJQServlet: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Collection
[Fri May 02 13:43:19 PDT 2008] unexpected problem running servlet net.sourceforge.sagetv.sjq.SJQServlet: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Collection
[Fri May 02 13:43:49 PDT 2008] unexpected problem running servlet net.sourceforge.sagetv.sjq.SJQServlet: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Collection
[Fri May 02 13:44:19 PDT 2008] unexpected problem running servlet net.sourceforge.sagetv.sjq.SJQServlet: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Collection
[Fri May 02 13:44:28 PDT 2008] /sage/SJQ: destroy

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Plugin: Sage Job Queue (SJQ) v3	Slugger	SageTV Customizations	1355	07-25-2013 07:44 AM
Sage Job Queue (SJQ) new release notifications	Slugger	Customization Announcements	3	12-17-2009 09:59 AM
Sage Job Queue Completed tasks problem	raffmanlt	SageTV Customizations	2	08-18-2009 07:34 PM
Comskip Monitor VS Sage Job Queue SJQ	personalt	SageTV Customizations	6	03-02-2009 10:27 AM
Plugin: SJQ v1.1.0RC1 Available - Testers Needed	Slugger	SageTV Customizations	35	04-21-2008 08:12 AM

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)