Plugin: Sage Job Queue (SJQ)

[gveres]

Slugger,

This works great. As nyplayer said, it is zippy now. Definitely ignoring the 15000 photos did the trick, because I am now trying to open the log for the last run under 1.3.0 and it is chugging away - has been for 5 minutes.

I have a couple of questions:

1) is there a way to enable and disable rulesets? For instance, right now there is a bug on the HD extender such that it can not play back any mpg that has been processed by VideoReDo - so I want to disable my videoReDo rule for now until this bug is fixed. But since I have gone through the trouble of getting it setup correctly and debugged, I don't want to delete the rule.

2) what is the filter link for at the top of the log page? Actualy, what do those check boxes do at the top of the log page? They don't seem to change the output at all.

BTW, I think I have the videoReDo rules working now. The next step is ComSkip.

Great work!!!
Greg

[Slugger]

Originally Posted by gveres

Slugger,

This works great. As nyplayer said, it is zippy now. Definitely ignoring the 15000 photos did the trick, because I am now trying to open the log for the last run under 1.3.0 and it is chugging away - has been for 5 minutes.

I suspect the processing of the 15,000 photos isn't the issue, but it's the logging of everything to the SJQ database that is the bottle neck. The media library on my system is only about 350 entries (video/music) so I just have no way to replicate an environment with that kind of media library size in order to try and increase the performance of the database operations. If you have an SQLite client available on the system, I'd highly recommend manually deleting the entire log table because the first time SJQ tries to purge the logs itself is likely going to cause performance issues as SJQ's method of purging logs is a brute force method required due to limitations in SQLite. I would highly recommend doing a simple 'DELETE FROM log' against the SJQ database. SJQ's purge method should work, but it will be incredibly slow. If you don't have an SQLite client handy then I'd recommend purging manually via the servlet at a time when you know the system won't be doing anything important for some time. When purging via the servlet, use values of 1 as 0 means, "don't purge anything."

Originally Posted by gveres

I have a couple of questions:

1) is there a way to enable and disable rulesets? For instance, right now there is a bug on the HD extender such that it can not play back any mpg that has been processed by VideoReDo - so I want to disable my videoReDo rule for now until this bug is fixed. But since I have gone through the trouble of getting it setup correctly and debugged, I don't want to delete the rule.

Not at this time. I do have a feature request in the project tracker for this as you're not the first person to ask for this. I've kind of been avoiding that ticket as my initial investigation suggests it won't be as easy as I originally thought. Now that I'm able to muster the strength to get out of bed, I'm starting a delayed vacation tomorrow so I'll probably tackle this ticket next time I sit down to play with the code, which will probably be towards the end of the month. I'll definitely try to include this feature in the final 1.4.0 build.

Originally Posted by gveres

2) what is the filter link for at the top of the log page? Actualy, what do those check boxes do at the top of the log page? They don't seem to change the output at all.

The filters determine what type of messages are displayed when you pick a log to view. Unless you're trying to track a problem, you really only need to have the 'Plugin' and 'Info' boxes checked. Everything else can be left unchecked. Here's what the various log messages mean:

Plugin: Show log messages produced by the plugin class
Servlet: Show log messages produced by the webserver servlet

Info: Show general info messages (can't be disabled)
DataStore: Show debug messages from the database (JDBC/SQL)
Debug: Show general debug messages
Exceptions: Show any Java exceptions caught by the plugin/servlet
Stdout: Show the stdout of any action ran by SJQ on a media file
Stderr: Show the stderr of any action ran by SJQ on a media file

Note that all of these messages are always logged to the SJQ database, the filter only filters their display within the log tab of the servlet. The items in bold are the ones you should leave on for typical log viewing, the rest are really only there for me while I develop or for people trying to track down a problem with SJQ. After modifying the checkboxes you need to click filter. The checkboxes are only modified after clicking the filter link.

Originally Posted by gveres

BTW, I think I have the videoReDo rules working now. The next step is ComSkip.

Great work!!!
Greg

Thanks, definitely making strides in the right direction. As I said, I'm heading out of town for the remainder of the week and probably won't be looking at any of this again until the end of the month, but please keep testing with the 1.3.0a build and report any problems/requests in the project tracker. It's ok to post messages and request features here, but they're likely to get lost, especially since I won't be keeping up with my email while on vacation (a personal no Internet policy while on vacation). Things added to the tracker are guaranteed to be seen. Here is a link to the trackers for the SJQ project.

[gveres]

Glad you are feeling better. Have a good vacation.

As an interesting data point, the SQLite database file is now 2.5 Gig.

Greg

[Slugger]

SJQ v1.4.0 is now available.

v1.3.0 -> v1.4.0
================

* 1962002: Allow for dynamic arithmetic in TimeUntilNextRecording test
* 1962455: Allow SJQ to ignore specified types of media (i.e. only scan tv files, etc.)
* 1965446: Make delay time before each SJQ run configurable

[nyplayer]

Slugger,

Is it possible to add a rule..if the Show Is Archived? I would like to move a show to a different drive if it is Archived.

[Slugger]

Originally Posted by nyplayer

Slugger,

Is it possible to add a rule..if the Show Is Archived? I would like to move a show to a different drive if it is Archived.

I don't see an API call to tell me if a recording is archived. So, assuming that's the case, then I'd have to say that this particular test isn't possible. There is an API call IsLibraryFile() which is about the closest thing I could find, but the description for that method in the API docs doesn't mention anything about the archive status of a recording. So my initial thought on this is that it doesn't look possible, but when I have a free hour some evening this week I'll dive into the API docs a little deeper and see what I can find. If there's an API call that gives the archive status of a recording then adding the test to SJQ is trivial - literally five minutes to do it. It's just a matter of finding an API call that returns the required information. Feel free to have a look at the API docs and let me know if you see something that I may have overlooked.

[BobPhoenix]

Originally Posted by Slugger

I don't see an API call to tell me if a recording is archived. So, assuming that's the case, then I'd have to say that this particular test isn't possible. There is an API call IsLibraryFile() which is about the closest thing I could find, but the description for that method in the API docs doesn't mention anything about the archive status of a recording. So my initial thought on this is that it doesn't look possible, but when I have a free hour some evening this week I'll dive into the API docs a little deeper and see what I can find. If there's an API call that gives the archive status of a recording then adding the test to SJQ is trivial - literally five minutes to do it. It's just a matter of finding an API call that returns the required information. Feel free to have a look at the API docs and let me know if you see something that I may have overlooked.

Try TWO. If the file is a library file (IsLibraryFile) and is a TVFile (IsTVFile) then it should be an archived file.

IsArchivedFile = IsLibraryFile(MediaFile) && IsTVFile(MediaFile)

[toricred]

I'm just setting this up and I've run into some issues I don't understand. First off I can't get my rule for running SA to work on only .mpg files. Every time it evaluates the files it tries to run on the .avi files as well.

Second I'm wondering if it will run multiple copies of SA at once or only 1.

[Slugger]

Originally Posted by toricred

I'm just setting this up and I've run into some issues I don't understand. First off I can't get my rule for running SA to work on only .mpg files. Every time it evaluates the files it tries to run on the .avi files as well.

Second I'm wondering if it will run multiple copies of SA at once or only 1.

I'd have to see your rules.sjq file to see how your rules are setup.

For the second part, SJQ will only run one process at a time and it waits for that process to complete before triggering another. I've thought about allowing multiple processes to be triggered by SJQ, but haven't had a personal need for it so haven't bothered implementing it.

[Slugger]

Originally Posted by BobPhoenix

Try TWO. If the file is a library file (IsLibraryFile) and is a TVFile (IsTVFile) then it should be an archived file.

IsArchivedFile = IsLibraryFile(MediaFile) && IsTVFile(MediaFile)

Seems to make sense. I'll add the test some evening this week and get a 1.5.0 release out maybe before next weekend.

[toricred]

Here are is the contents of my rules.sjq for right now. After I have it working well, I'll need to add rules to use different .ini files for certain shows and a rule to not process on the movie channels.

###__ShowAnalyzer
$ Filename: mpg
!= FileExists: %d%\%b%.edl
SA_NOT_5XX "c:\program files\dragon global\ShowAnalyzer\Showanalyzer.exe" "%f%"

[Slugger]

Originally Posted by toricred

Here are is the contents of my rules.sjq for right now. After I have it working well, I'll need to add rules to use different .ini files for certain shows and a rule to not process on the movie channels.

###__ShowAnalyzer
$ Filename: mpg
!= FileExists: %d%\%b%.edl
SA_NOT_5XX "c:\program files\dragon global\ShowAnalyzer\Showanalyzer.exe" "%f%"

I'm stumped, but change the "mpg" to ".mpg" (add a period in front) and try again. As is, the rule still shouldn't match avi files, but if adding the period actually fixes it then I have some debugging to do.

[toricred]

I had it like that before, but it didn't seem to work. Now it seems to be working with the .mpg. I'm also trying to tell it to not do it if an .edl file exists. Should this work with the rules.sjq I posted earlier? It seems to keep running on files that already have a .edl file.

[Slugger]

Originally Posted by toricred

I had it like that before, but it didn't seem to work. Now it seems to be working with the .mpg. I'm also trying to tell it to not do it if an .edl file exists. Should this work with the rules.sjq I posted earlier? It seems to keep running on files that already have a .edl file.

Everything should work as you think given the rule file you pasted in before. Stop Sage, replace the sjq jar in the JARs folder with this one, then restart Sage. Reproduce the unexpected results then go the log viewer, enable debug messages in the view then send me the output from a file that you didn't expect to run along with the ruleset file. With the debug data it shouldn't take me long to figure out what's going on.

[toricred]

The wife and kids are watching now so I'll have to do it tonight once I can shut down Sage for a few minutes without risking my life.

[toricred]

One more thing. Do the jobs get run at a lower priority or something? It seems to be running much slower than when I was using DirMon2.

[Slugger]

Originally Posted by toricred

One more thing. Do the jobs get run at a lower priority or something? It seems to be running much slower than when I was using DirMon2.

Nope, I do not modify process priorities at all.

[nyplayer]

Originally Posted by toricred

The wife and kids are watching now so I'll have to do it tonight once I can shut down Sage for a few minutes without risking my life.

What I do to avoid this is setup SJQ to run on the client on my server. I run Sage as a service the I run the Sage UI in low priority. That way I never have to stop the service to make changes to SJQ.

[toricred]

OK, I used a command line to get SA to not throttle.

It turns out that it still isn't looking at only .mpg files so I'm trying a different rule as follows:

% Filename: *.mpg

[nyplayer]

toricred
It should look like below... I Use "ends with"

$ Filename: .mpg