Plugin: Sage Job Queue (SJQ)

[nyplayer]

Originally Posted by Slugger

Hmm... I only use FF. Though this laptop still has FF2 (work requirement), are you using FF3 by any chance? Though I thought I tried FF3 once with SJQ on my desktop, but may not have.

yes Version 3

[perfessor101]

Ouch all this headache over ...

Yes, Internet Explorer 7.0 works

I created a rules.sjq manually (with a rule in it) and it read it in and displayed it in Firefox on the SJQ page and now I know why it wouldn't let me update the rules. Firefox 3 ;-)

The manually entered rule is processing things quite nicely ...

I had tried IE on the SageTV server and it didn't work, but I'm not sure which version it is.
Installing SP3 on a fresh install of MCE 2005 is a gamble it seems from what I've been reading.

I'll be trying IE7 on my main machine. I only use that for windows updates though

my sjq.sqllite file is around 2MB's

With that missed line it may have been something other than the installers fault.

I'm building a new server (separate from the current) and I'm running in service mode on that machine using a HD-100 extender to watch things on and I only use SageTV in Program Mode/Client when I'm testing DVD Burning.
So all the changes I make are in the SageTV directory on that machine and to the Sage.properties file

Thanks for all the help everyone,
Bobby

[Slugger]

Indeed, Firefox 3 is the culprit of the "unable to update ruleset" issue discussed above. This is a known issue with Firefox 3. The underlying issue is that the web server plugin doesn't like the default Content-Type header provided and simply ignores it as a valid HTTP POST request. My previous fix was to override the header with a value that the web server would recognize. IE and FF2 have no problems with the header override, but FF3 does not allow it. More details, with links to more info, are available in the bug I've opened for the issue in the SJQ project.

Until I can come up with something to work around it, the SJQ ruleset editor will not be able to save changes under Firefox 3, but clicking save will incorrectly tell you that the changes were successful (but they really aren't). Saving from Firefox 2 and IE does continue to work as expected.

[CollinR]

Feature Request: Multiple jobs!

It would be killer if you could have a limit per rule that generated the job, that way comskip or SA could be run #clients+1 so all can have realtime commercial detection.

As it sits right now if you try for near realtime detection you must wait until the show has finished before comskipping something else. I have mine tied up on a 2 hour show right now.

[Slugger]

Originally Posted by CollinR

Feature Request: Multiple jobs!

It would be killer if you could have a limit per rule that generated the job, that way comskip or SA could be run #clients+1 so all can have realtime commercial detection.

As it sits right now if you try for near realtime detection you must wait until the show has finished before comskipping something else. I have mine tied up on a 2 hour show right now.

As it turns out, I just upgraded my server recently to a quad core and was thinking the exact same thing.

This particular feature is a bit of an undertaking as it requires a complete architectural change in the way this thing works so it's probably not going to happen while the weather stays nice - my employer keeps me in front of my laptop more than my fair share during these summer months. But I will say this particular feature is coming at some point. If you wanted to add a feature request to the project on SF with your vision of how it would work then that'd be much appreciated.

Off the top of my head, I'm planning to turn SJQ into a client/server setup where you install SJQ on one machine to act as the server and you'll install a client package on each machine you want to be able to run jobs on. You then configure each client to say how many jobs it can run simultaneously, etc. and the server will assign jobs to clients as needed. In the case of a single physical machine, you'll just install the server and the client packages on the same machine. The other thought was to just keep SJQ as is and add the ability to have it start multiple jobs through a config option. This is much easier to implement, but one of the other things I'd like to do is to take advantage of all the hardware sitting on my LAN that could otherwise be transcoding or comm flagging, etc. To facilitate that wish, I'd have to go with the first idea (SJQ client/server setup). But nothing is decided on yet, so I'm quite interested to hear how others would like to see this kind of thing implemented (or if you don't think it'd be useful at all, etc.). I suspect it'll be early fall (late Sep/early Oct) before I dive into this one.

[nyplayer]

Slugger,

I was able to run multiple Showanalyzer across clients by modifing my batch job. I have the server and 2 clients running SJQ and Showanalyzer. I create the edl during start of processing and the vprj after completion. If one of the clients sees a working file then it exits with a non zero code and continues to the next job. Till the next scan if the working file does not exist then it will find the VPRJ and just end and mark it as completed. If showanalyzer fails then it will not mark it as completed.

I just set the scans to run at end of recording 3, 5, 7 minute intervals.



IF EXIST "%~d1%~p1%~n1.working" exit 1
IF EXIST "%~d1%~p1%~n1.vprj" goto end
echo "Commercial Skip">>"%~d1%~p1%~n1.working"
cd "C:\Program Files\Dragon Global\ShowAnalyzer"
c:
start /w /low ShowAnalyzer.exe "%~f1"
del "%~d1%~p1%~n1.working"
IF NOT EXIST "%~d1%~p1%~n1.vprj" exit 1
:end
exit



###__ShowAnalyzer
$ Filename: .mpg
RunShowAnalyzer "s:\sageencode\SJQshowanalyzer.bat" "%f%"

[Slugger]

Originally Posted by nyplayer

Slugger,

I was able to run multiple Showanalyzer across clients by modifing my batch job. I have the server and 2 clients running SJQ and Showanalyzer. I create the edl during start of processing and the vprj after completion. If one of the clients sees a working file then it exits with a non zero code and continues to the next job. Till the next scan if the working file does not exist then it will find the VPRJ and just end and mark it as completed. If showanalyzer fails then it will not mark it as completed.

I just set the scans to run at end of recording 3, 5, 7 minute intervals.



IF EXIST "%~d1%~p1%~n1.working" exit 1
IF EXIST "%~d1%~p1%~n1.vprj" goto end
echo "Commercial Skip">>"%~d1%~p1%~n1.working"
cd "C:\Program Files\Dragon Global\ShowAnalyzer"
c:
start /w /low ShowAnalyzer.exe "%~f1"
del "%~d1%~p1%~n1.working"
IF NOT EXIST "%~d1%~p1%~n1.vprj" exit 1
:end
exit



###__ShowAnalyzer
$ Filename: .mpg
RunShowAnalyzer "s:\sageencode\SJQshowanalyzer.bat" "%f%"

That's a pretty smart way of doing it. Certainly the best, maybe only, way to have multiple machines able to do jobs given the current constraints of how SJQ works. The idea of having 3 SJQ databases on three different machines and not knowing where to look for log output of the actual exe run really makes me want to eliminate the need for this method.

Under my proposed client/server model, you'd only have to install the SJQ "server" package once as a plugin with SageTV (since the "server" needs to be able to access the Sage database, etc.). Then you'd install the SJQ "client" package on each machine you want to run jobs - these machines won't have to have Sage on them nor will the clients have to be installed as Sage plugins - they'd be independent clients that talk to the SJQ "server". One "server" means just one SJQ database file, which means easy management of tracking which jobs have been done, which need to be done, etc. Of course, clients would need to have the media content available via a mapped drive, etc. if the command to be run were going to work on the actual media file.

It's an ambitious project, but one I want to pursue since shortly after upgrading to my quad core I caught myself saying, "why am I not comm flagging all recordings in real time?" Then as I thought about it more I thought why am I not using the other hardware on my LAN to offset other Sage tasks like transcoding, etc.?

[Slugger]

Originally Posted by Slugger

Indeed, Firefox 3 is the culprit of the "unable to update ruleset" issue discussed above. This is a known issue with Firefox 3. The underlying issue is that the web server plugin doesn't like the default Content-Type header provided and simply ignores it as a valid HTTP POST request. My previous fix was to override the header with a value that the web server would recognize. IE and FF2 have no problems with the header override, but FF3 does not allow it. More details, with links to more info, are available in the bug I've opened for the issue in the SJQ project.

Until I can come up with something to work around it, the SJQ ruleset editor will not be able to save changes under Firefox 3, but clicking save will incorrectly tell you that the changes were successful (but they really aren't). Saving from Firefox 2 and IE does continue to work as expected.

After upgrading to FF3 on my laptop, it was time to fix this bug once and for all. Unfortunately, the fix had to be done on the web server plugin side. I have built a version of the web server plugin that contains two fixes:

1) Allow post requests from Firefox 3 (i.e. the bug referenced in the quote above). This will allow the SJQ ruleset editor to properly save changes to disk from FF3.

2) Increase the maximum size of a post request from 10,000 bytes to 1,000,000. This fixes an issue reported some time ago where if you have large enough rulesets then you wouldn't be able to save them via the servlet as you'd hit this limit. 1 million bytes should be more than enough for even the largest of ruleset files.

As I said, I've built my own version of the web server that contains these fixes and I've installed it and am running it. Everything seems to be fine, but I only did limited testing. I've also submitted my changes as a patch to the web server project on SF and my hope is that they'll be applied and the next official release of the web server will include these fixes.

I am not posting my web server build publically as I do not want to get into the business of providing "special" releases of others' plugins, however these particular fixes greatly impact the usability of SJQ so if you require this patched web server please PM me and I'll send you the installer for my patched version "v2.22ddb01".

[CollinR]

I am having some trouble with a rule, anyone wanna show me the fix?

Processing media object: 'Color Correction'
	Checking if media object '2828284' has been processed by action type 'COMMFLAG'
	'E:\TV\ColorCorrection-TheRedZone-2773228-0.edl' does exist; checking next segment...
	All segments passed
	Rule processing stopped: Rule 0.2 returned false
	Checking if media object '2828284' has been processed by action type 'TRANS'
	'E:\TV/ColorCorrection-TheRedZone-2773228-0.edl' does exist; checking next segment...
	All segments passed
	Rule processing stopped: Rule 1.2 returned false
Processing media object: 'Cities of the Underworld'
	Checking if media object '2832385' has been processed by action type 'COMMFLAG'
	Rule processing stopped: Rule 0.1 returned false
	Checking if media object '2832385' has been processed by action type 'TRANS'
	'E:\TV/CitiesoftheUnderworld-BeneathVesuvius-2775612-0.edl' does not exist; test failed.
	Rule processing stopped: Rule 1.1 returned false
Processing media object: 'Trucks!'
	Checking if media object '2832386' has been processed by action type 'COMMFLAG'
	'H:\\Trucks-BodyBagPart1-2772163-0.edl' does exist; checking next segment...
	All segments passed
	Rule processing stopped: Rule 0.2 returned false
	Checking if media object '2832386' has been processed by action type 'TRANS'
	'H:\/Trucks-BodyBagPart1-2772163-0.edl' does exist; checking next segment...
	All segments passed
	Rule processing stopped: Rule 1.2 returned false

EDIT: Oh and the Comskip side appears to work but it never transcodes.

[nyplayer]

CollinR,

I do not see your transcode rule all I see is the Comskip. Maybe you posted Comskip 2x? Rule 2 pic is also comskip.

[CollinR]

TY and Fixed.

[nyplayer]

In the Compress SD the fileexists edl I believe has a / instead of a \ .

Should be %d%\%b%.edl aslo check the filename mpg

[CollinR]

An embarassing oversight however that didn't fix it either. The logs still say Rule 1.2 returned false but I have no clue what rule 1.2 is.

[nyplayer]

Colin

On your check for mpg. You should change that to filename ends with .mpg... that is the extension plus the period before it.

[Slugger]

Originally Posted by CollinR

An embarassing oversight however that didn't fix it either. The logs still say Rule 1.2 returned false but I have no clue what rule 1.2 is.

All of these messages are zero-indexed, so Rule 1.2 means second rule, third test, which is the "Filename equals" test. I had to take a quick look at the source to confirm, but as I suspected the Filename test does not do the media variable expansion. As nyplayer suggested, you'll want to change this test to "ends with .mpg", which should accomplish the same thing.

[CollinR]

That fixed it, thanks guys.

[CollinR]

Well that didn't quite fix everything, it is not replacing the original file even though the command ends with true.

Should this also be in quotes?
Should it have it's own quotes?
Should it be True rather then true?

http://forums.sagetv.com/forums/atta...2&d=1217804416

[nyplayer]

True means to keep the original. False means to replace the original.

Sample from online Help.

Here is an example rule that triggers a transcode job in the SageTV core:

= IsTV: true
= FileExists: %d%\%b%.edl
TRANS __TRANSCODE "MPEG4-High Quality AVI" true
This example will transcode all tv recordings that have an edl file. Since true was specified on the command line the source file will be kept by SageTV after transcoding.

[CollinR]

Geeze I'm on a roll with this plugin. I think I'll shut up until I can help someone else.

Once I get this lined out as I like it I'll post screen caps from the web interface so others have it a cakewalk for common functionality.

[antplugger]

Could anyone please put the install instructions in this thread ?

Thanks