Plugin: Sage Job Queue (SJQ)

[perfessor101]

Having fun ... XP changed my UNC shared directory permissions ... yet again

Sun Feb 15 11:49:42 PST 2009: SQLite driver implementation: 'native'
Sun Feb 15 11:49:42 PST 2009: QueueLoader started

[wayner]

Originally Posted by Slugger

The more complicated the syntax of the ruleset, the tougher it is to maintain. I managed to avoid writing a recursive decent parser for the current syntax and instead managed to write it using the StreamTokenizer class. Neither solution is particularly easy, especially since the whole area of parsers, etc. doesn't interest me at all. Introducing something like this only makes things more complicated and isn't likely to happen anytime soon (i.e. ever ) On the other hand, if someone wanted to introduce a more robust ruleset syntax I'd be more than willing to accept a proper patch.

However, with the use of the media variables, you should be able to reduce things to a single if block, something like this should work (untested):

Understandable on the ruleset. I will give your suggestion a try the only downside is that I may have to slightly rename some of my folders but that is no big deal.

It looks like it is only the Canadians participating in this thread this weekend!

[Beefcake550]

This may not be related to SJQ per say, but I thought I'd ask anyway.

Is it possible to somehow pause or scale back a comskip run that is started when no media is being streamed, but then an extender or two connect?
I've got the following rules to make it not start one while anything is connected or recording, but I'm unsure if it's possible to "pause" a comskip run while it's in progress.....

// Comskip on MPEG
if [IsViewingMedia == "false" && IsTV == "true" && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.edl" && IsActivelyRecording == "false" && IsScheduledRecording == "true"] {
:PRIORITY 100
COMSKIPMPEG
}

// Comskip on H264
if [IsViewingMedia == "false" && IsTV == "true" && FileExists == "%d%\\%p%.ts" && FileExists != "%d%\\%p%.edl" && IsActivelyRecording == "false" && IsScheduledRecording == "true"] {
:PRIORITY 80
COMSKIPH264
}

-Brian

[Slugger]

Originally Posted by Beefcake550

This may not be related to SJQ per say, but I thought I'd ask anyway.

Is it possible to somehow pause or scale back a comskip run that is started when no media is being streamed, but then an extender or two connect?
I've got the following rules to make it not start one while anything is connected or recording, but I'm unsure if it's possible to "pause" a comskip run while it's in progress.....

-Brian

Once SJQ queues a task, it's going to be assigned to the next client that calls in for work. At this stage you'd have to prevent the client from calling in and getting the task assigned to it (by stopping it). Once a client is given a task it simply runs it until it's done - no way to pause it or otherwise slow it down, etc. (within SJQ). Perhaps comskip has a way to pause an active run, but I doubt it - either way it's outside the scope of SJQ. The clients are "dumb". They simply call the server for work, if they get work they do what they're told until done, if they don't receive work they go to sleep and try again a few minutes later.

[Slugger]

Snapshot 280 addresses some performance issues with the task log cleaner thread that would also cause lengthy locking of the db file, which would in turn manifest itself as apparent hanging, etc. in the GUI (the "Request timed out after 5000ms" errors).

If those having these problems (perfessor101, wayner) could upgrade to 280 (along with anyone else willing to help out with testing) and let me know how things are going that'd be great.

[perfessor101]

I just installed snapshot 280 ... I'll let you know how it runs ...

I've found the more i leave it alone ... the better ...

With Firefox I'm finding that if I clear the 'cache' and possibly the 'off line website data' then the "live updates" work smoothly.
If I don't clear the cache it doesn't seem to pull in the new snapshot.
So I think that's just a browser cache thing ... nice to be able to update SJQ without restarting SageTV.

Thanks for all the work,
Bobby

[Slugger]

Originally Posted by perfessor101

With Firefox I'm finding that if I clear the 'cache' and possibly the 'off line website data' then the "live updates" work smoothly.
If I don't clear the cache it doesn't seem to pull in the new snapshot.
So I think that's just a browser cache thing ... nice to be able to update SJQ without restarting SageTV.

Thanks for all the work,
Bobby

I've never had to restart Sage to upgrade, just unzip the new zip file and away it goes. You will have to refresh the GUI if currently connected, but the server component of SJQ should upgrade just fine simply by unzipping the new package while Sage is running. You can confirm that by looking at the server logs in SJQ GUI. You should see a message saying the QueueLoader was stopped and then another one shortly after saying it was restarted.

As for the GUI, holding shift while clicking refresh (in FF) or holding ctrl while clicking refresh (in IE) should force the browser to ignore its cache and reload the GUI based on the new installation. Alternatively, clearing your entire cache should also achieve the same effect.

[madas]

Slugger,

Having a lot of lockups when I try to view the various log files.

I can view the server log when debug is off. But I cannot view any of the larger client logs or task logs. The webviewer just hangs and remains white if i click. The DB is running about 50Mb. I've tried leaving it for 30 mins and it doesn't help

Any thoughts?

Thanks

[Slugger]

Originally Posted by madas

Slugger,

Having a lot of lockups when I try to view the various log files.

I can view the server log when debug is off. But I cannot view any of the larger client logs or task logs. The webviewer just hangs and remains white if i click. The DB is running about 50Mb. I've tried leaving it for 30 mins and it doesn't help

Any thoughts?

Thanks

Which version are you running? How large is the media set that SJQ is scanning?

[wayner]

I am having major problems again with SJQ after running fine for a week. Last night I noticed that my extenders were very sluggish and that SageTVService was using 50%+ of the CPU. So I stopped the service and removed jetty from the startup.

This morning I added jetty back in and while I can log into SJQ immediately upon restarting the service all screens are blank. After a minute or two you can't even log in anymore.

Jetty seems fine as SRE and Sage Mobile Web work ok. Note that I am running SJQ 2.1.1.221. Debugging is off due to the previous problem.

I have not changed anything recently other than adding in a new client task and rule to support comskipping of H.264 files.

[Slugger]

Originally Posted by wayner

I am having major problems again with SJQ after running fine for a week. Last night I noticed that my extenders were very sluggish and that SageTVService was using 50%+ of the CPU. So I stopped the service and removed jetty from the startup.

This morning I added jetty back in and while I can log into SJQ immediately upon restarting the service all screens are blank. After a minute or two you can't even log in anymore.

Jetty seems fine as SRE and Sage Mobile Web work ok. Note that I am running SJQ 2.1.1.221. Debugging is off due to the previous problem.

I have not changed anything recently other than adding in a new client task and rule to support comskipping of H.264 files.

Remove the new rules you added. Does that fix the problem? If so, then you may have found a parser bug, which usually results in the SJQ server getting stuck in an infinite loop parsing the ruleset or client config.

You may also want to upgrade to the latest snapshot (280), as it includes a lot of performance fixes. If it's not a parser issue then the fixes contained in 280 may help.

[wayner]

I can't even get to the Edit Rules screen to delete that rule - but even at that this rule was in for 24 hours without causing the problem. I will upgrade to snapshot 280. What files do I have to change to upgrade by the way?

[Slugger]

Originally Posted by wayner

I am having major problems again with SJQ after running fine for a week. Last night I noticed that my extenders were very sluggish and that SageTVService was using 50%+ of the CPU. So I stopped the service and removed jetty from the startup.

This morning I added jetty back in and while I can log into SJQ immediately upon restarting the service all screens are blank. After a minute or two you can't even log in anymore.

Jetty seems fine as SRE and Sage Mobile Web work ok. Note that I am running SJQ 2.1.1.221. Debugging is off due to the previous problem.

I have not changed anything recently other than adding in a new client task and rule to support comskipping of H.264 files.

You say it's been a week, has it been exactly a week? If so, then it would be about time for SJQ to purge your completed task logs. And if this is what's causing the problems then snapshot 283 (built just minutes ago) should definitely fix it up. 283 will automatically wipe your log table out, which should fix the problem. Newer snapshots add some fixes to increase performance of the log purging. You may also consider changing your completed task log purge settings to 'Daily' so it's having to purge much less data on each run.

[Slugger]

Originally Posted by wayner

I can't even get to the Edit Rules screen to delete that rule - but even at that this rule was in for 24 hours without causing the problem. I will upgrade to snapshot 280. What files do I have to change to upgrade by the way?

Grab 283 - it will auto wipe your entire log file, which I believe is the problem. Simply unzip the zip just like an install and overwrite everything (i.e. unzip it from \SageTV\SageTV\ and maintain dir structure).

You might be better off stopping SageTV before unzipping (you usually don't have to, but since SJQ is probably locked up again the hot deploy upgrade probably won't work properly).

[wayner]

Hot deploy didn't work for going to 280 so I had to restart the Sage service. 280 seemed to work better for a little while but now I can't login again - I get to the SJQ login screen but after you enter the password and say login it times out.

I will try 283 as soon as I can - having to stop the Sage service can affect the rest of the family when they are using the extenders so I will have to look for a window!

[wayner]

Things are looking good now under 283. It did a few transcodes plus a comskip. I did have a failure of a H.264 Comskip but that is not a SJQ problem!

[Slugger]

Originally Posted by wayner

Things are looking good now under 283. It did a few transcodes plus a comskip. I did have a failure of a H.264 Comskip but that is not a SJQ problem!

How big is your sjq.sqlite file?

[wayner]

Originally Posted by Slugger

How big is your sjq.sqlite file?

38.3MB

[wayner]

I have been putting together some docs from the perspective of a new user of SJQ. Some things that are obvious to those who have used it for a while are not so obvious to the new user. I have also attempted to include other info of use - such as the problems with the return code for Comskip that is now somewhat obsolete given the recent enhancement that deals with this issue.

Sorry that it has taken me so long but I have only really been totally up and using SJQ for a week or so now.

[Slugger]

Originally Posted by wayner

I have been putting together some docs from the perspective of a new user of SJQ. Some things that are obvious to those who have used it for a while are not so obvious to the new user. I have also attempted to include other info of use - such as the problems with the return code for Comskip that is now somewhat obsolete given the recent enhancement that deals with this issue.

Sorry that it has taken me so long but I have only really been totally up and using SJQ for a week or so now.

Don't mean to spoil your fun, but if you've been referring to elements of the GUI in your docs then please download a recent snapshot and install it. I've merged my gui branch into trunk and so now the new vanilla GWT GUI has been delivered and is active. This will be the GUI starting with the next release of SJQ. There are still some cosmetic changes to happen over the next week or two, but all the current functionality is in place and all the key elements of the GUI are visible (though some of them still need to be cleaned up a little).

I think most will see this new GUI as a step backwards from the current one, however it address things like issue 50 along with other quirks and oddities introduced by the use of gwt-ext (and other 3rd party libs wrapped in GWT). Application behaviour is much more predictable and consistent when using vanilla GWT with no external helper libs so that's why I've made the switch back. If I had a more artistic side then the vanilla GWT could be made to look as polished as gwt-ext, unfortunately I don't have much of an artistic side.