Plugin: Sage Job Queue (SJQ)

[wayner]

Originally Posted by personalt

What does everyone use SJQ for? Right now I only use it for comskip.

No one responded to this other than me. What do people use SJQ for, other than commercial skipping and transcoding?

[Peter_h]

Originally Posted by Slugger

Yeah, if you get errors about files being locked when trying a hot deploy then simply stop Sage, unzip, restart Sage.

Hey Slugger.

This is still not working. I stopped the sageTVservice.exe. I then unzipped to "c:\program files\sagetv\sagetv" and it extracted fine. I then restarted sage.

However, when it up, it is still SJQ v2.1.0.176.

Also, I can not get the clientntsc to extract. I stoppe sage service and it still will not extract.

Edit: Got the Server to work. I had to open the jetty xml and save it.

Still can't get the client to extract. It is throwing a page full of errors.

[nyplayer]

Originally Posted by wayner

No one responded to this other than me. What do people use SJQ for, other than commercial skipping and transcoding?

Moving files by title to different drives... for ex: my wife has her own storage and I move titles she wants to keep over to her own storage. I move archived files over to none recording storage.

[Slugger]

Originally Posted by Peter_h

Hey Slugger.

This is still not working. I stopped the sageTVservice.exe. I then unzipped to "c:\program files\sagetv\sagetv" and it extracted fine. I then restarted sage.

However, when it up, it is still SJQ v2.1.0.176.

Also, I can not get the clientntsc to extract. I stoppe sage service and it still will not extract.

To upgrade the task client, you'll have to stop the Windows task client service before unzipping that. These details are already outlined in the task client installation wiki doc.

As for the server, you may have to clear your browser cache in order for the new version to load. Alternatively, you can load the GUI and hold the shift button while clicking reload (for Firefox) or the CTRL button while clicking reload (for IE).

[Peter_h]

Originally Posted by Slugger

To upgrade the task client, you'll have to stop the Windows task client service before unzipping that. These details are already outlined in the task client installation wiki doc.

As for the server, you may have to clear your browser cache in order for the new version to load. Alternatively, you can load the GUI and hold the shift button while clicking reload (for Firefox) or the CTRL button while clicking reload (for IE).

Hi Slugger. The wiki says to stop the service but it is unclear as to which service I should be stopping.

"To upgrade a Windows service installation of the task client simply stop the service, overwrite your existing installation with the new version, then restart the service. "

I assumed this was the sagetvservice.exe. So that is what I did.

After that didn't work I reread the entire install and went into control panel -> services -> sjqclient task and stopped it there. I then extracted and it worked.

You might want to clarify that in the install wiki.

Thanks again for all your hard work on this.

-Peter

[Peter_h]

Originally Posted by nyplayer

Moving files by title to different drives... for ex: my wife has her own storage and I move titles she wants to keep over to her own storage. I move archived files over to none recording storage.

I use it for comskip SD and HD jobs and launching EP's mediascraper.

I noticed some new features in the new version and this weekend I'm planning on adding deleting of comskip associated files for recordings that have already been deleted.

[Slugger]

Originally Posted by Peter_h

Hi Slugger. The wiki says to stop the service but it is unclear as to which service I should be stopping.

"To upgrade a Windows service installation of the task client simply stop the service, overwrite your existing installation with the new version, then restart the service. "

<snip>

You might want to clarify that in the install wiki.

Now that I've just released the latest stable versions of all my plugins and have the CI build environment setup the way I want it, I'm planning on spending some time writing a complete user guide for SJQ including examples, screenshots, etc. showing how to use SJQ for the most common setups, along with some examples of more advanced setups.

Basically a consolidation of all the various wiki docs into one complete guide. (No, there's no time line for its completion. But it is started - has been in the 'started' state for awhle - and I'm really going to try and stay away from the coding until it's done - wish me luck. )

EDIT: Hey, just noticed this is my 500th post! Not quite the same as 500 home runs or 500 goals, but still noteworthy none the less. [This milestone achieved without the assistance of PEDs ]

[wayner]

Good luck - I will send you what I have. I was just waiting until I had a little more knowledge using SJQ. For example, I don't seem to have the file cleaner working properly and I have only used one client. But if you want me to be a proof reader I am willing to play that role as well or instead.

[Opus4]

I can't resist the aside...

Originally Posted by Slugger

EDIT: Hey, just noticed this is my 500th post! Not quite the same as 500 home runs or 500 goals, but still noteworthy none the less. [This milestone achieved without the assistance of PEDs ]

Congratulations! And, I'm glad you aren't using any Posting Enhancement Droids... the spammers have been using them a LOT lately to register many new accounts, so PEDs aren't really welcome here.

- Andy

[bnh]

i've got this up and going and been running for awhile now with good results. Comskip processing has been offloaded to my other clients as opposed to running on the server, and my file cleanup task is now behaving as i intended...so all is good. This is so cool to have 3 machines performing tasks in concert with each other, disbursing the load. thank you Slugger for sharing!

Questions:

1) Can the media tests work within the client rules? Can i have a test in the client rule to see if the media object is HD or perhaps in the server rule to direct the job to a specific client if it is an HD object? i'm just thinking outloud since my clients have different CPU's. Why send a big transcode job to an underpowered client?

2) My BIG wish list item isn't so much about SJQ, as much as Sage modifying the internal transcode task, so that we could use SJQ to call _TRANSCODE in order to call our own SageTVTranscoder replacement utility, offloading that processing to another client, and still monitor it using Sage's internal screens.

3) Sometimes, mostly after power failures, SJQServer doesn't seem to see all the clients. I haven't rigorously tested it, but it seems i have to restart the client service for ALL clients to be recognized FROM all clients. Am i impatient?

[oshapir]

Hi,
I'm new to SJQ...well I haven't even installed it yet... but I have few questions so I will know if it's the right route to go:

1. I need to send a channel change (IR code) to the relevant STB after a recording (nt that STB) ends.
2. I need to send an IR code to my STB, based on the time and channel of the show beeing recorded, immediately after SageTV sends the IR code to change the channel on this STB.

Are those possible using SJQ?
(I really hope so )

Thanks,
Oren.

[Slugger]

Originally Posted by bnh

1) Can the media tests work within the client rules? Can i have a test in the client rule to see if the media object is HD or perhaps in the server rule to direct the job to a specific client if it is an HD object? i'm just thinking outloud since my clients have different CPU's. Why send a big transcode job to an underpowered client?

This is (currently) best achieved by defining two separate tasks for HD and SD comskip (if you haven't already). Then to force the HD comskip to a specific client you just ensure that only the specific client is configured to run HD comskip tasks.

If you want to be able to specify which client should be assigned a task in the ruleset then you'll have to open an issue ticket as that's not possible at the moment, but I could see the benefit of adding such a feature.

2) My BIG wish list item isn't so much about SJQ, as much as Sage modifying the internal transcode task, so that we could use SJQ to call _TRANSCODE in order to call our own SageTVTranscoder replacement utility, offloading that processing to another client, and still monitor it using Sage's internal screens.

Sounds like a feature request for SageTV.

3) Sometimes, mostly after power failures, SJQServer doesn't seem to see all the clients. I haven't rigorously tested it, but it seems i have to restart the client service for ALL clients to be recognized FROM all clients. Am i impatient?

Probably impatience. If the machines with the clients boot up quicker and start the SJQ task client before the machine running the server starts the SJQ server then the client will not register with the server on its first attempt. It will then wait five minutes before trying again. Though I haven't thoroughly tested this, if the server is not responding at all when the client tries to connect it might error out and exit. Though there should be no reason to reboot your system to get it to reconnect in these cases. If the client is running on Windows and you're using the Windows service version of the task client then just run:

net restart SJQTaskClient

from a command prompt once the server is up and running then the task client should reconnect with no problems. Another issue ticket to remind me to check on that connection code wouldn't hurt either.

[Slugger]

Originally Posted by oshapir

Hi,
I'm new to SJQ...well I haven't even installed it yet... but I have few questions so I will know if it's the right route to go:

1. I need to send a channel change (IR code) to the relevant STB after a recording (nt that STB) ends.

Possibly.

2. I need to send an IR code to my STB, based on the time and channel of the show beeing recorded, immediately after SageTV sends the IR code to change the channel on this STB.

No.

#1 is only possible if you have an executable or batch file that you can call to trigger the sending of the IR codes. And it could be up to five minutes after the recording ends before such a command would be executed.

#2 is not possible because there is no way to trigger an SJQ task immediately after a SageTV event, especially a channel changing event. I've submitted a feature request to SageTV in the past to be able to attach external Java code listeners to various SageTV events, but it has not been implemented. If it were, then a whole new set of possibilities would open up for how one could define tasks to be done. The big one for me is I'd love to be able to set the QueueLoader thread to run every time the recording schedule is updated or every time a recording has ended, etc. However, at this time such things aren't possible.

[Wheemer]

After upgrading to the latest version I am unable to reach the web page. It seemed to work at first but over the course of a couple days it has become totally unresponsive.

Sometimes the password box shows up but it will never accept the password.

Also isn't there any way this plug in can use a standard password box, so that firefox can remember it?

[Slugger]

Originally Posted by Wheemer

After upgrading to the latest version I am unable to reach the web page. It seemed to work at first but over the course of a couple days it has become totally unresponsive.

Sometimes the password box shows up but it will never accept the password.

What error do you get when you do get the prompt? Invalid password? Request timed out after 5000ms? Something else? Does downgrading back to the previous version you had installed fix the problem? If so, what version are you going back to? Is your system's CPU usage spiking when SJQ becomes non-responsive? If so, is it the SageTV process that has spiked? Some other process? Does it remain high until you stop SageTV? Are other Jetty apps that you may have installed (mobile web server, remote API, SRE, etc.) still responsive when SJQ no longer is? Does removing the Jetty plugin fix the spike in CPU from the SageTV process if it is indeed spiking? If so, does reenabling Jetty, but uninstalling SJQ relieve the CPU spike?

I love to squash bugs, but what you've provided is way too vague and doesn't point me in any direction as to what the problem could be.

Also isn't there any way this plug in can use a standard password box, so that firefox can remember it?

Unfortunately, the way it's currently implemented, no.

[Peter_h]

Originally Posted by Wheemer

After upgrading to the latest version I am unable to reach the web page. It seemed to work at first but over the course of a couple days it has become totally unresponsive.

Sometimes the password box shows up but it will never accept the password.

Also isn't there any way this plug in can use a standard password box, so that firefox can remember it?

I thought it was just me.

I just came into this thread to see if there was an easy way to revert back to the previous version.

This worked fine for 3 days and now the main page is intermittently avialable. Half of the time the page times out. The other times, I enter the password and it will crash the browser.

I also tried accessing the web interface from my laptop and it hangs the browser as well.

[Slugger]

Originally Posted by Peter_h

I thought it was just me.

I just came into this thread to see if there was an easy way to revert back to the previous version.

This worked fine for 3 days and now the main page is intermittently avialable. Half of the time the page times out. The other times, I enter the password and it will crash the browser.

I also tried accessing the web interface from my laptop and it hangs the browser as well.

Downgrading is the same as upgrading, just grab an older version and do the same thing you'd do for an upgrade.

If you're able to provide more information then I'd love to try and find the problem and fix it.

EDIT: The first thing to always try is to stop SageTV, make a backup of your current sjq.sqlite db file (found in C:\Program Files\SageTV\SageTV\) then delete it so that SJQ creates a fresh one on restart. Can you reproduce the problem on a fresh, empty db? If not, then add back your ruleset and client configs one at a time? Does one of those cause the problem? If so, then you may have found a parser bug. If not, continue operation with the new db file, if things remain stable then one the of db functions might be bad in which case I would love to get my hands on your db file for debugging. Also answering as many of the questions as possible from my above post also helps me to try and narrow down where in the code the problem might be.

[Peter_h]

Originally Posted by Slugger

Downgrading is the same as upgrading, just grab an older version and do the same thing you'd do for an upgrade.

If you're able to provide more information then I'd love to try and find the problem and fix it.

EDIT: The first thing to always try is to stop SageTV, make a backup of your current sjq.sqlite db file (found in C:\Program Files\SageTV\SageTV\) then delete it so that SJQ creates a fresh one on restart. Can you reproduce the problem on a fresh, empty db? If not, then add back your ruleset and client configs one at a time? Does one of those cause the problem? If so, then you may have found a parser bug. If not, continue operation with the new db file, if things remain stable then one the of db functions might be bad in which case I would love to get my hands on your db file for debugging. Also answering as many of the questions as possible from my above post also helps me to try and narrow down where in the code the problem might be.

Sounds good slugger. It will have to be this weekend but when I get a chance, I'll do some troubleshooting and try to reproduce the problem for you.

[Slugger]

For those having problems with v2.1.2, please try the latest snapshot (355). It's labeled as 2.1.3.355, but it's just 2.1.2 plus some debug output. When you install 355 enable SageTV debug logging (will require stop/restart of SageTV if not already enabled) and you should start seeing output like the following as SJQ is running:

SJQ QueueLoader: Cycle started...
SJQ QueueLoader: MediaMask retrieved [T]...
SJQ QueueLoader: Media list generated [size = 225]...
SJQ QueueLoader: Media scan completed...
SJQ QueueLoader: Gone to sleep!

SJQ ClientMonitor: Started...
SJQ ClientMonitor: Done...
SJQ ClientMonitor: Sleeping!

SJQ LogCleaner: Started...
SJQ LogCleaner: Server purge completed...
SJQ LogCleaner: Task purge completed...
SJQ LogCleaner: Client purge completed... LogCleaner done!

SJQ FileCleaner: Started...
SJQ FileCleaner: Video directories scanned [3 file(s) eligible for deletion]...
SJQ FileCleaner: Done!

SJQ Servlet: Request processing started...
SJQ Servlet: Client restriction check completed...
SJQ Servlet: Processing 'showQ'...
SJQ Servlet: Request processing completed!

Depending on if you have everything enabled, then you might get slightly different looking output, but regardless of what you have enabled/disabled each thread will end with a debug statement that ends with an exclamation point (such as the ones in bold above). If a thread doesn't print a statement with an exclamation then it's stuck - and this is what I need to know about - most importantly, I need to know the last statement a thread did print. I've rearranged and grouped the messages together by thread, in reality these messages will be mixed together (as the five threads of SJQ run simultaneously), but each thread should produce the statements above. If SJQ becomes non-responsive go to your SageTV debug output and piece together these messages and tell me where SJQ got hung up - this will greatly assist me in tracking down a problem.

[Slugger]

A "pre first draft" of the SJQ User's Guide is available for viewing. Some sections are still empty, some aren't even listed yet, but sections for the installation and configuration for the server and task client are done.

I'm hoping some people can review what's there and let me know if I'm on the right track. I realize it's a little long (34 pages and counting), but many of the pages are reference tables, etc. Ideally, a new user will take a read of the doc and let me know if it's helping them get SJQ up and running.

The link above is "live" so as I continue to edit the document, the changes are immediately published as I save my changes so this link will always provide the latest version of the doc, which will typically reflect the latest development version of SJQ. I will take snapshots of the document at each official release and post a PDF version, which is much nicer for printing as it includes a numbered table of contents and numbered pages.

Feedback on the doc as it stands is greatly appreciated.