Plugin: Sage Job Queue (SJQ)

[Slugger]

Originally Posted by nyplayer

Slugger,

It's not the maximum delay it's RunDelay... I have the server to scan During record. I would like to delay the scan for at least 5 minutes after start of record.

Ah... that's in seconds. Unfortunately, it's not used in v2.x. Fortunately, I should be able to reenable that feature. Steaks are almost done, but I'll see what I can do after dinner. I'll post when a snapshot is ready - probably 2-3 hours from now.

[Slugger]

Originally Posted by nyplayer

Slugger,

It's not the maximum delay it's RunDelay... I have the server to scan During record. I would like to delay the scan for at least 5 minutes after start of record.

Originally Posted by Slugger

Ah... that's in seconds. Unfortunately, it's not used in v2.x. Fortunately, I should be able to reenable that feature. Steaks are almost done, but I'll see what I can do after dinner. I'll post when a snapshot is ready - probably 2-3 hours from now.

Snapshot 207 has the fix that uses the RunDelay setting again. Setting is in seconds and will default to 10 if a number lower than 10 is specified, there is no maximum check on the value.

[scott2020]

Originally Posted by nyplayer

scott2020,

I encode to XVID using mencoder and I use the edl from the original mpg and that works fine. I do not change the framerate when I encode to avi I use the original FPS. Maybe you can post your mencoder command line.

Shoot!!

Another stupid move on my part. I forgot an old install of Dirmon2 was still running around, which I do still use for Videotools on occasion. It has file cleanup running that deleted the txt, edl, and so on if there was no mpg file. So when I deleted the mpg and replaced it with the avi, Dirmon2 erased the files Comskip created!

I have since removed that from Dirmon2 and set SJQ to clean up files if there is no mpg or avi file instead of just mpg. I assume it is formatted like:

mpg|avi

in the "where there is no matching file" part of the file cleaner.

Anyway, after fixing that the comskip works on the AVI files.

I am also trying Handbrake 0.9.2 to see if that has the goofy audio sync time message stuff. Handbrake seems to process faster than mencoder does.


thanks again everyone!

[chrishallowell]

Ok, I got everything setup and I noticed the next scheduled queueloader is for 10:30PM. (It's 11:19PM right now) So it's going to wait a whole day before it loads anything into the queue? The last run was at 10:26PM.

I'm using SJQ 2.1.0.176

What are the following settings and what units are they in:
MaxSleep
RunType
RunDelay
IgnoreFailedTasks

I basically want to run the queueloader every 10 minutes. How do I do this?

Also, is there a setting to get the client to look at the queue every 30 seconds? I noticed that once something gets in the queue, it sits there for awhile before it's assigned to a client.

I like to process recording shows that are still recording. We like to start watching a show 25 minutes after it starts and it's nice to have the commerials already marked.

Thanks for all of your work on this! It's great.

[chrishallowell]

How do I get rid of the *.log and the *.logo.txt when there is an edl?

I see these three options:
Delete files with this extension:
When there is no matching file with this extension:
Never delete files matching this regex:

Thanks,
Chris

[Slugger]

Originally Posted by chrishallowell

Ok, I got everything setup and I noticed the next scheduled queueloader is for 10:30PM. (It's 11:19PM right now) So it's going to wait a whole day before it loads anything into the queue? The last run was at 10:26PM.

Negative. There is a bug with the display on the GUI. If the timer to refresh it gets out of sync then it stops refreshing. The display stops refreshing, but rest assured the QueueLoader will run periodically as expected, which, by default, is a few seconds after the end of each scheduled recording.

I'm using SJQ 2.1.0.176

What are the following settings and what units are they in:
MaxSleep
RunType
RunDelay
IgnoreFailedTasks

I basically want to run the queueloader every 10 minutes. How do I do this?

Indeed you've caught me behind on my wiki writing. All of this needs to be placed into an appropriate wiki doc, but for now here's the details:

MaxSleep: The maximum amount of minutes between runs of the QueueLoader; the QueueLoader is guaranteed to run at least once during this time frame

RunType: How should runs of the QueueLoader be managed. One of three settings:

• AfterRec: Run the QueueLoader RunDelay seconds after the end of each scheduled recording
• DuringRec: Run the QueueLoader RunDelay seconds after the start of each recording
• Interval: Run the QueueLoader on a constant interval of MaxSleep minutes

Note, replace RunDelay with the hardcoded value of 10 for all 2.x builds prior to snapshot 207.


RunDelay: The amount of time to delay, in seconds, before each run of the QueueLoader; ignored when RunType is set to Interval

Note: RunDelay is ignored and not used in 2.x for all builds prior to snapshot 207.

IgnoreFailedTasks: When false, tasks that fail are placed into the failed task list; tasks put in this list will not be requeued. When true, a failed task is ignored and therefore eligible to be requeued on the next run of the QueueLoader

To run the QueueLoader every 10 mins, set the RunType to Interval and MaxSleep to 10.

Also, is there a setting to get the client to look at the queue every 30 seconds? I noticed that once something gets in the queue, it sits there for awhile before it's assigned to a client.

Not at this time. If it's something you really need, open an issue ticket on the project web site and I'll consider it. As it stands, clients are hardcoded to poll the server every five minutes. However, it is worth noting that clients are greedy and will request as much work as they can possibly handle each time they talk to the server (i.e. if the client is configured for parallel task execution then the client will ask for multiple tasks each time it talks to the server).

A hardcoded value of 30 secs will never be done as that's just too frequent with no real benefit. If an issue ticket were raised that asked for the polling to be dynamically set to a few seconds after the next QueueLoader run, I could see the benefit in that. Though I will admit this particular issue doesn't strike me as something that would be done in the immediate future as the implementation I'm thinking of would require some changes to the client/server protocol, but an issue ticket and some wishful thinking never hurts, again if it's really something you think is needed.

I like to process recording shows that are still recording. We like to start watching a show 25 minutes after it starts and it's nice to have the commerials already marked.

Thanks for all of your work on this! It's great.

RunType = DuringRec will allow for this

Originally Posted by chrishallowell

How do I get rid of the *.log and the *.logo.txt when there is an edl?

I see these three options:
Delete files with this extension:
When there is no matching file with this extension:
Never delete files matching this regex:

Thanks,
Chris

Currently the file cleaner doesn't support deleting files based on the existence of other files. It's main purpose is to remove edl files when the corresponding mpg or ts file has been deleted. This is a candidate for another issue ticket, but I will say that I'll have to be inspired to rewrite the admittedly simple, yet sufficient for my needs, file cleaner into a more robust file cleaner that takes multiple rules, etc. Not saying I wouldn't ever do it, but wouldn't hold my breath on it either. Certainly wouldn't hurt to open a ticket for it.

[Slugger]

Originally Posted by Slugger

To run the QueueLoader every 10 mins, set the RunType to Interval and MaxSleep to 10.

If you're only scanning tv files with SJQ, then a constant interval run of the QueueLoader just puts an unnecessary strain on your system. Unless new recordings start every 10 minutes, most runs of the QueueLoader are just going to do a lot of computations, comparisons and calculations, but never add anything to the queue.

[nyplayer]

Originally Posted by Slugger

Ah... that's in seconds. Unfortunately, it's not used in v2.x. Fortunately, I should be able to reenable that feature. Steaks are almost done, but I'll see what I can do after dinner. I'll post when a snapshot is ready - probably 2-3 hours from now.

This works great... Thanks

[chrishallowell]

Thanks Slugger for the documentation on these settings!
It is making a lot more sense now.

I'll try using the new info you gave me to see if I can make it work the way I want.

In the file cleaner setup. What does regex mean?
"Never delete files matching this regex:"

I'm thinking I could just create a task to delete these files.

[Slugger]

Originally Posted by chrishallowell

Thanks Slugger for the documentation on these settings!
It is making a lot more sense now.

I'll try using the new info you gave me to see if I can make it work the way I want.

In the file cleaner setup. What does regex mean?
"Never delete files matching this regex:"

I'm thinking I could just create a task to delete these files.

Using my current settings as an example:

Delete files with this extension: edl|log|txt
When there is no matching file with this extension: mpg|ts
Never delete files matching this regex: .*\.logo\.txt$

Consider what happens during a comskip run with file name 'Seinfeld.mpg'...

In your recording dir you have Seinfeld.mpg and when comskip starts it creates Seinfeld.log, Seinfeld.logo.txt, Seinfeld.txt, and Seinfeld.edl. All the files are safe from the file cleaner except Seinfeld.logo.txt since there is no corresponding file named Seinfeld.logo.mpg nor Seinfeld.logo.ts. So if the file cleaner were triggered while comskip was running it would delete the logo.txt file that comskip needs. So that last input says, regardless of anything never delete files that match the given regular expression, and in this case I'm protecting the comskip generated *.logo.txt files. In my comskip.ini I tell comskip to cleanup the logo files after it's done. Once I delete the Seinfeld recording then the file cleaner will delete the edl, log and txt files.

You could create a task to delete the files as desired, but you'll either end up queuing the task once for each recording you have or you'll have to craft a very smart rule to ensure that the task is only queued once per QueueLoader run. If you find a way to do the latter then I'd like to know what you did as it would be worth adding to the wiki.

[scott2020]

Originally Posted by edbmdave

These are working for me. I convert to mp4 (Ipod)

// Transcode
if [IsTV == "true" && IsActivelyRecording == "false" && FileExists == "%d%\\%p%.mpg" && FileExists == "%d%\\%p%.edl" && FileExists != "%d%\\%p%.AUTOCOMPRESS"]
{
:PRIORITY 5
CONVERT
}

I wanted to make sure I don't have something backwards. I use SageMC, and have turned off the Sage native compression (not sure the exact wording) and in my series settings set it to autocompress. I do this on a show by show basis if I want to compress something. When the recording is done, the .AUTOCOMPRESS file gets created in the recording directory.

From how I read your rule, it says if the .AUTOCOMPRESS file is NOT there, then run the compression job.

I think I want to say if the .AUTOCOMPRESS is there, then I want the compress job to run. Easy enough to do, but I wanted to verify I have the Sage thought accurate, where Sage will create the .AUTOCOMPRESS file if I tell it in the series setting to compress. Then I want to look for that file (remove the ! from your rule) to trigger the job to run.

With the ! (not) in your rule, it tried to compress everything I had recorded which I don't want to do.

Scott

[edbmdave]

I am compressing everything now.. I left out anything with the AutoCompress file incase the built in transcoder was runnng. I have since disabled it.

[wayner]

I am just getting around to using SJQ and I have read through all of the posts since Nov or so when SJQ2 was in beta. I will be using SJQ to run Comskip and Transcode (and maybe with VideoReDo to cut files) and I have seen several examples of those. But there are a few things that I haven't seen, or that I may have missed.

How do I move a file to a different folder? (i.e from \server\tvshows to \server\videos\kids shows\dora)

How do I rename a file to remove the Title from the filename?

Here is what I want to do - If the show is Dora the Explorer then transcode, move to folder z:\\kid shows\dora\, and rename to eliminate title from the filename (i.e rename "DoraTheExplorer-Boo-1234567-0.avi" to "Boo-1234567-0.avi") and maybe even eliminate the show ID as I won't really be using these from the MyTV, just from MyVideos.

[perfessor101]

Hello,
I was curious what where acceptable values for the clearserverlogs and clearcompletedtasklogs setting.

SJQ has been running for me quite smoothly ... but the database seems to grow quickly on my system (about 1200 recordings and three tasks).
In two weeks with the default settings (clearing logs weekly) I'm at 48.5MB for the database.

I was curious if there was an option for 3 days? or do the possible values follow the time multipliers for clearing the logs?

Thanks for your time,
Bobby

[wayner]

Sorry for this stupid question, it will be obvious once someone answers, but how do you manually kick off SJQ rather than having to wait for a recording to finish?

[perfessor101]

Hello,
that's something I might be able to help with ...

To restart the queue click on the time in the lower right corner of the SJQ Window

Here's a jpg with a black rectangle around the manual start Queue loader button ...

It takes about one to two minutes for the scan to complete on my system before the Active Queue updates

Hope this helps,
Bobby

[CollinR]

Originally Posted by wayner

I am just getting around to using SJQ and I have read through all of the posts since Nov or so when SJQ2 was in beta. I will be using SJQ to run Comskip and Transcode (and maybe with VideoReDo to cut files) and I have seen several examples of those. But there are a few things that I haven't seen, or that I may have missed.

How do I move a file to a different folder? (i.e from \server\tvshows to \server\videos\kids shows\dora)

How do I rename a file to remove the Title from the filename?

Here is what I want to do - If the show is Dora the Explorer then transcode, move to folder z:\\kid shows\dora\, and rename to eliminate title from the filename (i.e rename "DoraTheExplorer-Boo-1234567-0.avi" to "Boo-1234567-0.avi") and maybe even eliminate the show ID as I won't really be using these from the MyTV, just from MyVideos.

You will need to build a batch file that does the move and rename, then use SQJ to launch this batch file. Really the windows scheduler may work just as well as your batch file will probably be whats looking for filename and not SJQ.

[Slugger]

Originally Posted by perfessor101

It takes about one to two minutes for the scan to complete on my system before the Active Queue updates

Really? Are you scanning just tv or are you scanning all media? How large is the media set that you're scanning? For comparison, I only scan tv files and my tv library is usually around 220-250 files. A scan takes 1-2 seconds to complete on my server (specs in sig).

[Slugger]

Originally Posted by perfessor101

Hello,
I was curious what where acceptable values for the clearserverlogs and clearcompletedtasklogs setting.

Indeed, I need to write a quick user guide for SJQ with the various settings explained. Admittedly, most of my SageTV play time over the last week and a half has been spent on my latest plugin project (apologies for shameless plug). I will make an effort to write that doc sometime this week.

Accepted values for the log cleaners are 'Daily', 'Weekly' and 'Monthly'.

SJQ has been running for me quite smoothly ... but the database seems to grow quickly on my system (about 1200 recordings and three tasks).
In two weeks with the default settings (clearing logs weekly) I'm at 48.5MB for the database.

I was curious if there was an option for 3 days? or do the possible values follow the time multipliers for clearing the logs?

The db size should peak at your typical usage and then stay steady. Even when the logs are purged SQLite, by default, does not shrink the db file size, but instead marks the space as free for future use. I could shrink the db file size back down after each log purge, but it seemed kind of useless. If you've been running for two weeks now then the logs should have purged once already, meaning your typical db file size will be 48.5MB. The db should not grow any larger (unless you change your setup/increase task load, etc.). For me, my db file is 70MB, but has not grown any larger since it first hit that peak. Again, I could shrink the db file back down for you, but it's only going to grow to that size again each week. If the size keeps ballooning upwards then we have something to investigate.

[Slugger]

Originally Posted by wayner

I am just getting around to using SJQ and I have read through all of the posts since Nov or so when SJQ2 was in beta. I will be using SJQ to run Comskip and Transcode (and maybe with VideoReDo to cut files) and I have seen several examples of those. But there are a few things that I haven't seen, or that I may have missed.

How do I move a file to a different folder? (i.e from \server\tvshows to \server\videos\kids shows\dora)

How do I rename a file to remove the Title from the filename?

Here is what I want to do - If the show is Dora the Explorer then transcode, move to folder z:\\kid shows\dora\, and rename to eliminate title from the filename (i.e rename "DoraTheExplorer-Boo-1234567-0.avi" to "Boo-1234567-0.avi") and maybe even eliminate the show ID as I won't really be using these from the MyTV, just from MyVideos.

Originally Posted by CollinR

You will need to build a batch file that does the move and rename, then use SQJ to launch this batch file. Really the windows scheduler may work just as well as your batch file will probably be whats looking for filename and not SJQ.

When moving SageTV media files around while SageTV is running (which it must be if triggering tasks via SJQ), you must be very careful about what you're doing. Details are here and here. It's worth noting that those FAQ entries are quite clear in saying that SageTV must be stopped before attempting to move/rename/etc. recordings. So with that, it basically means you shouldn't be performing moves of recordings via SJQ. If you were going to try then CollinR's suggestion is the way you'd do it, but, again, I'd refrain from trying given the info from the FAQ items I've linked to.