Plugin: Sage Job Queue (SJQ)

[wayner]

Originally Posted by Slugger

When moving SageTV media files around while SageTV is running (which it must be if triggering tasks via SJQ), you must be very careful about what you're doing. Details are here and here. It's worth noting that those FAQ entries are quite clear in saying that SageTV must be stopped before attempting to move/rename/etc. recordings. So with that, it basically means you shouldn't be performing moves of recordings via SJQ. If you were going to try then CollinR's suggestion is the way you'd do it, but, again, I'd refrain from trying given the info from the FAQ items I've linked to.

Can this be done as a Sage internal command? I am switching to SJQ from JREKiwi's Autocompress plugin and it allowed you to specify the output folder for the Autocompressed file.

So perhaps what I need is not a move but an additional parameter for _TRANSCODE that would allow you to enter the output path for the compressed file. Any chance of that happening?

In an ideal world you would also be able to specify the filename and there would be a media variable that would just contain the episode name (so that I can have the filename be Boo.avi rather than DoraTheExplorer-Boo-123456-0.avi) but I can live with doing a manual rename every once in a while.

[Slugger]

Originally Posted by wayner

Can this be done as a Sage internal command? I am switching to SJQ from JREKiwi's Autocompress plugin and it allowed you to specify the output folder for the Autocompressed file.

So perhaps what I need is not a move but an additional parameter for _TRANSCODE that would allow you to enter the output path for the compressed file. Any chance of that happening?

In an ideal world you would also be able to specify the filename and there would be a media variable that would just contain the episode name (so that I can have the filename be Boo.avi rather than DoraTheExplorer-Boo-123456-0.avi) but I can live with doing a manual rename every once in a while.

Looks like this is possible via the Sage API. If you wouldn't mind opening a ticket in the issue tracker for this, I'd appreciate it.

What would be even better is if there was a transcode option that did nothing but rename the file, though that doesn't seem possible based on what I'm reading.

[perfessor101]

Originally Posted by Slugger

Really? Are you scanning just tv or are you scanning all media? How large is the media set that you're scanning? For comparison, I only scan tv files and my tv library is usually around 220-250 files. A scan takes 1-2 seconds to complete on my server (specs in sig).

So, I'm assuming that the 1-2 minutes for the active tasks page to update is related to the Queueloader's run time.

I have 1,100 video files stored in three UNC paths on the SageTV Server and 150 video files in a UNC path on an unRaid server that I have been slowly experimenting with moving week old recordings from Sagetv to unRaid.
623 files are mpeg-2 standard setting encoding and 609 xvid avi's are a mix of SageTV encoded and SageTV Tools 3.0 encoded

Some of those recordings have been migrated from my previous SageTV installations (via wiz.bin and nielm's xml generator along with SageMC's my files)

MaxSleep is 20 minutes because I have 3 tasks I run on each file
RunType AfterRec
RunDelay 2 minutes
ScanTV true
ScanMusic False
ScanVideos False
Scan DVDs False
ScanPictures False
IgnoreFailedTasks True
ClearServerLogs Weekly
ClearCompletedTaskLogs Weekly
ClearClientLogs Weekly
Debug false

My sig is pretty current except I am also using SJQ and SageTV's latest beta with an HD-100

One problem may be Windows XP MCE 2005 and Windows XP Service Pack 3.
At the time of install Windows Updates installs Service Pack 3 which was (or is still) incompatible ... I only wish I knew that before buying MCE 2005. (they even gave away free support for MCE 2005 and SP3 it worked so well together)

Anyway ... Sorry for the MS rant at the end ...

I may just need to simplify everything ... which is what I am trying to do with unRaid.

Eventually I want to record on two 250GB hard disks with the busiest tuners split between each (sort of balancing the tuner priority settings and which drive they record to) and then doing all the file gymnastics on the unRaid with SJQ.

Bobby

[wayner]

Originally Posted by Slugger

Looks like this is possible via the Sage API. If you wouldn't mind opening a ticket in the issue tracker for this, I'd appreciate it.

What would be even better is if there was a transcode option that did nothing but rename the file, though that doesn't seem possible based on what I'm reading.

Done. I can live without the renaming but moving to the destination folder would be nice.

[Slugger]

Originally Posted by perfessor101

So, I'm assuming that the 1-2 minutes for the active tasks page to update is related to the Queueloader's run time.

I have 1,100 video files stored in three UNC paths on the SageTV Server and 150 video files in a UNC path on an unRaid server that I have been slowly experimenting with moving week old recordings from Sagetv to unRaid.
623 files are mpeg-2 standard setting encoding and 609 xvid avi's are a mix of SageTV encoded and SageTV Tools 3.0 encoded

Some of those recordings have been migrated from my previous SageTV installations (via wiz.bin and nielm's xml generator along with SageMC's my files)

MaxSleep is 20 minutes because I have 3 tasks I run on each file
RunType AfterRec
RunDelay 2 minutes
ScanTV true
ScanMusic False
ScanVideos False
Scan DVDs False
ScanPictures False
IgnoreFailedTasks True
ClearServerLogs Weekly
ClearCompletedTaskLogs Weekly
ClearClientLogs Weekly
Debug false

My sig is pretty current except I am also using SJQ and SageTV's latest beta with an HD-100

If you use tests like FileExists and you're checking UNC/mapped drives from the network for the existence of files then I could see where maybe processing could slow down, depending on the responsiveness of your network (which may go to your MCE/XP discussion). That's the only time SJQ would even go looking for the physical files. Most of the tests talk to the SageTV wiz.bin for the details.

Tasks are put in the queue as soon as they match all conditions of a test and they are queued individually (not batch queued at the end of the QueueLoader run) and the active queue page is auto refreshed every 10 seconds so you should be seeing jobs queue up gradually during the course of a scan (if there were to be many jobs queued). I mean, as long as it's working, there are no problems, I was just a little concerned about the performance of the QueueLoader. I don't have any mapped drives, everything is local to my server, so that probably explains the bulk of the difference (though I am starting to plan for some type of NAS as I'm desperately low on disk space and the server is recording content faster than I'm able to watch it these days).

[wayner]

One more question - is this a valid test:

ChannelNumber != "300|301|302|303|304|305|306|307|221|429"]

Or can I do this to make it even shorter

ChannelNumber != "30[0-7]|221|429"] - does this imply AND or OR as the operator for channels 301,..,307

[CollinR]

Good points on the transcoder being able to move the file, after this is complete you could rename with I'm think pretty good safety as your end goal is the file no being in the database. I'm pretty sure thats what happens when you move stuff while sage is running. If you have rescan import directories enabled it should eventually find them and show them in imports as you desire. Also the rename probably isn't that bad assuming you ax everything after the second hyphen then everything before the first hyphen. This should result in a filename that is only the episode.

The nice part about having SJQ handle the transcode is as you desire you can set the destination folder in your rules so each title can have it's own folder.

***Warning I dunno for sure that losing it form the DB is the worst case senario if renaming while Sage is running. ***

[Slugger]

Originally Posted by wayner

One more question - is this a valid test:

ChannelNumber != "300|301|302|303|304|305|306|307|221|429"]

Or can I do this to make it even shorter

ChannelNumber != "30[0-7]|221|429"] - does this imply AND or OR as the operator for channels 301,..,307

Those tests will not do what you're expecting. The inequality operator (!=) does literal comparisons of the string, so the test is looking for the channel number to not equal that huge string, which every channel would pass that test as written. If used with the regex operator (replace != with !%) then they'll do what you're expecting and in the context of a regular expression both do the same thing. Your comfort level with regular expressions will dictate how you write the rule (i.e. some people would find it much easier to read and understand what the first one does as opposed to the second, but as written they both do the same thing). Have a quick read of the types and operators doc for more details on what the various operators do.

[wayner]

I have installed sjqc on my Sage server which is running WHS. I have not installed the client on this PC. I was able to use SJQ to initiate a transcode but I am getting an error when I try to use the following. Why is this?

The error I am getting is this: java.lang.RuntimeException: Unexpected '"' on line 3

if [IsTV == "true" && Filename =$ ".mpg" && FileExists != "%d%\\%p%.edl" && IsActivelyRecording == "false" && ChannelNumber != "300|301|302|303|304|305|306|307|221|429"]
{
"C:\\Comskip\\comskip.exe \"%c%\""
}

One more thing - am I supposed to also be running sjqc on my server? When I try to run sjqc.bat it I get "java is not recognized as a command". Doesn't Sage install Java? I have not installed Java on this system and the Sagetv.exe app runs as do several Sage plugins including SRE, nielm's Sage Webserver, the Sage Mobile Web Server, etc. Is there an environment variable that has to be set for Java to run? Java does not show up on my list of installed programs but it also doesn't show up on my Vista machine where I can run sjqc.bat without problems. Note that it could also be due to security settings as WHS is much more locked down.

[Peter_h]

Originally Posted by wayner

I have installed sjqc on my Sage server which is running WHS. I have not installed the client on this PC. I was able to use SJQ to initiate a transcode but I am getting an error when I try to use the following. Why is this?

The error I am getting is this: java.lang.RuntimeException: Unexpected '"' on line 3

if [IsTV == "true" && Filename =$ ".mpg" && FileExists != "%d%\\%p%.edl" && IsActivelyRecording == "false" && ChannelNumber != "300|301|302|303|304|305|306|307|221|429"]
{
"C:\\Comskip\\comskip.exe \"%c%\""
}

One more thing - am I supposed to also be running sjqc on my server? When I try to run sjqc.bat it I get "java is not recognized as a command". Doesn't Sage install Java? I have not installed Java on this system and the Sagetv.exe app runs as do several Sage plugins including SRE, nielm's Sage Webserver, the Sage Mobile Web Server, etc. Is there an environment variable that has to be set for Java to run? Java does not show up on my list of installed programs but it also doesn't show up on my Vista machine where I can run sjqc.bat without problems. Note that it could also be due to security settings as WHS is much more locked down.

I just finished setting everything up in WHS and it is working great.

I'll try to help you out with what I understand so far.

First off, the sagejobque SERVER assigns tasks. That's all it can do.

Therefore your code.

if [IsTV == "true" && Filename =$ ".mpg" && FileExists != "%d%\\%p%.edl" && IsActivelyRecording == "false" && ChannelNumber != "300|301|302|303|304|305|306|307|221|429"]
{
"C:\\Comskip\\comskip.exe \"%c%\""
}

will not work. It can not execute that line and that is why you are getting the error on line 3. The SJQServer can only assign a task so in place of the comskip.exe line, you need to call a task ID, ie "COMSKIP" like this.

if [IsTV == "true" && Filename =$ ".mpg" && FileExists != "%d%\\%p%.edl" && IsActivelyRecording == "false" && ChannelNumber != "300|301|302|303|304|305|306|307|221|429"]
{
:PRIORITY 10
COMSKIP
}

This also means, that for your server to process tasks it will also need to be a client, so you will also need to install the client onto your WHS box.

Once you do this, your WHS will appear in the clients list. Click on the client and you will then need to write the client rules. This is where you will define what the assigned task ID's from the server do. ie, the "COMSKIP" task.

This is what I have for the client.

// TASK definitions
:MAXPROCS 2
COMSKIP{
:MAX 2
"C:\\comskip\\comskip.bat \"%c%\""
}

I set a max of 2 process for now, with a max of 2 instances of comskip.

FYI: Please note that I am not calling comskip.exe but instead a batch file. The reason for this is because by default, comskip will return the wrong variable and the job will show up as failed in SJQ when it is actually completed. Therefore a simple batch file is needed to get around this.

Read a couple pages back in the thread for the batch file as well as the explanation.

Give that a try.

[ccourtney]

I got SJQ/SJQC 2.1 with comskip setup this weekend on my WHS based SageTV server. I started with base setup and went through the standard comskip returns 1 instead of 0 batch file shenanigans. Once that was fixed I was getting successfully completed comskip runs after each recording. I setup my client with MAXRUNRATIO 0.5 in order to detect hung comskip jobs.

Now I wanted to setup the system to scan for commercials during recordings. I changed to "DuringRec" removed IsActivleyRecording clause on my ruleset. I also changed MAXRUNRATIO to 1.5 but jobs timed out during the comskip processing.

How does the client calculate MAXRUNRATIO? Does it use length of the recording at the time of job start or the scheduled length of the recording?

[wayner]

Originally Posted by ccourtney

I got SJQ/SJQC 2.1 with comskip setup this weekend on my WHS based SageTV server.

Did you have to install Java on your WHS box? I cannot get the client to run as it says that it can't find java but I thought that java must be installed in order to use Sage?

[gplasky]

The .msi installed for WHS does not install java. You need to install java on your WHS server. The service will run. But other things (like the client) will not.

Gerry

[Slugger]

Originally Posted by wayner

Did you have to install Java on your WHS box? I cannot get the client to run as it says that it can't find java but I thought that java must be installed in order to use Sage?

SJQC expects that java is in the system path. If it's not then you must explicitly tell SJQC where to find java on your system. If you installed the NT Service version of SJQC then edit conf\wrapper.conf and edit the wrapper.java.command property to point to the exact location of java on your system. If you installed the command line version then edit sjqc.bat and modify the last line and replace 'java' with the full path to java on your system.

[wayner]

Thanks gusy - I guess the first step is to install Java and take it from there.

[Slugger]

Originally Posted by Peter_h

I just finished setting everything up in WHS and it is working great.

I'll try to help you out with what I understand so far.

First off, the sagejobque SERVER assigns tasks. That's all it can do.

Therefore your code.

if [IsTV == "true" && Filename =$ ".mpg" && FileExists != "%d%\\%p%.edl" && IsActivelyRecording == "false" && ChannelNumber != "300|301|302|303|304|305|306|307|221|429"]
{
"C:\\Comskip\\comskip.exe \"%c%\""
}

will not work. It can not execute that line and that is why you are getting the error on line 3. The SJQServer can only assign a task so in place of the comskip.exe line, you need to call a task ID, ie "COMSKIP" like this.

if [IsTV == "true" && Filename =$ ".mpg" && FileExists != "%d%\\%p%.edl" && IsActivelyRecording == "false" && ChannelNumber != "300|301|302|303|304|305|306|307|221|429"]
{
:PRIORITY 10
COMSKIP
}

This also means, that for your server to process tasks it will also need to be a client, so you will also need to install the client onto your WHS box.

Once you do this, your WHS will appear in the clients list. Click on the client and you will then need to write the client rules. This is where you will define what the assigned task ID's from the server do. ie, the "COMSKIP" task.

This is what I have for the client.

// TASK definitions
:MAXPROCS 2
COMSKIP{
:MAX 2
"C:\\comskip\\comskip.bat \"%c%\""
}

I set a max of 2 process for now, with a max of 2 instances of comskip.

FYI: Please note that I am not calling comskip.exe but instead a batch file. The reason for this is because by default, comskip will return the wrong variable and the job will show up as failed in SJQ when it is actually completed. Therefore a simple batch file is needed to get around this.

Read a couple pages back in the thread for the batch file as well as the explanation.

Give that a try.

Everything here is exactly spot on, I just wanted to add that the ChannelNumber test is incorrectly used in the original ruleset - review my reply from yesterday for details (you want to use the regex operator !% instead of the inequality operator !=).

[wayner]

Thanks Derek - I did change to use the regex operator !%. The reason that I was confused is that it does appear that you can assign internal Sage tasks from the server as if you replace the "c:\comskip..." line with "_TRANSCODE" then that works.

[ccourtney]

Originally Posted by wayner

Did you have to install Java on your WHS box? I cannot get the client to run as it says that it can't find java but I thought that java must be installed in order to use Sage?

I did install Java 6 Update 11 on my WHS server. In addition I renamed the jre directory under the SageTV directory so that SageTV service itself used U11. This was required for SageMC to run correctly for placeshifters. I also used the nt service version of the SJQC.

[Slugger]

Originally Posted by ccourtney

I got SJQ/SJQC 2.1 with comskip setup this weekend on my WHS based SageTV server. I started with base setup and went through the standard comskip returns 1 instead of 0 batch file shenanigans. Once that was fixed I was getting successfully completed comskip runs after each recording. I setup my client with MAXRUNRATIO 0.5 in order to detect hung comskip jobs.

Now I wanted to setup the system to scan for commercials during recordings. I changed to "DuringRec" removed IsActivleyRecording clause on my ruleset. I also changed MAXRUNRATIO to 1.5 but jobs timed out during the comskip processing.

How does the client calculate MAXRUNRATIO? Does it use length of the recording at the time of job start or the scheduled length of the recording?

Seems as though I am using the length of the file and not the length of the recording when I calculate the timeout value. And it seems the API call for the length of the file uses the length at the time of the call, which kind of makes sense. Can you please open an issue ticket for this bug?

[Slugger]

Originally Posted by wayner

Thanks Derek - I did change to use the regex operator !%. The reason that I was confused is that it does appear that you can assign internal Sage tasks from the server as if you replace the "c:\comskip..." line with "_TRANSCODE" then that works.

Yeah, those task ids are just special ids that the server interprets (any task id that starts with an underscore is just a special id that the server uses for internal tasks. All external tasks must have their command lines configured on a client).