Plugin: Sage Job Queue (SJQ)

[Slugger]

Originally Posted by moothekow

Slugger - is it at all possible to expose more media variables than are currently in the file? (Such as the episode title, episode number, season number, etc?). Looking at what you have so far - great works... Being a programmer myself I know how many hours you must have put into this thing... thanks.

Episode title is already exposed as a media variable (%s% - right or wrong, I commonly interchange the terms "subtitle" and "episode title" when discussing media files/airings/shows).

The other info isn't available in SageTV (episode number, season number, etc.). You may have external tools running providing metadata that does contain that info, but there is no Sage API call that provides it so I can't provide it either.

[moothekow]

Originally Posted by Slugger

Episode title is already exposed as a media variable (%s% - right or wrong, I commonly interchange the terms "subtitle" and "episode title" when discussing media files/airings/shows).

The other info isn't available in SageTV (episode number, season number, etc.). You may have external tools running providing metadata that does contain that info, but there is no Sage API call that provides it so I can't provide it either.

I was browsing through the SageTV API and noticed this call in the mediaDataAPI -- could you call this and try to get custom information from it by checking some various set of metaproperty names??:

public java.lang.String GetMediaFileMetadata(sage.MediaFile MediaFile,
java.lang.String Name)

Returns a string for the corresponding metadata property in the MediaFile's format. These are set during format detection/import. Only names set in the property "custom_metadata_properties" (which is a semicolon/comma delimited list) will be available; and any names used must not match standard attributes that SageTV imports (i.e. title, description, actor, etc.)

[Slugger]

Perhaps, if there were a well known/well defined list of metadata properties that I could lookup.

[moothekow]

Originally Posted by Slugger

Perhaps, if there were a well known/well defined list of metadata properties that I could lookup.

Check out this post (linked to from the Phoenix API discussion):

http://forums.sagetv.com/forums/show...24&postcount=2

Perhaps supporting those lists would be a good start?

[Slugger]

Maintaining that list doesn't appeal to me so this won't happen in SJQ v2. However, fear not! There is a decent chance that the ability to query for and use metadata values in your command lines will be a part of SJQ v3. No promises, but if things go as they're currently drawn up on my whiteboard then it's a strong possibility.

In the mean time, I'd recommend opening up an issue ticket at the project site as a reminder for me.

Usable SJQ v3 betas are currently targeted for late Q4 2009 or early Q1 2010.

[b-morgan]

I'm a bit confused about the setting:

SJQ should ignore failed tasks (allows for automatic requeuing):

I've just run a batch of comskip jobs and I have some complete and some failed (the server blue-screened in the middle).

If the above setting is no (the default), what will happen to the failed jobs?

If the above setting is yes, what will happen to the failed jobs? I think the answer is they will get queued again automagically.

Since my setting is currently no, how do I get those failed jobs processed again? Is it as simple as changing the setting to yes or am I too late?

Regards,

Brad

[nyplayer]

Just go into failed tasks delete the entries or clear the grid. Ignore yes means it will reque automatically.

[loonsailor]

In sqqc, what actually executes the command? Is it the shell? Specifically, I need to set an environment variable for wine, which is running comskip, because there is some confusion about where it looks for its disk drive and who owns it. So, I'd like to make the command something like:

"WINEPREFIX=/root/.wine; export WINEPREFIX; /wine/comskip %c%"

Would that work? If not, how can I set an environment variable for wine from within sjqc? I've been doing it on the line from which I run sjqc (sudo WINEPREFIX=~root/.wine ./sjqc.sh &) but that's kind of ugly.

Thanks for any help with this.

[Slugger]

Originally Posted by loonsailor

In sqqc, what actually executes the command? Is it the shell? Specifically, I need to set an environment variable for wine, which is running comskip, because there is some confusion about where it looks for its disk drive and who owns it. So, I'd like to make the command something like:

"WINEPREFIX=/root/.wine; export WINEPREFIX; /wine/comskip %c%"

Would that work? If not, how can I set an environment variable for wine from within sjqc? I've been doing it on the line from which I run sjqc (sudo WINEPREFIX=~root/.wine ./sjqc.sh &) but that's kind of ugly.

Thanks for any help with this.

The command is passed to java.lang.Runtime.exec() Not sure if it invokes a shell on Linux or not, you'll have to test (on Windows it does not). What I would do in this case is just create a shell script wrapper that sets the env var and then calls your command. Then use this wrapper shell script in SJQ.

[Slugger]

SJQ v2.1.5 released. All future release announcements will be made exclusively in the plugin release announcement thread located here. Please visit that thread for further details on the new release.

[Slugger]

As was stated in the last release announcement, SJQ v2.x has ceased development and I've started work on SJQ v3. If you look at the open/accepted issue tickets for SJQ on the project site you'll see what features are planned for implementation. The two "big" ones I'm planning for are:

• Add support for scanning other types of "objects" besides media files (i.e. system messages and "events")
• Add ability to cancel active tasks

As a bit of a preview, I was able to finish the "cancel active tasks" feature this evening and have checked in the code changes to subversion. The attached screen shot shows the new "cancel" item in the context menu. It's working, but I've disabled the automatic snapshot builds right now b/c everything I'm now working on in the v3 stream is bleeding edge and I feel obligated to give it some thorough testing before I release it into the wild.

With that said, if you want to join me on the edge, I'm always happy to have as many testers as possible. PM me and I can point you to the latest server and task client builds for testing.

Finally, this is the time to voice your wishes for any new feature requests. I can promise you I won't be able to do anything and everything as I'm on a sort of self-imposed deadline for SJQ work as I'm looking at tacking some other projects this fall, but I will definitely consider every idea brought forward. Ideally, you will add your idea/wish list items to the project issue tracker (linked to above) for easy management.

[b-morgan]

I have a Windows XP64 system with a dual-core processor that I'd like to use as an SJQ Client. I have installed the Java Runtime Environment 6 and when I open a command prompt window and type "java -version" I get a reasonable response.

When I try to start the SJQ Service, wrapper.log contains:

STATUS | wrapper  | 2009/09/27 10:46:05 | --> Wrapper Started as Service
STATUS | wrapper  | 2009/09/27 10:46:05 | Java Service Wrapper Community Edition 3.3.1
STATUS | wrapper  | 2009/09/27 10:46:05 |   Copyright (C) 1999-2008 Tanuki Software, Inc.  All Rights Reserved.
STATUS | wrapper  | 2009/09/27 10:46:05 |     http://wrapper.tanukisoftware.org
STATUS | wrapper  | 2009/09/27 10:46:05 | 
STATUS | wrapper  | 2009/09/27 10:46:06 | Launching a JVM...
FATAL  | wrapper  | 2009/09/27 10:46:06 | Unable to execute Java command.  The system cannot find the file specified. (0x2)
FATAL  | wrapper  | 2009/09/27 10:46:06 |     "java" -Djava.library.path="../lib" -classpath "../lib/wrappertest.jar;../lib/wrapper.jar;sjqc.jar;json.jar" -Dwrapper.key="AJTv8Q3lpyGV3g5j" -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=1612 -Dwrapper.version="3.3.1" -Dwrapper.native_library="wrapper" -Dwrapper.service="TRUE" -Dwrapper.cpu.timeout="10" -Dwrapper.jvmid=1 org.tanukisoftware.wrapper.WrapperSimpleApp com.google.code.sagetvaddons.sjqc.TaskClient
FATAL  | wrapper  | 2009/09/27 10:46:06 | Critical error: wait for JVM process failed

Any suggestions?

Regards,

Brad

[Slugger]

Try the non-Windows service client. Does that work? If so, then my guess is that the NT service wrapper lib I use isn't supported on Windows 64bit, if not then let's get the non-service client working before trying the service one.

[Skirge01]

I'm running the 64 bit Windows 7 Ultimate RC using the service without any issues, so it would have to be an issue specific to 64 bit Windows XP. The biggest problem I usually run into with the service is using an account with the necessary access.

[b-morgan]

I solved the problem with the large hammer...

I wiped the disk and installed XP Professional 32-bit instead.

This motherboard (ASUS A8V Deluxe) has struggled running a 64-bit OS and this was just the last straw.

I've got 3 systems with AMD Athlon X2 3800+ processors and 4 GB of memory (3.5 usable). All are connected with gigabit Ethernet and I've got 5 comskip jobs running. I only configured one processor on the SageTV server just in case.

Regards,

Brad

[loonsailor]

I've made start, stop and boot scripts for sjqc, very much like those included with sagetv. The existing sjqc.sh is not ideal, because it starts sjqc in-line, not as a daemon. That makes it difficult to start from an init script at boot time. Also, there's no straighforward way to stop it, other than a kill.

Startsjqc and stopsjqc start and stop sjqc as a daemon. startsjqc stores the pid so it can be easily stopped. I also included, in startsjqc, a read from /etc/default/sjqc to pick up any local environment variables or whatever. That way, localization can be in that standard place rather than modifying the script. I needed it because wine (which runs comskip) was getting confused about its initialization file so I made it explicit by setting the environment variable WINEPREFIX in the default file. Startsjqc and stopsjqc need to be in the same directory at the rest of the sjqc distribution, and to have execute permissions.

There's also a third file, sjqc, which should be in /etc/init.d to be run at boot time. It calls starsjqc and stopsjqc to do the real work. It does make one assumption, which is that sjqc is located in the directory /opt/sagetv/sjqc. If not, one variable needs to be changed. After putting it in /etc/init.d, run update-rc.d to make it run at the appropriate run levels. It also needs execute permissions.

I've only tested them on ubuntu, but they should be pretty general for most linux distros.

After installing these, start sjqc either with 'sudo /opt/sagetv/sjqc/startsjqc' or with 'sudo /etc/init.d/sjqc start', and stop with 'sudo /opt/sagetv/sjqc/stopsjqc' of 'sudo /etc/init.d/sjqc stop'. 'etc/init.d/sjqc restart' also works as expected. DON'T use sjqc.sh anymore if you want to use stopsjqc or /etc/init.d/sjqc, because it doesn't write the pid file and will therefore confuse things.

Here are the files:

----------------- startsjqc -------------------

#!/bin/bash

# startsjqc - Start Sage Job Control Queue Client
#
# Starts sjqc as a daemon.  Stores its process id in /var/run/sjqc.pid, so it can be easily
#	stopped later.  Before doing any work, it reads commands from /etc/default/sjqc if it
#	exists to pick up any local environment variables or whatever may be there.
#
# This file must be located in the same directory as the sjqc distribution.
#
# Returns 1 if the daemon was already running, 0 otherwise.



PIDFILE=/var/run/sjqc.pid
LOCALDEFAULTS=/etc/default/sjqc

# Read configuration variable file if it is present
[ -r $LOCALDEFAULTS ] && . $LOCALDEFAULTS

case $0 in
/*)
   SJQCPATH="$0"
   ;;
*)
   SJQCPATH="$PWD/$0"
   ;;
esac

SJQCPATH=`dirname $SJQCPATH`
echo Changing to directory $SJQCPATH
cd $SJQCPATH

# See if the daemon is already running.

if [ -e PIDFILE ] ; then
   sjqcpid=`cat $PIDFILE`
   running=`ps --pid $sjqcpid | grep java`
   if [ "$running" != "" ]; then
      echo Already running
      exit 1
   fi
fi


echo Starting server mode
java -cp json.jar:sjqc.jar com.google.code.sagetvaddons.sjqc.TaskClient "$@" &
javapid=$!
echo -n $javapid > $PIDFILE
exit 0

------------ stopsjqc --------------

#!/bin/bash

# stopsjqc - Start Sage Job Control Queue Client
#
# Stops sjqc daemon.  It looks for the pid in /var/run/sjqc.pid, confirms that the daemon is actually running, and kills it.
#
# Returns 1 if the pid file is missing or the daemon is not running or can't be stopped, 0 otherwise.


PIDFILE=/var/run/sjqc.pid

if [ -e $PIDFILE ] ; then
   PID=`cat $PIDFILE`
   running=`ps --pid $PID | grep java`
   if [ "$running" = "" ]; then
      echo sjqc not currently running
      exit 1
   fi
   kill $PID
   killcount=0
   while [ "$killcount" -lt "30" ]
   do
      running=`ps --pid $PID | grep java`
      if [ "$running" = "" ]; then
         rm $PIDFILE
         exit 0      
      fi
      killcount=`expr $killcount + 1`
      sleep 1
      echo .
   done
   echo sjqc service did not stop within 30 seconds
   exit 1 
fi
exit 1

------- /etc/init.d/sjqc ------------

#! /bin/sh
### BEGIN INIT INFO
# Provides:          sjqc
# Required-Start:    $sagetv
# Required-Stop:     $sagetv
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Sage Job Queue Client
# Description:       Starts the client that executes jobs for the sage job queue.
### END INIT INFO

# Author: Jerry Fiddler
#

# Do NOT "set -e"

BASE=/opt/sagetv/sjqc
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="Sage Job Queue Client"
NAME=sjqc

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service
#
do_start()
{
	$BASE/startsjqc
}


#
# Function that stops the daemon/service
#
do_stop()
{
	$BASE/stopsjqc
}

case "$1" in
  start)
	do_start
	;;
  stop)
	do_stop
	;;
  restart|force-reload)
	#
	# If the "reload" option is implemented then remove the
	# 'force-reload' alias
	#
	do_stop
	do_start
	;;
  *)
	#echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
	echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
	exit 3
	;;
esac

:

[Beefcake550]

Quick question, I only have extenders (HD200s) and no clients, and I just upgraded and changed my ruleset from having "IsSomethingRecording" to "IsClientConnected" since my main purpose was to not have new comskip jobs started if any HD200 is currently connected to the server. Well, last night, I know that my extender was connected since I was watching TV, yet SJQ started a comskip job.

Here's my ruleset

// Comskip on MPEG
if [IsClientConnected == "false" && IsTV == "true" && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.edl" && IsActivelyRecording == "false" && IsScheduledRecording == "true"] {
:PRIORITY 100
COMSKIPMPEG
}

// Comskip on H264
if [IsClientConnected  == "false" && IsTV == "true" && FileExists == "%d%\\%p%.ts" && FileExists != "%d%\\%p%.edl" && IsActivelyRecording == "false" && IsScheduledRecording == "true"] {
:PRIORITY 80
COMSKIPH264
}

Now, I just realized I should change IsActivelyRecording to IsSomethingRecording, but beyond that, does IsClientConnected detect HD200s being connected, or is there another test for that?

Thanks

[Slugger]

Just took a quick look at the source code. It seems in my infinite wisdom , I only chose to check for PC clients and not extenders/placeshifters with the IsClientConnected test so it will only detect if PC clients are connected to the server. As an alternative, the IsViewingMedia test will test if a connected extender is viewing media and return true if so (but it will return false if the extender is connected, but isn't watching anything). I know it's not quite exactly what you are looking for, but it's the best that's available.

I'd suggest opening an issue ticket on the project site for this. I'll only end up fixing this one in the v3 code stream so you won't see a fix for it until v3 snapshots start appearing on Google Code. (As an update to that, I'm now actually running the first v3 betas myself right now so I'm ahead of schedule, but it's rather alpha-quality and still probably at least a few weeks away before I start sharing these builds on Google Code.)

[Beefcake550]

Thanks for the heads up. I'll try to log the issue tonight when I get home.

[Slugger]

This is a request for feedback. I've implemented an initial version of the new SJQ v3 test and command syntax. Before I pat myself on the back, I'm going to explain what I've done and solicit some feedback on it.

The quick overview: Basically, I'm removing most of the tests in SJQ and replacing them with (limited) direct access to the underlying object being processed by SJQ. This accomplishes two key things:

• Allow immediate access to new tests without the need to manually add the test and rebuild SJQ
• Provides a general mechanism that is the basis for the addition of new object types that can be scanned by SJQ

Basically, I've introduced this new object called '$' that you will use in your tests to test against object specific info. The same '$' object can also be used in your command lines to easily allow the passing of any^ type of data to your command lines.

^ Only access to methods that are named Is*, Get*, and Has* is permitted and the return value of these methods must be a boolean, String or number; any other type is converted to a String via the object's toString() method.

So, here's a basic ruleset from SJQ v2.x:

if [IsTV == true && IsActivelyRecording == false && IsWatched == false && FileExists != "%d%/%p%.edl" && Filename =$ ".mpg"] {
  COMSKIP
}

Now, in SJQ v3 your ruleset will look like this:

if [$.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && $.IsWatchedCompletely{} == false && FileExists != "%d%/%p%.edl" && Filename =$ ".mpg"] {
  COMSKIP
}

Some SJQ tests will remain like FileExists, Filename, DayOfWeek, etc. (all tests that aren't really related to the actual object being processed). But if data can be queried from the object itself then that's where you should be getting it now. And the object being queried is referred to as '$' in SJQ. Where am I pulling these method names from? Directly from the SageAPI (via gkusnick's API wrapper). All the tests that you call on an object can be had right from the SageAPI docs. However, as stated above, only a subset of the object methods are available to be called. Namely, only those methods that return data and must be named Get*, Has* or Is*. When dealing with media files, the '$' object knows about all methods for the media file object, it's associated Airing object, Show object, and Channel object. You can call any of the methods from those APIs and they will be processed by SJQ.

On the client task side, nothing has changed except that the $ object is available for use in your command lines and some of the media variables (ones related to providing info about the media file like title, subtitle, etc.) have been removed since the info is now available via the $ object.

So for the comskip example, you probably don't need to change anything:

COMSKIP {
   "C:/comskip/comskip.exe \"%c%\""
}

But if you were running commands and passing the media's title to the command line like this:

SOMECMD {
   "C:/tools/mytool.exe \"%t%\""
}

Then you would now do it like this (because the %t% media variable has been removed):

SOMECMD {
   "C:/tools/mytool.exe \"$.GetShowTitle{}\""
}

This allows you do insert any kind of data about the media file into your command line (as long as that data is available via an API call). You can now pass custom metadata like this:

MYCMD {
   "C:/tools/mytools.exe \"$.GetMediaFileMetadata{my_custom_field}\""
}

I'm looking for thoughts on what I've done. Is there something missing? Some way to make it easier? Ultimately, you're going to be able to process System message objects via SJQ as well (and have full access to the SysMsg object via '$' in SJQ). So you could write a rule that resets a device when a system message tells you it's broken (this example is theoretical, I haven't even started on introducing new object types into SJQ yet, but this is what I'm expecting to happen):

// Must determine what type of object you're scanning first before calling API calls with $
if [IsSystemMessage == true && $.GetSystemMessageLevel{} == 3 && $.GetSystemMessageString{} =* "HD-PVR"] {
   RESETHDPVR
}

So this test processes a system message and if it's an error message and it's about the HDPVR then queue up a "RESETHDPVR" task. Then your task client will call a bat file or whatever to reset your HDPVR. Again, this system message business is theoretical, but it's in the plans so I'd welcome feedback on this idea as well. Everything I've discussed is working and I'm running it as my current SJQ install except for the system message stuff so it's close to coming, however, not too close as I have some cleanup to do and some heavy testing. Probably a few weeks before any builds become available on Google Code.