Plugin: SJQv4

[SteveW]

Originally Posted by Slugger

@SteveW I didn't hear back from you so I went ahead and whipped up the periodic scan and queue example (as opposed to the event based handler). This script simulates the old SJQv3 way of doing things. Basically, configure this script to run via the SJQv4 crontab once every 10 mins, an hour, etc. and it will scan and process each media file, queuing up tasks for matching files. Read the script carefully and configure it to your needs before letting it loose on your media collection!

The script is available here.

The event based scan and queue script is, therefore, left as an exercise for the reader.

Happy scripting!

So a few questions:

1) So I created a task called "SCANFORMEDIA". I have the test script set as "media_file_scanner.groovy". What would be the Executable and Exec Args that I would add in "Add SJQ Task" for "SCANFORMEDIA"? Or are there even any? Is media_file_scanner.groovy itself calling the "COMSKIP" task which I would also need to setup as well, as both would be required to complete a COMSKIP task?

2) Am I supposed to be passing any Test Script Args to "media_file_scanner.groovy"? I don't believe I should be..

3) In the SJQ UI, if I decide I want to remove the test script arguement "ts", after I remove it and save it, it comes right back. Best I can remove it down to is a single character (ie: "t"). That field won't let you resave it as blank....


Tnx

[Slugger]

Originally Posted by SteveW

So a few questions:

1) So I created a task called "SCANFORMEDIA". I have the test script set as "media_file_scanner.groovy". What would be the Executable and Exec Args that I would add in "Add SJQ Task" for "SCANFORMEDIA"? Or are there even any? Is media_file_scanner.groovy itself calling the "COMSKIP" task which I would also need to setup as well, as both would be required to complete a COMSKIP task?

No test, use the media_file_scanner.groovy as the exe (prefix it with script: so SJQ knows it's a script for the exe). You don't need to define a test, test args, nor exe args. Just make the script the exe. You will need to define the COMSKIP task. You could modify that script to actually just run comskip for you, but it's not desirable. If you still wanted to, it's left as an exercise for the reader.

2) Am I supposed to be passing any Test Script Args to "media_file_scanner.groovy"? I don't believe I should be..

No. The script is the exe, not the test. There is no test. You don't need to pass args to the script (as currently written).

3) In the SJQ UI, if I decide I want to remove the test script arguement "ts", after I remove it and save it, it comes right back. Best I can remove it down to is a single character (ie: "t"). That field won't let you resave it as blank....


Tnx

Open a ticket on the project site for tmiranda. I caught that bug on another form this morning, but not that one. You can just directly edit the sjqagent.properties file in SageTV\plugins\sjq-agent\conf\ in the meantime.

[Slugger]

Server 4.0.0.1396 and task client 4.0.0.1395 are now in the repository. One key change in this release:

Tasks can now have the sysmsgonfail property attached to them:

task.comskip.exe=C:/comskip/comskip.exe
task.comskip.sysmsgonfail=true
...

When true, the sysmsgonfail option will generate a system message (warning level) anytime the task exe fails. A message is only generated when the exe fails, not when the test fails. This option is optional and defaults to false if not specified. Details for this option are updated in the user guide wiki.

The UI plugin currently does not support this task option. Until an update is provided, you will have to manually modify this task option by directly editing the sjqagent.properties file.

[Slugger]

Just checked in some significant bug fixes for the media_file_scanner.groovy script. The original version would not properly queue up tasks that match the needsProcessing() function when testMode = false.

Please grab the latest version and update your local installations.

[tmiranda]

An API update will be coming as well but probably not until later next week. I'll be at Daytona all this weekend and will have little time to devote to Sage.

[toricred]

Originally Posted by Slugger

My latest Groovy script release is the port of the old SJQv3 file cleaner to a Groovy script suitable for periodic execution from the SJQv4 crontab. You can download the script from here:

http://sagetv-addons.googlecode.com/...cleaner.groovy

Simply edit the list of file extensions you wish to have cleaned up by the script and then schedule it for periodic execution from the SJQv4 crontab. You could also run it on demand through an SJQv4 task client or manually via SageGroovy.

After some thought, I think this is a better approach than my Sage Artifact Delete plugin (SAD), which reacts to media file deletion events. Periodically running this Groovy script will not only handle new media deletions, but will also find orphaned artifacts left prior to a SAD plugin installation. However, both solutions work and can even work together. Either way, this Groovy script provides the port of the old SJQv3 file cleaner, as promised.

Happy scripting!

I'm finally getting around to trying this script and I'm getting errors in the sjqagent.log file:

2011-02-21 09:00:08,012 INFO  [192_168_1_2-23347-1858]: Starting process runner for: Task[id=CLEANER, reqRes=100, maxInst=1, sched=ON, maxTime=86400, maxTimeRatio=1.0, rc=0-0, exe=script:C:/SJQ4/sjqv3_file_cleaner.groovy, exeArgs=, test=, testArgs=]
2011-02-21 09:00:08,012 ERROR [ServerClient]: IOError
java.io.EOFException
 at java.io.DataInputStream.readUnsignedShort(Unknown Source)
 at java.io.ObjectInputStream$BlockDataInputStream.readUnsignedShort(Unknown Source)
 at java.io.ObjectInputStream$BlockDataInputStream.readUTF(Unknown Source)
 at java.io.ObjectInputStream.readUTF(Unknown Source)
 at com.google.code.sagetvaddons.sjq.agent.network.ServerClient.getExeArgs(ServerClient.java:68)
 at com.google.code.sagetvaddons.sjq.agent.ProcessRunner.run(ProcessRunner.java:158)
 at java.lang.Thread.run(Unknown Source)

I'm pretty sure I've got my task set up wrong. The attachment shows what I've got for it. What did I miss?

Once I get this working I want to make changes to my SAV task for all segments. I've tried using the quick code slugger wrote earlier and I'm getting the same problem so I figure if I get one then I'll get the other.

[Slugger]

You can ignore that IOError (and most IOErrors). The script should still work fine. There's a bug in the SJQ server that doesn't take too kindly to trying to read exe args when they're null/not set. It doesn't affect functionality, it just causes some IOErrors on the network socket sometimes. Should they be fixed? Of course. Do they affect program functionality? No. I'm using the exact same script from svn and I get the same IOErrors, but the script works fine. Is it not deleting artifacts as expected?

[toricred]

The artifacts seem to be deleting fine. The reason I noticed that was when I tried running the sample script you posted for running comskip (SAV in my case) for all segments. That doesn't seem to be running.

Here's the script I'm trying to use:

String cmd = "C:\\Program Files (x86)\\Dragon Global\\ShowAnalyzerSuite\\ShowAnalyzer.exe"
boolean failed = false;
for(int i = 0; i < SJQ4_METADATA["SJQ4_MAX_SEGMENT_INDEX"].toInteger(); ++i) {
   String segment = SJQ4_METADATA["SJQ4_SEGMENT_${i}"]
   String[] exe = [cmd, segment]
   println "** Running ${exe}... **"
   Process p = exe.execute()
   p.waitFor()
   int rc = p.exitValue()
   println p.out.text
   println p.err.text
   if(rc < 0 || rc > 1) // Task failed, but run against the rest of the segments anyways...
      failed = true
   println "** DONE SEGMENT **"
}
if(failed)
   return 1
return 0

and attached is my task in the client.

[Slugger]

What is (or isn't) happening when you run that script?

[toricred]

I get entries in the log file like the ones I posted earlier, the SJQ completed task log shows success, but SAV never receives the files to process.

Edit: There also aren't any entries in the log file from the script.

[Slugger]

You use SAV1? If so, calling ShowAnalyzer.exe will just queue up the file with SAV's job queue and immediately return (successfully) in SJQ. I use SAV and I call ShowAnalyzerEngine.exe, which actually calls the processor and the task will wait in SJQ until it's done. That's probably what you want to do.

Also, that initial Groovy script is a bit buggy. The for loop condition would never run if there was only one segment and it would always skip the last segment of multi-segmented files. I've fixed that in the version below. It shouldn't matter for SAV, but the original would probably also deadlock if using comskip (or anything that dumps tonnes of output to stdout/stderr). That is also addressed below.

def cmd = "C:\\Program Files (x86)\\Dragon Global\\ShowAnalyzerSuite\\ShowAnalyzerEngine.exe"
def failed = false

for(int i = 0; i <= SJQ4_METADATA["SJQ4_MAX_SEGMENT_INDEX"].toInteger(); ++i) {
   def segment = SJQ4_METADATA["SJQ4_SEGMENT_${i}"]
   def exe = [cmd, segment]
   println "** Running ${exe}... **"
   def p = exe.execute()
   p.consumeProcessOutput(System.out, System.err)
   p.waitFor()
   def rc = p.exitValue()
   if(rc < 0 || rc > 1) // Task failed, but run against the rest of the segments anyways...
      failed = true
   println "** DONE SEGMENT **"
}
return failed ? 1 : 0

[Slugger]

There appears to be a bug in the latest sagex-services release (7.0.23.7). If you upgrade to this version it will fail and SJQv4 will then fail to start (since it depends on sagex-services). I've been talking to stuckless, he's aware of the problem and is looking into it.

[toricred]

This script seems better. At least the attempts seem to be getting logged. I think it is missing the path to the file and so it isn't working yet. I'll try to take it from here and respond when I've got it.

Edit:

Got it! The path was the problem.

[Spectrum]

Originally Posted by Slugger

There appears to be a bug in the latest sagex-services release (7.0.23.7). If you upgrade to this version it will fail and SJQv4 will then fail to start (since it depends on sagex-services). I've been talking to stuckless, he's aware of the problem and is looking into it.

Stuckless has pushed out 7.0.23.8. Is it safe to upgrade now?

[Slugger]

Originally Posted by Spectrum

Stuckless has pushed out 7.0.23.8. Is it safe to upgrade now?

Upgrading to .8 will allow SJQ to start up again. If you already upgraded to .7 then go ahead with .8, but if you're not on .7 then wait because .8 has another bug that prevents the HTTP services from starting properly.

[phelme]

Originally Posted by Slugger

Yes, lobs.db is used by SJQ. That's how H2 stores CLOB columns and CLOB columns is what's used to store task output. 10GB is crazy unless your tasks are dumping tonnes of output (and I mean tonnes) and the tasks are failing. By default, failed tasks will retain their output for 21 days (so as to allow time for investigation, if necessary). Successful tasks retain output for 7 days and skipped tasks for 1 day. That's all configurable in the SJQ plugin config options. I'd be having a look at the contents of some of those files, what process/task is dumping that quantity of output? Hopefully, you're not just toasting off that directory? I would assume that would corrupt your H2 database and then who knows what happens in that case. For me, my lobs.db dir contains 14KB of data in it, I've seen it get up to 500KB, but then it cleans itself up. 10GB is beyond comprehension (to me).

This just happened to me after noticing the server was sluggish. I have 20 GB of lob.db data.

Now I have to figured out how to get rid of it.

[Slugger]

Originally Posted by phelme

This just happened to me after noticing the server was sluggish. I have 20 GB of lob.db data.

Now I have to figured out how to get rid of it.

20GB in SageTV\.lobs.db\ right? If so, just stop Sage and toast of that directory.

You should never see that kind of disk usage in SageTV\plugins\sjq\sjq4.lobs.db\ - that's something I need to know about.

You may want to upgrade the sagex-h2 plugin to the latest version. It seems the latest H2 driver may fix this problem with SageTV\.lobs.db\ filling up. Somewhere buried in this thread is the discussion about why this is happening (it's an issue with the H2 tcp client caching lob columns locally). SageTV\.lobs.db\ can be toasted off completely (as long as Sage isn't running). SageTV\plugins\sjq\sjq4.lobs.db\ cannot be toasted. However, the latter should never get that large as SJQ is constantly cleaning log entries from the db. The former fills up because of issues with the H2 tcp client lib caching stuff and not cleaning up after itself, but I think this is fixed in the latest H2 driver and is available by upgrading the sagex-h2 plugin.

[KarylFStein]

This post is to add some search keywords for people that may have similar problems that I did, (and had to read/search a bit to solve along with trial-and-error).

So, I recently upgraded to SageTV 7 (finally) and SJQ/Agent/UI were the first plugins I installed for automatic commercial skip. (BTW, I've read about how great the new plugin manager was, but never really understood until I used it—wow.) I had some issues with SJQ, so read all the documentation (great job) and searched this thread, but even then it took a bit to get things running as desired.

1) Selecting Show Clients in the UI hangs. Cause: Squeezebox Server. Unlike someone earlier in this thread, I didn't get any entries in my logs about H2 DB connectivity issues. I spent a lot of time testing connectivity, turning off firewalls, etc. I couldn't figure out why SJQ and SQJAgent (on the same computer) weren't communicating. When I finally found the post about port conflicts and configured the H2 Database to use port 9093 instead of 9092 that was resolved, (as well as some weird issues with my players suddenly stopping playback randomly). Did something change with the log file entries? Because it wasn't clear there that I even had a connectivity issue. Pointing to an H2 DB connection problem would have given me more information to go on.

2) UNC paths with spaces. I recently moved from WHS to Windows 7, but wanted to keep the Recorded TV share that's a default in WHS. (I don't remember exactly, but it seems like when I went from XP to WHS and used the "\\SageTV\Recorded TV" default share instead of "\\SageTV\TV" that I used before, I had to do something with wiz.bin. (I think I read somewhere that this wasn't the case anymore, but figured it would be better to just keep things the same as far as UNC paths were concerned.) Anyway, the current united_comskip_test.groovy download didn't like the space when adjustExeArgs was set to true, but that was easily fixed by adding a couple \" characters to the Tools.setExeArgs line. I'm not sure if this will break other scenarios as my server is the only client, but it works for me.

3) Using sjqv3_file_cleaner.groovy as an executable. OK, I'm dense, but it took me a little to figure out how to run a Groovy script as an Executable instead of just a Test Script. (FYI: add script: to the front—it's in the documentation, but you have to view the SJQv4 Scripting Details section to get it). IMHO, this cleanup step should be added to the comskip tutorial as if you're using comskip you probably want to clean up the related files too after deleting the video, no? Anyway, My scripts are on a UNC drive, but no spaces there so no \" needed—not sure if spaces would cause issues.

[Slugger]

Originally Posted by KarylFStein

This post is to add some search keywords for people that may have similar problems that I did, (and had to read/search a bit to solve along with trial-and-error).

So, I recently upgraded to SageTV 7 (finally) and SJQ/Agent/UI were the first plugins I installed for automatic commercial skip. (BTW, I've read about how great the new plugin manager was, but never really understood until I used it—wow.) I had some issues with SJQ, so read all the documentation (great job) and searched this thread, but even then it took a bit to get things running as desired.

1) Selecting Show Clients in the UI hangs. Cause: Squeezebox Server. Unlike someone earlier in this thread, I didn't get any entries in my logs about H2 DB connectivity issues. I spent a lot of time testing connectivity, turning off firewalls, etc. I couldn't figure out why SJQ and SQJAgent (on the same computer) weren't communicating. When I finally found the post about port conflicts and configured the H2 Database to use port 9093 instead of 9092 that was resolved, (as well as some weird issues with my players suddenly stopping playback randomly). Did something change with the log file entries? Because it wasn't clear there that I even had a connectivity issue. Pointing to an H2 DB connection problem would have given me more information to go on.

No changes to the logging, not sure why you wouldn't have seen the H2 errors for this problem.

2) UNC paths with spaces. I recently moved from WHS to Windows 7, but wanted to keep the Recorded TV share that's a default in WHS. (I don't remember exactly, but it seems like when I went from XP to WHS and used the "\\SageTV\Recorded TV" default share instead of "\\SageTV\TV" that I used before, I had to do something with wiz.bin. (I think I read somewhere that this wasn't the case anymore, but figured it would be better to just keep things the same as far as UNC paths were concerned.) Anyway, the current united_comskip_test.groovy download didn't like the space when adjustExeArgs was set to true, but that was easily fixed by adding a couple \" characters to the Tools.setExeArgs line. I'm not sure if this will break other scenarios as my server is the only client, but it works for me.

Thanks for the fix, I just committed it to svn.

3) Using sjqv3_file_cleaner.groovy as an executable. OK, I'm dense, but it took me a little to figure out how to run a Groovy script as an Executable instead of just a Test Script. (FYI: add script: to the front—it's in the documentation, but you have to view the SJQv4 Scripting Details section to get it). IMHO, this cleanup step should be added to the comskip tutorial as if you're using comskip you probably want to clean up the related files too after deleting the video, no? Anyway, My scripts are on a UNC drive, but no spaces there so no \" needed—not sure if spaces would cause issues.

Just reviewed this script, doesn't appear to be any problems with spaces in this one. I didn't mention this script in the tutorial because this script never existed at the time that tutorial was written. Perhaps I'll add it in next time I'm editing the tutorials.

[SteveW]

Originally Posted by Slugger

... Basically, configure this script to run via the SJQv4 crontab once every 10 mins, an hour, etc. and it will scan and process each media file, queuing up tasks for matching files....

So if this is running as a cron job, I take it we don't have the ability as we did in v3 to run an immediate scan at the click of a button? I ask because sometimes I use that having had the task client off temporarily to reduce the load on my server, and then decide to fire it up and forced it to run right away. There have been a few other scenarios for me where that has been a handy feature...