Plugin: SJQ v1.1.0RC1 Available - Testers Needed

[bluenote]

Hey slugger, thanks for all the great work!

I've run into another GUI issue (I can't help it .. when there's two choices available, I'd rather the program take care of syntax and file format for me).

In trying to add my action of "TRANS __TRANSCODE" "CustomH264-LowBRAVIAmp3" false
the gui complains the same way it complained for the java path, but since this is supposed to be internal, I can't imagine how I need to qualify this?

(I tried TRANS __TRANSCODE with and without quotes)
oops, I seem to have figured this one out, I think. no TRANS needed?

I have a couple other thoughts, I leave it to you to decide whether they might be useful if included. One is being able to check at evaluation time the full path (I'd like to be able to do certain things based on where the files are stored local or network for example).

Multiple actions would be a nice feature too -- for things like file renames, file flag setting, cleanup actions, etc. I imagine this after transcoding, for example, (probably if NOT using the internal transcode) but really it has many uses.

The other is, I think it may be useful to be able to say .. this rule is memoryless .. in that it will continue to execute actions (assuming criteria continue to evaluate true) until criteria change. ? I imagine this might be necessary with transcoding, sometimes my jobs get aborted? the only thing is, queuing up multiple actions if SJQ is executed in quick succession wouldn't be good. Maybe this is more trouble than it's worth.

Right now I'm trying to get my first evaluation run to work with the transcode action so I'll let you know how I make out. Thanks a lot for all the great work, I love the possibilities here.

Cory

[Slugger]

Originally Posted by cnovak

Hey slugger, thanks for all the great work!

I've run into another GUI issue (I can't help it .. when there's two choices available, I'd rather the program take care of syntax and file format for me).

In trying to add my action of "TRANS __TRANSCODE" "CustomH264-LowBRAVIAmp3" false
the gui complains the same way it complained for the java path, but since this is supposed to be internal, I can't imagine how I need to qualify this?

(I tried TRANS __TRANSCODE with and without quotes)
oops, I seem to have figured this one out, I think. no TRANS needed?

Am I to understand that the __TRANSCODE exe does not work from the servlet and that you had to manually edit the ruleset in order to get it to work? I'd expect that to be the case since I know I don't check for this special exe name when verifying the path. Assuming this is the case, I'll open a bug and fix it for the next build.

Originally Posted by cnovak

I have a couple other thoughts, I leave it to you to decide whether they might be useful if included. One is being able to check at evaluation time the full path (I'd like to be able to do certain things based on where the files are stored local or network for example).

Something like a IsMediaFileLocal test?

Originally Posted by cnovak

Multiple actions would be a nice feature too -- for things like file renames, file flag setting, cleanup actions, etc. I imagine this after transcoding, for example, (probably if NOT using the internal transcode) but really it has many uses.

What I've been doing for times when I want multiple actions to be run is to create a batch file and use the batch file as the action.

Originally Posted by cnovak

The other is, I think it may be useful to be able to say .. this rule is memoryless .. in that it will continue to execute actions (assuming criteria continue to evaluate true) until criteria change. ? I imagine this might be necessary with transcoding, sometimes my jobs get aborted? the only thing is, queuing up multiple actions if SJQ is executed in quick succession wouldn't be good. Maybe this is more trouble than it's worth.

I've been debating on this 'feature' for awhile. I'm really considering just removing this restriction altogether and evaluating each rule/action combo for every media object regardless of if a previous action was run. This may or may not happen in the next build.

As for the next build, I decided to merge my db logging code into main after RC3 and so I want to stabilize that and provide all the log functions for purging logs, etc. And the state of the code is basically at a point of no return. I've invested too much time into it to back out the db logging code so now I've got to stabilize it. As I said previously, using SQLite as the db engine provides many challenges in concurrent programming that aren't an issue when using a typical client/server RDBMS. Everything was looking good until it was time to delete logs. I have some work to do in this area before I'm comfortable releasing another RC. If you don't mind being a guinea pig then PM me and I can send you builds directly as I make progress, but I probably won't post any new RC builds publicly for a week or two.

Originally Posted by cnovak

Right now I'm trying to get my first evaluation run to work with the transcode action so I'll let you know how I make out. Thanks a lot for all the great work, I love the possibilities here.

Cory

Do keep me posted on your progress.

[bluenote]

Originally Posted by Slugger

Am I to understand that the __TRANSCODE exe does not work from the servlet and that you had to manually edit the ruleset in order to get it to work? I'd expect that to be the case since I know I don't check for this special exe name when verifying the path. Assuming this is the case, I'll open a bug and fix it for the next build.

I did get it to work, it was just in the format __TRANSCODE , not TRANS __TRANSCODE that I saw on the webpage example ruleset file.
I had some problems with my custom conversion profile names, but I think I've got that squared away now.

Originally Posted by Slugger

Something like a IsMediaFileLocal test?

That would be damn awesome.

Originally Posted by Slugger

What I've been doing for times when I want multiple actions to be run is to create a batch file and use the batch file as the action.

That's what I will end up doing, too, but I think the transparency is a lot better if its all right there in the file. Troubleshooting command.com lines, and so forth the first time you do it is a pain in the ass if you just want to add one command.

Originally Posted by Slugger

I've been debating on this 'feature' for awhile. I'm really considering just removing this restriction altogether and evaluating each rule/action combo for every media object regardless of if a previous action was run. This may or may not happen in the next build.

I don't have any final conclusions about this one myself. I like the 'history' feature for sure (because having little text files serving as flags isn't that elegant) and in my case for example, keeping an EDL around so commercial skip doesn't run again, and another flag file to signal commercial cutting has been done, etc ... isn't elegant. It can be worked out though, possibly with renames because sage doesn't seem to care about the text name of a file, but rather just its ID. Implementing it without history does bring up the multiple instance issue though, in the case of jobs that take longer to complete than the time in between SJQ evaluations?

Originally Posted by Slugger

As for the next build, I decided to merge my db logging code into main after RC3 and so I want to stabilize that and provide all the log functions for purging logs, etc. And the state of the code is basically at a point of no return. I've invested too much time into it to back out the db logging code so now I've got to stabilize it. As I said previously, using SQLite as the db engine provides many challenges in concurrent programming that aren't an issue when using a typical client/server RDBMS. Everything was looking good until it was time to delete logs. I have some work to do in this area before I'm comfortable releasing another RC. If you don't mind being a guinea pig then PM me and I can send you builds directly as I make progress, but I probably won't post any new RC builds publicly for a week or two.

I don't mind testing if you don't mind what may be irregular progress. I have time today and tonight but I can't vouch for regularity after that. I'm certainly interested though, I'll send you a PM.

Do keep me posted on your progress.[/QUOTE]

Success, at least for one phase of my project so far. I now have +500 entries in my convert videos list !
(I wonder if this should be limited after some abritrary number though?) to consider the queue 'full' after say 10 - 15? and wait for the next execution?

thanks

Cory

[bluenote]

My installation is working pretty well, but necessarily uncomplete for now. How are you making out slugger?

Cory

[Slugger]

Originally Posted by cnovak

My installation is working pretty well, but necessarily uncomplete for now. How are you making out slugger?

Cory

I was actually just putting the final touches on RC5. The current changelog is below (hopefully I'll post the build before 5pm today). You say "necessarily [in]complete for now", why's that? Does the current changelog for RC5 address those issues?

RC4 -> RC5
==========

FIXES:

* Stabilize database logging code
* Add ability to purge logs from servlet
* Remove unused tabs from servlet interface
* Move all plugin specific settings to database instead of using Sage properties

REMAINING ISSUES (for RC6/final release):

* Complete/enable log filtering (buttons are there now, but they do nothing)
* Add ability to have plugin (optionally) auto purge logs daily
* Add ability for plugin to (optionally) auto remove edl files for deleted recordings daily (remove [my personal] need for dirmon2 to be running anymore)

CANNOT FIX ISSUES:

* IsMediaFileLocal test: I don't see a reliable way to provide such a test using pure Java/Sage API and I don't want to get into writing JNI code or branching off into system calls to handle this.

Am I forgetting anything?

[bluenote]

Before I run everything together (comskip/SA launch, commercial cut, compression launch) I need to solve my remote storage issue -- because my compressions need to only happen on locally stored files.

I'm hoping you wouldn't find it too troublesome to give me a feature similar to the "filename", but instead be "filepath" upon which I could run a "contains" "doesnotcontain" or what have you much like other text fields you are evaluating? I think that will allow for much flexibility, to identify by drive letter, or by UNC, or anything else under the sun. I imagine I can use this to filter commands by the location of the particular file.

The other 'nice to have' I was hoping for was a secondary/multiple action line -- primarily for me to do things like rename/delete a flag file, move "cut" file from temporary workspace back to original storage, that sort of thing. As you pointed out I can do that in a batch file, but that necessitates having *all* commands inside which is a bit awkward (although, batch files do have the benefit of taking care of concurrency/blocking/contention problems because they execute serially). I haven't started to write these in hopes some kind of solution might be implemented, but I will if this isn't something that will make it in.

So far my server is choking down all compressions queued up, but Im still running dirmon for commercial detection and cutting , but in a semi-manual way because I don't have any flag maintenance batch files implemented, so files can enter the queue twice (for example).

I'll go ahead and start implementing the whole chain and see what happens and what the challenges are when it comes down to the bits and bolts. I'll keep you updated.

thanks

Cory

[Slugger]

Originally Posted by cnovak

Before I run everything together (comskip/SA launch, commercial cut, compression launch) I need to solve my remote storage issue -- because my compressions need to only happen on locally stored files.

I'm hoping you wouldn't find it too troublesome to give me a feature similar to the "filename", but instead be "filepath" upon which I could run a "contains" "doesnotcontain" or what have you much like other text fields you are evaluating? I think that will allow for much flexibility, to identify by drive letter, or by UNC, or anything else under the sun. I imagine I can use this to filter commands by the location of the particular file.

The filename test will use the complete absolute path of the file in the next build so this should work as you're hoping (in RC5):

# Is this recording in the M: drive?
^ Filename: M:

From the servlet, this rule reads as 'Filename starts with M:'

Originally Posted by cnovak

The other 'nice to have' I was hoping for was a secondary/multiple action line -- primarily for me to do things like rename/delete a flag file, move "cut" file from temporary workspace back to original storage, that sort of thing. As you pointed out I can do that in a batch file, but that necessitates having *all* commands inside which is a bit awkward (although, batch files do have the benefit of taking care of concurrency/blocking/contention problems because they execute serially). I haven't started to write these in hopes some kind of solution might be implemented, but I will if this isn't something that will make it in.

Perhaps down the road, but this will not happen for the 1.1.0 final release. I'm still partial to using batch files for multiple actions myself so 'by down the road' you could interpret that as 'very unlikely to happen'.

Originally Posted by cnovak

So far my server is choking down all compressions queued up, but Im still running dirmon for commercial detection and cutting , but in a semi-manual way because I don't have any flag maintenance batch files implemented, so files can enter the queue twice (for example).

I'd modify the rule to limit recordings added to the trascode queue by using the Age test or some other means of restricting the number of recordings being added to the transcode queue. I have no plans on artificially restricting that number from SJQ.

Originally Posted by cnovak

I'll go ahead and start implementing the whole chain and see what happens and what the challenges are when it comes down to the bits and bolts. I'll keep you updated.

thanks

Cory

Again, keep me updated.

[Slugger]

RC5 is now available.

v1.1.0RC4 -> v1.1.0RC5
======================

* Bug 48: Use complete, absolute path of file names in 'Filename' test
* Bug 54: Log arguments used in transcode action lines
* Bug 61: Store plugin settings in data store instead of using Sage properties
* Bug 62: Stabilize database logging; move back to native JDBC driver

Enhancements to the ongoing servlet development in this build:

* Add ability to purge logs from servlet
* Remove unused tabs from servlet interface

[Slugger]

RC6 is now available.

v1.1.0RC5 -> v1.1.0RC6
======================

* Bug 53: Finalize DB logging issues
* Bug 57: Should quote everything in command lines printed to servlet
* Bug 59: Fix issue where some processes triggered could hang if they exhausted the OS output buffer
* Bug 63: Add new test: TranscodeQueueSize
* Bug 64: Log all stdout and stderr output from external commands to the database
* Bug 65: Fix synchronization issue in Logger class

[brzez]

I've been using SJQ for moving movie files from a local drive to a network drive and it works very well. One thing I'd like to use it for as well is to move normal tv recordings if or when the local drive gets filed to a certain point. It would e very useful to have SJQ check or be aware of how much space is on a local disk.

You might ask why not just have the recordings be done to the network drive?
Well, the reason for that is that I'm running a raid array on the network drive and the drives spin down when the array is not in use so it saves power that way. Unfortunately, spinup can take a while a times so recordings are not immediately accessible and that causes problems. Also when multiple recordings and playback are occurring to the network drive, the playback can stutter and suffer. So I've relegated the network drive for movies and playback only while my local drive is used for all recordings as well as playback of tv recordings.

Thanks..

-- John

[Slugger]

RC7 is now available.

RC7 should basically be relabeled as the official 1.1.0 release in the next week or two as I'm not planning any more functional changes to the code. There are a few cosmetic changes I'll probably address, but nothing major. Of course, should something be brought to my attention then that's a different story.

v1.1.0RC6 -> v1.1.0RC7
======================

* Bug 49: Add link to docs from online rule editor
* Bug 50: Make action line text input larger on form
* Bug 55: Update online docs
* Bug 56: Add ability to run SJQ on demand (wake it up early and run it from servlet)
* Bug 66: Add live status message to servlet showing what SJQ is up to
* Bug 67: New test: IsSomethingRecording - returns true if Sage is currently recording anything
* Bug 68: New test: FreeSpacePercentage - allows checking the amount of free space available on the disks used for recording

[bcjenkins]

Is there anything in SJQ which prevents java 1.5 from being ok? Not an issue if it hasn't been tested, but don't want to bother if it won't work.

When I run 1.6 I have to restart SageTV frequently.

B

[Slugger]

Originally Posted by bcjenkins

Is there anything in SJQ which prevents java 1.5 from being ok? Not an issue if it hasn't been tested, but don't want to bother if it won't work.

When I run 1.6 I have to restart SageTV frequently.

B

Unfortunately, I use parts of the java.io.File API that are only available in
JRE6 so SJQ will only build and run under JRE6.

[perfessor101]

Hello,
I was curious is there a way to have SJQ not launch a process if there will be a recording starting?

Ideally I would like to have SJQ not launch processes if recordings will be starting within a configurable amount of time ... I wasn't sure if I saw this option in the list (It may be lack of sleep).

Eventually it might be cool if SJQ had the ability to keep track of the time to complete a task (ETA / minutes per gigabyte on average?) so that it would launch jobs based on when it estimates there will be enough time to complete the task before a recording starts. (I think I'm dreaming already)

Thanks for your time,
Bobby

[Slugger]

Originally Posted by perfessor101

Hello,
I was curious is there a way to have SJQ not launch a process if there will be a recording starting?

Ideally I would like to have SJQ not launch processes if recordings will be starting within a configurable amount of time ... I wasn't sure if I saw this option in the list (It may be lack of sleep).

This currently isn't in SJQ, but doesn't seem like a bad idea.

Originally Posted by perfessor101

Eventually it might be cool if SJQ had the ability to keep track of the time to complete a task (ETA / minutes per gigabyte on average?) so that it would launch jobs based on when it estimates there will be enough time to complete the task before a recording starts. (I think I'm dreaming already)

Thanks for your time,
Bobby

This one seems like it would require a little more work and I don't really see this one surfacing to the top of my todo list in the immediate future. With that said, feel free to add it to the feature request list on the project page. I just moved my plugins (SJQ and SRE) into SourceForge so please go ahead and add a feature request so at least I'll be reminded of it. I'd actually appreciate adding both items into the feature tracker in SourceForge since these things tend to get lost otherwise as the threads get larger.

[perfessor101]

The first one would definitely help ...

I checked my logs and found that most times showanalyzer or videoredo quick stream fix take less than 7 minutes for 1 hour of video

the second one I knew was sorta dreaming ... and if I had that Quad core right now ...


Bobby