Plugin: SJQv4

[BobPhoenix]

Originally Posted by gabrielcab

What for is primarily used? to schedule TV recording? what else can a home user could use it for an HTPC experimental?

I will be using it to move my recordings to my unRAID server once I get everything setup the way I want.

[ewelin]

Slugger thanks for the replies. I'll give it a wirl as outlined above, but that's later this weekend. Right now I'm trying to get a 2nd client going. Both the server and client (slave) are in the basement right next to each other. I downloaded and installed the windows service. I then copied over the properties file and put the following at the end:

agent.port=23344
agent.schedule=* * * * *
agent.resources=100
task.HDCOMSKIP.exe=C:/Tools/comskip/comskip.exe
task.HDCOMSKIP.args="--ini=C:\Tools\comskip\comskip.ini" "$SJQ4_PATH\$SJQ4_LAST_SEGMENT"
task.HDCOMSKIP.test=\\63-server\SageTV\SJQscripts\comskip_test.groovy
task.HDCOMSKIP.testargs=ts
task.HDCOMSKIP.schedule=* * * * *
task.HDCOMSKIP.resources=51
task.HDCOMSKIP.maxprocs=1
task.HDCOMSKIP.maxtime=28800
task.HDCOMSKIP.rcmin=1
task.HDCOMSKIP.rcmax=1

I then installed the service, and started it. From there I registered the client on the server via hostname and it said it was offline and I couldn't get it to show up as online. I also tried using IP instead of hostname and the same result. I'm at a loss, any idea why the slave isn't showing up as online?

[Slugger]

Firewall blocking access? Either a firewall on the client or the server? Could be either side. Have a look at sjq.log in plugins/sjq on the server. It'll log the exception everytime it tries to contact the new client. What's the exception? If it's "connection timed out" it's probably a firewall thing. If it's "connection refused" then the client didn't start properly so head over to the client host and look at the logs there. If it's something else then I'll need to see the exception stacks. Also, realize it might take up to 10 mins (using default SJQ server settings) for a newly registered client to show up as online (maybe, thought I fixed that before public beta 1).

[phelme]

Originally Posted by Slugger

I'm not sure what you mean by this? If SJQ kills your process (or says it did) it shouldn't be running again. Unless you mean you run the script again and it finds the same media file to process again because it never properly completed the first time? I could understand that, but that is avoidable by tagging the media file with metadata to say you've already processed it so as not to repeat processing of the same media file.

I'm only adding the task (manually) once. The task is marked as "Running" (green), the groovy script is executed, mediashrink starts to run under the java process, then at some point it relaunches itself and is no longer a sub-process, mediashrink completes and I have my MKV, the task turns orange (but is stlil marked "Running"), my SJQ test script log on the task is empty and it says SJQ killed the process. There is a pause for a few seconds , the task goes back to green, script is called and mediashrink is relaunched. The created time on the task stays the same so I guess it never left the queue.

After further testing, it appears my script is being returned to after MediaShrink completes (yeah!), because I tested writing a tag to the original MPG media file (and it's in the .properties file) after the .waitFor() for MediaShrink. Even though I do have a "return 0" at the end of the script, for some reason the log is never displayed and the task thinks things are still in flux. :/

I'll get you a copy of the script when I can. Currently it's a mess and hard to read.

[Slugger]

Originally Posted by phelme

I'm only adding the task (manually) once. The task is marked as "Running" (green), the groovy script is executed, mediashrink starts to run under the java process, then at some point it relaunches itself and is no longer a sub-process, mediashrink completes and I have my MKV, the task turns orange (but is stlil marked "Running"), my SJQ test script log on the task is empty and it says SJQ killed the process. There is a pause for a few seconds , the task goes back to green, script is called and mediashrink is relaunched. The created time on the task stays the same so I guess it never left the queue.

After further testing, it appears my script is being returned to after MediaShrink completes (yeah!), because I tested writing a tag to the original MPG media file (and it's in the .properties file) after the .waitFor() for MediaShrink. Even though I do have a "return 0" at the end of the script, for some reason the log is never displayed and the task thinks things are still in flux. :/

I'll get you a copy of the script when I can. Currently it's a mess and hard to read.

I'm guessing that mediashrink fork()s itself off and does its thing in a separate process, which coincides with what you're telling me about the process tree you're seeing. When the parent returns control back to groovy I'm also guessing it returns an error that the groovy process handler isn't liking and the script is inadvertently returning 1 and if it returns 1 (or any value that isn't 0 or 2, i.e. 255, which might be happening depending on what mediashrink is doing for the fork()) then SJQ will assume the script failed and put the task back in the queue, which is why it seems to auto restart itself. To fix this:

1) Convince evilpenguin to experiment with mediashrink + SJQv4; he has way more knowledge about the interactions between the two system than I do. My test system is Linux, I don't run MS on my production system and I don't believe it runs on Linux so I really have no way to investigate it.

OR...

2) Don't run your groovy script as a test script, but instead run it as the exe. Doing that means only 0 and non-zero return values are processed and the task will never be pushed back into the queue (only pretest scripts have the ability to push the task back to the queue, which sounds like what's happening here)

I actually recommend you do #2 regardless... you don't need to define a test for a task, only an exe. So don't define a test and move your current groovy script to the exe (prefix it with script: when doing that so SJQ knows it's a groovy script you're wanting to execute for the task exe).

[ewelin]

Originally Posted by Slugger

Firewall blocking access? Either a firewall on the client or the server? Could be either side. Have a look at sjq.log in plugins/sjq on the server. It'll log the exception everytime it tries to contact the new client. What's the exception? If it's "connection timed out" it's probably a firewall thing. If it's "connection refused" then the client didn't start properly so head over to the client host and look at the logs there. If it's something else then I'll need to see the exception stacks. Also, realize it might take up to 10 mins (using default SJQ server settings) for a newly registered client to show up as online (maybe, thought I fixed that before public beta 1).

Firewall it was. Thanks again. Now I've got my 2nd agent setup and running with the same task, hdcomskip. What's strange is that if I view the queue, it shows one item running on the server, and the rest of the tasks are marked as returned. What does returned mean?

[Slugger]

Returned means the test script failed and returned 1... have a look at the output for clues as to the reason.

[ewelin]

Where would that be located? If I view the logs via the GUI it says "No log information for this task"

[tmiranda]

I just uploaded a new UI Mod Plugin which *should* fix the issues reported. I can't reproduce the "Tasks disappearing" issue but I think I know why it was happening. I'd appreciate feedback, positive or negative, on the issue.

Tom

[Slugger]

Originally Posted by ewelin

Where would that be located? If I view the logs via the GUI it says "No log information for this task"

If that's happening then there's a problem with the script itself... look in the log file of the task client that ran it.

[phelme]

Originally Posted by Slugger

2) Don't run your groovy script as a test script, but instead run it as the exe. Doing that means only 0 and non-zero return values are processed and the task will never be pushed back into the queue (only pretest scripts have the ability to push the task back to the queue, which sounds like what's happening here)

I actually recommend you do #2 regardless... you don't need to define a test for a task, only an exe. So don't define a test and move your current groovy script to the exe (prefix it with script: when doing that so SJQ knows it's a groovy script you're wanting to execute for the task exe).

This did the trick! Everything executes now as expected. Additionally, now that I have Eclipse setup I can make faster progress.

Thanks Slugger!!

[ewelin]

Originally Posted by Slugger

If that's happening then there's a problem with the script itself... look in the log file of the task client that ran it.

The groovy script? It's the exact same script. I configured both agents to have the exact same HDCOMSKIP task. The groovy script is in a network location which both agents can access. The comskip is installed in the same location on both machines.

I'm also not sure which log file you want me ot look at. Is it the sjqagent.log file on the agent computer in the logs folder? If so that file isn't much help:

2011-01-14 15:47:17,285 INFO  [Config]: Watching 'C:\sjqagent\bin\..\conf'
2011-01-14 15:58:14,390 INFO  [Config]: Watching 'C:\sjqagent\bin\..\conf'
2011-01-14 16:13:54,213 INFO  [Config]: Watching 'C:\sjqagent\bin\..\conf'
2011-01-14 16:21:14,209 INFO  [Config]: Watching 'C:\sjqagent\bin\..\conf'
2011-01-15 18:20:09,905 INFO  [Config]: Watching 'C:\sjqagent\bin\..\conf'

[Slugger]

That's the log... I might turn to the sjq.log on the server. If there's no help there then you might have to crank the log level up - start on the task client. Edit the sjqagent.log4j.properties file. Increase logging from INFO to DEBUG, restart agent. (The log will get real big, real fast.)

EDIT: Once you crank up the logging, reproduce the scenario (tasks keep going back to returned). Send me the debug level task client log along with the server's sjq.log (at INFO level). I need to compare side by side to see how the two are interacting.

[ewelin]

Something is funky... log did not get big at all..
client:

2011-01-15 19:55:46,839 INFO  [Config]: Watching 'C:\sjqagent\bin\..\conf'

sjq.log:

2011-01-15 19:54:41,963 INFO  [TaskQueue]: Running queue processor now!
2011-01-15 19:54:51,404 INFO  [ListenerClient]: Disconnected from 63-server:23344
2011-01-15 19:54:51,404 INFO  [ActiveTaskManager]: Validated 1 running task(s)!
2011-01-15 19:55:03,972 INFO  [TaskQueue]: Scheduling queue processor for ~8 seconds from now!
2011-01-15 19:55:11,975 INFO  [TaskQueue]: Running queue processor now!
2011-01-15 19:55:33,984 INFO  [TaskQueue]: Scheduling queue processor for ~8 seconds from now!
2011-01-15 19:55:41,987 INFO  [TaskQueue]: Running queue processor now!
2011-01-15 19:55:51,412 INFO  [ListenerClient]: Disconnected from 63-server:23344
2011-01-15 19:55:51,412 INFO  [ActiveTaskManager]: Validated 1 running task(s)!
2011-01-15 19:56:03,996 INFO  [TaskQueue]: Scheduling queue processor for ~8 seconds from now!
2011-01-15 19:56:05,809 INFO  [AgentManager]: Pinging Client[host=63-server:23344,state=ONLINE,lastUpdate=2011-01-15 19:54:05.808]
2011-01-15 19:56:05,809 INFO  [ListenerClient]: Disconnected from 63-server:23344
2011-01-15 19:56:05,809 INFO  [AgentManager]: Pinging Client[host=63-slave:23344,state=OFFLINE,lastUpdate=2011-01-15 19:54:26.785]
2011-01-15 19:56:06,434 INFO  [ListenerClient]: Disconnected from 63-slave:23344
2011-01-15 19:56:11,999 INFO  [TaskQueue]: Running queue processor now!
2011-01-15 19:56:14,969 INFO  [Listener]: Received connection from: /192.168.15.10
2011-01-15 19:56:17,876 INFO  [Handler]: CMD: ADDTASK :: PEER: /192.168.15.10
2011-01-15 19:56:18,189 INFO  [TaskQueue]: Scheduling queue processor for ~8 seconds from now!
2011-01-15 19:56:18,595 INFO  [Handler]: CMD: QUIT :: PEER: /192.168.15.10
2011-01-15 19:56:26,192 INFO  [TaskQueue]: Running queue processor now!
2011-01-15 19:56:26,208 WARN  [TaskQueue]: Client does not have enough free resources to perform this task! [49 < 51]; skipping: Client[host=63-server:23344,state=ONLINE,lastUpdate=Sat Jan 15 19:56:26 EST 2011]
2011-01-15 19:56:26,208 INFO  [ListenerClient]: Disconnected from 63-server:23344
2011-01-15 19:56:27,442 INFO  [TaskQueue]: Assigned task 237 of type 'HDCOMSKIP' to Client[host=63-slave:23344,state=ONLINE,lastUpdate=Sat Jan 15 19:56:24 EST 2011]
2011-01-15 19:56:27,646 INFO  [ListenerClient]: Disconnected from 63-slave:23344
2011-01-15 19:56:32,913 INFO  [Listener]: Received connection from: /192.168.15.11
2011-01-15 19:56:33,070 INFO  [Handler]: CMD: UPDATE :: PEER: /192.168.15.11
2011-01-15 19:56:33,304 INFO  [TaskQueue]: Scheduling queue processor for ~8 seconds from now!
2011-01-15 19:56:33,679 INFO  [Handler]: CMD: QUIT :: PEER: /192.168.15.11
2011-01-15 19:56:34,008 INFO  [TaskQueue]: Scheduling queue processor for ~8 seconds from now!
2011-01-15 19:56:41,307 INFO  [TaskQueue]: Running queue processor now!
2011-01-15 19:56:42,011 INFO  [TaskQueue]: Running queue processor now!
2011-01-15 19:56:42,011 WARN  [TaskQueue]: Client does not have enough free resources to perform this task! [49 < 51]; skipping: Client[host=63-server:23344,state=ONLINE,lastUpdate=Sat Jan 15 19:56:42 EST 2011]
2011-01-15 19:56:42,011 INFO  [ListenerClient]: Disconnected from 63-server:23344
2011-01-15 19:56:42,761 INFO  [Listener]: Received connection from: /192.168.15.11
2011-01-15 19:56:43,042 INFO  [Handler]: CMD: UPDATE :: PEER: /192.168.15.11
2011-01-15 19:56:43,042 INFO  [TaskQueue]: Assigned task 237 of type 'HDCOMSKIP' to Client[host=63-slave:23344,state=ONLINE,lastUpdate=Sat Jan 15 19:56:39 EST 2011]
2011-01-15 19:56:43,246 INFO  [ListenerClient]: Disconnected from 63-slave:23344
2011-01-15 19:56:43,261 INFO  [TaskQueue]: Scheduling queue processor for ~8 seconds from now!
2011-01-15 19:56:43,746 INFO  [Handler]: CMD: QUIT :: PEER: /192.168.15.11
2011-01-15 19:56:51,264 INFO  [TaskQueue]: Running queue processor now!
2011-01-15 19:56:51,421 INFO  [ListenerClient]: Disconnected from 63-server:23344
2011-01-15 19:56:51,421 INFO  [ActiveTaskManager]: Validated 1 running task(s)!

[ewelin]

so i have a 2nd win7 machine that I have used as a client with sjq3 in the past but wasn't planning on it with sjq4. Anyways, I installed the windows service on it. copied over the properties file from the agent that keeps marking tasks returned and added it to SJQ. It's doing the exact same thing with returning each task as returned.

[dadavo]

Originally Posted by Slugger

If you're running via cron then no env vars are ever going to be available. You'll have to find the media file of interest via groovy then pull all the info from the Sage API. So assuming you've found the media file of interest and stored it in a variable called mf:

String desc = MediaFileAPI.GetMediaFileFormatDescription(mf)

Then use the desc variable as an argument to the batch file you're calling (presumably from groovy). You'd pull the path and file names in similar fashion using the MediaFileAPI and pass those as arguments to the batch file as well.

I have resolved my issue. The problem was the location set in the client had .GROOVY in uppercase. Changing that resolved my issue. I also had this problem with the Test Script Arguments being in uppercase. I am sorry if I misunderstood that you meant the extension in the client settings should be lowercase or uppercase, not the actual file extension.

[Slugger]

@ewelin I don't see any DEBUG log messages being printed anywhere. Did you restart the agent after changing the log level settings?

Alternatively, change the script to something dead easy and see if it runs successfully. I'm just not sure what the issue is at this point.

[ewelin]

Yup, I had restarted the server at 54, watched the queue and the tasked was returned at 55 so I stopped the log at 56. I'll re-run it and provide a longer timeframe later this morning. I just don't get it cause the same script is running on the server's agent just fine.

[phelme]

Originally Posted by awhenry

...as I don't think there is a way to leverage resource allocation and multiple clients using the cron method alone since cron only runs on the server. Once queued, presumably clients capable of running those tasks would pick them off the queue and run them as resources are available.

Slugger: Need a clarification based on awhenry's statement. Are cron based tasks' resources monitored like trigger based tasks? If not, I'd have to take a differing direction on what I'm doing as currently I'd be relying on a cron task not being executed because there wouldn't be enough resources would be available.

yes, I could just try this out and see what happens but I figured it would be easier to simply ask first.

[Slugger]

Originally Posted by phelme

Slugger: Need a clarification based on awhenry's statement. Are cron based tasks' resources monitored like trigger based tasks? If not, I'd have to take a differing direction on what I'm doing as currently I'd be relying on a cron task not being executed because there wouldn't be enough resources would be available.

yes, I could just try this out and see what happens but I figured it would be easier to simply ask first.

Tasks queued up via cron are treated the same way as tasks queued up via Sage events. But it depends on how you implement your cron based task. Let's take the recently popular "scan and process" implementation everyone wants to see return from SJQv3.

In v4, to accomplish this you'd have to have a groovy script that looks like this:

for(Object mediaFile : MediaFileAPI.GetMediaFiles()) {
   if(<Whatever conditions you want to check for...) {
      // Do your task
   }
}

It's how you implement the "do your task" that is important. If you just call your exe in that block then there is no resource management/client distribution for the task. The loop simply runs and does the exe on each file it finds that matches.

However, if inside that if block you queue up the tasks you want to do (i.e. COMSKIP) and put COMKSKIP tasks into the queue for each matching file then SJQ will process all of those additional tasks you've queued up using all of the resource/client rules/allocations as defined.

The difference might seem subtle, but hopefully it's clear?