Plugin: Sage Job Queue (SJQ)

[perfessor101]

Hello Slugger ...
I finally wrote some blank batch files so I could process all my pre-SJQ files through and see if that helped.

It did, SJQ was running very smoothly for about two weeks without crashes.
Then I jinxed it, I was about to write and tell you about how much it helped...

I hadn't really noticed the hang times at first ... because well ... you know SageTV ... and I was moving more and more of my media collection to my UnRaid and the hang times were getting longer.

In the last few days the hang times were getting worse ... but I removed sagex.jetty.starter.Main from the runnables line after trying a few other things and my hang times dropped considerably.

I have 1,431 recorded files on the unRaid Nas.

I have the latest SJQ and SJQC snapshots installed with Jetty 1.5

I was curious if there was anything I could send to help find out how to speed SageTV & SJQ back up with crazy large video sets?

ps I was also curious about this error

MediaVariables.getVars(File f): Cannot get MediaFile object from database!

is there anyway for me to find out more information for this?

Thanks for your time,
Bobby

[Slugger]

Originally Posted by perfessor101

I was curious if there was anything I could send to help find out how to speed SageTV & SJQ back up with crazy large video sets?

Install snapshot 355 and tell me where it's hanging when it does by following the debug output as described in the above post. Typically, hanging is going to be caused by one of the threads hanging on to a db lock, which will block the other threads from progressing. I need to know which threads are hanging/blocking then I can start to investigate ways to address it.

1400+ is a larger sized media set, but not massive. Is SJQ locking (i.e. required a restart of SageTV to get running again) or is it just taking a long time to run? If you use file tests (such as FileExists) and your recordings are on a NAS then that could be the reason why it takes a longer time for the QueueLoader to run (as the QueueLoader must constantly check for files on your NAS for each scan).

I'm not really sure why removing the remote API plugin helped, I tend to lean on the side of coincidence, but if you can tell me which thread is taking a long time/hanging then that might give me some more answers.

[perfessor101]

I'll put the Jetty Server (sagex.jetty.starter.Main) back in before going to sleep and see what it does by wake up time ... and then I should have lots of log information.

If I process the last few recordings manually in the morning ... and then change the logic I use in my batch files to check if the task is completed ... then I don't have to perform as many file exists checks in SJQ.

if I have three tasks to perform on an object ... and use a priority on each task will SJQ sort them out for me without ever running them in the wrong order?
ie:
Battlestar Galactica.mpg
VRDQSF-300
then ShowAnalyzer-200
then VRDCUT-100
then I would only have to check for file exists Battlestar Galactica.working so it doesn't try to do two things at once.
Currently each rule checks a tag file exists from the last process.

Does SJQ re-check the rule that 'fired' a task before preforming that task?
or after it queues all three tasks for a media object will it skip the second and third task for the file when Battlestar Galactica.working pops up from the first task.

too much rambling ...

and oh, yes I have to kill the SageTV process to stop it ...

I'll have the logs by midnight ...
Bobby

[perfessor101]

I know part of what I was trying to ask ... is there any way to ask SJQ if a previous task has completed as a condition to running the next ?

Bobby

checking documentation ...

[Peter_h]

Originally Posted by perfessor101

I know part of what I was trying to ask ... is there any way to ask SJQ if a previous task has completed as a condition to running the next ?

Bobby

checking documentation ...

It's in the wiki somewhere. The easiest way is to have the first task output "somefile" and then the next task searched for that.

For example:

Job 1 renames and outputs renamefinished.txt.

Job 2 shrinks and has to look for renamefinished in order to run otherwise it skips over it.

[Slugger]

Originally Posted by perfessor101

I know part of what I was trying to ask ... is there any way to ask SJQ if a previous task has completed as a condition to running the next ?

Bobby

checking documentation ...

Originally Posted by Peter_h

It's in the wiki somewhere. The easiest way is to have the first task output "somefile" and then the next task searched for that.

For example:

Job 1 renames and outputs renamefinished.txt.

Job 2 shrinks and has to look for renamefinished in order to run otherwise it skips over it.

Peter's got it right, SJQ itself doesn't support conditional running of tasks based on the success of others, but you can simulate it through the use of trigger files that your command lines look for before running.

[Slugger]

Originally Posted by perfessor101

I'll put the Jetty Server (sagex.jetty.starter.Main) back in before going to sleep and see what it does by wake up time ... and then I should have lots of log information.

Yeah, I completely misread what you wrote the first time. I thought you were referring to the remote API plugin, but in fact you were saying that removing the Jetty plugin startup returned stability. That's expected, but the next step is to reenable Jetty, but remove SJQ. Is the system stable? Does it remain stable? If so, reinstall SJQ. Does the system become unstable? Does removing SJQ return stability? However, I'm willing to stipulate that SJQ is causing your problems, what I really need to know is where the problems are being caused and the debug output from snapshot 355 will help get me pointed in the right direction.

You're going to have to piece together the debug output and rearrange it, etc, but when SJQ becomes non-responsive then one or more of the threads should be stuck and telling me the last statement printed is going to get me started towards finding the problem.

If I process the last few recordings manually in the morning ... and then change the logic I use in my batch files to check if the task is completed ... then I don't have to perform as many file exists checks in SJQ.

We're assuming here that the problem is indeed with the number of reads to the NAS - that's only speculation at this point. Though I could see it being a cause for a perf issue, though I don't see how it would cause SJQ to completely lock up. Again, debug output is hopefully going to answer a lot of questions when we get some.

if I have three tasks to perform on an object ... and use a priority on each task will SJQ sort them out for me without ever running them in the wrong order?
ie:
Battlestar Galactica.mpg
VRDQSF-300
then ShowAnalyzer-200
then VRDCUT-100
then I would only have to check for file exists Battlestar Galactica.working so it doesn't try to do two things at once.
Currently each rule checks a tag file exists from the last process.

You should be creating a signal file for each process if each depends on the prior one succeeding. Alternatively, you could put all three command lines in the same task block - this ensures that each command runs one after the other and command lines only run if the one that precedes it succeeds.

So instead of something like this:

# Server Ruleset
if [<whatever conditions>] {
   :PRIORITY 30
   TASK1
   :PRIORITY 20
   TASK2
   :PRIORITY 10
   TASK3
}

-------

# Client config
TASK1 {
   "C:/whatever.exe \"%c%\""
}

TASK2 {
   "C:/something.exe \"%c%\""
}

TASK3 {
   "C:/foobar.exe \"%c%\""
}

You could set it up like this:

# Server Ruleset
if [<whatever conditions>] {
   :PRIORITY 10
   ONEBIGTASK
}

---------

# Client config

ONEBIGTASK {
   "C:/whatever.exe \"%c%\""
   "C:/something.exe \"%c%\""
   "C:/foobar.exe \"%c%\""
}

So with ONEBIGTASK each command only runs if the previous one returns a success code - once a command fails then the remaining ones are not ran and the task is marked as failed in SJQ.

Does SJQ re-check the rule that 'fired' a task before preforming that task?
or after it queues all three tasks for a media object will it skip the second and third task for the file when Battlestar Galactica.working pops up from the first task.

No, once a task is queued no further checks are done on it - it simply sits in the queue until a client pulls it out to work on it.

too much rambling ...

and oh, yes I have to kill the SageTV process to stop it ...

I'll have the logs by midnight ...
Bobby

If SJQ is locking up/getting caught in an infinite loop then something weird is happening. Hopefully some debug output gives me some answers.

[perfessor101]

I'm already doing that but it results in lots of file exists tests ...

and I'm trying to streamline things as SageTV on my quadcore was hanging more than playing

the UI Hang times had numbers something like about 18320 (I might have an extra zero or be missing one) in blocks of fifteen together in the log

Hang times almost disapear with Jetty and SJQ removed

VRDQSF.bat

now Starting Quick Stream Fix %~dpnx1 >>%~dpn1.working
rem Video Redo QuickStreamFix
cscript //nologo "c:\testing\vp.vbs" "%~dpn1.mpg" "%~dp1temp\%~n1.mpg.bak" "/t1"

IF %ERRORLEVEL% NEQ 0 goto done 

if not exist %~dp1temp\%~n1.mpg.bak goto error
rem swap original to temp directory 
move /y %~dpn1.mpg %~dp1temp\%~n1.mpg
rem swap QSF'd file to original name and location
move /y %~dp1temp\%~n1.mpg.bak %~dpn1.mpg

if not exist %~dpn1.mpg goto error

now Finished Quick Stream Fix %~dpn1 >>%~dpn1.vrdqsf

del %~dpn1.working

exit

:error

now Error %~dpn1.vrdqsf >>%~dpn1.error

Then Show Analyzer is kicked off when the *.vrdqsf file is created.

which creates an *.Vprj file in which I confirm the commercial cuts ...

which finally launches VideoRedo to cut the commercials

then I'm considering automating compression and moving to the NAS via SJQ

Bobby

[perfessor101]

I could combine VideoRedoQuickStreamFix and ShowAnalyzer in one task ...

and then put VideoRedoCut and Move / compress in another task.

but I confirm the commercial cuts in between ShowAnalyzer and VideoRedoCut as I can never get SA or comskip tweaked just quite right for every show on every channel.


OR if I move files to the NAS in between recordings with SJQ ...
once they are on the NAS I can run all the other tasks via DirMon from another computer.

Bobby

[Slugger]

Originally Posted by perfessor101

I'm already doing that but it results in lots of file exists tests ...

and I'm trying to streamline things as SageTV on my quadcore was hanging more than playing

the UI Hang times had numbers something like about 18320 (I might have an extra zero or be missing one) in blocks of fifteen together in the log

Hang times almost disapear with Jetty and SJQ removed

VRDQSF.bat

now Starting Quick Stream Fix %~dpnx1 >>%~dpn1.working
rem Video Redo QuickStreamFix
cscript //nologo "c:\testing\vp.vbs" "%~dpn1.mpg" "%~dp1temp\%~n1.mpg.bak" "/t1"

IF %ERRORLEVEL% NEQ 0 goto done 

if not exist %~dp1temp\%~n1.mpg.bak goto error
rem swap original to temp directory 
move /y %~dpn1.mpg %~dp1temp\%~n1.mpg
rem swap QSF'd file to original name and location
move /y %~dp1temp\%~n1.mpg.bak %~dpn1.mpg

if not exist %~dpn1.mpg goto error

now Finished Quick Stream Fix %~dpn1 >>%~dpn1.vrdqsf

del %~dpn1.working

exit

:error

now Error %~dpn1.vrdqsf >>%~dpn1.error

Then Show Analyzer is kicked off when the *.vrdqsf file is created.

which creates an *.Vprj file in which I confirm the commercial cuts ...

which finally launches VideoRedo to cut the commercials

then I'm considering automating compression and moving to the NAS via SJQ

Bobby

Do your commands produce a lot of output? If so, try disabling task output logging and see if that helps as well? Doing so means your output will not be viewable via the SJQ GUI, but that might be ok if it means SJQ doesn't bork itself.

Maybe 1400 is too large a media set as well, the first time I remember reading about such an issue I thought it was a set of 10,000 and most files were MP3s, that's when I implemented the use of the media mask, which from what I remember solved that user's problem.

I only scan TV files and I'm always hovering around 200-250, I've never had a media list even close to 1000 to test the load/scalability of the various execution threads of SJQ. Each run of the QueueLoader will do approximately n*t*r queries of the wiz.bin in rapid succession, where n is the number of media objects to be scanned, t is the number of tests in a single rule block and r is the number of rule blocks defined in your ruleset. I have no idea, of course, what kind of locking SageTV does when something is reading from wiz.bin, which might explain your UI lock ups. My 225 recordings are scanned by the QueueLoader on my Q6600 quad core in 0.8-1.3 seconds. My ruleset consists of two ruleblocks, each with 6 or 7 tests. I have no NAS for it to be reading (at this time), everything is local. Seems like 1400 recordings would take 5-8 seconds (assuming the time function here is linear, which seems like a rather safe bet). So does SageTV lock down wiz.bin while SJQ repeatedly queries it for those 5-8 seconds? Is that long enough for the Sage GUI to get upset? The plot thickens... hopefully the debug output you provide later this weekend answers some of these questions.

[perfessor101]

Here's some logs ...

SageTV without Jetty ...
SageTV with Jetty
and SageTV with Jetty and SJQ

I didn't see much with the older logs ...

the SageTV with Jetty and SJQ is the largest and only covers a few hours ...

Hope they help ...

thank you,
Bobby

ps -- actually a rar file ... bad form I know to rename ... but the zip file was 1.9MB so unless I split it three ways ...

[Slugger]

Originally Posted by perfessor101

Here's some logs ...

SageTV without Jetty ...
SageTV with Jetty
and SageTV with Jetty and SJQ

I didn't see much with the older logs ...

the SageTV with Jetty and SJQ is the largest and only covers a few hours ...

Did SJQ hang up while you were logging it? Does the logged session represent an SJQ hang?

[Slugger]

Very interesting... I can see that the log provided is a session where SJQ hung... and the debug output is rather interesting:

$ grep "SJQ FileCleaner" sagetvwithJettyandSJQ.txt
Mon 3/16 6:31:29.730 SJQ FileCleaner: Started...
Mon 3/16 8:44:32.943 SJQ FileCleaner: Started...
Mon 3/16 10:35:49.645 SJQ FileCleaner: Started...

The FileCleaner thread starts at 6:31, but never finishes and then another one is kicked off at 8:44, then another one at 10:35, but none of them ever complete. Can I assume you have setup the file cleaner to clean files for you? What are your settings? Are any (or all) of your registered recording dirs in SageTV on a network drive (i.e. the API call GetVideoDirectories() returns one or more directories on a network drive)?

Then I look at the trace points for the QueueLoader

$ grep "SJQ QueueLoader" sagetvwithJettyandSJQ.txt
Mon 3/16 6:31:29.205 SJQ QueueLoader: Cycle started...
Mon 3/16 6:31:30.953 SJQ QueueLoader: MediaMask retrieved [T]...
Mon 3/16 6:31:30.955 SJQ QueueLoader: Media list generated [size = 1489]...
Mon 3/16 8:38:32.237 SJQ QueueLoader: Media scan completed...
Mon 3/16 8:38:32.947 SJQ QueueLoader: Gone to sleep!
Mon 3/16 8:44:32.940 SJQ QueueLoader: Cycle started...
Mon 3/16 8:44:33.074 SJQ QueueLoader: MediaMask retrieved [T]...
Mon 3/16 8:44:33.075 SJQ QueueLoader: Media list generated [size = 1488]...
Mon 3/16 10:35:49.376 SJQ QueueLoader: Media scan completed...
Mon 3/16 10:35:49.641 SJQ QueueLoader: Gone to sleep!
Mon 3/16 10:35:49.642 SJQ QueueLoader: Awakened...
Mon 3/16 10:35:49.642 SJQ QueueLoader: Cycle started...
Mon 3/16 10:35:49.768 SJQ QueueLoader: MediaMask retrieved [T]...
Mon 3/16 10:35:49.768 SJQ QueueLoader: Media list generated [size = 1487]...

A scan of your 1489 recordings is taking approximately two hours to complete. This is obviously the problem. Between this and the FileCleaner thread not ever returning, I'm guessing there is a problem with your network drives (or at least a problem reading them from the server running SJQ), assuming you have network drives registered as recording dirs.

Assuming you have network drives registered, the first thing to try is to disable use of the FileCleaner in SJQ. Presumably, this is the thread that is getting caught in a loop or getting stuck somewhere, and disabling it may relieve the rest of the problems.

I do not have network drives to test performance, hopefully someone else running SJQ with network recording dirs can chime in with some results of their own. The code in the FileCleaner thread is so small and basic that I don't see what I could even change to try to "fix" it. It's standard java.io.File stuff. What version of Java are you running? If not the latest, perhaps an upgrade is in order?

[Peter_h]

Hey Slugger,

I did not have a chance to install the latest snapshot or do any troubleshooting this weekend (marathon of the wifes shows) but I did notice additional details when the hang occurred.

We sat down to watch some recorded TV and i noticed that no programs in the last 12 hours had commercial skipping processed. I went to the sjq main page and sure enough, i got the login prompt but after entering password, it just sat there and eventually hung the browser.

I did a quick restart and then SJQ was working again and all of the programs that did not get processed were now in queue.

So, it's not just the login prompt that is hanging but it's not even queueing up jobs.

There are no scheduled recordings after 5pm today so I'll take it down and do some troubleshooting.

-Peter

EDIT: I noticed that there are 4 snapshot builds now, which one should I be installing to troubleshoot?

[Peter_h]

I'm back again. I'm trying to get the file cleaner working and i'm not sure on the syntax. I checked the guide (great job BTW) and there isn't much in the file cleaner section.

What I want it to do is to delete .edl files for recordings that have been deleted. I have both .ts and mpg. recordings so is this correct?

Delete files with an extension matching this regex: .edl

When no corresponding file has an extension matching this regex: .ts

Never delete files matching this regex: .mpg

-Peter

[tomfisk]

I'm having the same issue, on build 361. Type in the password and it just sits there. If I navigate away from the page "Invalid password!" is displayed. This has happened twice now. I have to restart the Sage Service in order to get things working again.

[Slugger]

Originally Posted by tomfisk

I'm having the same issue, on build 361. Type in the password and it just sits there. If I navigate away from the page "Invalid password!" is displayed. This has happened twice now. I have to restart the Sage Service in order to get things working again.

Enable debug logging in SageTV, reproduce then get me the output (see a few posts up for what I'm looking for). Are any of your recording directories on a network drive? How large is the media set SJQ is scanning?

[Slugger]

Originally Posted by Peter_h

EDIT: I noticed that there are 4 snapshot builds now, which one should I be installing to troubleshoot?

Latest one available should be fine.

[Slugger]

Originally Posted by Peter_h

I'm back again. I'm trying to get the file cleaner working and i'm not sure on the syntax. I checked the guide (great job BTW) and there isn't much in the file cleaner section.

What I want it to do is to delete .edl files for recordings that have been deleted. I have both .ts and mpg. recordings so is this correct?

Delete files with an extension matching this regex: .edl

When no corresponding file has an extension matching this regex: .ts

Never delete files matching this regex: .mpg

-Peter

Yeah, that section of the doc will be filled shortly. Everything is right except remove the periods (it checks against everything after the period). Also, you don't need to tell it to not delete mpg files, it will never delete a file registered with SageTV, even if you tell it to (so you can leave that last box blank).

EDIT: If you want to delete the edl files for both mpg and ts recordings then the values should be:

Delete files with an extension matching this regex: edl
When no corresponding file has an extension matching this regex: mpg|ts
Never delete files matching this regex: .*\.logo\.txt$

Assuming you use comskip, then you might want to use the value I've provided for the third box, it will prevent the FileCleaner from delete the *.logo.txt files that comskip generates while it's processing a recording. You can flip a switch in your comskip.ini to have comskip auto delete those logo files when it completes.

[Peter_h]

Originally Posted by Slugger

Yeah, that section of the doc will be filled shortly. Everything is right except remove the periods (it checks against everything after the period). Also, you don't need to tell it to not delete mpg files, it will never delete a file registered with SageTV, even if you tell it to (so you can leave that last box blank).

EDIT: If you want to delete the edl files for both mpg and ts recordings then the values should be:

Delete files with an extension matching this regex: edl
When no corresponding file has an extension matching this regex: mpg|ts
Never delete files matching this regex: .*\.logo\.txt$

Assuming you use comskip, then you might want to use the value I've provided for the third box, it will prevent the FileCleaner from delete the *.logo.txt files that comskip generates while it's processing a recording. You can flip a switch in your comskip.ini to have comskip auto delete those logo files when it completes.

If a program is being processed, the logo and txt files shouldn't be in danger because the mpeg/ts file is still there right?

I have comskip set to "cleanup" after it is finished. However, it still outputs 3 files, .edl, txt. and .log.

Can I have the job clear all 3 or do I need separate instances of the cleanup?

For example:

Delete files with an extension matching this regex: edl|txt|log
When no corresponding file has an extension matching this regex: mpg|ts
Never delete files matching this regex:

Is this syntax correct?

Thanks,
Peter