Comskip for Sage Linux..... :)

[ChePazzo]

I just keep the Sage server in the basement, turn on smb, and run comskip over the network from a windows machine..... Works fairly well. I run showanalyzer that way as well.

[lewispm]

I have a WinXP Server that only runs comskip overnight to keep the load down while we're watching tv. I have two linux laptops that are on most of the day with some cpu to spare, so I looked into using these computers to run comskip while they're on.

I modified the script I found here to run comskip if a new .mpg file is created in the recording directory. It uses inotify in the linux kernel, so there's no polling of the directory involved. It does nothing until a new file appears, then it runs comskip.

#!/usr/bin/python

import os
import pyinotify
import time

wm = pyinotify.WatchManager()
mask = pyinotify.IN_DELETE | pyinotify.IN_CREATE

class PTmp(pyinotify.ProcessEvent):
  def process_IN_CREATE(self, event):
    #test for .mpg extension
    if os.path.join(event.name)[-4:] == '.mpg':
      # wait 3 mins before processing new file
      time.sleep(180)
      # run comskip on the new file
      os.system("/usr/bin/wine 'C:\Program Files\comskip\comskip.exe' 'C:\\tv\%s'" % os.path.join(event.name))
 

notifier = pyinotify.Notifier(wm, PTmp())

wdd = wm.add_watch('/media/tv', mask, rec=True)

while True:
  try:
    notifier.process_events()
    if notifier.check_events():
      notifier.read_events()
  except KeyboardInterrupt:
    notifier.stop()
    break

change the line:

os.system("/usr/bin/wine 'C:\Program Files\comskip\comskip.exe' 'C:\\tv\%s'" % os.path.join(event.name))

to match your setup. ie - location of wine, location wine will look for comskip.exe, and your recording directory. I have a symlink named tv in my wine drive_c to the recording directory.

[lewispm]

Originally Posted by lewispm

wdd = wm.add_watch('/media/tv', mask, rec=True)

Also change this line to match your recording directory.

and here's how to install pyinotify.

[loonsailor]

Thanks, lewispm!

Like several other scripts that I've seen, this will run comskip on all new recordings, even those on channels with no commercials (HBO, PBS, ...). Is there any way to know, other than from inside sage, what channel a recording came from? I guess there's no problem just running comskip anyway, other than a few wasted watts, but it would be nice to be able to be more selective. Has anybody had issues with comskip finding "false positive" commercials on shows where no commercials exist?

[Slugger]

Originally Posted by loonsailor

Thanks, lewispm!

Like several other scripts that I've seen, this will run comskip on all new recordings, even those on channels with no commercials (HBO, PBS, ...). Is there any way to know, other than from inside sage, what channel a recording came from? I guess there's no problem just running comskip anyway, other than a few wasted watts, but it would be nice to be able to be more selective. Has anybody had issues with comskip finding "false positive" commercials on shows where no commercials exist?

Since Sage doesn't put the channel name/number in the file name, there's no way to know which channel the recording came from other than to ask SageTV.

But if you want to be more selective about which recordings get comskip run on them, then SJQ can definitely help with that. The task client will also run on Linux so you can use a Linux client to run comskip tasks.

[bcjenkins]

you can also query the filename against a webservice as well.

B

[lewispm]

Originally Posted by loonsailor

Thanks, lewispm!

Like several other scripts that I've seen, this will run comskip on all new recordings, even those on channels with no commercials (HBO, PBS, ...). Is there any way to know, other than from inside sage, what channel a recording came from? I guess there's no problem just running comskip anyway, other than a few wasted watts, but it would be nice to be able to be more selective. Has anybody had issues with comskip finding "false positive" commercials on shows where no commercials exist?

I have found a few false positives, which led me to disable the auto commercial skipping feature. Having said that, I'm very impressed at the accuracy of comskip. It works most of the time for me.

Aside from the suggested SJQ, the only way I can think of selectively running comskip would be recording in different directories (I don't think this is possible), or creating symlinks to a separate, monitored directory, from only those files you want to run comskip on. If you are parsing the name of the file to make the decision, that could be accomplished in this script, but it would probably be more work than its worth.

[drewg]

Originally Posted by bcjenkins

you can also query the filename against a webservice as well.

B

Thats what I do. FWIW, here is my script which runs from a cron job at 5 minutes past the hour, and the 1/2 hour. I'm pretty sure it was based on another script I found here. Maybe yours?

Drew

#!/usr/bin/perl

sub commercial_free
{
  $file=$_[0];
  $url="http://127.0.0.1:8080/sage/DetailedInfo?FileName=" . $file;
  use LWP::UserAgent
  $ua = LWP::UserAgent->new;
  $req = HTTP::Request->new(GET => $url);
  $req->authorization_basic('sage', 'frey');
  $r = $ua->request($req)->as_string;

# Look for a string which identifies a show as commercial free.  For
# me, I just look for shows recorded on WUNC, my local PBS affiliate

  $i = index($r, "WUNC");
  if ($i > 0) {
#    print "$file is a PBS show\n";
    return 1;
  }
#  print "$file is not a PBS show\n";
  return 0;
}

### CHANGE THESE VALUES FOR YOUR SYSTEM
# You have to use 4 \'s per one /, since perl changes \\\\ to \\
# and shells change \\ to \
#$inifile = "Z:\\\\usr\\\\local\\\\etc\\\\comskip.ini";
@directories = qw (/var/media/tv);
#@directories = qw (/tmp/tv);

foreach(@directories) {
  $dir=$_;
  opendir(DIR, "$_");
  @files = readdir(DIR);
  closedir(DIR);
  foreach $file (@files) {
    if ($file =~ /mpg/) {
      $comskip_file = $file;
      $comskip_file =~ s/\.mpg/\.txt/;
      $exists = 0;
      foreach $name (@files) {
        if ($name eq $comskip_file) {
          $exists = 1;
          break;
        }
      }
      if (! $exists) {
        $pathname_mpg = $dir . "/" . $file;
        if (!commercial_free($pathname_mpg)) {
          print "Will comskib $file ($comskip_file missing)\n";
          system "(cd $dir ; wine comskip $file )\n";
        } else {
          print "Will create .txt file $comskip_file\n";
          system ("cd $dir; touch $comskip_file\n");
        }
      }
    } 
    else {
      if ($file =~ /\.edl$|\.txt$/) {
        # look for comskip .txt or .edl files left over after .mpg file has
        # been deleted.  These seem to mainly come from live tv.
        $mpg_file = $file;
        $mpg_file =~ s/\.[et][xd][tl]/\.mpg/;
        $mp4_file = $file;
        $mp4_file =~ s/\.[et][xd][tl]/\.mp4/;
        $avi_file = $file;
        $avi_file =~ s/\.[et][xd][tl]/\.avi/;
        $exists = 0;
        foreach $name (@files) {
          if ($name eq $mpg_file ||
              $name eq $mp4_file ||
              $name eq $avi_file) {
            $exists = 1;
            break;
          }
        }
        if (! $exists) {
          $pathname = $dir . "/" . $file;
          print "mp4 = $mp4_file";
          print "mpg = $mpg_file";
          print "rm $pathname\n";
          unlink $pathname;
        }
      }
    }
  }
}

[bcjenkins]

The new script (link above) now has the ability to load the CPU based on file type. TS files require more CPU that MPEG-2 files for instance.

B

I'll have to add in the web lookup feature though.

[bcjenkins]

I installed jetty and sagex.api from jreichen and stuckless. I chose curl to parse because it is standard on OS X (my script can run on that platform too)

Using that I can determine the channel id fairly easily.

curl --user user:pass --silent http://localhost:8080/sagex/api?c=GetMediaFileForFilePath\&1=$VIDEO | grep AiringChannelName | cut -d[ -f3 | cut -d] -f1

I'll put out a script update which covers this as well later. The channel ID will be in an array format.

New variables will be:

• server
• port
• server_user
• server_pass
• No_scan_channels

I would be open to further enhancements as well such as being able to specify a comskip ini on a per show/channel basis which would help for tuning specific shows

Thoughts?

B

[Slugger]

I don't mean to hijack and if you tell me to I'll shut up and move on, but is there something about SJQ that is preventing you from using it? If so, I'd really like to know if it's something I could address. It seems to me you're writing scripts to parse out show info to help make decisions on what gets processed. SJQ does it already and all the hard work of having to parse out details to make decisions is done for you. You want to comskip everything, but stuff from PBS and HBO? The ruleset for SJQ can be as simple as this:

if [IsTV == true && IsScheduledRecording == true && ChannelName !% "HBO.*|WPBS" && FileExists != "%d%/%p%.edl"] {
   COMSKIP
}

Basically, if the file is a tv recording, was scheduled (i.e. don't comskip live tv recordings since it's likely a waste of time), was not recorded from any HBO or WPBS (replace with your actual PBS callsign) and there's no edl already then run the COMSKIP task. Plus there are 42 other tests available meaning you can make your decision to run based on just about anything.

Then configure a task client to run comskip for the COMSKIP task:

COMSKIP {
   :CPU "LOW"
   /usr/local/bin/comskip "%c%"
}

Again, if SJQ just isn't what you're looking for (and there's nothing missing/can be fixed/etc.) then just say the word and I'll quietly vanish. However, if you're not using it b/c there's something missing, etc. then I'd love to hear your ideas.

[bcjenkins]

Well honestly I never looked at it. Off the top after skimming the docs (still learning to read you see)

• Does SJQ support multiple executions of comskip?
• Support throttling of executions based on file type?
• Clean up when a media file has been deleted?
• Run without SageTV?

  (Until my proposed change above, there isn't an explicit need for SageTV nor will the script cease to function when I add it in if unavailable)
  

Given the last is only mildly useful to me, but I do post this on my blog as well.

B

[Slugger]

Originally Posted by bcjenkins

Well honestly I never looked at it. Off the top after skimming the docs (still learning to read you see)

• Does SJQ support multiple executions of comskip?
• Support throttling of executions based on file type?
• Clean up when a media file has been deleted?
• Run without SageTV?

  (Until my proposed change above, there isn't an explicit need for SageTV nor will the script cease to function when I add it in if unavailable)
  

Given the last is only mildly useful to me, but I do post this on my blog as well.

B

[*]Does SJQ support multiple executions of comskip?

Yes. SJQ supports multiple, distributed clients and clients can run multiple tasks simultaneously.

[*]Support throttling of executions based on file type?

Yes, sort of. You can assign CPU priority to defined tasks. If you defined an MPG comskip task and a TS comskip task then you could set the CPU priority to be lower for the TS task (if that's what you're getting at?).

[*]Clean up when a media file has been deleted?

Yes, there is a built in file cleaner thread. You can tell it to do things like "delete *.edl files when there is no corresponding *.ts or *.mpg file to go with it."

[*]Run without SageTV?

The SJQ server must run within SageTV (and it actually runs within the Jetty plugin), but the task client does not run within Sage/Jetty, it's a completely standalone self contained package. Run it on the Sage host or any other host - it doesn't need Sage at all - the task client only talks to the SJQ server looking for work to do. Once it gets work it does it and reports back to the server.

[bcjenkins]

Can SJQ determine if preexisting tasks were running and delay execution until tasks complete? I have 7 tuners; if all 7 start recording at once or within 30 minutes, how is that handled for an external application like comskip? Would it fire off 7 comskips regardless?

You say you can affect prioritization, you're executing threads at a lower priority via what mechanism?

My script or most other scripts on this page are independent of SageTV. Granted there is little value on these forums regarding that aspect, but there is still value outside.

B

[Slugger]

Originally Posted by bcjenkins

Can SJQ determine if preexisting tasks were running and delay execution until tasks complete? I have 7 tuners; if all 7 start recording at once or within 30 minutes, how is that handled for an external application like comskip? Would it fire off 7 comskips regardless?

Yes, the server keeps track of what's running and honours the various client settings about max number of tasks a client can execute simultaneously. Clients can be configured to run at most x number of tasks simultaneously and you have finer control on top of that by being able to say things like, "this client can run no more than 4 total tasks simultaneously and no more than 2 h.264 comskip tasks simultaneously."

You say you can affect prioritization, you're executing threads at a lower priority via what mechanism?

You can set the priority of the thread created to run the command and that setting is passed to java.lang.Thread.setPriority()

My script or most other scripts on this page are independent of SageTV. Granted there is little value on these forums regarding that aspect, but there is still value outside.

B

[bcjenkins]

Download here
Support here

Added the ability to filter based on channel id.

B

[loonsailor]

I tried running comskip with SJQ and it works, though I have a couple of questions / issues.

First, in order to help the next person, here's what I did. In SJQ, I entered the following rule:

if [IsTV == true && IsScheduledRecording == true && ChannelName !% "KQED.*|KTEH.*|KRCB.*|WORLD|CSPAN.*|CALCHAN.*|EDAC.*|KVIE.*|STARZ.*|STZ.*|HBO.*|.*MAX.*|SHO.*"] {
COMSKIP
}

I also set the file cleaner to delete files with extensions "edl|log|txt|logo.txt", if there's no .mpg or .ts present.

Then I started SJQC:

cd /opt/sagetv/sjqc
sudo WINEPREFIX=~root/.wine ./sjqc.sh &

and entered the following into it:

:MAXPROCS 2

COMSKIP {
:CPU "LOW"
:MAX 2
"/usr/bin/wine comskip %c%"
}

When I run the queue, the right stuff gets put on to the active queue, and it runs, and it works. As I specified, it runs two comskips at a time (using 2 cores of my quad-core CPU). The old .edl and other files all get nicely deleted. The ability to queue all this work is really great!

The .mpg's go really fast - about 90 seconds for a one hour show. The .ts files take much longer - 20-90 minutes for an hour show. CPU is an AMD Phenom II 905e X4. Because of the time discrepancy, I don't care about the loading that bcjenkins' script uses. The .mpg's are in and out so fast it doesn't matter.

Several issues / questions

• The startup script sjqc.sh wouldn't work, because it had CR-LF, the DOS convention, instead of just LF as linux expects. I had to filter the CR's out using SED.
  
• The startup script isn't set up to start the java process as a daemon. I ran it from my shell and it works, but it's not in the right form to start sjqc at system boot time. There needs to be a script in /etc/init.d that can start and stop sjqc properly, like the equivalent sagetv scripts. It's not a big deal to write it, but it needs to be done. Does it exist yet? It's not in the distribution.
  
• The comskip processes, running under wine, work fine, but they almost always wind up in the "failed tasks" queue, instead of in "completed tasks". Is this a problem with the value being returned by wine, or did I configure something wrong? How does sjq decide about failure / success? Not a huge deal, but a bit annoying.
  
• Just curious - what happens if I stop / restart sage without restarting sjqc? Will it reconnect properly? If not, there should probably be some way of restarting sjqc automatically when sage restarts.

[Slugger]

I'll briefly address this post in this thread, but after this all future questions/posts about SJQ should not be hijacking this thread, but rather should be posted in the SJQ thread. Maybe Opus4 will be kind enough to merge the last post and this reply into that thread?

Originally Posted by loonsailor

Several issues / questions[*]The startup script sjqc.sh wouldn't work, because it had CR-LF, the DOS convention, instead of just LF as linux expects. I had to filter the CR's out using SED.

Can you open an issue ticket for this, please?

[*]The startup script isn't set up to start the java process as a daemon. I ran it from my shell and it works, but it's not in the right form to start sjqc at system boot time. There needs to be a script in /etc/init.d that can start and stop sjqc properly, like the equivalent sagetv scripts. It's not a big deal to write it, but it needs to be done. Does it exist yet? It's not in the distribution.

I typically don't write init.d scripts b/c each Linux distro is a little different. An init.d script that works on Red Hat rarely works on SuSE, etc. Usually each distro has a skeleton in /etc/init.d and you can whip up one that will work on your distro in like 5-10 mins. And when I moved this summer my Linux box was decommissioned so I don't have access to a Linux box at home to play with this kind of stuff (though I'm thinking about setting up a NAS this fall).

[*]The comskip processes, running under wine, work fine, but they almost always wind up in the "failed tasks" queue, instead of in "completed tasks". Is this a problem with the value being returned by wine, or did I configure something wrong? How does sjq decide about failure / success? Not a huge deal, but a bit annoying.

This is talked about in the SJQ thread. The solution is here. Basically, comskip returns 1 on success, but SJQ expects the standard return value of zero. You can override the expected return code as discussed in the post linked to above.

[*]Just curious - what happens if I stop / restart sage without restarting sjqc? Will it reconnect properly? If not, there should probably be some way of restarting sjqc automatically when sage restarts.[/LIST]

On Windows, the service recovery option is available. On Linux, I'd write the init.d script and start/stop it as necessary via that mechanism or if your distro supports an inittab type setup then I'd look into that as inittab supports respawning. With that said, the task client should reconnect just fine, but it will give up and die if the SJQ server is unreachable for too long.

[loonsailor]

Thanks, slugger!

[dshields]

Originally Posted by laurenglenn

First, get a copy of comskip.exe (I used version 0.64) and put the files in /var/media/tv

emerge wine
cd /var/media/tv

----------------------------------------

Please tell me Wine is not the only way to get commercial skip on Linux. There is no way I am installing wine on my sagetv server. (Nothing against Wine, and I'm running it on another computer...)