Commercial Advance Beta Release

[Jere_Jones]

Comskip version 0.42 is now available.

Bugfixes
- setting version=0 will not create a log file anymore
- increased number of possible black frames to avoid overflow problems
- score threshold can be set by percentile
- black frame threshold can be set by percentile
- if a logo block was open at the end of the show, then that block wasn't marked. Fixed
- improved gracefulness if there are problems with logo detection
- improved gracefulness if there are problems with the ini file
- exefilename and inifilename should *NEVER* be blank anymore
- modified default settings to look for logos longer
- added more options for the ini file (see here)

Enjoy,
Jere

<EDIT>
As requested, I have also attached the modified source files.
</EDIT>

[laurenglenn]

Originally Posted by Jere_Jones

I'm going to post a new version tonight and then the Navy is sending me to a school for the next two weeks.

What this means to you is that if you are seeing a bug, I need to know about it before I post my next version or it will have to wait a couple of weeks. I expect to get a lot of coding time, but no internet time.

This also means that you can expect a major update (with command line arguments) in two weeks.

Jere

Here's an excerpt from the LOG file that was created when I tried to do a manual COMSKIP detection from Sage:

logoMinX=541 logoMaxX=617 logoMinY=55logoMaxY=408
Using commandline -

################################################################
Attempting to run second pass using E:\VIDEO\AndromedaDoubleHelix-9066869-0.logo.txt
Time at end of first run:
Thu Oct 07 22:46:02 2004
################################################################
Closed 3 files
Successfully deleted batchfile - comskip.bat


Once in a while when doing manual detection, it would not run a Batch file. Either that, or it would start and then stop in 37 seconds. In my INI, I turned off the 2-pass Logo detect and this seems to happen. It appears to be a fluke thing as it doesn't happen all the time.

Lauren

[laurenglenn]

If Sage is doing Logo detection, starts the first logo check, and then a second instance of COMSKIP is started, COMSKIP appears to stop the manual instance of COMSKIP when Logo check has completed.

I don't know if that's the reason, but that's what seems to be happening. My STV is running COMSKIP during background while watching LiveTV, so whenever I want to detect commercials in two different files at once, COMSKIP appears to be stopping without finishing.

Lauren

[Jere_Jones]

Originally Posted by laurenglenn

Using commandline -


Closed 3 files
Successfully deleted batchfile - comskip.bat
Lauren

Oops, my fault. That was fixed for 0.42. Comskip was zeroing some strings before it had determined if it needed to.

Oh, and I have good news! The base where I'm going to school didn't have any room in the barracks, so I'm staying in the local Hampton Inn and they have wireless internet.

Basically, the updates will keep coming... and coming... and coming...
Enjoy,
Jere

[heffe2001]

Originally Posted by Jere_Jones

That won't be possible because comskip only has access to the information in the mpeg file itself and the filename. Hence, you could (in the future) set it to use a different ini for Nova than for SG-1. Of course that is after I give comskip the ability to accept command line arguments.

Jere

Could it be done in such a way that you can set a list of stuff to comskip via the ini file? We only really watch a few shows that we want comskipped (currently anyway), basically I watch some Star Trek reruns from Spike (which comskip nails the commercials every time now), all the CSI's, and Forensic files mostly. Basically have a front end that you can run on a non-sage PC, that at X:XX time scans the storage area for any shows with xxxxxxxxxx name, and comskips them? I know I can do this manually with a batch script file, but would be nice to have a windows program with a gui (maybe that shows some stats or something .

[Humanzee]

I haven't checked back in on this thread for a while so when I saw a new release decided to try out 0.42 this afternoon. And well, didn't get very good results. Before I overwrote the comskip files I backed up all the .txt files to another folder just incase I wanted to go back to 0.32 which I have been using to date. After restarting Sage I set it to sleep while I went and ate my lunch. When I came back it had finished scanning a few half hour shows. I then wanted to watch "Myth Busters" so I found it in my nielm1.5 Malore recordings menu and told it to run comskip while I watched another show. After some time it said that comskip was done, so I switched over to "Myth Busters".

It found some commercials but didn't mark the beginning and ending of them very accurately at all. In fact it missed some all together, and others it would skip the first commercial but not the next 2 or 3 in the same segment. I then went and watched some of the half hour shows that had scanned during lunch and found the same issues.

After reading the info file from Jere I wonder if 70% score threshold minimum marked as show is a good idea or not. If there are more than 30% commercials does it just not mark some in favor of maintaining the 70% show ratio? For instance some shows on the cartoon network are like 7 minutes long, and marked in the guide as 15 minutes. Thus nearly half the recording is commercial. I dunno, maybe its just my machine. I'm going to lower the threshold to 60% and let it run while I'm away for the weekend and see if new recordings have the same issues. Else I think there is still some tweaking that needs to be done to get all the detection methods to work together. Has anyone had similar issues? Has anyone resolved them via an ini file tweak? If so could we post some ini files to try out? I love Comskip I just don't have a lot of time to test different settings. Especially when it takes a good 15-20 mins to scan one hour of media.

Just my 2 cents.

[Jere_Jones]

Originally Posted by Humanzee

I then wanted to watch "Myth Busters" so I found it in my nielm1.5 Malore recordings menu and told it to run comskip while I watched another show. After some time it said that comskip was done, so I switched over to "Myth Busters".

MythBusters is one of my test shows because it is VERY difficult to identify correctly.

Originally Posted by Humanzee

After reading the info file from Jere I wonder if 70% score threshold minimum marked as show is a good idea or not.

I use a variety of shows for testing, but (of course) they are also shows that I personally like. It has been my experience that one hour shows are fairly consistant at having 75% as show and 25% as commercial (give or take a little). Half hour shows tend to be a little less show (72/28). But this is my experience. That is why I put it in the ini file.

Originally Posted by Humanzee

If there are more than 30% commercials does it just not mark some in favor of maintaining the 70% show ratio?

Pretty much. Actually it tries to find the 30% that is least likely to be a show and marks it as commercial. I'm going to update the Comskip logic page soon to explain exactly how comskip works.

Originally Posted by Humanzee

Else I think there is still some tweaking that needs to be done to get all the detection methods to work together.

I am open to any ideas that anyone has!

That being said about the score threshold percentile, you may want to look at the black_percentile. This determines how much of the show is labeled as black frame. This setting will (oddly enough) have an immense effect on the quality of the detection. 3/4 of 1 percent seemed to work the best for me.

And use the verbose setting. That will give you some information about why portions of the recording were marked or not marked. Thus tweaking the ini will be much more educated and not just guessing.

I've also opened my gmail account to anyone who wants to send me their log files and I'll make suggestions on how to modify the ini or it will give me ideas of how to improve the coding itself.

Jere

[krutaw]

Originally Posted by Jere_Jones

Cool. Always looking for help.

Take a look at this thread. That is where I orginally posted it and I recently posted an update that tries to handle the sync problem of the captions.

Enjoy...
Jere

Sweet! Finally something I can do to help in a language I already know.

[malore]

Jere_Jones,

I modified my version of comskip to allow the creation of VideoRedo projects and ZoomPlayer cutlists. I was going to release it after I finished making improvements to the commercial identification using black frames, but my changes proved to be wasted effort and you've since made a bunch of improvements of your own. I'll send you the code.

I think a big improvement could be made to comskip if you were able to use the v-chip rating square to identify the beginning of the show.

[Jere_Jones]

Originally Posted by malore

I modified my version of comskip to allow the creation of VideoRedo projects and ZoomPlayer cutlists. I was going to release it after I finished making improvements to the commercial identification using black frames, but my changes proved to be wasted effort and you've since made a bunch of improvements of your own. I'll send you the code.

Outstanding!!! For those following development, this will be made available via command line arguments. I've often thought about zoomplayer cutlist, but didn't dive any further into it.

Originally Posted by malore

I think a big improvement could be made to comskip if you were able to use the v-chip rating square to identify the beginning of the show.

I'll look into this. I might be able to use a template logo to detect it. Or are you talking about something that isn't visible?
Then I could back up to the black frame before it and mark as show. Good idea!

Jere

[malore]

I figured your logo detection software might be modified to look for the black box with white text in the upper left corner.

The zoomplayer player cutlist is nice because it's near frame accurate so it jumps at the right place, but if the cutlist is wrong you have to disable it inorder to view the skipped content, because it jumps every time.

[krutaw]

I realize this isn't directly about the commercial advance functionality itself, however, it is about the STV that Malore made in general so please don't yell at me for posting semi-offtopic.

Does Malore's Commerical Advance custom STV support the viewing of closed caption information, and if so, how do you configure it to be viewed?

I've already configured the mpg2srt to scan each of the recordings to get the closed caption text, I just can't seem to figure out how to make it actually appear while I'm watching the shows.

[vigor]

Once you have the closed caption data extracted you should be able to right click while watching a movie and enable them. The option is only available when the text file with CC data is present.

[krutaw]

Originally Posted by vigor

Once you have the closed caption data extracted you should be able to right click while watching a movie and enable them. The option is only available when the text file with CC data is present.

AHA! That makes alot more sense.

[krutaw]

Jere_Jones~

Do you happen to know what all (if any) errorlevels are returned from the comskip program? I ask because I didn't like the idea of parsing through the text output from the program to find out if it worked, and checking for the existance of a non-zero byte file is pretty lame.

Anywho, I've got the basics of the script in place and working. Here are the features that I'm finishing up for v0.1 of the script:


- Automatic determination of the SageTV directory
- Automatic determination of the video directories
- Checking of only files that don't have a pre-existing .txt file
- Checking if the file is already in use
- Reporting on all features
- Logging to either flat text files or event viewer

Once I've got all of those features working, I'll hit you with a copy for testing if you don't mind.

I'm hoping that v0.2 can have determination on the success of the comskip outcome depending on the errorlevel returned (or if you have any ideas on what to check for.)

[Jere_Jones]

Originally Posted by krutaw

Do you happen to know what all (if any) errorlevels are returned from the comskip program?

What kind of errors would you like? Right now it errors out with a 1 if something drastic happens. Otherwise 0. Did you want an error level indicating if commercials were detected or what? I'm sure I can accomodate whatever.

Originally Posted by krutaw

-Reporting on all features

Huh? Can you elaborate on that. I don't understand.

Originally Posted by krutaw

Once I've got all of those features working, I'll hit you with a copy for testing if you don't mind.

Heck no I don't mind.

Originally Posted by krutaw

I'm hoping that v0.2 can have determination on the success of the comskip outcome depending on the errorlevel returned (or if you have any ideas on what to check for.)

Can you elaborate on this, too. I don't quite follow.

Jere

[Jere_Jones]

I hate to do this to everyone, but version .43 will be out shortly. In version .42, I introduced a bug that would cause comskip to only mark one block as a commercial instead of all the blocks it was supposed to. This basically means that .42's detection is horrible. I was trying to change the marking from the first black frame in a group to the middle black frame and didn't notice that it threw the entire algorithm out of whack.

Anyway, I'm testing it now and should post it shortly. My apologies...

Jere

[Deadbolt]

...Jere_Jones...just about to post this! Great catch...my last two days of comskip have been as you say "horrible." I had changed a few things in the ini...but then when I changed them back it still didn't fix it! I'm so glad you found the bug...I was bumming...because prior to the 8th your comskip was killer! I can't wait for .43! I'll just delete all the .42 .txt files and let it all re-detect. I'm averaging about 10 min 30 seconds per detection…and have two machines detecting (server, plus a client). Even with 4 tuners blazing away I can catch up pretty quick.

Again, thanks for such effort! You and Frey have changed my TV experience forever! If I'm away in a hotel, and have to suffer commercials...I just break out a book. I'm forever spoiled!

[Jere_Jones]

Is there any interest in a version of comskip that will run on Linux?

The original author of the commercial detection algorithms contacted me and suggested he could port it to Linux. All that was needed is a makefile and comskip would compile under GCC.

For those of you that use Linux/Samba to keep the txt/smi files cleared, you could also use it to process your shows for commercials.

Personally, I don't use Linux, so it wouldn't benefit me, but hey, I'm not everyone.

Jere

[krutaw]

Originally Posted by Jere_Jones

What kind of errors would you like? Right now it errors out with a 1 if something drastic happens. Otherwise 0. Did you want an error level indicating if commercials were detected or what? I'm sure I can accomodate whatever.
Huh? Can you elaborate on that. I don't understand.
Heck no I don't mind.
Can you elaborate on this, too. I don't quite follow.

Jere

Actually, I was thinking that you already had something built in. If not, then I see no reason why we can't just check to see that a non-zero byte file is created, and assume everything went okay since there's really no other way to check the content.

With that being the case, disregard the parts of my post you had questions about. I will get you a copy of the script as soon as I've got it adequately commented.

Oh, and don't be offended, but I started completely from scratch when I built it, simply because it's often faster to do that than to follow someone else's logic. Not to mention that I wanted to add features to the core functionality such as finding the sage directory automagically etc.