ComSkip XVid batch HQ 1200 bitrate

[davephan]

The two-pass process worked with Windows XP Pro!

Thank you Pkadare for catching the missing Msharpen.dll. Sure enough, it was missing on my computer. I downloaded the DLL, and the process ran successfully.

Also, thank you Roxy99 for helping with the troubleshooting and creating this great commerical eliminating and compressing script!

I tried running the same Suzi Orman video. The input mpeg2 file was 3280 megs, the converted AVI file was 442 megs. The computer has an Althon XP+2800 CPU and 1 gig ram (my eariler posting said 2100, but I checked again and the CPU is a bit better than I thought). The computer is running Windows XP Pro. The processing time for the 60 minute program was 146 minutes.

The quality looks good on the 27 inch television. I did not have a chance to try it on the 60 inch standard projection TV. I would like to test some video that has motion to evaluate the compressed video quality.

I was looking for a compression utility that I can offload to another computer. However, it might take a fast CPU to handle the CPU intensive task!

I think that Show Analyzer does a better job at detecting commericals than Comskip. So, I might try using Show Analyzer instead of Comskip. Or, maybe removing the commerical detection and process only programs that are already edited with VideoReDo.

I am having hardware probems with my old Windows 2000 'clunker' computers, so I won't be able to test the process on those computers for awhile. Maybe I'll be able to upgrade my main computer and use this unit for the compression later.

Again, thank you for the help to resolve the problems I was having with your video compression / commerical reduction process.


Dave

[pkadare]

Originally Posted by roxy99

If I haven't yet lost all my credibility, please bear with me and try this ver 1.3. All I did was include the Msharpen plugin. Or you can also simply maually download the Msharpen plugin from Msharpen.zip.

Hey, you haven't lost any credibility at all! In fact, this is an awesome tool and I'd like to thank you very much for taking the time and effort to create it in the first place and to share it with all of us!

[roxy99]

Jansmith,

Eric is the author of comskip and you may wish to pm him about that. I found that comskip is either bang on for an entire show or else consistently leaves a couple of seconds between when the program skips to commercial. However, just 1 or 2 seconds beyond the black screen fade. Check also for the latest version of the comskip files. The project.zip file may include an older version by now as I now Erik is constantly working on it. Don't forget to change the comskip.ini line "vcf_output=0" to "vcf_output=1".

Davephan,

I re-read your first post as to when you were having problems and you did mention the lack of MSharpen.dll. I should have read more closely.

Regarding offloading to a slower Windows 2000 computer: Install everything to the slow computer and place mapped network drives to the Sage server. Dedicate a Source directory (instead of c:\project\source) for mpg processing and a Converted directory (instead of c:\project\converted) on the Sage server. Also, create a Temp directory on the server (instead of c:\project\misc).

Change variables in the batch file for your server directory, CAPDIR, CUTDIR, SDIR. Do not change WKDIR since this will still be c:\project on the local Windows 2000 system.

See also previous posts for setting low prioirity.


Pkdare,

Thank you for the kind words

[jan smit]

Thanks for the advice, Roxy.

I have experimented a bit with comskip and come to the conclusion that on my Dutch TV-channels the way they treat ads di9ffers from program to program. There are programs where the ad changes to the program that I want to record without any black (or even just darker) frame.

My conclusion is that I will have to keep on cutting commercials manually with VideoRedoPlus, which however of course also doesn't mark a scene-change without a dark frame.

jan

[erik]

What channels are causing problems

[roxy99]

Originally Posted by jan smit

Thanks for the advice, Roxy.

I have experimented a bit with comskip and come to the conclusion that on my Dutch TV-channels the way they treat ads di9ffers from program to program. There are programs where the ad changes to the program that I want to record without any black (or even just darker) frame.

My conclusion is that I will have to keep on cutting commercials manually with VideoRedoPlus, which however of course also doesn't mark a scene-change without a dark frame.

jan

Jan,

If you already HAVE Videoredo, then modify the batch file to execute videoredo instead of comskip. I find that using the comskip method is much more elegant since you are not double processing the source mpg with comskip. With videoredo, you are reprocessing the mpg into a cut mpg, a lossy process, and then processing a second time for compression, also a lossy process. Is Videoredo processing the mpg in a lossless manner and is audio synch maintained? Maybe maybe not. Comskip is only making markers into a vcf file and the mpg is only processed at compression time so there is only one lossy process (lossy not lousy)

Comskip can be tailored to the type of show and Erik will tweak its detection parameters accordingly. So theoretically, comskip is as acurate as videoredo.

Regarding Videoredo removing the black fade, that's a preference and not a fault of comskip. I like fades as long as the fade out leads into a fade in and not a 2 second click of a commercial. Now if Dutch broadcasters are cutting to commercials in a retarded fashion (-eg in the middle of a car scene chase) as a opposed to a natural scene change where a fade out is logical, then thats' another matter and comskip won't be able to decide whether to keep the fade or get rid of it.

I would say comskip is 98% accurate for everything I've used. I prefer 98% commercial free to not doing any commercial removal at all. It's silly to do it manually just for 100% accuracy.

Erik,

Does comskip detect blackscreens? Ever commercial break I have seen has a black screen fade. However, sometimes there are fades inside a show so you don't want a false positive.

[erik]

Originally Posted by roxy99

Does comskip detect blackscreens? Ever commercial break I have seen has a black screen fade. However, sometimes there are fades inside a show so you don't want a false positive.

Comskip uses MANY parameters including blackscreens. It can even work without blackscreens.
Heuristics try to automatically compensate for blackscreens during shows. This normally works, even for shows like Lost which are VERY dark.

[jan smit]

What channels are causing problems

NL1,2,3 often have no black frame betwen the "STER" commercial and the start of a program.
RTL4,5,7 and SBS6 sometimes have the movie continue after the black fade.

Is Videoredo processing the mpg in a lossless manner and is audio synch maintained? Maybe maybe not.

As far as I know VideoRedo does not do a lossy recompress (if I am wrong I would like to know it); I never had any sync problems after a VideoRedo cut.

theoretically, comskip is as acurate as videoredo

I did not state that videoredo is more accurate. As far as I have seen, they are both not sufficiently accurate to allow automatic cutting for movies that I want to keep.

Regarding Videoredo removing the black fade, that's a preference and not a fault of comskip.

You must have misunderstood me. I am not at all agains black fades. I just believe that comskip and VideoRedo need them for their detection, so if the black fade is missing they have a problem.

It's silly to do it manually just for 100% accuracy.

You may call that silly, I think it is worth the extra effort for movies I want to keep. I just don't like to get a piece of a commercial in the middle of a movie.

By the way, don't get the impression that I have no respect for all the good work that you guys are doing!

jan

[roxy99]

Originally Posted by jan smit

By the way, don't get the impression that I have no respect for all the good work that you guys are doing!

jan

No offence taken. I think I can speak for Erik as well and say that we all just want to perfect these tools to make the most users happy. For sure a movie for keeps sakes I too would try to get it perfect. I still think comskip has the potential to be perfect, but parameters need to be tweaked. Its all about automation.

Black fades and Logo detection are both detection parameters. I wonder if successive 30 second changes in scenes are detected? I think comskip looks at that as well.
I wonder if Erik can place the parameters in the comskip.ini file in a manner that assigns weighting so that certain conditions more strict detection rules can be used.

[erik]

Originally Posted by jan smit

NL1,2,3 often have no black frame betwen the "STER" commercial and the start of a program.
RTL4,5,7 and SBS6 sometimes have the movie continue after the black fade.

Did you enable aspect ratio based cutting? (add method 32)
This often helps when there are no black frames and is rather safe.
Otherwise you may have to enable silence based cutting (add method 64).
This also helps when there are no black frames but needs more tuning.

[erik]

Originally Posted by roxy99

I wonder if Erik can place the parameters in the comskip.ini file in a manner that assigns weighting so that certain conditions more strict detection rules can be used.

Its already done.
I have one ini file that I use for regression testing on a very large set of reference recordings coming from many places all over the world.
This ini file exploits the automatics adaptation to various conditions.
But you do have to enable more methods then only blackframe based cutting.
And you have to be aware that you could be the unlucky person that has a broadcast channel that is so out of the regular that even the automatic adaptation fail.
The reason I do not distribute this ini file is that is does generate a little more false positives compared to the rather safe behaviour of the current default ini file. People don't like to see some commercial but they hate cutting parts of the show

As an example.
I have two recordings with exactly the save paterns of logo, black frames, silence, etc...
At the border between commercial and show is a 10 second block that for one recording should be show and for the other it should be commercial.
This can never be solved by any automation, it simply requires manual intervention and tuning.
I have been working on certain "high level" tuning parameters to catch these differences in interpretation but there are simply too many different patterns possible so for the moment you have to work with tuning the rather low level interpretation parameters. These have been proven so powerfull that, apart from less then five occasions, I have been able to tune Comskip to live up to the expectations of all the people that have asked for help.
That is why I like to receive questions for help because solving those problems improves the automatic tuning!

[jan smit]

Sorry, but at the moment I am just not in the position to spend more time on this. As I mentioned before, I have decided to cut manually for the time being, which takes me 1-2 minutes hands-on time per movie. I may come back to this subject later.
Thanks for the help and for the offer to do even more.

jan

[roxy99]

You can modify this batch file to do some work on .ts files. Check out HDTV2Mpeg2 at http://www.videohelp.com/tools?tool=HDTVtoMPEG2.

I have no experience with it but anyone with HD ts files should try it.

The other thing to be aware of is audio. Presently, the batch file assumes a stereo audio stream and produces an mpa which then gets converted to a stereo mp3. If you would like to produce a stereo mp3 file then the only other change required is the besweet line in the batch. This is because the digital audio will be ac3 format and besweet is expecting mpa format. So the besweet line need to be modified to produce and AC3 to Wav.


If you want to maintain the digital audio stream then the line for DGIndex needs to be modified to demux audio and not decode to mpa. This way you'll end up with an AC3 file instead of a mpa file.

Delaycut can work on ac3 files so this line does'nt need to be changed.

You need to modify avisynth and remove the line for audio=..
This is because we don't need audio until we remux the avi (video only) with the ac3.

Keep in mind that you'll need the comskip produced vcf file to select non-commercial ranges to remux to the avi.

For this you will need VirtualdubMod instead of Virtualdub and you will have to do a manual remux and not use the remux section of the batch file.

Remuxing the digital mpa file to the avi file requires a manual process of using Virtualdubmod, a different program. Or do a search on line and see what else there is.

[bobtom]

Roxy99, having trouble using the program. Uninstalled everything then re-installed. Went through all the batch files line by line. Can't find anything wrong.So far all the program does is move the mpg from the Source folder to the misc folder. Here is a copy of the batch process:

C:\Project>FOR %f IN (c:\Project\Source\*.mpg) DO call Convert2pass.bat %f c:\Pr
oject\misc\%~nf.mpg

C:\Project>call Convert2pass.bat c:\Project\Source\60Minutes.mpg c:\Project\misc
\60Minutes.mpg

C:\Project>setlocal

C:\Project>move c:\Project\Source\60Minutes.mpg c:\Project\misc\60Minutes.mpg

C:\Project>REM The variables below need to be changed to match your setup

C:\Project>REM Please don't change the presence or absence of quotation marks as
it may interfere with the functionality of the batch file

C:\Project>Set CUTDIR=c:\Project\Converted

C:\Project>set CAPDIR=c:\Project\misc

C:\Project>set VDir=c:\VirtualDub

C:\Project>SET WORKDIR=c:\PROJECT

C:\Project>set SDIR=c:\Project\source

C:\Project>set DG="c:\PROJECT\DGIndex.exe"

C:\Project>set LM="c:\PROJECT\Lame.exe"

C:\Project>Set WV="c:\PROJECT\Wavemp3.exe"

C:\Project>set COMSKIP="c:\PROJECT\comskip.exe"

C:\Project>set VDUB="C:\VirtualDub\vdub.exe"

C:\Project>Set DC="c:\PROJECT\delaycut.exe"

C:\Project>Set BS="c:\Besweet\BeSweet.exe"

C:\Project>SET AVIWAV="c:\PROJECT\AVI2WAVCMD.exe"

C:\Project>REM Set Program Locations

C:\Project>set AVISYNTH1=LoadPlugin("c:\PROJECT\DGDecode.dll")

C:\Project>set AVISYNTH2=LoadPlugin("c:\PROJECT\SimpleResize.dll")

C:\Project>set AVISYNTH3=LoadPlugin("c:\PROJECT\Leakkerneldeint.dll")

C:\Project>set AVISYNTH4=LoadPlugin("c:\PROJECT\MSharpen.dll")

C:\Project>set AVISYNTH5=audio = WAVSource("c:\Project\misc\60Minutes.wav")

C:\Project>set AVISYNTH6=AudioDub(video, audio)

C:\Project>REM Replace these example AVISynth commands here with your commands.
Add more variables if required.

C:\Project>"C:\VirtualDub\vdub.exe" /c
VirtualDub CLI Video Processor Version 1.6.17 (build 24469/release) for 80x86
Copyright (C) Avery Lee 1998-2006. Licensed under GNU General Public License

Any ideas?

[roxy99]

Originally Posted by bobtom

Roxy99, having trouble using the program. Uninstalled everything then re-installed. Went through all the batch files line by line. Can't find anything wrong.So far all the program does is move the mpg from the Source folder to the misc folder. Here is a copy of the batch process:


Any ideas?

First of all, try the 'HQ' version since you are using the standard version.

Did even comskip start to run or did it crash at that stage?

Have you tried disabling the lines that delete the vcf, waw,d2v and other temp files? That is, place 'rem' in front of all Del commands.

Then when the batch crashes, tell me what files are present in your c:\project\misc folder?

[bobtom]

Should it be empty? I thought the project file you uploaded has files in the misc folder.
EDIT: Re-downloaded the project folder. Has lame scripts in the misc folder.

Tried the HQ bat file. Exzact same thing. Nothing runs. Execpt for vdub but it locks up looking for the file.

Re-downloading on another computer will retry. Just waiting on the file to transfer to my pc now.

UPDATE: Roxy, looks like it is working on my other computer. Thanks. Would like to get it to work on my HTPC though. Will re-reinstall vdub.

UPDATE2: Re-installed vdub had to delete the old directory. Works fine now. Thanks for some awesome scripts man.

[roxy99]

Originally Posted by bobtom

Should it be empty? I thought the project file you uploaded has files in the misc folder.
EDIT: Re-downloaded the project folder. Has lame scripts in the misc folder.

Tried the HQ bat file. Exzact same thing. Nothing runs. Execpt for vdub but it locks up looking for the file.

Re-downloading on another computer will retry. Just waiting on the file to transfer to my pc now.

UPDATE: Roxy, looks like it is working on my other computer. Thanks. Would like to get it to work on my HTPC though. Will re-reinstall vdub.

UPDATE2: Re-installed vdub had to delete the old directory. Works fine now. Thanks for some awesome scripts man.

Your welcome. I haven't done anything for HD but theoretically it should work the same. Try the tips I suggested above and have fun experimenting. Please share your discoveries with us here.

You'll need the utility HDTV2Mpeg ( see above link 3 posts above). As a first test, allow dgindex to produce a regular mpa and make a normal mp3.

Later you can experiment with just using the ac3 stream without producing an mp3.

[bobtom]

Hey roxy99 after running your program the mpg file cam out a xvid file larger than the mpg. Ever heard of this?

[roxy99]

Originally Posted by bobtom

Hey roxy99 after running your program the mpg file cam out a xvid file larger than the mpg. Ever heard of this?

was it a hd file? The audio stream may not be mp3. However I never encountered the xvid ending up larger.

[bobtom]

The audio is 192 mp3 and the video is 512x384 29fps xvid