STV Import: SageMC Video Tools v3.0

[evilpenguin]

This plug-in is dead, please see MediaShrink for all your encoding needs.


(This plug-in is just for SageMC)



Well, it only took about 10 months, but I finally have something worth releasing

This plug-in acts as a replacement for the default SageTV compression engine to enable some things that I felt were sorely missing.

Features:

• Use alternate encoders such as mencoder and handbrake.
• Automatically detect videos that have been telecined and compensate
• Compress DVD's that you have ripped in your video library using handbrake
• Automatically crop videos to remove black bars (mencoder, handbrake)
• Cut commercials from TV shows using comskip/SA .edl files
• Create video DVD's from recordings that are playable in regular set top players (This is currently WORKING)
• Auto-compress favorites with multiple profiles (ex: In Place x264 & iPod Compatable to a folder)
• Specify detailed settings such as bitrate, deinterlace, codecs, container, number of passes, etc.
• Support for subtiltes/CC.
• Easily modifiable/distributable encoding profiles

Download And Install Instructions
(Note: Download is a self-extracting 7zip archive)

I added a "How's and Why's section to the instructions that will hopefully answer some questions for people who have more than a simple single computer setup (i.e. multiple clients, recording stored on NAS, have non SageTV computer encoding videos, etc.)

Usage:
If you just select one of the profiles you'll end up with an in-place 2-pass deinterlaced xvid/mp3 in an avi @ 1500 kb/s. However, if you toggle the "Enable/Disable Advanced Options" and then select either the Handbrake or Mencoder profiles, you'll get a popup window where you can create a much more customized encode.



My Encoding Settings:

Check out the AutoEncode profile which will set these options up automagically. I highly recommend using it, I do

SDTV

• Handbrake
• Auto-Crop
• Deinterlace
• Denoise
• 2-Pass
• H.264 Video
• MP3 Audio
• 1500 kb/s
• avi container

HDTV

• Mencoder (Remember, only mencoder works for HDTV)
• Auto-Crop
• Deinterlace (only 1080i)
• Auto Telecine Detection
• Vertical Scale: 688 (Better compression if divisable by 16)
• 2-Pass
• H.264 Video
• 5.1 AC3/Stereo MP3
• 1500 kb/s
• MKV container

What's the "Auto Encode" profile and why are you so jazzed about it?

That's a good question. Lets say you were to just pick a random recording and try to compress it manually using mencoder. Well, the first thing you'd do is open it up and see what the heck it is.

Is it SDTV or HDTV? Is telecined? Is it interlaced? What's the frame rate? Does it have black bars? Are there commercials? Is the audio stereo or 5.1?

Then, once you know everything about the video, next you have to figure out the best way to encode it.

What codec should you use? Should you resize or crop it? What video filters should you apply? What codecs can go in which containers? What options make the encode go faster? Does mencoder support a feature you want? Is mencoder even the tool you should be using?

That's a lot of tough questions and thay have a lot of complicated answers. Wouldn't it be nice is you could just give mencoder a video and let it do all of the thinking for you? That's where the "Auto Encode" profile comes in.

Basically, when you give it a video, it automatically figures out everything it needs to know about the video and then makes smart encoding decisions based off of that information. Basically, it's like having a little evilpenguin in your machine looking over all of your videos and telling you the best way to handle each of them based off of my years of encoding experience. Haunting and reassuring at the same time, isn't it?

Thanks:
* flachbar/Mlbdude for their hard work on the SageMC STV
* JREkiwi/flachbar for all their Studio help
* Gplasky for some updated/new graphics which will be added soon
* The hardworking artists from MediaPortal for letting us Sage people use their excellent graphics
* ffmpeg, mencoder, and handbrake for their fantastic encoding tools
* Sourceforge for hosting all of my data/downloads

History:
2/12/2008 - v3.0k

• Feature – Added support for attaching subtitle tracks/burning subtitles into videos from embedded closed captions in mpg’s or from smi/srt files.
• Feature – Added Horizontal Scale, Frames Per Second, and Auto Telecine Detection to advanced options.
• Improvement – Auto-Crop adjustments are now evenly distributed between the top/bottom and left/right edges of video. Previously, in the worst case, about 15 lines of resolution were being taken from just the bottom/left. Now it would be evenly distributed to 7 lines from top/right and 8 lines from bottom/left.
• Improvement – Renamed some profiles to remove ambiguity. I highly recommend deleting your current “Encoding Profiles” folder and replacing it entirely with the one in this version.
• Improvement – The Auto Encode profile will use the mkv container if it is using 5.1 AC3 audio as 5.1 AC3 in avi is pretty unstable.
• BugFix – Vertical and Horizontal scale were reversed in mencoderVideoFilters.func.
• BugFix – Added random 2 digit number to qdvd files so that multiple files with the same name won’t over write each other.
• BugFix – Hard coded reference to multiple options theme in STVi to temporarily fix issue where the advanced options pop-up is showing up blank.
• BugFix – Fixed issue where using Auto-Crop with Handbrake/AutoEncode would cause videos to be improperly scaled.
• BugFix – Reversed the slashes in some of the menuGen commands to make it work with UNC paths.

1/19/2008 - v3.0j

• BugFix - Fixed issue where comparing kilobits to kilobytes was causing videoDVD's to try and stream copy when they shouldn't have. This was resulting in oversided DVD's.
• BugFix - I changed autoEncode profile to use AC3 sound for HDTV instead of AAC 5.1. I may change it back once the STX-1000 can handle it good enough.
• Improvement - Changed autocrop detection to use longer samples in order to take advantage of comskip's logo detection. This should help discovery shows like Mythbusters where the logo is halfway in the black bars.
• BugFix - Removed SageMV_169_ep.xml, you should *not be using it anymore*.

12/21/2007 - v3.0h

• Feature: Added Auto Telecine detection option for compression
• Bugfix: Fixed multiple profile issues including 5.1 AAC w/mencoder, commercial cutting from mpegs, splitting a/v, and a few more I can't think of.
• Bugfix: Fixed issue where adding multiple files to a videoDVD would randomly ignore videos
• Bugfix: Title,Episode,Description now show up in DVD menus
• Workaround: Added new option to select PAL in SageMC Options->Video Tools->DVD.

v3.0f1

• Note: I added some features to SageMC_169_ep.xml as well so you'll need to start off with my new one as your base.
• Feature: Enabled DVD burning
• Feature: Added Commercial Cutting for compressed videos
• Feature: ReAdded way more awesome autocrop thanks to comskip
• Feature: Added AutoEncode profile which basically inspects the video sets the encoding options based off of that
• Feature: Added "Wait for Comskip" and "Wait for Archived" to Auto-Compress favorites options (thanks JREkiwi!)
• Feature: MKV now works for mencoder
• Feature: AAC 5.1 audio now works for Mencoder
• Profile Enhancement: Added ability to set CLI switches from within profiles (this is how the AutoEncode profile works)
• Profile Enhancement: Added ability to inspect the video properties of temporary videos in the encoding process and make encoding decisions based off of this (this is how AAC 5.1 profile works)
• Bugfix: If you try and pass Handbrake a video it can't handle it will automatically be passed to mencoder
• Bugfix: Changed "Advanced Options" text in STV to be clearer
• Known Issue – Handbrake still can't handle HDTV-PS videos, but they've isolated the problem and will patch it when they get a chance.
• Known Issue – Commercial cutting only when compressing to .avi
• Known Issue – I forgot about commercial cutting in video DVD's

v3.0d
* Added creation and deletion of custom profiles
* Added setupCentralWorkFolder.exe for people who want to use this feature
* Added new video filtes: Reverse Telecine, Denoise, and Vertical Scale
* Made qdvd/status folder a server property so it only needs to be entered once.
* Added preference for using a single encodingProfile folder for multiple computers.
* Added "Wait For Watched" option in auto compressing favorites.
* Removed autoCrop as it was too unstable. I'm hoping to bring this back asap.
* Removed ffmpeg profile as it didn't work very well before and I won't be supporting it in the future. (Note: You'll need to manually remove ffmpeg.profile from your encodingProfiles folder)
* Completely reworked the encoding profiles system to support more reusable code

What still isn't working:
* Handbrake currently only supports SDTV and HDTV MPEG-TS, but not the HDTV MPEG-PS that Sage makes. I've already sent the handbrake dev's a small clip to see if they can sort out where the issue is.

[JREkiwi]

Big breath and rest...

John

[mikesm]

Looks great! Thanks for the hard work! Some questions:

1) Does the h.264 profile you are using enable playback using hardware acceleration of the video card like ATI and Nvidia do for hd-dvd etc... content?

2) Is the encoding able to be run on a different server than the one sage is running on like the last version did?

3) How do you select using an sd vs hd profile?

Why not always de-interlace for hd content like you do for hd?

thx
mike

[toricred]

I'm a little confused (big surprise there). I import the sagemcvideotools_3.0.xml file, but I don't ever see the advanced options. How do I bring them up?

[evilpenguin]

There's a button you have to click at the top of the "Select Profile" window that says "Enable Advanced Options". If you don't click it all, everything will just default to 2-pass Xvid/mp3 in an avi @ 1500 kb/s.

[evilpenguin]

Originally Posted by mikesm

Looks great! Thanks for the hard work! Some questions:

1) Does the h.264 profile you are using enable playback using hardware acceleration of the video card like ATI and Nvidia do for hd-dvd etc... content?

2) Is the encoding able to be run on a different server than the one sage is running on like the last version did?

3) How do you select using an sd vs hd profile?

Why not always de-interlace for hd content like you do for hd?

thx
mike

1) That's a good question and I don't know.
2) Yep, UNC paths are handled much better in this one so you don't even need to translate the paths.
3) I'm not sure understand your question
4) Trial and error. I've found that all SDTV content looks better deinterlaced, but some HDTV content doesn't need it. I don't really have a good reason for it.

[toricred]

OK. I was reading that button backwards. When the title of it said "Enable," I thought you had to "click" it to enable the advanced options. It seems to be much better now.

This is much better and I really appreciate your patience with my constant asking when it would be done. You've done a fantastic job on this one.

[toricred]

2 Questions.

1. How do I create my own profiles so that I don't have to re-create it each time?

2. ffmpeg compression doesn't seem to work. Is there something I need to do to make it work?

[evilpenguin]

Originally Posted by toricred

OK. I was reading that button backwards. When the title of it said "Enable," I thought you had to "click" it to enable the advanced options. It seems to be much better now.

This is much better and I really appreciate your patience with my constant asking when it would be done. You've done a fantastic job on this one.

Yeah, when you asked that question, I realized that button was a bit ambiguous, i'll make it clearer in the next update.

Originally Posted by toricred

2 Questions.

1. How do I create my own profiles so that I don't have to re-create it each time?

2. ffmpeg compression doesn't seem to work. Is there something I need to do to make it work?

1) Right now there's no easy way to create your own profiles/defaults from within Sage. I'm working on ways to correct this, but for now you either have to modify/create a new profile to set the defaults to what you want. If you tell me what you want your defaults to be I can show you how to make a new profile out of them.

2) Just by default, or with the extra options? Also, ffmpeg is by far the worst option of the 3, possible just because my profiles need improvement. I highly recommend Handbrake if your show is regular SDTV.

[toricred]

Good then I wasn't being stupid reading the button. I can see how it could be either way. The profile I'm trying to create would be mencoder, 1-pass, divx, mp2 audio.

I thought handbrake was only for DVDs. I really want to convert to either to keeping the original sound, mp2, or ac3. I really want to avoid mp3 and I'm not sure that Sage can handle AAC.

[lobosrul]

Penquin, I've spent many... too many... hours of experimenting with mencoder; I'd consider myself an expert at encoding HDTV in x264 with it.

Some tips:

1) I've found its always best to pass the mpg source thru VideoRedo. BobPhoenix's mod is great for this. I used to get audio sync issues all the time before this. I never could get accurate cuts on edl lists with mencoder.

2) As far as encoding quality goes. I always use two passes. Also the newer builds of mencoder have x264 built in, not sure if your using it seperately or not.

I've found the following to be a very good trade-off in speed/quality.
pass1: deblock:qcomp=.65:bframes=3:turbo=2:b_pyramidass=1
pass2: deblock:qcomp=.65:subq=5:8x8dct:frameref=2:bframes=3:b_pyramid:weight_bass=2

The qcomp setting is a bit subjective. Be sure and add "threads=2" to both passes if the cpu is dual core. I don't like any denoise filters, again thats subjective.

You'll also have a more efficient encode if you crop it to compensate for over-scanning. I crop by 10%.
For 720p: crop=1152:648:64:36,scale=<whatever>
For 1080i: crop=1728:972:96:54,scale=<whatever>

This would not work correctly for any "HD-lite" content.
Always crop before scaling.

3) REVERSE TELECINE. The is the main reason I decided against using the Sage Transcoder. All movies, and nearly all weekly drama's and sitcoms on TV are filmed and are therefore 24 (23.976 fps).

I've found no reliable, automatic way of detecting whether a program is 24fps or not. To manually find out, simply framestep by hitting pause in SageTV. You should see a clear 2:3 pattern in 720p, and 1080i if you have decoders that perform IVTC.

For 720p use -vf decimate=2:1000:1600:.001. Do this before cropping and scaling. Also add -ofps 23.976*

For 1080i (and 480i) use -vf pullup,softskip again before cropping and scaling. Also add -ofps 23.976*

4) For interlaced content I have been using yadif to de-interlace it. Recently I've been experimenting on simply leaving it as interlaced and letting my decoder do the work after the fact. I'll give you more details later if you want.

5a) It looks like your supporting AAC audio. Which implies you arent using mencoder itself to mux the 264 and aac file. If so, ALWAYS uses the harddup filter at the end of the vf tag, or you risk losing sync. I use mp4box for muxing BTW.

5b) The only way I found to create AAC(5.1) audio is to use besweet and create a wav with the tag " -6chwav" and then use neroAACenc.exe with the tag "-ignorelength" (because the wav file is sometimes over a GB). A bitrate of 192000 sounds as good as AC3 to my ears.


* I use decimal numbers instead of fractions (ie 24000/1001) even though the mencoder manual suggests it because they are more accurate. This is because mp4box uses decimals.

Whoa... didn't mean to ramble quite that much. Anyways, I hope this is useful to you, or somebody out there.

[TechBill]

I imported this tool in my SageMC 6.3.5.

When I went to SageMC Extra setting.

It put a big gap between the SageMC Video Tools and the other import tools I added in SageMC. Like this

SageMC Video Tools Open









Captions Playback Options Modify





should it been something like this in Extra with no gap?

SageMC Video Tools Open

Captions Playback Options Modify



Also when you go to the Extra and under "Compress" it all blank with just up and down arrow. Is there suppose to be any setting in this one?

[evilpenguin]

Originally Posted by toricred

Good then I wasn't being stupid reading the button. I can see how it could be either way. The profile I'm trying to create would be mencoder, 1-pass, divx, mp2 audio.

I thought handbrake was only for DVDs. I really want to convert to either to keeping the original sound, mp2, or ac3. I really want to avoid mp3 and I'm not sure that Sage can handle AAC.

Well, if you can live with just creating it manually for a bit, i'll crank out a way to create custom profiles from within Sage. Oh, and i'll also throw in an mp2 audio option for mencoder.

As for handbrake, it used to be just for DVD's but recently they have added some code specifically for supporting MPEG-PS/TS, which is fantastic for us I don't know what they do behind the scenes but the encodes are very high quality, everything is multi-threaded to take advantage of multiple cpu's (it just *smokes* on my quad-core Q6600), and the CLI is very clean and easy to use.

[evilpenguin]

Originally Posted by lobosrul

Penquin, I've spent many... too many... hours of experimenting with mencoder; I'd consider myself an expert at encoding HDTV in x264 with it.

Some tips:

1) I've found its always best to pass the mpg source thru VideoRedo. BobPhoenix's mod is great for this. I used to get audio sync issues all the time before this. I never could get accurate cuts on edl lists with mencoder.

2) As far as encoding quality goes. I always use two passes. Also the newer builds of mencoder have x264 built in, not sure if your using it seperately or not.

I've found the following to be a very good trade-off in speed/quality.
pass1: deblock:qcomp=.65:bframes=3:turbo=2:b_pyramidass=1
pass2: deblock:qcomp=.65:subq=5:8x8dct:frameref=2:bframes=3:b_pyramid:weight_bass=2

The qcomp setting is a bit subjective. Be sure and add "threads=2" to both passes if the cpu is dual core. I don't like any denoise filters, again thats subjective.

You'll also have a more efficient encode if you crop it to compensate for over-scanning. I crop by 10%.
For 720p: crop=1152:648:64:36,scale=<whatever>
For 1080i: crop=1728:972:96:54,scale=<whatever>

This would not work correctly for any "HD-lite" content.
Always crop before scaling.

3) REVERSE TELECINE. The is the main reason I decided against using the Sage Transcoder. All movies, and nearly all weekly drama's and sitcoms on TV are filmed and are therefore 24 (23.976 fps).

I've found no reliable, automatic way of detecting whether a program is 24fps or not. To manually find out, simply framestep by hitting pause in SageTV. You should see a clear 2:3 pattern in 720p, and 1080i if you have decoders that perform IVTC.

For 720p use -vf decimate=2:1000:1600:.001. Do this before cropping and scaling. Also add -ofps 23.976*

For 1080i (and 480i) use -vf pullup,softskip again before cropping and scaling. Also add -ofps 23.976*

4) For interlaced content I have been using yadif to de-interlace it. Recently I've been experimenting on simply leaving it as interlaced and letting my decoder do the work after the fact. I'll give you more details later if you want.

5a) It looks like your supporting AAC audio. Which implies you arent using mencoder itself to mux the 264 and aac file. If so, ALWAYS uses the harddup filter at the end of the vf tag, or you risk losing sync. I use mp4box for muxing BTW.

5b) The only way I found to create AAC(5.1) audio is to use besweet and create a wav with the tag " -6chwav" and then use neroAACenc.exe with the tag "-ignorelength" (because the wav file is sometimes over a GB). A bitrate of 192000 sounds as good as AC3 to my ears.


* I use decimal numbers instead of fractions (ie 24000/1001) even though the mencoder manual suggests it because they are more accurate. This is because mp4box uses decimals.

Whoa... didn't mean to ramble quite that much. Anyways, I hope this is useful to you, or somebody out there.

Wow, a ton of good info in there

1) Yeah, I know passing everything through videoredo is the way to go but i'm torn by the fact that its not free so I hesitate to support it. I'm really hoping to find a free way to duplicate their quick stream fix.

2) I'm using svn mencoder builds from

http://oss.netfarm.it/mplayer-win32.php

and using the built in x264 encoder. As for my x264 options, I'm not particularly attached to any of them as I just grabed them off of the mencoder man page. I'll give yours a shot.

For cropping, right now i'm using mplayer to detect the optimal crop settings but it doesn't remove the junk at the top and bottom of a lot of HDTV streams. It shouldn't be to hard to just manually add in a little extra to what mplayer detects to handle this.

3) Ahh, this is the info I was really hoping to get because I couldn't find any definitive answer on the net. It shouldn't take much to add in a resolution dependent reverse telecine option. Now, when you say 480i are you refering to analog captured programs or digitally recorded SDTV streams? Also, should this go before the deinterlace filter?

4) Yeah, I did a little experimenting too and, atleast for HDTV it really didn't make that much of a difference when I watched it on my Sage Client. About the only thing it did noticably was slow down my encode by ~30 fps

5a) Mencoder can't mux x264 and AAC into an avi? I'm confused by what you mean there.

So for mp4box do you just encode the a/v with mencoder, then use mencoder to dump the individual streams into files, then use mp4 box to put them into mp4 container?

5b) That's definetely do-able. Can you PM me some full example commands? And does processing the audio and video seperately create sync issues?

Also, i'm hoping that we can get handbrake working with our Sage created HDTV streams because they support support creating 5.1 AAC (and Ogg) straight from AC3. That would make everything much easier

6) Just curious, where do you cut commercials in your process? I've found that if I try and cut before/during the encode it sometimes randomly which is unacceptable for an automated process. About the only reliable way i've found is to cut after the video is encoded. This does have a few problems though...

a) mencoder -oac copy -ovc copy -edl .... will only work if you're muxing it into an avi, which rules out alternate containers like mp4 which i'd much rather use.

b) Since its just copying and not encoding the cuts usually fall between I-frames which can create weird video errors until the next I-frame is reached.

[evilpenguin]

Originally Posted by TechBill

It put a big gap between the SageMC Video Tools and the other import tools I added in SageMC.

What's the other plug-in that your using?

Originally Posted by TechBill

Also when you go to the Extra and under "Compress" it all blank with just up and down arrow. Is there suppose to be any setting in this one?

Nope, there are just no options there, I moved them all into the per video advanced options screen and forgot to remove the button.

[lobosrul]

Originally Posted by toricred

I really want to avoid mp3 and I'm not sure that Sage can handle AAC.

It definitely can! Sages built in player is simply mplayer in disguise; it's one of the most robust video players out there.

[TechBill]

Originally Posted by evilpenguin

What's the other plug-in that your using?

Nope, there are just no options there, I moved them all into the per video advanced options screen and forgot to remove the button.

The other plugin I am using is Closed Caption Playback plugin which can be found at http://forums.sagetv.com/forums/showthread.php?t=23752


Ah ok I was wondering about that Compress option so I will ignore it.


Thanks for making a great tool plugin possible!
Bill

[evilpenguin]

Unless someone can give me a very compelling reason not to, I think i'm going to dump ffmpeg from the profile list. I'll keep the .exe around because I use it for other reasons behind the scenes, but I'd really rather focus on developing the mencoder and handbrake profiles as they have many more available options and in my testing are producing *much* higher quality encodes.

[toricred]

My compressions aren't having the commercials removed. I just had some problems with my dirmon2 install for comskip cleaning up the .edl files. I changed it to clean up in subdirectories as well. Could that cause the problem?

[evilpenguin]

Commercial cutting isn't currently implimented because I didn't have enough time to dedicate to testing it. Its on my todo list this week. Don't worry, if you still have you're .edl files you'll be able to cut the commercials out of the compressed avi's.