Transcoding Confusion

[deria]

Hi,

Does Sage offer any way to tune/alter the transcoding system?

I've tried converting a few videos from MPEG-2 into AVI High Quality. It works, but the "high quality" part of the experience seems to be missing compared to (for example) the digital downloads that are...around. In those videos, a 400mb file is one hour of high definition video with crystal clear video and audio. With my transcoded shows, made from my recordings, I get less than stellar results. In particular, in high motion scenes I notice blocky-ness and tearing. I don't expect miracles (since I'm not using a high definition source) but I -would- like to see the quality be at least equal to the source material.

Any thoughts?

Ideally, what I'd like to do is transcode my shows that my fiance records but seems to never watch into something smaller and yet retain the same image quality. In MPEG-2, a low bit-rate is usually the cause of blocky-ness. In AVI, I have no idea.

[stanger89]

Originally Posted by deria

Any thoughts?

A couple...

The options used for encoding are specified in the properties file (server I believe). I think you should be able to specify any ffmpeg options in there to tweak it.

In those videos, a 400mb file is one hour of high definition video with crystal clear video and audio.

At 400MB, those are NOT actually HD, they may be HD sourced, but they've been downconverted. Best case for HD is about 1GB/hr.

Second, these are often/usually from pristine sources, and quite often run through avisynth scripts to get the best possible results.

With my transcoded shows, made from my recordings, I get less than stellar results.

What's the source of these recorings? Analog, digital SD, HD? That will have a large bearing on the required bitrate. If you've got a nice clean source, it can usually be shrunk pretty small with good results, if it's noisy (analog cable) it becomes exceedingly difficult to get good results with significant space savings and minimal effort.

In MPEG-2, a low bit-rate is usually the cause of blocky-ness. In AVI, I have no idea.

Same with most any codec, macroblocking is usually caused by insufficient bits.

[lobosrul]

Does the Sage transcoder do any resizing? The less pixels an encoded file has, the less artifacting/blocking (assuming the same compression setting/bitrate).

I resize HD files down to 1024x576 (actualy I might try 960x540), and encode to 2.5mbps divx. No its not hi-def, but still significantly better than standard def. Those files are around 800 MB's for a 42 minute show, AC3 sound. Theres no way your going to get a really great pic quality at 400MB/hour, not even with MP4/AVC.

[stanger89]

Originally Posted by lobosrul

Does the Sage transcoder do any resizing?

It can. Looks like the MPEG-4 settings don't.

[deria]

Originally Posted by stanger89

At 400MB, those are NOT actually HD, they may be HD sourced, but they've been downconverted. Best case for HD is about 1GB/hr.

True, I should have said that they were extremely high quality, rather than HD. They're noticeably better than anything I can record, thats for sure.

Originally Posted by stanger89

What's the source of these recorings? Analog, digital SD, HD? That will have a large bearing on the required bitrate. If you've got a nice clean source, it can usually be shrunk pretty small with good results, if it's noisy (analog cable) it becomes exceedingly difficult to get good results with significant space savings and minimal effort.

My source is analog-cable. As for the noise level, I would say its minimal. I had some signal problems up until a few days ago, but I've since re-arranged my cabling and all my recordings look "clean". (I realize thats not a very scientific statement, but they're free from ghosting, artifacts, wierd flickery lines, or other noticeable defects.)

Originally Posted by stanger89

Same with most any codec, macroblocking is usually caused by insufficient bits.

I'll have to look into those properties that you mentioned and see what I can do. Even if I have to sacrifice space to get acceptable quality, its still going to be better than the 3GB/hour recording size I currently have.

[lobosrul]

As far as standard def sources go, I use 1.5mpbs, no resizing (unless its letterboxed). At the distance I view my TV from, I cant even tell a difference from the source and encoded file.

But thats still a digital source, I do almost nothing in the way of analog.

[agover]

Originally Posted by deria

True, I should have said that they were extremely high quality, rather than HD. They're noticeably better than anything I can record, thats for sure.

There are a number of.... tricks that improve their encodes - most notably they likely use multipass encoding (which doubles to triples the encoding time, but decreases the file size without impacting the quality).

My source is analog-cable. As for the noise level, I would say its minimal. I had some signal problems up until a few days ago, but I've since re-arranged my cabling and all my recordings look "clean". (I realize thats not a very scientific statement, but they're free from ghosting, artifacts, wierd flickery lines, or other noticeable defects.)

You are double to triple encoding your programs using a low res encoder which is why you are getting so so results. It works like this:
Source station receives origin signal - likely compressed using mpeg2 high bit rate
Your tuner card then takes the analog copy of the source signal and then re-encodes it at a lower bit rate
Then you are taking a lower bit rate mpeg2 source and converting it again.

The XVID 450mb files based off of HD are sourced at 1080i or 720p then downconverted to 480p or less. Since the origin is a high bit rate mpeg2 file the impact of re-encoding to a lower resolution is much less noticable.

If you want/need higher quality encodes you will either need to adjust the encoding settings to a higher bit rate (compromising file size) or switch out from Sage altogether and utilize a higher quality encoder that supports multipass encoding. Another option (for video break up during high motion scenes) is to increase the motion sensitivity (this too will increase file size)./

I'll have to look into those properties that you mentioned and see what I can do. Even if I have to sacrifice space to get acceptable quality, its still going to be better than the 3GB/hour recording size I currently have.

[Narflex]

BTW, 2-pass encoding is in the next build

[deria]

Excellent to hear! I have a feeling that using that, and/or increasing the bitrate will solve my quality issues.

[ugh]

SWEEEET!!!!!

[Enormous]

Originally Posted by Narflex

BTW, 2-pass encoding is in the next build

That's excellent news. Thank you!
Will there be more encoding options included by default as well?

[dadams]

Just an experience I recently had...

When I first went to V6 I couldn't transcode any HD material to MPEG-4.

I tried it a couple of days ago and it worked(finished) except when I played the file back the audio was playing fine, but the video was playing in FF mode, finishing in just a couple of minutes. The audio was still in the first few minutes. When I tapped rewind the video would play normally for a few seconds then resume to the FF mode. The audio never was in sync. I will try it several more times to see if I can reproduce it.

[popechild]

Originally Posted by dadams

Just an experience I recently had...

When I first went to V6 I couldn't transcode any HD material to MPEG-4.

I tried it a couple of days ago and it worked(finished) except when I played the file back the audio was playing fine, but the video was playing in FF mode, finishing in just a couple of minutes. The audio was still in the first few minutes. When I tapped rewind the video would play normally for a few seconds then resume to the FF mode. The audio never was in sync. I will try it several more times to see if I can reproduce it.

First problem was a bug in 6.0.11. Second problem is a bug in 6.0.12. Supposedly both are fixed in the next build.

[Patilan]

Originally Posted by Narflex

BTW, 2-pass encoding is in the next build

Although 2-pass encoding is quite important for mpeg4 encoding, it is not the biggest problem in the current SageTV beta. The biggest problem as I see it is that it is impossible to hook any decent filter in ffmpeg (aka: SageTVTranscoder).

The reason why I say this is that 95% of what Sage records for me is material that had been originally shot in 24 fps progressive, and then later has been hard-telecined to 30 fps interlaced for television. This includes all the movies, and all kinds of shows like The Simpsons, Futurama, South Park, you name it. The transcoding kills the interlacing, and this makes any telecined material look bad, no matter if we deinterlace while transcoding or not: If we do any kind of transcoding without an inverse-telecine filter, we end up with stuttering videos which have dropped/duplicate frames. And that's annoying to watch, no matter how high bitrate you chose for the transcoding.

I see 2 ways of fixing this problem:

1. Integrate Mencoder in Sage for the transcodings instead of ffmpeg. (or along with ffmpeg if you want). Mencoder is more versatile than ffmpeg, has a variety of good filters, and it is part of the Mplayer package which Sage is already using anyway (SageTVPlayer.exe=Mplayer=www.mplayerhq.hu). So go for the real thing!

OR (/and)

2. Give us an option to enter our own command line for the transcoding. (In this case Sage should only append the file name of the selected file to our command line) This will give us a way to use any encoder of our choice, and it will streamline things a little bit for those of us who are now going through the trouble of writing scripts, changing time stamps, and all kinds of unnatural tricks.

And also, it will be nice if the transcoded shows are marked with some icon in the lists, so we can more easily see what's already been transcoded and what's not.

Thanks!

-=patilan=-

[Patilan]

Originally Posted by deria

the "high quality" part of the experience seems to be missing compared to (for example) the digital downloads that are...around. [...] In particular, in high motion scenes...

That's exactly what I was talking about in my previous post. If you look at your transcode frame by frame, you'll notice that every 5-th frame is a duplicate. That's annoying to watch not just in high motion scenes but with any kind of motion. Try transcoding to 24 fps WITH inverse telecine filter (pullup) and you'll be amazed by the result! Also, for high-motion source, particularly source with lots of small moving objects, enable the optional 8x4, 4x8 and 4x4 subpartitions in the macroblocks.

Originally Posted by deria

In MPEG-2, a low bit-rate is usually the cause of blocky-ness. In AVI, I have no idea.

AVI is just a container, and it can hold a bunch of different audio/video formats. For the video codex I'd say try h264 (if your computer can handle it), it does a remarkable job.

-=patilan=-

[autoboy]

Hey, thats some nifty stuff you know. Can you point me to sites that explain the specifics of video? I've always been mystified by inverse telecine and other similair terms. I see all kinds of things in video that bug the crap out of me and can't determine what is causing them. The every 5th frame thing you descirbe is one on them. I've identifiend tearing and feathering but that is just about as advanced as most sites get. I've been trying to track down info like that on video and how different video cards with purevideo and Avivo handle the video streams and I can't find any info on it that really explains it well.

thx.

[Enormous]

Originally Posted by Patilan

Mencoder is more versatile than ffmpeg, has a variety of good filters, and it is part of the Mplayer package which Sage is already using anyway

Great suggestions.

Could you provide us with a sample command you would use to encode a show (and also a short explanation of what the parameters you're using are)?

Thanks.

[Patilan]

Originally Posted by autoboy

Can you point me to sites that explain the specifics of video? I've always been mystified by inverse telecine and other similair terms. I see all kinds of things in video that bug the crap out of me and can't determine what is causing them. The every 5th frame thing you descirbe is one on them

Originally Posted by Enormous

Could you provide us with a sample command you would use to encode a show (and also a short explanation of what the parameters you're using are)?

OK guys, here is what I am doing:

At first I was very excited with the new mpeg4 transcoding in SageTV beta. I spent endless hours tweaking the ffmpeg options in the Sage.properties file.

Frustrated by the results, I decided to track down the the source of ffmpeg (AKA SageTVTranscoder.exe) and Mplayer (AKA SageTVPlayer.exe) so I can learn about the options. I found: www.mplayerhq.hu

There I found some must-read documentation that opened my eyes about many things: http://www.mplayerhq.hu/design7/info.html#docs

Me being new at video encoding and stuff, I decided to look for some GUI frontends to get me started: http://www.mplayerhq.hu/design7/projects.html

I played with quite a few of them, but most were either too buggy or too limiting for my taste. The one I ended up using was: http://mulder.dummwiedeutsch.de/home...ml#mencoder264

That's still not the greatest software I've ever seen, but it was very useful to tweak a few options, start encoding, and watch the log window to see what the Command line looked like. Very educational!

Once I got some idea about the command line options, I trashed that GUI, and started writing my own scripts. The two documents I used the most are:
http://www.mplayerhq.hu/DOCS/HTML/en...ing-guide.html
http://www.mplayerhq.hu/DOCS/man/en/mplayer.1.html

I tried to write a .bat file that passes command line to mencoder.exe (the one I got from the mencoder264 package), but that gets you too much white hairs, and I don't recommend it even to my enemies. Then it dawned on me that Windoze has vbscript built-in! Just save the text of a script to a file with name ending on ".vbs" and Windows will execute it when you double-click on it, or when you drop files on top of it, just like a ".bat" file. I even downloaded a nice little shareware vbscript editor: http://www.vbsedit.com but you don't really need that.

Below is a simplified version of the script I am currently using. Just copy/paste it in a text file, and save it as ".vbs" file in the same directory where you have the mencoder.exe. Then you drag an alias of this .vbs file to your desktop, and you can drag-drop SageTV recordings on top of the alias. The resulting output will be in "x:\Transcoded\", where X will be the same drive letter as the original: This way later, after fixing the time stamp, we'll just move the new file to the SageTV recordings directory instead of copying it.

'###############################################################################################
'Limited-purpose version, still in development. Use at your own risk.
'You can get the latest Windows build of MEncoder from: http://ffdshow.faireal.net/mirror/mplayer/
'Remember to save this ".vbs" script in the same directory where you have MEncoder.exe
Option Explicit
Dim progPath, outPath, InFile, OutFile, LogFile, strEncoder, strDuration, strFilter, strOptions, strCmd1, strCmd2, cmd, n 
Const BitRate = 800 ' <-- You can play with this.
Const fTelecined = " -vf crop=704:464:8:8,pullup,softskip,hqdn3d " 
Const fProgressive = " -vf hqdn3d=2:1:2 "  
'strDuration = " -frames 120 "
progPath = Left(WScript.ScriptFullName,Len(WScript.ScriptFullName)-Len(WScript.ScriptName))
set cmd = WScript.CreateObject("WScript.Shell")
For Each InFile In Wscript.Arguments
	Select Case Lcase(Right(InFile, 4))
		Case ".vob" strFilter = fProgressive : Transcode '<-- For progressive dvd movies
		Case ".mpg" strFilter = fTelecined : Transcode '<-- For hard-telecined SageTV recordings
		Case Else MsgBox "Ignoring file: " & vbCr & InFile
	End Select
Next
Sub Transcode ' InFile 
	outPath = Left(InFile,2) & "\Transcoded\"
	n = InstrRev(InFile, "\")
	OutFile = outPath & Mid(InFile, n+1, Len(InFile)-n-4) & ".avi" 
	LogFile = OutFile & ".stats"
	InFile = Chr(34) & InFile & Chr(34) : OutFile = Chr(34) & OutFile & Chr(34) : LogFile = Chr(34) & LogFile & Chr(34) : 
	strEncoder = Chr(34) & progPath & "MEncoder.exe" & Chr(34) 
	strOptions = " -passlogfile " & LogFile & " " & strDuration & " " & strFilter & " -ofps 24000/1001 -ovc x264 " _
	 & " -x264encopts frameref=4:bframes=2:trellis=1:subq=6:8x8dct:keyint=120:weight_b:b_adapt:bime:brdo:nodct_decimate:nofast_pskip:bitrate=" & BitRate
	cmd.Run "CMD /c MD " & outPath,0,True 
	strCmd1 = strEncoder & strOptions & ":pass=1:turbo=1 -o NUL -nosound " & " " & InFile 
	strCmd2 = strEncoder & " -forceidx " & strOptions & ":pass=2 -o " & OutFile & " -oac copy " & " " & InFile  
	cmd.Run "cmd /c " & Chr(34) & strCmd1 & " && " & strCmd2 & " && DEL " & LogFile & Chr(34),,True 
	'MsgBox "The transcoded video was saved as:" & vbCr & OutFile
End Sub
'###############################################################################################

Of course, the above options are only my personal subjective preferences. There is no one "right" answer about what the optopns should be. It depends greatly on your source, and on your goals, and on your limitations. I can go into some detail why I chose which particular option, but this post got too long already.

Now that we have the transcoded file, all we have to do is fix the time stamp, and switch it with the original in the SageTV recordings directory. I'll add a few lines to the above script to do that. In the mean time, follow this guide by nielm: http://forums.sagetv.com/forums/showthread.php?t=10646

That's all I've got so far. I hope this helps to get you started.

Now I only pray that someone from Sage will notice my earlier post, and either integrate mencoder in Sage instead of [or along with] ffmpeg. Or at least give us a command line option in the Sage.properties file.

Yours,
-=patilan=-

P.S. Note that I live in the USA, and the video problems that I am talking about may not apply to people in other countries. I don't know.

[Enormous]

Patilan, thank you very much for your reply!

[blade]

Originally Posted by Patilan

The reason why I say this is that 95% of what Sage records for me is material that had been originally shot in 24 fps progressive, and then later has been hard-telecined to 30 fps interlaced for television.

Same here. I was very excited about built in transcoding, but right now it is not very useful for saving space on the server. I'm sure it is great in other situations, but not for playback on a primary client. The only way it will be of use to me is if the quality is on par with my 6600gt.