HDPVR Commercial Detection

[razrsharpe]

mmmph... weird... do these recordings have padding? ... i ask bc my understanding of GetMediaTime() is that it is relative to the airing start time (not the beginning of the mediafile)

[Taddeusz]

Originally Posted by razrsharpe

mmmph... weird... do these recordings have padding? ... i ask bc my understanding of GetMediaTime() is that it is relative to the airing start time (not the beginning of the mediafile)

Nope, no padding at all.

[blade]

I've never run commercial processing on HD-PVR files, but I do know that showanalyzer used to be more accurate when run on completed recordings than those in progress because there was more information to check against.

Comskip was just as accurate on live recordings because after the processing was complete it ran a quick check (took a couple of seconds at most) against the complete recording and adjusted the commercial blocks as necessary.

Is it possible that showanalyzer does this check as well now? If so maybe the final check against the completed file is causing the markings to become less accurate instead of improving detection.

Has anyone actually opened the edl file during processing of an accurately marked show and made note of when the beginning and end of commercials were marked and then compared it to what the edl file shows after the recording is complete and the show is not accurately skipping? If the blocks have changed then something in the final check would be throwing it off. If not then the problem should be somewhere else.

[stanger89]

Originally Posted by Taddeusz

But I usually have it do live detection using SJQ which is how I discovered that the segments were dead-on while still recording.

How are you checking that they're "Accurate" as the recording is taking place? Is it by watching where SA flags commercials vs the "live" point, ie that it finishes flagging the commercial right when it ends?

I ask because my running theory is that Sage's skipping/timing is inaccurate on HD PVR recordings by some multiplier. Marked commercials seem to be further and further off as the show progresses.

Essentially I'm noticing/theorizing the same thing as razrsharpe.

[Fuzzy]

Originally Posted by blade

I've never run commercial processing on HD-PVR files, but I do know that showanalyzer used to be more accurate when run on completed recordings than those in progress because there was more information to check against.

Comskip was just as accurate on live recordings because after the processing was complete it ran a quick check (took a couple of seconds at most) against the complete recording and adjusted the commercial blocks as necessary.

Is it possible that showanalyzer does this check as well now? If so maybe the final check against the completed file is causing the markings to become less accurate instead of improving detection.

Has anyone actually opened the edl file during processing of an accurately marked show and made note of when the beginning and end of commercials were marked and then compared it to what the edl file shows after the recording is complete and the show is not accurately skipping? If the blocks have changed then something in the final check would be throwing it off. If not then the problem should be somewhere else.

I think it's been 'confirmed' to not really be SAV's inaccuracy (as the files DO have perfect skip point sin VRD). It definately seems to be a playback issue, in sage, but one that is NOT there if the file is being played back 'live'. I agree it seems to be something to do with the airing time, and that it is different when live vs. recorded.

[Taddeusz]

stanger is right. I received this from SageTV support last night:

Yes the developers know why it's happening but there's no configuration
setting in V6 that'll fix it. They will add a property for V7 called:
"videoframe/disable_timescaling_on_seek" which will default to false, but
you can set it to true to resolve the problem.

Apparently after it completes the recording it adds some kind of timescaling on seek and the feature to disable it will be in version 7. At least we know it will be fixable.

[bialio]

I wonder why it's going to be a property rather than just fixing it - you would think inaccurate seeking would be undesirable all the time.......

[stanger89]

Presumably they added it for some reason in the first place (not sure what it is) I suppose there's a reason some might still want it.

[Fuzzy]

I'm guessing the 'time-scaling' is a feature that is adjusting the timeline reported to the UI such that an hour long show is exactly an hour long (timeline being 0:00 to 60:00). In actuality, most recording are NOT exactly an hour long, due to variances in the time it takes to start and stop the graphs, and for some sources, waiting for a B-Frame, etc. If they used actual length for display (0:00 to 59:48), they would get complaints like "Sage is cutting off the last 12 seconds of my show!" It seems obvious, now, that they figure out a scaling factor after the show is complete (and they have the 'real' length), and use that for the UI time based calls, which are what the comskip plugins use.

[Fuzzy]

After a little more thinking, there should be a way to determine what this scaling factor is (by looking at the media's actual length, and comparing it to the airing length) and reverse 'correction' this in the comskip import. I'm not sure if it's possible to even read the real recording length from the UI, or if all time based calls are 'corrected'.

This really should be considered a bug, and fixed in a minor release. I don't want to wait for the sage7 beta to get accurate skipping back...

[Fuzzy]

Well, did some further digging, and it does look like all time-based api calls have the 'correction' in them. So, to find the REAL file length, I'll have to figure out an external method of getting the real duration from the files.

[ericscottf]

Wandered over here from another thread. I'm seeing this too, have been as long as i've used sage, but wanted to be sure before i posted.

If you want to do a back calculation, i would recommend breaking out ffmpeg. If you do an ffmpeg -i, you'll get some info on the file, one piece of which will show the duration. Here's what i get from a "60 minute" show recorded from sage:

FFmpeg version Sherpya-r11050, Copyright (c) 2000-2007 Fabrice Bellard, et al.
libavutil version: 49.5.0
libavcodec version: 51.48.0
libavformat version: 51.19.0
built on Nov 18 2007 09:00:58, gcc: 4.2.1 [Sherpya]
Input #0, mpegts, from 'NOVA-AbsoluteZeroTheConquestofCold-1170004-0.ts':
Duration: 00:59:48.4, start: 0.387044, bitrate: 2594 kb/s
Program 1
Stream #0.0[0x1011]: Video: h264, yuv420p, 720x480 [PAR 8:9 DAR 4:3], 29.97
tb(r)
Stream #0.1[0x1100]: Audio: liba52, 48000 Hz, stereo, 384 kb/s
Must supply at least one output file


It is 11.6 seconds short.

Next i'll poke around with a media player that uses the actual timeline and shows EDL data and determine how that offset gets woven into the system to produce the problem.

One thing i've noticed, however, at least on my setup is that sage shows pieces of commercials on both sides, it isn't a solid offset that shows some commercials and eats some tv.
That could be because i put some padding into comskip (i *want* to see a piece of a commercial on each side so i can be sure the show didn't cut early) to deal with the problem at hand...

..more later, after thinking and poking.

[ericscottf]

I posted the problem to Erik (of comskip fame)'s forum, maybe he'll have something clever to say about it.

I'm trying to find a media player that will read .ts files properly (smplayer thinks they're all 0:00:00 length and will play them, but won't let me jump around in them and won't read an EDL file).

I've seen VRD mentioned here, but can't find a media player by that name. My google-fu is weak today.
Can someone point me to a windows based media player that likes .ts and .edl files so i can ascertain how .edl data needs to be pushed around in order to fix this issue?

thanks!

[razrsharpe]

Originally Posted by ericscottf

I've seen VRD mentioned here, but can't find a media player by that name.

VRD = Videoredo

[Taddeusz]

If you're dealing with H.264 HDPVR recordings you have to go to the VideoRedo forums and download the V4 Beta. The old program that is still downloadable from the front page only supports MPEG2 video.

[Fuzzy]

Did some digging last night guys. I did confirm that it is because sage is 'faking' the time numbers to make it look like a full length recording. I've got a 'solution' of sorts (just have to polish it up, and I'll get it shared on here).

I've got SJQ set up to run the CLI version of MediaInfo on the recordings (after they've finished) to create a simple text file containing the REAL duration of the media.

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && FileExists != "%d%/%p%.duration"]
{
   :PRIORITY 20
   DURATION
}

DURATION
{
   :RESOURCES 25
   :MAX 1
   :CPU LOW
   "C\:\\SJQC\\MediaInfo\\MediaInfo.exe --Inform=General;%Duration% --LogFile=\"%d%\\%p%.duration\" \"%c%\""
}

I then modified the Comskip 1.5.4 STVi to read in this file, and use the real duration from there, and the false reported duration from Sage to come up with a 'correction factor' that is then used when parsing the .edl file. So far, I've tested it on 4 or 5 shows, and the skips are near perfect (within a second).

I'm at work right now, but when I get home this afternoon, I'll finish polishing up the STVi mods and get it posted up. Hopefully it'll straighten this out until Sage gets things straight on their end.

[ericscottf]

So what is doing the actual correction? The comskip plugin or are you using something else to modify the .edl file?

thanks for the effort!

[Fuzzy]

The correction is done in the Comskip plugin.

[ericscottf]

Interesting.
So you'll read in the output from mediainfo if and only if it is present -- so there shouldn't even be a need for a setting in the STVI, right?

I'll do the mediainfo processing with my own software, but i should be able to figure that out from your post above.

I'm looking forward to testing this asap!

thanks!

[Fuzzy]

yes, basically I check if the .duration file exists, and if it does, use the ratio of that duration and the duration provided by Sage to generate the correction factor. If the .duration file doesn't exist, or if the show is flagged as LiveTV (which doesn't have the false times reported) then if uses a correction factor of 1.0, meaning it skips exactly as the EDL contains.