How do you archive?

[Dargason]

I'm sure there are lots of different ways that people use to shrink recordings that they want to keep around. Over the years I've played with Mediashrink and Handbrake, and I've never been very happy with either. But it wasn't really a big issue because I could always dial back the HD-PVR recording quality for programs where it didn't really matter to me.

Now that I have a cablecard tuner I don't have that option... I have to either start deleting stuff faster, buy another hard disk, or bite the bullet and figure out how to get this 20GB recording of starwars down to a more manageable size.

In my case, I have a few requirements (or preferences):

- I want to keep closed captions.
- I would like to be able to comskip, or even edit out the commercials to save space, if it isn't too time consuming.
- I'd like to keep the DD 5.1 audio.
- I'd like it to play on my extenders just like my TS streams do.... the m4v and mkv files that I've produced using handbrake don't seem to skip forward/backward correctly. Maybe I'm using it wrong.

I guess what I'm saying is that I'd like to have a ts file, just a smaller one. (If someone can tell me how to make an mkv file act like a ts file, that's ok too...)

[wayner]

Most of my stuff does come from an HD-PVR so I don't bother archiving that. But kids shows do get archived - I comskip them and then run them through handbrake to convert to mp4 and finally copy them to a folder structure based on the show name. This also makes it easy to use the files on iDevices. I have written an SJQ process to handle all of this. I haven't gotten around to actually cutting out the commercials but that shouldn't be too hard to do with VRD.

I am pretty sure that I have no problems skipping around in mp4s in exactly the same way that I do in .ts files. My files don't have 5.1 audio but that shouldn't be an issue in Handbrake. I don't know about CCs as I never use them.

But to be honest the best option might just be to buy another hard drive now that prices have come back down. And a 2TB or 3TB drive holds a lot.

[Dargason]

So you run comskip on the .ts file and then convert it? Will the edl file still work for the mp4 file?

[wayner]

In my case these are generally initially MPG files as they are generally SD, but yes, I run comskip on that and then the edl file still works on an mp4 file.

[Fuzzy]

Sage's UI is what interfaces with the .EDL file, not the decoder, so it will work with and media type.

[fidget]

I copy my files onto slower (5900 RPM) long-term storage drives (2 TB). For TV series, they get replaced by the DVDs (encoded with Handbrake) when they are released. The other files get deleted, eventually.

[Dargason]

Originally Posted by Fuzzy

Sage's UI is what interfaces with the .EDL file, not the decoder, so it will work with and media type.

Good to know. I didn't know what the numbers in the file mean... are they time markers, or frame numbers, or something else?

I played around with handbrake a bit more last night. Either it has improved or I've become a bit more knowledgeable, because it wasn't quite as cryptic as I remember. I figured out how to add the right audio and CC tracks, and I'm working on improving the quality. The presets seem too focused on compression and the resulting quality suffers.

I've been mainly using the "Normal" and the "High Profile" profiles as my starting point. I like the "high profile" better, but it takes much longer to process.

To improve video quality, it looks like I can use "constant quality" and modify the quality setting. I tried moving this from 20 down to 10 and got video glitches in my test file. So now I'm trying to set an average bitrate. The default is 1500, and I tried 3000 which seem OK on my test file. I increased it again to 6000 and I'm going to try that next.

What settings do you guys use?

[wayner]

I think that I am pretty much using the iPad default settings with the excpetoin of turning on deinterlacing. Note that after you figure out what works for you if you run this from a batch file you will use HandbrakeCLI.exe and you will likely have to manually specify all of the parameters as (and this isn't well documented) the CLI does not accept custom presets.

Here is some info on edl files from the mplayer FAQ (presumably it is the same for Sage):

The current EDL file format is:

[begin second] [end second] [action]
Where the seconds are floating-point numbers and the action is either 0 for skip or 1 for mute. Example:

5.3 7.1 0
15 16.7 1
420 422 0

This will skip from second 5.3 to second 7.1 of the video, then mute at 15 seconds, unmute at 16.7 seconds and skip from second 420 to second 422 of the video. These actions will be performed when the playback timer reaches the times given in the file.

[Fuzzy]

That's correct. Sage, however, does not utilize the 'mute' flag, just the start-skip and end-skip times.

[Dargason]

Thanks! Eventually I might want to do all this with scripts, but I'm not that far along yet... I need to understand it much better.

I just read a thread on avsforums where they seem to be recommending a constant quality (RF) setting of 16, so maybe I was too aggressive when I set mine to 10. I'll try again with 16 when I get home and see how that turns out.

There were also some comments about audio sync. I didn't run any issues with that, so I'm keeping my fingers crossed.

That's interesting information on the edl file. So even after comskip does its thing, if I want to adjust a marker a second forward or backward, I can just edit the text file.

[wayner]

Originally Posted by Dargason

That's interesting information on the edl file. So even after comskip does its thing, if I want to adjust a marker a second forward or backward, I can just edit the text file.

Yep, and you can also have comskip produce .vrpj which are used by VideoRedo so you could automate comskipping, actually cutting commercials from the file (which might be dangerouse if comskip is inaccurate) and then reencoding.

One thing to try - if you have VRD then I believe you can also use it to reencode from a ts file in MPEG2 encoding to a H.264 file type such as mp4. And VRD also let's you do the editing or cutting. And it can also be run from a batch file.

In fact, here is a thread where someone has posted their automated process:http://forums.sagetv.com/forums/show...highlight=vprj

I occasionally have minor synch issues on my Handrake reencoded files but this doesn't always happen.

[Dargason]

I don't have VideoReDo, but I was thinking about getting it, which is one of the reasons I started this thread. Being able to automatically cut out commercials would be great.

But while comskip works great on some channels/recordings, it doesn't do nearly well enough to automate, especially on movies, which is where I'd want it most. But I think VRD does have an editor of its own that would let you edit the .vprj files that comskip produces and 'fix' the problems that show up before you remove the commercials. I'm not sure how easy that is, but it would be a manual process in any case. And VRD isn't cheap.

[graywolf]

If you end up getting VRD, be sure to check the forum there and look into VAP add-on (user written/supported).

It helps automate many of the steps and make it more batch like.
Have it run the comskip to generate the vprj files, then at some time you can go ahead to check/modify the cut points on a bunch of the files. Then you start processing again and it'll perform the cuts on all the files where you have verified the cutpoints.

[wayner]

VRD costs about $75 but it is a very good program. It is quite easy to make edits and resave the file which it can do quickly without reencoding if you are saving to the same file type. If you are changing to a new file type then it may take a while but it may work for you as your encoding program as well.

[ranger]

I use VRD to edit the files and then compress with Handbrake. Using VRD is easy, but it can take some time if you have a large number of files to process. I make the required changes to the .vprj file generated by comskip and then use the batch function in VRD to save new files.

For Handbrake, I use the Normal setting modified just a bit. First I drop the framerate down to 29.97. If I leave it at 60, some of my clients will stutter a bit during playback. I use a constant quality between 19 and 21 depending on whether or not it is a favorite show. Typically don't touch the audio settings.

I have been usng mkv format, but am considering switching to mp4. Anybody have an idea of which might be more practical. Main use of the files is for playback on TV connected devices so I have been using mkv. Just wondering mp4 would give me any more flexibility.

[Dargason]

I'm glad you say that, because that's the direction I'm thinking of going.

So it would go like this:
Record show -> Comskip -> VRD to fix and remove commercials -> compress with VRD or Handbrake.

I tried the High Profile with quality setting RF=16 last night and the image looked good, but I had a corrupted timeline. I'm trying it again. The file was about 2/3s of the original size, which I think is pretty good. 5.1 Audio and CC were intact.

Is there a way to reconstruct a corrupted timeline?

Edit: Just did a quick search, and it sounds like VRD has a feature to fix a corrupt timeline... +1 for that!

[stanger89]

FWIW, to the poll question, I don't archive, my TV recordings I just let roll off after I watch them (ie they're deleted), for movies, I get more space.

I find anything worth "archiving" isn't of "archival quality" from any broadcast source. It's either cropped/pan & scanned, edited, has logos burned in, poor audio quality (dynamic range compression) etc. I find anything I have worth "archiving" is worth buying on Blu-ray and not having to deal with cutting commercials and all the above issues.

The other thing is with the advent of services like Netflix/Amazon/etc, it's even less of an issue. I used to "save" (turned off auto delete) Top Gear, but since Netflix and Amazon have every episode from seasons 2-17, there's really no point in me keeping my (inferior) copies anymore.

[fidget]

My archiving is shows that we are waiting to get on DVD and programs that we have yet to watch. Since it is so easy to record things, my wife's list of things yet to watch is growing and growing (probably about 1.5 TB right now, all HD Dish shows).

[ranger]

I do find that I am saving fewer and fewer shows because the availability through other sources like Netflix, but I still save a few because I'm too cheap to buy the DVD sets until they get discounted.

[david1234]

Originally Posted by stanger89

I find anything worth "archiving" isn't of "archival quality" from any broadcast source. It's either cropped/pan & scanned, edited, has logos burned in, poor audio quality (dynamic range compression) etc. I find anything I have worth "archiving" is worth buying on Blu-ray and not having to deal with cutting commercials and all the above issues.

The floating logos and the pop-up commercials over the top of shows drives me nuts. The worst offender is the ratings square that's stuck right in the middle, since they put it over the picture for a 4x3 crop. The other day we were watching something and it was right over the head of the guy who was talking!

If it weren't for the pop-ups and logos, what we record OTA is almost always better than what I get from DVD or Netflix's streaming service. Plus, as much as I really like Netflix, the I can't rely on it to ever have a specific show when we want to watch it.

For the couple shows we do hold onto, I just buy more disks for unraid, and move things over to the NAS.