Recording Sage Recordings to DVD Burner

[taltman]

Hi Kjake,

Dude you rock! Let me know if you need help with any scripts. I don't really have that much time to contribute but I am a full time sysadmin. I could probably help you write up a script (batch or vbscript) or even some java code if you need.

Also - someone mentioned early on in this thread that we should be cutting out commercials. Have you made progress in this front? It seems easy to do provided you have a tool that can parse the text files and cut out the frames. It seems to me the best way to cut the frames is from the end...in other words if there are commercials from segements 100 to 200, 500 to 900, 1520 to 1780 for example you should cut out the frames from the end and repeat the process, otherwise all your going to have to re-index all your frames....so I would do the following:

cut frames 1520 to 1780 first
then 500 to 900,
then 100 to 200....

Does that make sense? Hope all is well. Good luck and let me know how I can help - although moving at snail pace compared to your progress =)

[KJake]

Thanks taltman
I have thought about cutting out commercials. I do that by hand whenever I burn a DVD, and although MpegVCR makes quick work of it, I'd like to automate it too. I've breifly looked at the comskip application here in the forums and I'm thinking that we could adapt it easily to this project. I'll start playing with it tonight, but I've heard that it's not quite 100% accurate yet, so we'll see how testing goes.

Off to work

[justme]

I've also been thinking using Comskip...

I think it's a good idea. Even though it won't be 100% accurate i think we can eventually correct some of it's missmarks manually. What I was thinking of, is using Comskip as it at first. We know from Cayar's STv that we can mark sections as keep or skip based on comskip feedback. These would be the dafault edits.

Now as for maually correcting some small mismarks by Comskip. it seems to me that if we look at the comskip data more closely we should find lots of black frame marks that were not determined to be commercial breaks. If we can get this data into the Sage UI then a user could go to the area where comskip miscalculated and then allow the user to step the marker forward or backwards using the unused black frame detections. In most cases this should allow the user to find a point very close to the true start/end of a cut segment. It won't be perfect or dead accurate but if you want that kind of accuracy then you should go outside of Sage's UI, IMHO.

I haven't really looked into all the details of this. ie would comskip be able to provide the correct info in one pass or would a second specialized "editting" pass be needed. Anyway I just wanted to throw it out there for people to think about. It seems doable at a glance, rather it's worth the effort or not is another issue entirely. Still such a detailed/user controlled editting operation is most likely going to have to wait until the full basic of this DVD proccess becomes a reality.

my 2 cents,
justme

[KJake]

I think this was a bit of a challenge, but it seems to work so far.
I've not fully tested it all because I don't have the time/paitence to wait for it all to run through the entire job. All of my test files are small clips that don't have any commercials in them. I tested comskip on 1 ~800Mb file so far and it was almost correct. It wasn't dead wrong though either. It just missed a portion of a set of commercials, but it got the other 2 commercials in the 30min segment almost 100% dead on frame.

I've got this setup to call comskip, then feed this to videoredo, and then demux and rejig for DVD compliance. It should be that easy... I'm going to see how easy it will be to add in the 'cutlist' approach for womble mpegvcr too so that the users have a bit more choice since no one makes a good free mpeg2 cutter.

I'll finialize more tomorrow night to where it should be working essentially. Sorry if I dissappointed anyone today I just am tired after staying up so late last night.

Hey, and then let's go hog wild and throw in Rejig while we're at it and dynamically shrink the final product down to something that'll fit onto a DVD-R...yeah, no. Sorry, I looked at that when I began this. I'm not sure if we can make that part work.

[Shuyun]

I'm very excited about this project and think you’re doing great work KJake. Here is my 2¢ as a part time programmer and full time sys admin. Get the basic stuff done first. Get the menu system working, plugged into sage and burning. This is the core of your project, the other stuff like auto commercial deletion and auto compression is a great thing to have but it can be added later and these add-on parts can sometimes be more difficult than the original project. This way the program is a little tighter when you start bug hunting after release and once you get the bugs down you know any further problems are caused by the new additions. Also the commercial skip is a neat feature but it is no where near 100% as to finding all of the commercials. For some shows it misses commercials and others it marks part of the show as a commercial and this is something that a lot of people have spent a lot of time on as I’m sure you know. What about instead of auto deleting the detected commercial you put in chapter points as a start. This way when a commercial is detected you can chapter to the end of the commercial break. Later when commercial detection is 100% you could set them to auto delete.

This is just my 2¢ so feel free to disregard it and by no means do I mean any disrespect. I think you are doing a great thing and look forward to using your Sage recordings to DVD burner program.

Thanks,
Shuyun

[KJake]

Originally Posted by Shuyun

What about instead of auto deleting the detected commercial you put in chapter points as a start.

This was something I thought about today. Doing this would definately push commercial marking in the current alpha out of the release until we get it finalized. I like that idea because then we're not getting rid of commercials that should 'technically' stay in there.

Also, I'm using the untested .61 version of ComSkip, not the .2x version in the melorebeta.zip in the commercial detection thread. .61 is much better at detecting commercials, but I've only tested 1 file so far. If I don't see that it's going to be worth while, I'll scrap the idea for now and put it on the TODO list.

Thanks for your interest, I appreciate it. We're working hard to get something out to the public soon. As always, keep watching this thread for any updates.

[KJake]

Hey guys - wanted to give an update tonight.

I have the comskip/videoredo portion worked into the process, but I don't really like it, so I'm going to leave it commented out for now while we focus on the main thing at hand.

Since I last updated, headway has been made.

We now have an STV in the works that is outputting XML for the system to read in. The structure of things have been changed to make integration into Sage easier too.

Right now, I don't have to manually come up with XML files to test the system, so that part is working well. There is still some focus up front that needs to be spent on some more XML file work, but after that, it will mostly become a project to develop the STV integration to allow the user to interface to this system.

Also, I did think more about Rejig and this would be doable in certain cercumstances where it's easier to fit in (i.e. big giant TV movie that is 6Gb could be shrunk to fit on a DVDR). So, that's on the TODO list still...but probably won't be in the first public release.

No pretty pictures either Sorry

-KJ

[abasu2003]

may i ask who's STV?

[KJake]

Right now, it's just a mod to the OriginalV2. Hoping to get some help down the road to pull the mods out as a plugin/import. It is being made with portability in mind so that hopefully anyone can either use it as an Import STV or it won't be hard for the STV maintainer to include it in their next release.

I'm not sure if who's helping me wants it to be known yet...so, I'll leave it at that. They'll speak up if they want to.

[taltman]

KJake,

I was thinking, rather than cutting out the commercials - why not make dvd chapters at the ending of the commercial break? This way you can skip to the next section of the movie...

I would prefer burning to dvd-r. Dual layer burning might be interesting later on...for now I think a majority of people are interested in DVD-R recordings.

Maybe if we can focus on just creating the VOB, INFO and BUF files we can then leave it up to others to use their favorite dvd burning/authoring tool. Building the chapter indexes and cutting out the commercials might be a nice touch for phase 2, or phase 3 of the project.

/Tal

[KJake]

taltman, I totally agree.

I should be able to do the chapter skips in this first phase. I can just read in the output from comskip and then calculate the time code. (Should be 29.97 frames per second for NTSC, so 2000 frames should be 2000/29.97=00:00:66:73 HH:MM:SS:ss. This will be fun to do...) Then since I need to write out the dvdauthor.xml file dynamically anyways, I can just add in the chapter points. I'll see how it goes, if it ends up holding anything up, than I'll put it to the side for later. Also, as you can figure out here, 'phase 1' will be for NTSC. I'd like to get this to do PAL at some point too because I know PAL users are out there.

Also, yes, burning to a DVD-R is already possible with this. The structure is created, and ISO is made (UDF DVD-video standard) and then the ISO is burned to the user supplied drive letter (e.g. D:). So that part is done already :)

More to come soon!

-KJ

[KJake]

An update on how things are fleshing out.

Squashed a lot of bugs here and there and we're able to get some good XML data now, so the menu(s) have/has changed a little to provide some more info to the user to help them select their choice and for nicer archiving.

Basic XML generation of the show info is completed I'd have to say. The only thing that might need to be done is to tweak formatting down the road.

Moving on to more complicated XML writing this week to finish the automation of the scripts that make the DVD in the background.

Here's a look at the new menus (ignore the colors, I encoded this fast):
www.kjake.net/sage/newdvdmenu.wmv

You can see on the Top Level that now you see this:
Title: Episode (if exists) Date Recorded HH:MM duration

Also, at the SubMenu, Episode has been added and line breaks are put in automatically to format the description text. Duration was modified and the Quality setting was dropped due to a limitation. The entire block of details have moved to get out of the way of the description too.

you'll also see a short blip of a recorded video play. It is meant to show that when the video ends, it returns to the Top Level menu, but the encoding screwed it up.

[alon24]

Originally Posted by KJake

An update on how things are fleshing out.

Squashed a lot of bugs here and there and we're able to get some good XML data now, so the menu(s) have/has changed a little to provide some more info to the user to help them select their choice and for nicer archiving.

Basic XML generation of the show info is completed I'd have to say. The only thing that might need to be done is to tweak formatting down the road.

Moving on to more complicated XML writing this week to finish the automation of the scripts that make the DVD in the background.

Here's a look at the new menus (ignore the colors, I encoded this fast):
www.kjake.net/sage/newdvdmenu.wmv

You can see on the Top Level that now you see this:
Title: Episode (if exists) Date Recorded HH:MM duration

Also, at the SubMenu, Episode has been added and line breaks are put in automatically to format the description text. Duration was modified and the Quality setting was dropped due to a limitation. The entire block of details have moved to get out of the way of the description too.

you'll also see a short blip of a recorded video play. It is meant to show that when the video ends, it returns to the Top Level menu, but the encoding screwed it up.

I am guessing that you only support ISO-8859-1, would it be possible for you to support ISO-8859-8 also (both in xml header and in writing the output stream to disk).

without this I would loose all the info that sage holds for me since I use Hebrew.

Cayars Stv supports Changing of xml header only (for compression), but we (hebrew and other langs), need the file to be written in the same encoding, so that it can be read as hebrew and not only xml encoding header which still writes the hebrew as jiberish).

I could supply a small code sample that does this thing exactly in the hopes that you will incoparate this into you'r plugin,.

also I suppose that since creating the xml and only then after we have a set of files, and the xml will hold all data, it would be possible for me to edit both the xml file and the file names in order to make the file names more readble (on the dvd), is that a resnoble assumption???

[KJake]

Originally Posted by alon24

I am guessing that you only support ISO-8859-1, would it be possible for you to support ISO-8859-8 also (both in xml header and in writing the output stream to disk).

We could make the XML writing configurable, but you are correct...currently it is set to ISO-8859-1. What about UTF-8? That's supposed to be universal, right? Also, I'm not sure if the output stream is or needs to be set to a certain charset...or are you talking about the file names?

Originally Posted by alon24

also I suppose that since creating the xml and only then after we have a set of files, and the xml will hold all data, it would be possible for me to edit both the xml file and the file names in order to make the file names more readble (on the dvd), is that a resnoble assumption???

I'm not sure that I know what you mean here. Do you mean you'd like to have hebrew text on the DVD screen? I think that is possible because the text is created via a TTF font file, so just drop in your language's version of the TTF (if it exists), otherwise it will use the Arial font on your system.

[KJake]

I'm taking a small break from all of this for the holiday season. I'll start up on it again next week and I'm brain storming about changing this hodge podge of batch files into a single .exe with a .ini for configuring. The .ini could be seperate, but I think I'd like to integrate it into the sage.properties since a configuration menu could be built for the DVD stuff and then writing things out to the properties file would be easy.

Thoughts, comments?
-KJ

[alon24]

Originally Posted by KJake

We could make the XML writing configurable, but you are correct...currently it is set to ISO-8859-1. What about UTF-8? That's supposed to be universal, right? Also, I'm not sure if the output stream is or needs to be set to a certain charset...or are you talking about the file names?

I do believe that u need to write the output stream with the proper encoding or else it will not work, I've attached a small piece of code to illustrait this.
UTF-8 is ok, but again I believe u'd have to do the same trick with the outputstream, also, I input the data to dage with Cp1255 (equal to ISO-8859-8) and sage shows it very well.
File names is something I would love to change, I just don't think that its relevant since I guess u do a compress and create one video file, so file names should not matter - right?

Originally Posted by KJake

I'm not sure that I know what you mean here. Do you mean you'd like to have hebrew text on the DVD screen? I think that is possible because the text is created via a TTF font file, so just drop in your language's version of the TTF (if it exists), otherwise it will use the Arial font on your system.

Hebrew text is displayed well with Arial font. I was thinking that before the process starts(the actual burning), I could change the text that is going to be diplayed, if its giberish, I would love to be able to change it).

also would it be possible to support RTL (Right To Left) so I can read my text from right to left, since in sage today it's LTR and I flip everything.

If this is possible then I would parse the XML file before starting the actual job, and reverse the hebrew again(for RTL purposes) - also in reversing I replace "(" for ")" and other tricks.

Have a nice holiday and merry chrismas.
Ilan

[KJake]

yeah, this is a little over my head right now...but I would be more than happy to work with you to figure this out and implement it. I think a lot of this is fairly new to Sage since I think I8N just came about in version 2.1....or at least noticably.

I'm going to try and get this past Alpha state and then start releasing things to a new thread so that people can play with it. I'm more than happy to share my scripts and any Java that I'm writing. The more people interested in this, the better it will be.

[alon24]

any news???

[KJake]

Hi there Ilan. Sorry, no news. I ended up taking a vacation from the whole project and have not gotten back into it yet. I've got another big project outside of work right now that will be partly finished on Thursday...hopefully the 2nd part of it will be finished the week after. I'll try and find time in there to work on this, but for now, it's on hold.

[alon24]

Originally Posted by KJake

Hi there Ilan. Sorry, no news. I ended up taking a vacation from the whole project and have not gotten back into it yet. I've got another big project outside of work right now that will be partly finished on Thursday...hopefully the 2nd part of it will be finished the week after. I'll try and find time in there to work on this, but for now, it's on hold.

ok, i am waiting pattiently
is it time yet?
is it time yet?
is it time yet???

:-)