Automating Sage recompression: An idea!

[ptaylor]

Currently, I'm manually recompressing the files that I want to keep (mostly stuff for the kids) using AutoGK... It does an amazing job of keeping the quality up while giving me Xvid files about 1/5th of the original size. It would work great as an automated tool, except for the fact that it doesn't support the command line (and it sounds like they don't plan to).

Tonight I was thinking about that a bit, and thought... "Why should that stop us?"...

I found a free tool that might help in relation to the GUI aspect... AutoItScript (http://www.autoitscript.com/autoit3/) looks like it might be a good fit. I imagine it wouldn't be terribly difficult to write a C# program that would pick up MPEG files out of a directory that Sage writes them to and write an AutoIt script, then execute the AutoIt script... The AutoIt script would open AutoGK, then set up a "job" in AutoGK for each of the files, then just hit the Start button in AutoGK...

Does this sound feasible to anyone? What would be the best way to integrate this into Sage? Or, should this just be something that runs as part of a scheduled task using a config file (to indicate the shows to recompress, the compression percentages, etc.)?

Paul

[nielm]

There are other command line recompressors (Mencoder, FFMPeg), but nothing like AutoGK for automatic cropping/rescaling/deinterlacing/inverse TeleCine and chosing a bitrate to fit etc. (on the plus side, if you don't care about exact file sizes, and if you are keeping the recordings on the HDD, MEncoder single pass is a lot faster than AutoGK)

The problem (at the moment) for automatic recompression is that there is no automatic way of replacing the .mpg with a .avi and then get sage to recognise that the .avi is the same show -- keeping it in Sage Recordings with show info. (There is a manual way -- see the HowTo in the customisations link in my footer)

PS: I have a Girder script to automate adding recordings to AutoGK v1.60, attached; usage:

event.exe -silent AutoGkEncodeFile 18 infile outfile [customsize]

[ptaylor]

Yes, if Sage would let us transcode transparently, that would be the best scenario...... I believe ShowShifter has this built-in... Not sure about Beyond TV (I own both apps, as I found them before Sage)...

Forgive me if the rest of this looks like a rant or something along those lines... I don't really mean it to be that way...

I've been looking at other alternatives lately.. I mean, Sage won the last time I looked around about a year ago... I'm just seeing that transcoding would be a great addition... About the only other thing I can think to do to get that in Sage is get one of those hardware encoders that takes it directly to DIVX.

Most recently, I've been playing with MythTV and learning more than I wanted to know about Linux along the way... It appears to support automatic transcoding, but the fact that so many programs are integrated means that it is a mess to get everything working "together", if you know what I mean... My current issue is that every time I reboot I have to delete and re-add my encoder card to get it to work for watching TV for more than a few seconds... MythTV seems.... infuriating..

MCE is very good looking and the fact that it supports remote clients and native HDTV is very appealing, but I don't like the fact that you don't have as much control over your data... Transcoding and such... I mean, there are lots of programs that I'd like to keep from PBS for the kids to watch, but I don't need them at 1 GB per half hour. I can transcode them to XVID at about 150 MB per half hour and it still looks decent. Of course, then there is the cost.. There is no legal way for me to try it out. I've found it on one popular enthuiast website for about $130, but I don't want to spend that kind of money just to play with it, especially since I'm not sure that I would be happy with the feature set.

I guess what bugs me the most about Sage is that it doesn't appear to be moving forward very quickly... I got it just after version 2 came out and I really can't think of any new features added since then that I use... I think most new versions were bug fixes and support for additional devices. (I could be wrong here though.)

Anyhow, at this point I think Sage still wins out over the competition, but not by the same distance that it did when I first got it... I wish there were more that I could do directly to add features to Sage, or that the Sage team would hurry up with version 3, which I imagine will have all the major features we've all been clamoring for.. Ok, I can dream can't I?...

Paul

[TiltRod]

nielm,

I started with SeaMonkey's divx encoding batch. Of course I modded it quite a bit for personal preferences.

With it I use the following for encoding and file replacement

1. VirtualDub 1.5.10 (for mpeg video/audio parsing & and avi (divx/wmv3) encoding)
2. GnuWin32 FileUtils (for changing the avi timestamps using the "touch -r" command)

Now here's the trick!!
using the "move" or "copy /y" commands change the extension to ".mpg". I prefer "move" because it's faster and you don't end up with double the drive space used.

Ex.
move "s:\mpeg4working\SaturdayNightLive-75396-0.avi" "s:\sage\SaturdayNightLive-75396-0.mpg"

I NEVER HAVE TO STOP/START/RESTART SAGE USING THIS METHOD!!!

I think that this is possible because:
1. Because the reencoded file is placed back in the Sage Recordings directory with the EXPECTED extension and timecode.
2. The avi is really just a container for the directshow enabled reencoded file.

Anyway, that is my 2 cents. Any/All comments gladly accepted.

-d

[alon24]

Can you share with us the full script??

[TiltRod]

sure.
Hope you can follow it. I have some external calls, and I am still trying to "unbloat" it. There are still a lot of SeaMonkey's notes included that don't necessarily apply to my current iteration.

set vdub=C:\Program Files\VirtualDub-MPEG2
set idir=s:\sage
set wdir=s:\mpeg4working
set odir=s:\sage
set bdir=s:\sagebackup
set oFile=avi
set profile=video.vcf
set cs2vd=comskip2vdub.vbs
set ldir=s:\sage
set trigger=mpg.xml
set shows=shows

for /f %%i in (%shows%.txt) do for /f %%u in ('dir /b "%idir%\%%i-*.mpg"') do if exist "%idir%\%%u.xml" if not exist "%ldir%\%%u.log" >> Processing.log echo %%u

@REM -------------------------------------------------------------------------------------------------------------------
@REM Comskip Files in Directory
@REM -------------------------------------------------------------------------------------------------------------------

for /f "skip=1 tokens=1 delims=." %%i in (processing.log) do if exist "%wdir%\%%i.mpg" (if not exist "%wdir%\%%i.log" wscript comskip.vbs "%wdir%\%%i.mpg") else (if not exist "%idir%\%%i.log" wscript comskip.vbs "%idir%\%%i.mpg")
for /f "skip=1" %%i in (comskip.log) do comskip "%idir%\%%i"

@REM -------------------------------------------------------------------------------------------------------------------
@REM Create VirtualDub Commercial cutting vcf files
@REM -------------------------------------------------------------------------------------------------------------------
for /f %%i in (commercialcut.txt) do for /f %%u in ('dir /b "%wdir%\%%i-*.txt"') do wscript %cs2vd% %wdir%\%%u

@REM -------------------------------------------------------------------------------------------------------------------
@REM Process Video Files
@REM -------------------------------------------------------------------------------------------------------------------
echo. Processing Video Files. Please Wait...
echo. --------------------------------------
dir /b "%wdir%\*.mpg"
echo. --------------------------------------
echo. Please Wait...

for /f "skip=1 tokens=1 delims=." %%u in (Processing.log) do if exist "%wdir%\%%u.mpg" if exist "%wdir%\%%u.vcf" (start /WAIT /BELOWNORMAL virtualdub.exe /s"%wdir%\%%u.vcf" -q /h /r /x /c) else (start /WAIT /BELOWNORMAL virtualdub.exe /s"%profile%" -p"%wdir%\%%u.mpg","%wdir%\%%u.%oFile%" -q /h /r /x /c)

echo.
echo. Finished Processing Video Files!

@REM -------------------------------------------------------------------------------------------------------------------
@REM this section cleans up the old files and moves the avi/wmv to the odir\date folder and sets Sage Time codes
@REM -------------------------------------------------------------------------------------------------------------------
echo. 
echo. Applying time Sage time Codes. Please Wait...
for /f "tokens=1 delims=." %%i in ('dir /b "%wdir%\*.%oFile%"') do if EXIST "%idir%\%%i.mpg" (touch -r "%idir%\%%i.mpg" "%wdir%\%%i.%oFile%")
echo.
echo. Sage Time Codes Set!

for /f "tokens=1 delims=." %%i in ('dir /b "%wdir%\*.%oFile%"') do if EXIST "%idir%\%%i.mpg" if not EXIST "%bdir%\%%i.mpg" move "%idir%\%%i.mpg" "%bdir%\%%i.mpg"
for /f "tokens=1 delims=." %%i in ('dir /b "%wdir%\*.%oFile%"') do move "%wdir%\%%i.%oFile%" "%odir%\%%i.mpg"

Conversion/Replacement Batch Text

[ptaylor]

Interesting thing I just noticed.... I've been discussing issues with loading a new OS on a separate drive in a thread on another forum here, and made this interesting discovery:

Here's the situation:
My original OS was on an old 8 GB drive. My media files are on a 160 GB drive and a 250 GB drive. Yesterday, I removed the 8 GB drive from the system (after backing up various required files to one of the media drives) and installed a 40 GB drive where I loaded XP MCE. I tried reinstalling Sage but could not get video to work. After trying to get Sage to work (play live TV) for about an hour or so with no success, I left the Sage installation just after installing it and then copying over my the Sage directory off of my 8 GB drive. (When I loaded Sage, all the settings were preserved and everything looked good, except no live TV playback, and hence no recording)...

I was ready to go to bed, so I swapped my original 8 GB drive back in, then verified that everything was working properly and set up scheduled recordings for two shows that my 20 month old son loves. They were recorded on one of my media drives...

This afternoon, after finding out that MCE has services that must be stopped to capture with any other app, I popped the 40 GB drive back into the system.

Now, the part I find interesting... When I go to the list of TV recordings, it shows up the two kid's shows that were recorded with the other OS... I thought that all the info about programs you had recorded was stored in one of those files on the C drive (where Sage is installed). I didn't recopy that info over - It is still using the info copied over last night.

I don't know how Sage figures out this info.... I would have thought it would be in a file in the main Sage directory. The other files with the same name as the .MPEG file (just different extensions) don't seem to contain anything special...

So, perhaps it is simply the fact that the file has the properly formatted file name and time stamp....

[Opus4]

Your recently recorded files were still in the same database as what you have on the other drive because you copied your existing wiz.bin. The ID number at the end of the file thus matches the ID in the database. The recent recordings would not have been recognized by the newer installation had you used different wiz.bin files.

- Andy

[ptaylor]

Originally Posted by Opus4

Your recently recorded files were still in the same database as what you have on the other drive because you copied your existing wiz.bin. The ID number at the end of the file thus matches the ID in the database. The recent recordings would not have been recognized by the newer installation had you used different wiz.bin files.

- Andy

Andy,

I think one of us is misunderstanding the other... (Just not sure which one of us yet)...

I copied the wiz.bin over last night from the 8 GB drive to the new fresh install on the 40 GB drive. I put the 8 GB drive back in last night and recorded two more shows... This afternoon, when I put the 40 GB drive back in, I did NOT copy the wiz.bin from the 8 GB drive again... It was still using the wiz.bin file from last night. That is why I was so suprised to see the shows just recorded this morning in the listing.

Paul

[Opus4]

Yes, I knew what you meant. You are still dealing with the same wiz.bin with the same EPG listings for that day -- those airings would have already been in the EPG for wiz.bin when you first made the copy

- Andy

[ToxMox]

Just to clarify to get rid of any confusion this is what happened:
Two different wiz.bin files both have same EPG data with same show IDs. One wiz.bin records the show and stores info for that show. Other wiz.bin restored. Sage finds recordings that aren't in the recordings database then finds that it has info in the EPG for that ID and adds it to the recordings as a recovery feature of Sage if it can find the matching ID.

[ptaylor]

Originally Posted by TiltRod

sure.
Hope you can follow it. I have some external calls, and I am still trying to "unbloat" it. There are still a lot of SeaMonkey's notes included that don't necessarily apply to my current iteration.

TiltRod,

I'm trying to get this to work, but having tons of issues... I downloaded the linked file, but there appears to be much more in it than your post... Would the elements in your post be what we should splice into a batch file?

I'm asking, because I see things like "away.bat" being called, but yet don't know what away.bat is... I don't have one

Should this convert.bat file go in the virtual dub directory, or the Sage directory? (Or does it not really matter)

Thanks,
Paul

[ptaylor]

Originally Posted by ToxMox

Just to clarify to get rid of any confusion this is what happened:
Two different wiz.bin files both have same EPG data with same show IDs. One wiz.bin records the show and stores info for that show. Other wiz.bin restored. Sage finds recordings that aren't in the recordings database then finds that it has info in the EPG for that ID and adds it to the recordings as a recovery feature of Sage if it can find the matching ID.

Thanks, ToxMox... I was trying to understand what was happening here and this clears it all up..

Now if I can get TiltRod's script working, I'll have pretty much what I was looking for...

[ptaylor]

Tiltrod,

Questions:

What is the shows.txt file for? From looking at the batch file, I am assuming that this is a text file containing shows that you want to have automatically recompressed.

What is the commercialcut.txt file for? I initially thought it would be the shows that you want to cut commercials for, but figured that must be wrong because it looks like we are running comskip on every file before we access the commercialcut.txt file. I would think we would read this file first to see if we needed to run comskip if that was its purpose.

Paul

[TiltRod]

ptaylor,

1. The call for "shows.txt" refers to a txt file containing the shows that I want to recompress on any given day.

2.a. The call for "commercialcut.txt" refers to a txt file containing the shows that I am confident have "good" comskip detection and for which I want to generate a specific virtualdub ".vcf". If an alternate ".vcf" is found for a given ".mpg" then it is substituted in the virtualdub task and the commercials are cut from the recompress.

2.b. All files will be comskipped.

3. The call for "away.bat" refers to an external batch that will be called when the %code1% variable is set to "away". Not to get into the specifics, but basically, if I am going to be away from home for a couple of days, i set this variable, and all recordings get recompressed.

The lines in the post were just to demonstrate "some" of the logic in an in place recompress/replace for SAGE. I did include the entire file, and understand that it may be hard to follow, since it has so many external calls.

I am not an expert by any means. So I would not presume to say that this is a plug an play solution. I merely provided the entire batch because it was requested, and would only use it as reference material for code logic purposes.

P.S. -- I haven't cleaned this up too much because I am working on translating it all into a vbscript.

[TiltRod]

BTW,

A "Shout Out" to SeaMonkey420 for the great start with this one!!

[ptaylor]

Ok... Another question then... I see that you are looking in the idir a file ending in .mpg.xml.. If it exists, you see if a file matching that name (but ending in .log) exists in the odir. If it does, you add that file to Processing.log...

Sounds good, but what's the .mpg.xml file?

Here's a cut of what I'm getting:

C:\VDub>for /F %i in (shows.txt) do for /F %u in ('dir /b "F:\InDirTest\%i-*.mpg"') do if exist "F:\InDirTest\%u.xml" if not exist "F:\outDirTest\%u.log" echo %u 1>>Processing.log

C:\VDub>for /F %u in ('dir /b "F:\InDirTest\AmericasTestKitchenFromCooksIllustrated-*.mpg"') do if exist "F:\InDirTest\%u.xml" if not exist "F:\outDirTest\%u.log" echo %u 1>>Processing.log

C:\VDub>if exist "F:\InDirTest\AmericasTestKitchenFromCooksIllustrated-3002673-0.mpg.xml" if not exist "F:\outDirTest\AmericasTestKitchenFromCooksIllustrated-3002673-0.mpg.log" echo AmericasTestKitchenFromCooksIllustrated-3002673-0.mpg 1>>Processing.log

[TiltRod]

the ".mpg.xml" is a file generated by the cayars stv. It is only created after the scheduled recording has finished. So, this is the file that tells the batch the recording is completed and triggers it for recompression.

Any file that "signals" the recording is completed could act as the "trigger". For instance, if you were to allow one of the cusom stvs to do background commercial detection, you could use one of those result files as the trigger. i.e. ".txt", ".vprj", ".log"

[ptaylor]

I'm running Cayars, but don't have those .mpg.xml files... Perhaps that's a newer version of Cayars than I have. I do have some of the other files that I could use as a trigger.

I'll try it out some more...

Paul

[TiltRod]

I spent a better part of the weekend translating my batch file into a vbscript.

It is by no means complete and the documentation is practically non existent, but if anyone care to take a look, here it is.

I still haven't hashed out the vbscript logic for a clean recompression to sage directory file replacement, so i still have one section that generates an external batch to accomplish this.

I won't be available for questions for about a week, so that should give you fine people enough time to look it over first.

Also, the functions are not necessarily in the order they are executed, so keep that in mind when looking this over.

recompression vbscript