TV.com automatic scraper

[roxy99]

Originally Posted by dvd_maniac

I have about 2 dozen of them in my Music section that only have .my files...no .properties files created for them at all. I have Jerkiwi's STVI that takes the .my file and imports it into the Wiz.bin database when the video file is first scanned by Sage.
So Sage is seeing the .my file and somehow seeing it as a Music file.

Another possibility is that the timestamp of associated media is not changed by the the .my creator. So when you try to rescan, Sage doesn't even see the .my file because the video file already was already there to begin with and all you did was to create .my files. Sage doesn't know to rescan unless the timestamp on the video file is altered. I have no clue why its ending up in the music section, but at least that's a logical explanation on why the video library is not being updated with the .my file.

The solution if I am correct in my assumption is to generate .properties files for these .my files for the simple reason that I want the time stamp on the video to change and force the video library to rescan the video. (see the converter I posted earlier). You should shut down all Sage services when running the .my to .properties converter since it is changing the video's time stamp.

[dvd_maniac]

I renamed or moved the imported video directory just so that didn't happen.

I do not believe this is a problem with your scraper though, as everytime I fix the .my files, rename the Imported Videos directory and rescan it... It seems to find other videos and put them in the Music section. So I think this is a Beta issue with how it seperates Music and Imported Videos.

I will continue testing for a few days before filing a bug report.

[roxy99]

Just wanted to thank everyone who is using the scraper for your encouragement and to let you know that I am sorry that promised updates have been late in coming. I've had a few important projects on the go lately and haven't had as much hobby time to devote lately.

Therefor I am making a call to any venturous people who know Python script to take the ball and continue this project. I thing gui development and an online metafile browser is the next step.

If anyone is interested, please send me a pm and I'll email you all my source code.

Regards

Roxy

[roxy99]

Come on I know there are some programmers out there. If you're a Java developer, you can do a GUI in Java and then implement some hooks to the Python code.

[megamojo]

Is there any way we could get the episode name in the title space in the .my file? I name all my shows with the show name, episode number, and title in the filename, like this "CSI 7x01 Creepy Crawlies" for example. Without any properites file sage displays the filename and I know exactly what I'm watching, which is nice. After scanning a few directories with your scraper and generating properties files now the title is exactly the same for every episode of a given show.

[roxy99]

Originally Posted by megamojo

Is there any way we could get the episode name in the title space in the .my file? I name all my shows with the show name, episode number, and title in the filename, like this "CSI 7x01 Creepy Crawlies" for example. Without any properites file sage displays the filename and I know exactly what I'm watching, which is nice. After scanning a few directories with your scraper and generating properties files now the title is exactly the same for every episode of a given show.

What happens if you don't generate properties files and just use the given .my files? The .my files use the same file name as the source video file. I think that what is happening is the properties file is missing the title information. See if the error happens when you simply generate .my files.

If the problem only occurs with the properties stage then I know that the .my to .properties converter needs to be fixed. Please correct me if I am misunderstanding the problem you are having.

[megamojo]

I think you are misunderstanding me. It happens with just the .my files, doesn't matter if I make a properties file or not. In the .my file, the space for title just says the show name e.g. "Family Guy" instead of "Family Guy 1x01 Death Has a Shadow." When I convert the .my files to .properties, it just uses what .my has for the title, and all my episodes of any given show have the exact same name displayed by sage instead of a name which includes the episode number and/or episode title. The .my files do contain the correct information in the space for episode name, it's just that sage only displays the entry for title in the OSD during playback. This is why I would like the full episode name to be in the title space if there is any way to easily do that.

[megamojo]

I understand that you're busy. If you want to examine my problem great, but don't worry about it too much. I installed the .my reader STV import so I don't have to worry about creating .properties files. Also, with this method I still get my nice filenames displaying in the OSD, so all is well. Thanks for such a great scraper!

[roxy99]

Originally Posted by megamojo

I think you are misunderstanding me. It happens with just the .my files, doesn't matter if I make a properties file or not. In the .my file, the space for title just says the show name e.g. "Family Guy" instead of "Family Guy 1x01 Death Has a Shadow." When I convert the .my files to .properties, it just uses what .my has for the title, and all my episodes of any given show have the exact same name displayed by sage instead of a name which includes the episode number and/or episode title. The .my files do contain the correct information in the space for episode name, it's just that sage only displays the entry for title in the OSD during playback. This is why I would like the full episode name to be in the title space if there is any way to easily do that.

That's because the title field in the .my file is only 'Family Guy'. We could always merge the episode title with the show title however that doesn't seem right when there is already an episode field header in the .my file. I'm afraid that this is Sage issue in the way the .my plugin is reading the .my file.

[megamojo]

Originally Posted by roxy99

That's because the title field in the .my file is only 'Family Guy'. We could always merge the episode title with the show title however that doesn't seem right when there is already an episode field header in the .my file. I'm afraid that this is Sage issue in the way the .my plugin is reading the .my file.

Agreed. Your way is absolutely correct, since there's a tag for both title and episode. I was just disappointed when i tested it on a couple folders and suddenly lost my OSD info. But, using the direct .my RW STVi from JREkiwi is working just beautifully for me. If only there was a way to automagically create .my files for all my movies now that your great scraper has flawlessly made them for all my imported TV shows.

[megamojo]

Here's a nice little one line batch file I created tonight. Just put one of these in each of your directories where you put TV shows and you can generate the metadata files just by running it instead of going up one folder level and dragging the folder onto the metadata creator shortcut.

echo.| "D:\Program Files\TV Metadata Creator\videometadata.exe" -p (%~dp0)

The "echo.|" at the beginning is so you don't have to press enter to close the cmd window, it's the pipe character (same key as backslash), not a lowercase L. The "%~dp0" for the path just means that it will use the directory where the batch file is located.

[millers_35]

I'm not sure how to get it to scrape the info for "Special" episodes. I guess I dont have the file names setup right for those files. The ones I am trying to do are from Lost and Ghost Hunters. Could anyone lend a hand?

The way I have my eps setup is.

--Lost
----Season 1
-------01 - Pilot(1).avi
----Season 2

and so forth. Here is the command I am using:

"C:\Program Files\TV Metadata Creator\videometadata.exe" -p (G:\Imported\TV Shows\lost) -t 2 -s 1

Miller

[kricker]

I just tried out the app and it's pretty cool. I have a couple questions and request.

When I run the app it seems to find things ok, but in the DOS window it always shows episode:0 for every show even though it finds the right info. Why is that? For reference my structure is as follows:

..\..\TV\Two and a Half Men\Season 05\12-A Little Clammy and None Too Fresh.avi

and my batch file run from the TV folder is:

echo.| "D:\Utilities\TV Metadata Creator\videometadata.exe" -p (%~dp0) -t 2 -s 1

It had issues finding "Human Body: Pushing the Limits" most likely because due to windows restrictions I had to name it "Human Body- Pushing the Limits". I fixed this by using the showid.txt method. Would it be possible to implement a switch or something for character replacement before the search happens? So the app will know to use a : instead of - when searching for a show title? I can see this happening for various show and episode names.

Do you plan on adding the ability to scrape the http://thetvdb.com/?

[kricker]

How can I look up specials? like listed at the bottom of this page.

I also noticed the scraper will not recognize "Top Gear" but it has to be "Top Gear (UK)."

[mherce]

I'm a newbie to SageTV and I'm giving this plugin a try in hopes it will replace my beloved MyTV MCE plugin.

I have a test.bat which I put in the directory I am testing it out on. It looks like this:

"D:\Apps\TV Metadata Creator\videometadata.exe" -p (%~dp0) -t 2 -s 1

This is what the windows looks like:

H:\Media\TV\Tin Man\Season 1>"D:\Apps\TV Metadata Creator\videometadata.exe" -p
(H:\Media\TV\Tin Man\Season 1\) -t 2 -s 1
Importing modules
Starting with forced parameters: t=2 s=1

Reading show titles...please wait
Season level: 1
-->Tin Man
Tin Man - Episode 01 - Ep 01.avi
EP TYPE->

Season level: 1
-->Tin Man
Tin Man - Episode 02 - Ep 02.avi
EP TYPE->

Season level: 1
-->Tin Man
Tin Man - Episode 03 - Ep 03.avi
EP TYPE->





VIDEO :
H:\Media\TV\Tin Man\Season 1\Tin Man - Episode 01 - Ep 01.avi

Loading episodes...
Looking up show title...please wait

Now it's been like that for 4 hours and counting now. There is a text file called vidlist.txt in the directory with the Tin man video files but its blank.

My library is organized like this:

"Drive root (H)\Media\TV\Tin Man\Season 1\Tin Man - Episode 01 - Ep 01.avi"

Some of my files have the actual episode name instead of the repeat of the Ep number.

Can anyone help me? How to I debug this app or get a log of what its hanging on? I've tried this with 4 different shows now all hang. I figured maybe it was just slow and left it while I went out. Howevere, 4 hours seems like way too long. I'm on a 5mbps download DSL connection.

As an aside, that my TV plugin for MCE also uses the tvdb.com and I find it to be a very nice site which is made for scraping.

Thanks!!

Edit: Now its been 8 hours and I closed it. Videolist.txt is still empty.

[mherce]

2 months later and I've got the time to play with HTPC software again. I got this working now, using "D:\Apps\TV Metadata Creator\videometadata.exe" -p (H:\Media\TV) -t 2 -s 1" .

I have to say, this plugin gives metadata hope to Sage. I find I wish that TV.com did movie info as well so i could use this on my movie library. If I end up going with Sage, I'll be using your donte button if you got one for some beer.

I've only got two issues which are not major and, if I stick with SageTV, I will just spider the page and use a text scraper to create the my files. So this is just a bug report kind of thing.

First is that Ghost in the Shell: Stand alone complex season 2 (aka 2nd gig) does not import. I've stuck in a showid.txt and it finds the show but doesn't match the episodes (Episode does not exist). I've got it so that:

H:\Media\TV\Ghost in the Shell Stand alone complex\Season 2\episode 01.avi
H:\Media\TV\Ghost in the Shell Stand alone complex\Season 2\episode 02.avi
....

Second, is not a bug but a feature. Specials make a big mess of things, it would be nice if the app would read a special tag and skip episode processing on the file and search for the show title instead.

Otherwise a great app and one sorely need by this program, thanks!

[Fuzzy]

Hey there, I'm not a big TV archiving guy, but I am curious about your scraper. Is there a way to scrape the info with Title and Episode name, instead of Season and Episode number? How hard would something like that be to do? I'm asking because of the method SageTV uses for its recordings is to use Title-Episode-ID.mpg. Adding this feature would allow pulling of tv.com data for tv recordings, and even better, allow semi-automatic reintegration of recorded shows after a wiz.bin crap-out. Whitespace would have to be regenerated, but I have been successful doing a search on tv.com using title and episode in the general search box. for example:

TheWonderPets-HeresOllieSavetheVisitor-565430.mpg

some whitespace parsing (adding spaces before capital letters and minor words (the, and, or, and, etc.) gives:

The Wonder Pets Heres Ollie Save the Visitor

Using that in the tv.com search string:

http://www.tv.com/search.php?qs=The+Wonder+Pets+Heres+Ollie+Save+the+Visitor&type=11&stype=search&tag=search%3Bbutton&om_act=convert&om_clk=search

yields the correct page.

Haven't tried it with too many other shows yet though.

[roxy99]

Glad to see my little app has generated some interest.

Maybe I'll dust off the source code and see what improvements I can make.

Anyone else have a wants list about what they would like with this scrapaer?

I remember talking about a gui interface... I don't know if that is what people want.

Anyway,

Just wanted to say hi!

[dvd_maniac]

Hi Roxy,

I noticed that TV.com has been updated and when I tried to use the scraper it seemed to connect and work fine but then everything Errored...
Here is my videolist.txt

Does this just need an update? I hope you have time to look at it because I love this program...

[megamojo]

I'm seeing the exact same behavior.