TV.com automatic scraper

[roxy99]

Check it out at: http://forums.sagetv.com/forums/down...do=file&id=188

-Completely Media player independent.

This utility doesn't require Sage and will allow you to prepare .my files based on videos in your video library that use a season/episode based naming convention.

Once the .properties is added to this tool, we'll be all set.



Enjoy

[roxy99]

While I find command line utilities easy to work with via shortcuts, many prefer a windows look. So I'll get working on a simple gui interface.

Still to come is the .properties file generator.

[Armondole]

This is fantastic! Great job Roxy99!!

[UFGrayMatter]

I might recommend using www.thetvdb.com

It was built for these reasons - so we don't piss off the powers that be.

It has a whole API to use. Meedio uses as well as Media Portal and an MCE plugin

[roxy99]

Well I could say mine was built for the same reasons. I could say 'why use MacOS when Windows was built for the same reasons...

But the point is its a done deal and I've already built it to scrape directly from TV.com anyway. (Also spent a great number of hours and sleepless nights). Now you have a choice. You don't need sage or Meedio for that matter. Its an independent utility for anyone wishing to scrape data from TV.com. Nothing more nothing less and not intended to replace Meedio or tvdb or Sage.

As well, this is just the beginning. I'll be adding more and more to it. I am open to feedback for features and improvements.

I may in the future offer the option for the user to choose from where to scrape- TV.com or tvdb.com.

[Armondole]

Originally Posted by roxy99

As well, this is just the beginning. I'll be adding more and more to it. I am open to feedback for features and improvements.

I may in the future offer the option for the user to choose from where to scrape- TV.com or tvdb.com. However, I don't see the point since all the info should be at tv.com. Why be dependent on tvdb if you don't need to be.

One reason you may want to keep tvdb.com as a future option would be for the downloading of banners and graphics. If it were possible to incorporate wide icons (wide banners) into the SageTV imported videos section, then you could use this to take advantage of the great selection of icons from tvdb.com. There are banners also available on TV.com, but not as many.

here's a screen of the banners that have been scraped using XBMC scrapers for TV.com and TVDB.com...

[roxy99]

Originally Posted by Armondole

One reason you may want to keep tvdb.com as a future option would be for the downloading of banners and graphics. If it were possible to incorporate wide icons (wide banners) into the SageTV imported videos section, then you could use this to take advantage of the great selection of icons from tvdb.com. There are banners also available on TV.com, but not as many.

here's a screen of the banners that have been scraped using XBMC scrapers for TV.com and TVDB.com...

Excellent suggestion. It would be nice to eventually get banners along with a gui for the scraper. Maybe I'll contact tvdb.com and ask their advice on the best way to interface with them.

[dvd_maniac]

Tried the new 1.0 version and ran it through the Buck Rogers episodes I've been testing the betas on.
It got the title, season and episode right but...It says NO Show ID for all of them and no .my file was created anywhere in the Buck Rogers directory..

VIDEO :
t:\TV1\TV\SciFi\Buck Rogers\02\2x11 The Hand of Goral\the hand of goral.mp4

Buck Rogers=> No show ID found
-------TITLE----BUCK ROGERS
-------SEASON --2
-------EPISODE--11
=========================================

VIDEO :
t:\TV1\TV\SciFi\Buck Rogers\01\1x03 x04 Planet of the Slave Girls\planet of the slave girls.mp4

Buck Rogers=> No show ID found
-------TITLE----BUCK ROGERS
-------SEASON --1
-------EPISODE--3
=========================================

[Armondole]

Originally Posted by dvd_maniac

Tried the new 1.0 version and ran it through the Buck Rogers episodes I've been testing the betas on.
It got the title, season and episode right but...It says NO Show ID for all of them and no .my file was created anywhere in the Buck Rogers directory..

VIDEO :
t:\TV1\TV\SciFi\Buck Rogers\02\2x11 The Hand of Goral\the hand of goral.mp4

I think the "TIP" section of the download/instructions page answers this question...

[dvd_maniac]

Originally Posted by Armondole

I think the "TIP" section of the download/instructions page answers this question...

I'm pretty sure that I got the -t & -s parameters correct since the Title, Season and Episodes are all correct...

[JREkiwi]

Originally Posted by dvd_maniac

I'm pretty sure that I got the -t & -s parameters correct since the Title, Season and Episodes are all correct...

From the Tips
In naming your title folders, be careful to use proper full tiltle names. visit TV.com and see the exact title. For instance, naming a show folder 'Buck Rogers' will not work and the utility will fail to retrieve the show id. The proper show name is 'Buck Rogers in the 25th century'.

John

[roxy99]

Originally Posted by JREkiwi

From the Tips
In naming your title folders, be careful to use proper full tiltle names. visit TV.com and see the exact title. For instance, naming a show folder 'Buck Rogers' will not work and the utility will fail to retrieve the show id. The proper show name is 'Buck Rogers in the 25th century'.

John

Thank you JREkiwi for covering for me and thank DVD maniac for beta testing. That's not really a bug bug rather a limitation of the way the title search feature works which is exactly why I placed that tip in the docs and is also the reason for a speedy update coming( I'll discuss in a minute).

I could have made the program search for all names with Buck Roger's in them. But if there is also a show just called Buck Rogers, then it will make a false identification. The same applies for Hulk and The incredible Hulk. Hulk is cartoon show that exists so you need to use 'The incredible Hulk. I don't want the program to mistake 'hulk for the incredible hulk so I have to be specific.

Which brings me to my upcoming update. The next fix will allow you to place a tiny txt file in each title folder named: showid.txt. In that file you put the actual tv.com show id. This will force the utility when it sees a showid.txt to override it's title seach and use the title id provided.

[roxy99]

Ok version 1.0.1.0 released now. Now you can be flexible with naming your title folders as long as you create showid.txt with the actual showid# per TV.com.

Thanks DVD Maniac for uncovering the limitation


I'm becoming more curious about this tvdb.com scraper. Armonpole, where did you get thos slick library screen shots? Is that simply Sage's import library or are you using some slick tvdb plugin with or something? My Sage Video library looks so boring comared to that. Is the screen shot you posted this Xbox media center that everyone is talking about?

In a nutshell- can Sage's video library (nonMC) be configured to show banners and such?

[Armondole]

Originally Posted by roxy99

Ok version 1.0.1.0 released now. Now you can be flexible with naming your title folders as long as you create showid.txt with the actual showid# per TV.com.

Thanks DVD Maniac for uncovering the limitation


I'm becoming more curious about this tvdb.com scraper. Armondole, where did you get thos slick library screen shots? Is that simply Sage's import library or are you using some slick tvdb plugin with or something? My Sage Video library looks so boring comared to that. Is the screen shot you posted this Xbox media center that everyone is talking about?

In a nutshell- can Sage's video library (nonMC) be configured to show banners and such?

I got the screen from XBMC (Xbox Media Center). That would be XBMC's equivalent to "folder" view for SageTV. The difference is that you can assign any graphic you want, and they have many different views under the main "folder" style view... for example, they have DVD view, Wide Icon, Square icon, Large Square icon... etc.

The wide icons you see in the screen are the ones downloaded from TVDB.com and/or TV.com. If there was nothing downloaded, the screen layout would look the same except instead of a nice graphic you would see the TV series' name.

Do a google search for XBMC and you will find tons of info about it. They already have icon packs for tv shows, games, dvd's... they refer to these as "wide icons".

Unfortunately I don't know the answer to your main question, "Can Sage's video library (nonMC) be configured to show banners and such?"

[justme]

Just wanted to say thanks for this, even though I haven't yet tested myself. Will do it this weekend. I like the more automated/flexible approach over the previous users TV.com scraper(used that alot).

One question though... I read your comments in the download sections about files supported. Does your scraper do .OGM and .MKV? I saw .mp4 and .divx but not these two containers. I used to do alot of OGM(that I still have archived) and now use MKV for almost everything that is compressed, so support for these would be cool.

Ohh, just thought of one more question. Can your scraper handle or be made to handle filenames that don't use the season, but just increase the episode number as each season goes on? ie. Season 2 episode 1(aka 02x01) of a show that has 13 episodes a season and at least 2 seasons would be named like: Show Title -E14- Episode Title
Often I won't even put the "E" in the file title like: Show Title -14- Episode Title.

Anyway, thanks for all you time and effort.

[roxy99]

Originally Posted by justme

Just wanted to say thanks for this, even though I haven't yet tested myself. Will do it this weekend. I like the more automated/flexible approach over the previous users TV.com scraper(used that alot).

One question though... I read your comments in the download sections about files supported. Does your scraper do .OGM and .MKV? I saw .mp4 and .divx but not these two containers. I used to do alot of OGM(that I still have archived) and now use MKV for almost everything that is compressed, so support for these would be cool.

.OGM and .MKV not yet supported but a very easy fix. I'll add those 2 extensions to the list and submit an update. Any other video extensions I'm missing, please let me know.

Originally Posted by justme

Ohh, just thought of one more question. Can your scraper handle or be made to handle filenames that don't use the season, but just increase the episode number as each season goes on? ie. Season 2 episode 1(aka 02x01) of a show that has 13 episodes a season and at least 2 seasons would be named like: Show Title -E14- Episode Title
Often I won't even put the "E" in the file title like: Show Title -14- Episode Title.

Anyway, thanks for all you time and effort.

That's an interesting one. So Season 3 episode 5 would be epsiode 44? As a matter of fact, that is how they are listed in TV.com. You'll notice that scraped .my files refer to season numbers exactly like that.

Presently my utility expects to see a Season # identifier so it won't work. But I can add an option switch for that. It may take a little work though because the regex may get muddled and bugger up the other detection formats. For instance, the regex is designed to accomodate a file name like
Show Title -105- Episode Title. Where 1 is the Season # and 04 is the episode #. If I add the new regex detection format, then this can be confused with a season 8 episode 1 (assuming 13 episodes/ per season- NOT always the case though)

So it gets messy. There are two options: I can rethink my regex expressions but it will take some time. Or alternatively I can add parameter switch that forces this a special case. I think automating it is risky.

[roxy99]

Just a note - Ver 1.0.1.1 , just uploaded. New features:

1. added support for ogm and mkv
   
   
2. New parameters :

• -c title (in lieu of -t 1 -s 0)
• -c season (in lieu of -t 2 -s 1)

Its essentially a shortcut way of saying the same thing: either videos are classified in folders by title (-c title) or by title/season (-s season)


Still working on this bug:
BUG: -t 1 -s 1 nor -t 2 -s 2 work properly. The season folder cannot be the same as the title folder. So a subfolder such as .../CSI Miami - Season 1/.... will not work unfortunately. You need to break the folder down to the title and season subfolder.
Temporary work around: Use -t 0 -s 1 (or -t 0 -s 2) and refer to Tip 2 below for creating a showid.txt file and place it in the location where the video files are saved.

Also taking due note of justme's suggestion about sole episode # without mention of season. Once these things are fixed I'll work on the gui.

[dvd_maniac]

This is working out great so far.
I am finally starting to organize my Video Import library with a proper naming convention so this program will work wonders.
I hope I get finished by the time you add .properties support...


Thanx for all your effort Roxy.

[justme]

added support for ogm and mkv

Thanks, I'll let you know if I think of any other obvious extensions.

That's an interesting one. So Season 3 episode 5 would be episode 44?

Wouldn't that be season 4 episode 5, using the example of a show that has 13 episodes a season. 3*13=39(last episode of season 3) so 39+5=44(or the 5th episode of season 4) I don't know whether that was a typo or I'm just doing a bad job explaining myself. If a typo don't worry, but just in case I'm explaining it badly here's a real TV.com example from the show Inuyasha.

Jinenji, Kind Yet Sad is the 4th episode in the second season of Inuyasha. Since Season 1 of Inuyasha had 27 episodes, that makes this episode 31(27+4=31). I used the filename of:
"Inuyasha-31-Jinenji, Kind Yet Sad.mkv"

So it gets messy. There are two options: I can rethink my regex expressions but it will take some time. Or alternatively I can add parameter switch that forces this a special case. I think automating it is risky.

Totally understand, and I don't want to make you more work. I'd be fine with a parameter switch or even a switch that was stored in the showid.txt file. I know you said that showid.txt should only be a 2-5 digit code inside, but I thought I'd mention it that option anyway. Since the way I'll likely use your app will cause me to almost always make a showid.txt for any dir I process. Did, or do, you ever think you may develop something like an ini(expanded showid.txt maybe) file(for a fallback user desired behavior without any switches specified OR a directly specified ini file by a single switch) as an alternative to all the switch options? I'm fine with the switches and prefer the versatility of CLI apps(far easier to integrate into many different uses rather than the one the author directly saw or intended), but I find ini files can be a very useful option. Then again maybe just sticking with the idea of adding the desired switches to shortcuts is simpler. Now that I've written it out and considering your next step is a GUI wrapper maybe the ini is something for the GUI not the CLI app. I'll leave all those decisions up to you since I can't claim to be a coder.

Thanks for taking my requests/suggestions into account in your effort. I look forward to watching this app evolve.

[roxy99]

Originally Posted by justme

Thanks, I'll let you know if I think of any other obvious extensions.

Wouldn't that be season 4 episode 5, using the example of a show that has 13 episodes a season. 3*13=39(last episode of season 3) so 39+5=44(or the 5th episode of season 4) I don't know whether that was a typo or I'm just doing a bad job explaining myself. If a typo don't worry, but just in case I'm explaining it badly here's a real TV.com example from the show Inuyasha.

Typo and bad math on my part. I think we understand each other though.

Originally Posted by justme

Jinenji, Kind Yet Sad is the 4th episode in the second season of Inuyasha. Since Season 1 of Inuyasha had 27 episodes, that makes this episode 31(27+4=31). I used the filename of:
"Inuyasha-31-Jinenji, Kind Yet Sad.mkv"

I got the picture.

Originally Posted by justme

Totally understand, and I don't want to make you more work. I'd be fine with a parameter switch or even a switch that was stored in the showid.txt file. I know you said that showid.txt should only be a 2-5 digit code inside, but I thought I'd mention it that option anyway. Since the way I'll likely use your app will cause me to almost always make a showid.txt for any dir I process. Did, or do, you ever think you may develop something like an ini(expanded showid.txt maybe) file(for a fallback user desired behavior without any switches specified OR a directly specified ini file by a single switch) as an alternative to all the switch options? I'm fine with the switches and prefer the versatility of CLI apps(far easier to integrate into many different uses rather than the one the author directly saw or intended), but I find ini files can be a very useful option. Then again maybe just sticking with the idea of adding the desired switches to shortcuts is simpler. Now that I've written it out and considering your next step is a GUI wrapper maybe the ini is something for the GUI not the CLI app. I'll leave all those decisions up to you since I can't claim to be a coder.

Thanks for taking my requests/suggestions into account in your effort. I look forward to watching this app evolve.

Hmm. An ini file is faster to develop and less of a pain in the derier to debug. I want to do what users feel is more value added. Anyone else? Should I make an ini or a gui next? PS- someone is helpong me on the .properties addon. I figure a month or so before that's ready.