TV.com automatic scraper

[roxy99]

Originally Posted by Armondole

I'm sure most people just don't understand the potential behind it. Wait until you release the GUI version... anyone who knows how to use .my files and downloads TV shows will surely put this to good use.

Hopefully I can figure out the .properties file creation... i'm a pure newb with this stuff and I haven't programmed in 10 years, so it might take me a while just to get started.

gr8 that'll free me up some time to get the gui going.
PS- Congrats on your upcoming wedding.

[dvd_maniac]

I think that when the .properties file part is figured out it will have a little more appeal. Right now I have a whole bunch of .my files but they are not searchable, which is the holy grail of an Imported Video library(IMHO).

[justme]

Fixed timeout issue

Your fix worked in all 5 of my tests. The only thing is I feel like it's slower(especially at the start of a dir process, I guess your buffering), but that's no issue compared to a full/successfull .my file creation. Still maybe a message about what the app is doing when it seems to be stalled would put new users at ease. ie

"Started proccessing"
"Looking for episode matches this may take some time depending on your internet speed"

Another possible suggestion about more feedback: Have the app report if all valid video files got a .my created for each Dir. ie.

for an error(missed 2) on a dir of 26 episodes:
24 .my file created
2 video failed to be recognized

For a successful run on the same dir:
26 .my files created
All valid video files now have .my files

Of course the actual wording is best done by you. And for both this and the processing message above could be turned on or off by a verbose switch.

Fixed Capitalization of .my files

The Caps issue is also gone.

I noticed you're not using S__E__ syntax but only the episode # syntax. For now, the utility requires a season# to work. All of your episodes return season=0.

As for my wierd file structure... I know it's nothing like what you guys are setting up but, with your app assuming a season 0(all episodes listed) it seems to work as I'd like. So I'm very happy with it. I do need to run some tests on some mutliseason shows and see how it does. I'll get back to you on that, but I think it will be fine as long as it assume Season=0 like you stated.

If you are choosing one show folder at a time, then you're missing half the power of the app.

Understood and I'll likely get more adventurous as time goes on but for now I'm going slow.

I like you gui wrapper idea. The only thing is that remember the utlity will actually convert an entire folder tree including all subfolders. So If I have a parent folder for SciFi, then all shows within the parent (say u:\tvshows\SciFi\...) will be converted. So I don't want the user to drill down to the file level. The user will only click 'SciFi' and by default all subfolders are included (until I add a CLI option to surpress).

As for the GUI, your logic seems valid to me. I guess my thinking came about due to my different way of dir structure. I'll see where you go and maybe adapt if needed(none needed so far).

About the most recent version(I didn't check this type of dir on older versions) I did notice that it seems to fail on a Dir with spaces in the name.ie "M:\VIDEOS\Anime\El Hazard The Magnificent World"
Returns "Invalid Path-- Enter valid path :"
I don't know whether you'd call this a bug or if it's another issue of my unique file structure. Just thought I'd mention it.

As for the rate your app is being downloaded, I too think the GUI and properties files will change alot of that. I'll tell anyone I can think of about your app(both in the Sage comunity and out).

Another request/suggestion:
An option to allow the .my to be created in an alternate location. I know this sounds strange and would cause issues with the current STVs, but... I'm thinking of cases like where the video files have been burnt off to DVD and the app would have no write access. I know I have alot of these from years ago that I'd like to genrerate .my files for. I plan to ask(or maybe try myself) the author to Mod the STV import to handle .my files in an alt location if none are found in the current dir.

One question about your apps handling of the video files name. Does it determine what Episode Title's data to use for each file just based on the Season switches(or showid.txt in my case) and the episode number listed in the video file name? Or does it also take into account the actual video file's episode title name as listed in the filename? In other word does "Escaflowne-01-A Very Fateful Confession by the hero.ogm" produce the same episode title .my file(allowing for different names of resulting .my files, but the internal data being the same) as "Escaflowne-01-Fateful Confession.ogm"? In the example above the second one is the "real" episode title and the first one is a "made up" one I'm using as an example. I'm asking this because my anime shows all get their filenames from anidb's official english episode title database and this is occasionally diffferent from TV.com's episode title name. Still the episode number is the same.

Thanx for the new version.

[Armondole]

Roxy,

I tried the new version and it seems to work perfectly. I have only had a chance to use it on one show at a time so far (with 3 seasons in seperate folders)... all episodes were handled perfectly.

One thing I find interesting is the order in which they are processed... it appears to be somewhat random. ie: S01E23 was the first file processed, then came S01E16... This doesn't cause any problems for me, other than a little bit of searching for errors in the videolist.txt file... any reason for it?

-A

[JREkiwi]

Roxy99, I've had a bit of a play and it's looking good.

One minor thing is that in a list of multiple writers there is an extra space following the first name and before the comma delimiter.

TVWriter=Jonathan Lynn , Anthony Jay

All the other fields don't have that problem.

John

[dvd_maniac]

From a .my file produced by the scraper app.

Stardate: 47988.0Captain Picard is shifting through three time periods: just before theEnterprise-Dis sent

For some reason every .my file produced for ST-TNG has no blank space before or after the word Enterprise and after the stardate # . I checked the tv.com website here: and the spaces are fine. I've check a bunch of them and they're all the same.

I've only done a couple of shows so far.
Has anybody else run into this with other shows?

[roxy99]

Originally Posted by dvd_maniac

From a .my file produced by the scraper app.

For some reason every .my file produced for ST-TNG has no blank space before or after the word Enterprise and after the stardate # . I checked the tv.com website here: and the spaces are fine. I've check a bunch of them and they're all the same.

I've only done a couple of shows so far.
Has anybody else run into this with other shows?

That may be because of the italics formating of the 'Enterpise' . Very interesting probably it only happens with italics. The program doesn't see enterprise put only text strings and the only difference is that 'Enterprise' is in italics format. Keep an eye out for it and keep me posted.

[roxy99]

Originally Posted by Armondole

Roxy,

I tried the new version and it seems to work perfectly. I have only had a chance to use it on one show at a time so far (with 3 seasons in seperate folders)... all episodes were handled perfectly.

One thing I find interesting is the order in which they are processed... it appears to be somewhat random. ie: S01E23 was the first file processed, then came S01E16... This doesn't cause any problems for me, other than a little bit of searching for errors in the videolist.txt file... any reason for it?

-A

No reason for it other than that'he way windows is fetching the file list. I could have performed a pre-sort but that would add extra overhead for nothing.

[roxy99]

Originally Posted by JREkiwi

Roxy99, I've had a bit of a play and it's looking good.

One minor thing is that in a list of multiple writers there is an extra space following the first name and before the comma delimiter.

TVWriter=Jonathan Lynn , Anthony Jay

All the other fields don't have that problem.

John

Easy fix. Thanks.

[roxy99]

Version 1.1.1.4 just uploaded. No more pesky -t -s parameters.

The utility is getting smarter now. Just need a -p (path) parameter.

[dvd_maniac]

Hi Roxy,

I'm running into a problem with a few of the shows I've tried this on. South Park is one that is giving me tons of issues.

Basically, I'm still getting the Failed to generate TV data error message for some episodes. For some episodes it seems that I could run it 100 times and it still won't work.
I did ST-TNG all 7 seasons and no matter how many times I tried it. 3x03 and 7x17 would not work.
I've attach the videolist.txt file for your review.

[roxy99]

Originally Posted by dvd_maniac

Hi Roxy,

I'm running into a problem with a few of the shows I've tried this on. South Park is one that is giving me tons of issues.

Basically, I'm still getting the Failed to generate TV data error message for some episodes. For some episodes it seems that I could run it 100 times and it still won't work.
I did ST-TNG all 7 seasons and no matter how many times I tried it. 3x03 and 7x17 would not work.
I've attach the videolist.txt file for your review.

I think Ver 1.1.1.4 may work for you. As a bonus, no more silly -t -s parameters needed.

Edit: Make that. 1.1.1.5. I keep pumping them out!

[justme]

Version 1.1.1.4 just uploaded. No more pesky -t -s parameters.

The utility is getting smarter now. Just need a -p (path) parameter.

Just downloaded and ran the newset version. I like the feedback as it is processing the files. Thanx for that. This app is definitely getting better/smarter with each version.

I still do have an issue with spaces in a path. Maybe it's how I'm using the app. I'm just running the setup and using the shortcut it's creates on the desktop. I click the shortcut to run the app and then drag and drop a Dir to process. This works great as long as the dir doesn't have any spaces in the path name. However, it fails when the path contains spaces in the path name(it does put quotes around the path BTW). ie "M:\Videos\Anime\Cowboy Bebop"
cause the app to return
"Invalid path-- Enter valid path :"
in the cmd/dos window.
However if I drag the parent dir "M:\Videos\Anime" it seems to start to process the "Cowboy Bebop" dir and its files. I only say seems since this is a large dir and it's not done yet(processing while I type) but I've seen feedback on a few of the Cowboy Bebop files.

Much thanks and I'll post a log if you need it.

PS:Any feedback on my request for you to add an option for the creation of the .my files in a seperate location? If you're not interested don't worry. I only ask because JREkiwi was soo fast in posting a new version of his STV import, for Reading .my files, to handle this type of situation.

[roxy99]

Originally Posted by justme

Just downloaded and ran the newset version. I like the feedback as it is processing the files. Thanx for that. This app is definitely getting better/smarter with each version.

I still do have an issue with spaces in a path. Maybe it's how I'm using the app. I'm just running the setup and using the shortcut it's creates on the desktop. I click the shortcut to run the app and then drag and drop a Dir to process. This works great as long as the dir doesn't have any spaces in the path name. However, it fails when the path contains spaces in the path name(it does put quotes around the path BTW). ie "M:\Videos\Anime\Cowboy Bebop"
cause the app to return
"Invalid path-- Enter valid path :"
in the cmd/dos window.
However if I drag the parent dir "M:\Videos\Anime" it seems to start to process the "Cowboy Bebop" dir and its files. I only say seems since this is a large dir and it's not done yet(processing while I type) but I've seen feedback on a few of the Cowboy Bebop files.

Much thanks and I'll post a log if you need it.

PS:Any feedback on my request for you to add an option for the creation of the .my files in a seperate location? If you're not interested don't worry. I only ask because JREkiwi was soo fast in posting a new version of his STV import, for Reading .my files, to handle this type of situation.

Try 1.1.1.5 Just uploaded. But I don't have that issue. Please post your log because that one has me stumped about the spaces issue. the parenthesis surrounding the pathname (pathname) is supposed to take care of that.

Please modify your shortcut under target. You need to specify a -p (pathname) in the Target section of the shortcut to the .exe.
(No drag/drop for now. Coming soon though...)

[justme]

O.K. I found a difference in behavior. I did what you said and modified the shortcut and then did a real drag n' drop of a "problem" dir("E:Anime\Trinity Blood") and it started processing the dir as expected.

The behavior I was describing before occurs when I double clicked the shortcut. That then opened up the cmd/DOS window with the app displaying:
Please enter path to scan :
If I then drag the path with spaces to this open window I got the error. But using it's parent path worked. I'm sorry I wasn't clearer about how I was using(not drag n drop onto the shortcut but into the already active/open window) the app when it errored earlier. I've included a zip file with a log of a dir it processed and a screen cap of what is going on. The shortcut(the default install desktop shortcut) I double clicked is circled in red. It also has the cmd window that app opned and then the displayed error resulting from dropping(into the already active window) "E:Anime\Trinity Blood" and hitting "enter". It also has me then dropping(into the already active window) "E:\Anime" on it, hitting "enter" and it start proccessing. I noticed the app(OR was it WinXP) didn't add quotes to the second path, not that it needed to, just pointing out it is noticing(OR was it WinXP) a difference but not adjusting in some way to the first path. The log file is from the pictured run. Don't worry about it failing on the other show in the dir, I made a smaller test dir for this with a show known to TV.com and one TV.com had nothing on.

BTW:Small note the installer for 1.1.1.5 says 1.1.1.4 when it goes thru the setup. Bo biggy but it made me redownload the app 2 times just to be sure. Then I finally thought to check the file size/CRC with the 1.1.1.4 I downloaded earlier. It was then obvious this was a new version.

[roxy99]

Thanks for the screenshots. It will help solve a few misconceptions.

You cannot simply drag/drop onto the TV Metadata shortcut. The parameter can only be set in the shortcut under target. As you can see from my screen shot, I right clicked on the TV Metadata Creator shortcut and added the -p (path name) after.

By drag/drop the application never gets the -p switch that it needs. That's why you're having problems.

'RE: 'Mezzo Forte' try to find out the exact TV.com showid and create a showid.txt file. It seems TV.com doesn't have it listed under Mezo Fort.


PS- Sorry about forgetting to change the Ver # to 1.1.1.5. It was a last minute change.

[roxy99]

Thanks to Justme's idea, now drag and drop show folder onto the shortcut directly!

Thanks also to DVD Maniac for vigorous testing and everyone else.

[justme]

Thanks for the screenshots. It will help solve a few misconceptions.

One thing I've learned about myself... If there's a way to use a program in a way the author didn't intend, that's the default way I'll use it. I just have a knack for doing things differently than everyone else. So don't worry about misconceptions, I'm just weird.

Thanks to Justme's idea, now drag and drop show folder onto the shortcut directly!

Just downloaded and tested this version and drag n' drop onto the shortcut(default one your setup made) does work with paths that have spaces in their name.

One thing I did notice is that double left-clicking the shortcut(default one made by your setup) no longer opens a window for input, like 1.1.1.4 and earlier did. Or rather it does open a dos/cmd style window that promptly closes. This makes sense to a degree since there was no input(directory) to process, but is a change from the behavior of the earlier versions. Which would then ask for a dir to process. I'm not sure if this is a big deal, or just a side-effect of the new drag n' drop functionality or what. Just thought I'd mention it.

Anyway being the nosy person I am. I created a batch file that called your app inside a dos/cmd window and then added a pause to see what it was doing(when it was run without a dir to process). It's reporting this error:

Traceback (most recent call last):
  File "videometadata.py", line 34, in <module>
IndexError: list index out of range

Once again not that big of a deal but I thought you'd what to know it is throwing an error, rather than a simple close or asking for a dir to process like the earlier versions did.

Ok, while typing this and re-running the exact same tests I did earlier(to get steps+wording right) I just got a weird error. I did save the test's text from the DOS window(or rather XP's simulation), and will include it in a zip file attached to this message. The zip will also contain the simple batch I was using when the error occurred, but keep in mind it also worked on the same dir at least 4 other times. I even tested it again on the same dir right after the error and it did what was expected and ran properly. I don't know why the exact same situation would cause an error one out of 4 times. I'm also including the log from the successful run in the zip file.

Please keep in mind that this dir is just a selfmade test dir, with some weird filenames to stress your app under strange conditions. I didn't expect it to return any .my files for the files. Mostly because I didn't make a showid.txt for this show, but also because these filenames where intentionally made strangely formatted by me. However one thing to note is that this show does actually contain an episode with the name of "1+1=?"(obviously I can't put the "?" in the file title due to it being a forbidden char, so I left it out of the file name). I choose this show and these 3 files for that reason. I was curious to see how your app would handle numbers in the Episode Title. I realize that this is not the type of formatting your app was meant to handle and that's why I did it, so please don't think I'm expecting the app to handle these files and give proper results(with this formatting). I'll do some more tests(with and without showid.txt) later with the files renamed to match your official suggested naming and see what happens.

Thanx for the time and don't worry about the 1.1.1.5 build's version notice. I realized how it happened and besides I can't code so I can't criticize you for such a small issue. I just wanted to point it out.

/Peace

[roxy99]

Originally Posted by justme

One thing I've learned about myself... If there's a way to use a program in a way the author didn't intend, that's the default way I'll use it. I just have a knack for doing things differently than everyone else. So don't worry about misconceptions, I'm just weird.

Just downloaded and tested this version and drag n' drop onto the shortcut(default one your setup made) does work with paths that have spaces in their name.

One thing I did notice is that double left-clicking the shortcut(default one made by your setup) no longer opens a window for input, like 1.1.1.4 and earlier did. Or rather it does open a dos/cmd style window that promptly closes. This makes sense to a degree since there was no input(directory) to process, but is a change from the behavior of the earlier versions. Which would then ask for a dir to process. I'm not sure if this is a big deal, or just a side-effect of the new drag n' drop functionality or what. Just thought I'd mention it.

Anyway being the nosy person I am. I created a batch file that called your app inside a dos/cmd window and then added a pause to see what it was doing(when it was run without a dir to process). It's reporting this error:

Traceback (most recent call last):
  File "videometadata.py", line 34, in <module>
IndexError: list index out of range

Once again not that big of a deal but I thought you'd what to know it is throwing an error, rather than a simple close or asking for a dir to process like the earlier versions did.

Ok, while typing this and re-running the exact same tests I did earlier(to get steps+wording right) I just got a weird error. I did save the test's text from the DOS window(or rather XP's simulation), and will include it in a zip file attached to this message. The zip will also contain the simple batch I was using when the error occurred, but keep in mind it also worked on the same dir at least 4 other times. I even tested it again on the same dir right after the error and it did what was expected and ran properly. I don't know why the exact same situation would cause an error one out of 4 times. I'm also including the log from the successful run in the zip file.

Please keep in mind that this dir is just a selfmade test dir, with some weird filenames to stress your app under strange conditions. I didn't expect it to return any .my files for the files. Mostly because I didn't make a showid.txt for this show, but also because these filenames where intentionally made strangely formatted by me. However one thing to note is that this show does actually contain an episode with the name of "1+1=?"(obviously I can't put the "?" in the file title due to it being a forbidden char, so I left it out of the file name). I choose this show and these 3 files for that reason. I was curious to see how your app would handle numbers in the Episode Title. I realize that this is not the type of formatting your app was meant to handle and that's why I did it, so please don't think I'm expecting the app to handle these files and give proper results(with this formatting). I'll do some more tests(with and without showid.txt) later with the files renamed to match your official suggested naming and see what happens.

Thanx for the time and don't worry about the 1.1.1.5 build's version notice. I realized how it happened and besides I can't code so I can't criticize you for such a small issue. I just wanted to point it out.

/Peace

Wow. I did'nt realize there were still so many nagging bugs. I'm working through them one at a time so please bear with me. The gui wrapper will have to wait until the CLI is perfected. The gui will also be a metadata file browser so that you can see what's in each .my file. It will be essentially like looking in the vide library of Sage with some new features. This will help manage your collection.

RE: that weird error was an intermittent error caused most likely by your internet connection flaking for a split second while the app was trying to download from the web. There are a few places in the app where it goes to the internet. I placed another timeout loop to try a few times to get that data and gracefully fail to generate in case of bad connection and not barf up on your screen.

[garyellis]

I love what this application can do...but I am having strange problems getting it to work.

This is strange... I loaded it on my laptop and transferred a video file over to my laptop and tested it out. After a few false starts trying to get the name conventions right, it worked and worked great. So, I loaded the application on my home theater computer and used it on the exact same directory and video name as I had transferred to my laptop. But, I get "show not found"...I know the files are the same names, etc, because I did a copy from one machine to the other. I have a lot more security on the home theater computer and the hard drive is a USB drive...I doubt any of that could cause a problem...

Any ideas?

and thanks for your work on this application.

Gary Ellis