TV.com automatic scraper

[roxy99]

Originally Posted by justme

I did attach a zip with 4 videolist log files. Their names pretty much say what they are. I'm trying(not too good at it) to keep this post shorter. They really won't(at least I don't think) tell you anything about your scraper that you didn't already know. Besides given your excellent explanation of the scraper's logic, in post #63, I think all the behavior in these logs is exactly as it should be for the current(1.1.1.6) version. It definitely sounds like you know where all the issues I described(or posted about earlier) are coming from and have a fix in the works. One Note, about the logs, in case you do decide to check them:I let the 1st run go all the way thru, but I interrupted(escaped the batch) the last 3 runs after they had processed a good sample of files. Letting it run all 4 test on 163 files 4 times would have taken quite a while, so I cheated for the last 3 tests
/peace

Your tests confirm the known limitation with 1.1.1.6 : IE That 3 digit episode groups are interpreted as : 1st Digit: Season : 2nd-3rd: Episode.

The EP tag you added will now enable my next version of the software to properly interpret the single 3 digits as one sole data element, being simply the episode#.

Thanks for testing

[roxy99]

My to properties converter:

See:

http://forums.sagetv.com/forums/down...do=file&id=190

[roxy99]

http://forums.sagetv.com/forums/down...do=file&id=188

Hopefully you'll find it usefull. Also don't forget about the .mp to .properties file converter above.

There are many changes as discussed previously in this version and while the utility is very simple looking there is a lot going on behind the scenes to get it to do what it does.

I hope you find it useful.

[Halstead]

Originally Posted by roxy99

http://forums.sagetv.com/forums/down...do=file&id=188

Hopefully you'll find it usefull. Also don't forget about the .mp to .properties file converter above.

There are many changes as discussed previously in this version and while the utility is very simple looking there is a lot going on behind the scenes to get it to do what it does.

I hope you find it useful.

Will this and the .properties converter work with DirMon?

[roxy99]

Originally Posted by Halstead

Will this and the .properties converter work with DirMon?

Sure thing. It would work great as a scheduled task in Dirmon being a CLI. You can also create a windows batch file with a list of directories to process:

EG:

c:\Program Files\TV MetaData Creator\videometadata.exe -p (video path1)
c:\Program Files\TV MetaData Creator\videometadata.exe -p (video path2)
etc..

Set that batch to run on a scheduled basis in windows task scheduler.

[Halstead]

Originally Posted by roxy99

Sure thing. It would work great as a scheduled task in Dirmon being a CLI. You can also create a windows batch file with a list of directories to process:

EG:

c:\Program Files\TV MetaData Creator\videometadata.exe -p (video path1)
c:\Program Files\TV MetaData Creator\videometadata.exe -p (video path2)
etc..

Set that batch to run on a scheduled basis in windows task scheduler.

Cool-
I'm going to try setting them both up as automated processes, and I'll let you know how it goes.

[roxy99]

I just found a bug. Guest star is being doubled up with TV actors and TV actors doesn't show up on generated .my files. Please report any other important bugs.

For now, just tell me the important bugs and leave out the 'nice-to-have' request. IE- I know an .my viewer and gui is nice to have, however, I'd like to focus on getting stomping out the bugs first.

Any other bugs?

[justme]

EDIT>>

For now, just tell me the important bugs and leave out the 'nice-to-have' request. IE- I know an .my viewer and gui is nice to have, however, I'd like to focus on getting stomping out the bugs first.

Any other bugs?

Sorry I was composing this as you posted appearantly. So if I went to far sorry. /Peace.<<EDIT


Thanks roxy99,

Glad to have a new version to play with. I've only had a chance to do some very quick tests on a few directories. So far it is going pretty well. I even made sure my network was saturated to test it's ability to handle ISP issues and it seemed to eventually(after several automated retries) get the info. Of course without the artificially created saturation it works fine the first time.

Now the bad news...
This issue relates to the presence(or actually absence) of "EP" in the filename. If you try to process a directory(or tree) that has a file without "EP" in the name it crashes. Using my batch I caught the error(no videolist log gets ctreated). It's listed below. EDIT2:>>I just did a another test on my Inuyasha dir(I believe I uploaded a zero size copy in one of my old posts) and it seems it's not neccessarily the lack of "EP" that may cause this. I say this since the files in the Inuyasha dir all have a format like "Inuyasha- 002 - Seekers Of The Sacred Jewel.ogm". Since these files were proccessed(correctly below 99, as expected without the EP) without a crash, I suspect it may be the space (or lack there of) around the episode digits. Maybe it's just a different method being called for the 3 digit format, and only the 2 digit metjod is crashing with the "EP". <<:EDIT2

E:\Documents and Settings\Michael>REM @echo off

E:\Documents and Settings\Michael>"E:\Program Files\TV Metadata Creator\videomet
adata.exe" "\\Htpc\m on htpc\Zoids Fuzors\Zoids Fuzors[ZA]"
Importing modules

Reading show titles...please wait
No season level found, assuming slev=0
Zoids Fuzors-01-Knockover on Planet Zi.avi
SE+EP TYPE ### ->

Traceback (most recent call last):
  File "videometadata.py", line 174, in <module>
  File "videometadata.py", line 33, in mainprog
  File "getvideoname2.pyc", line 207, in viddic
  File "getvideoname2.pyc", line 184, in gettitepisode2
UnboundLocalError: local variable 'episode' referenced before assignment
test batch of .my file creation

E:\Documents and Settings\Michael>pause
Press any key to continue . . .

This format of course isn't the one you suggest, but I doubt you wanted the app to crash if it hits a files like this. I ran your batch renamer(Thanx again ) on the dir to add "EP" to all the files and it worked fine so that's good. Personally I'd love it if you could have the parser fall back to the old behavior(I realize it's less accurate) when it hits a file of this format. But I understand if you only want to fix the app to not crash and skip the parsing of such files. If you go this way please(if possible) have it provide clear feedback of where it didn't/couldn't make .my files, due to this. I guess I could always use the older version for easier my creation on these dirs(I have several where changing the filename is either not possible(write access) or not practical(other apps use them and are expecting that name).

I've included some logs from some other runs on this same dir, after I added "EP". Don't worry(except for episode 14) about the repeated "FAILED TO GENERATE TV DATA...TRYING AGAIN", as they come from the intentional clogging of my internet connection. There are 2 other things to note.
1) Episode 14 never seems to generate a .my file even without congestion. I suspect it has something to do with the fact that TV.com has this episode's original air date in 2008. This also causes a "Pre-show buzz" poll to be listed in that episode's page, maybe that's causing the parsing issues too/instead. All the others episodes have dates that make sense and lack the poll. I hope this clue helps figure out what is throwing the app with this ep.

2) Episode 2 does not get a .my created. This happens both without and with a showid.txt. To be fair however the episode title is evil for this kind of app. It's episode name is "1 + 1 = ?". This is one of the resaons I choose this show as a test. Two things I see in the logs are....The numbers seems in the name("Zoids Fuzors-EP02-1+1=.avi" is the actual name, after adding "EP", in my test dir) seem the make the app parse it as a different season than all the other episodes, so maybe this is it. But also since the "?" is an illegal char for a filename, it's not in the actual filename(see logs, or note above) on the drive. Still illegal chars could be common on some shows. If it's the illegal char, I don't know what to suggest. Part of me wants a fallback to using the same season# as the other files, but this could be hard in a mutliseason show with files in 1 dir(no subdir season stucture). I also thought about your fallback of title matching, maybe it could be made to make a match if a certian % of the chars match(say 85-90).

Anyway this is all I could find to bug you with. So good job. Thanx for all your effort, time and feedback.

/peace

[roxy99]

Originally Posted by justme

EDIT>>Sorry I was composing this as you posted appearantly. So if I went to far sorry. /Peace.<<EDIT


Thanks roxy99,

Glad to have a new version to play with. I've only had a chance to do some very quick tests on a few directories. So far it is going pretty well. I even made sure my network was saturated to test it's ability to handle ISP issues and it seemed to eventually(after several automated retries) get the info. Of course without the artificially created saturation it works fine the first time.

Now the bad news...
This issue relates to the presence(or actually absence) of "EP" in the filename. If you try to process a directory(or tree) that has a file without "EP" in the name it crashes. Using my batch I caught the error(no videolist log gets ctreated). It's listed below.

E:\Documents and Settings\Michael>REM @echo off

E:\Documents and Settings\Michael>"E:\Program Files\TV Metadata Creator\videomet
adata.exe" "\\Htpc\m on htpc\Zoids Fuzors\Zoids Fuzors[ZA]"
Importing modules

Reading show titles...please wait
No season level found, assuming slev=0
Zoids Fuzors-01-Knockover on Planet Zi.avi
SE+EP TYPE ### ->

Traceback (most recent call last):
  File "videometadata.py", line 174, in <module>
  File "videometadata.py", line 33, in mainprog
  File "getvideoname2.pyc", line 207, in viddic
  File "getvideoname2.pyc", line 184, in gettitepisode2
UnboundLocalError: local variable 'episode' referenced before assignment
test batch of .my file creation

E:\Documents and Settings\Michael>pause
Press any key to continue . . .

This format of course isn't the one you suggest, but I doubt you wanted the app to crash if it hits a files like this. I ran your batch renamer(Thanx again ) on the dir to add "EP" to all the files and it worked fine so that's good. Personally I'd love it if you could have the parser fall back to the old behavior(I realize it's less accurate) when it hits a file of this format. But I understand if you only want to fix the app to not crash and skip the parsing of such files. If you go this way please(if possible) have it provide clear feedback of where it didn't/couldn't make .my files, due to this. I guess I could always use the older version for easier my creation on these dirs(I have several where changing the filename is either not possible(write access) or not practical(other apps use them and are expecting that name).

I've included some logs from some other runs on this same dir, after I added "EP". Don't worry(except for episode 14) about the repeated "FAILED TO GENERATE TV DATA...TRYING AGAIN", as they come from the intentional clogging of my internet connection. There are 2 other things to note.
1) Episode 14 never seems to generate a .my file even without congestion. I suspect it has something to do with the fact that TV.com has this episode's original air date in 2008. This also causes a "Pre-show buzz" poll to be listed in that episode's page, maybe that's causing the parsing issues too/instead. All the others episodes have dates that make sense and lack the poll. I hope this clue helps figure out what is throwing the app with this ep.

2) Episode 2 does not get a .my created. This happens both without and with a showid.txt. To be fair however the episode title is evil for this kind of app. It's episode name is "1 + 1 = ?". This is one of the resaons I choose this show as a test. Two things I see in the logs are....The numbers seems in the name("Zoids Fuzors-EP02-1+1=.avi" is the actual name, after adding "EP", in my test dir) seem the make the app parse it as a different season than all the other episodes, so maybe this is it. But also since the "?" is an illegal char for a filename, it's not in the actual filename(see logs, or note above) on the drive. Still illegal chars could be common on some shows. If it's the illegal char, I don't know what to suggest. Part of me wants a fallback to using the same season# as the other files, but this could be hard in a mutliseason show with files in 1 dir(no subdir season stucture). I also thought about your fallback of title matching, maybe it could be made to make a match if a certian % of the chars match(say 85-90).

Anyway this is all I could find to bug you with. So good job. Thanx for all your effort, time and feedback.

/peace

While trying to fix that crazy naming format I ended up breaking something else, so lets stick to the general naming format and try to test it. What I need to know is that in general it's ok right?

[justme]

What I need to know is that in general it's ok right?

Yep.

While trying to fix that crazy naming format I ended up breaking something else, so lets stick to the general naming format and try to test it.

Sorry for the trouble. As a matter of fact I just editted my post above to say more and saw this new one by you. Just keep in mind I only had a chance to start testing today(maybe 2-3 hours of parsing time total). So don't think I've been pounding the app all weekend.

I'm going to rerun the test(Edit2 in post above) of the Inuyasha dir with the "EP###" format to make sure that episodes above 99 get the right .my files just to be safe. I haven't seen any errors yet. So looks good. I'll post if it does error, just keep in mind the whole test of 160 some files(in just that dir) can take a long time.

EDIT>>RESULTS OF TEST:Inuyasha dir was successfull for ALL files including the merged 2-parters(it used the info from the 1st episode of the 2 but that is perfect as I don't expect it to recognize the merge.). This dir used the "SHOW-EP###-Episode.ext" format(meaning no season info, so it assumes season=0).<<EDIT

[roxy99]

Justme,

Thanks for your help. Try the new 1.2.1.9 version. I think it may fix the stuborn filename errors.
If you leave out the 'EP' preface, the program just assumes an 'EP' naming method which is fine. However when there is a 3 digit episode # and you forget to put the EP, though the program will make a mistake and assume digit 1 is the season. So you still need the EP preface but it should'nt crash at least. It will only make a mistake. EP numbering assumes all seasons and matches the unique episode#.

The scan logic kicks in when no -t and -s parameters are provided. What takes place is a parse of the directory tree first for the Season identifier. Season identifiers are: se, season, or just a path with digits- eg ../04/.. is season 4. If the program finds a season folder, the program logically assumes that the parent of the season folder must be the title folder, unless there is a '-' in the season folder designating that the title-season are in one path.

The reason for using episode titles as a matching method is that if a match occurs, the program can be sure that its a good match. Whereas people often make mistakes on the episode count the title is very specific. An exact match is the most reliable because failing an exact match, the program looks at teh episode# anyway.

PS: Weighted relevence for episode matching by various criteria-- may be a bit beyond the scope of the program for now. Lets get the bugs stomped and then get fancy later.

[justme]

Originally Posted by roxy99

Thanks for your help. Try the new 1.2.19 version. I think it may fix the stuborn filename errors.

Done. I've posted in detail after I responded to your quotes.

Originally Posted by roxy99

If you leave out the 'EP' preface, the program just assumes an 'EP' naming method which is fine. However when there is a 3 digit episode # and you forget to put the EP, though the program will make a mistake and assume digit 1 is the season. So you still need the EP preface but it should'nt crash at least. It will only make a mistake. EP numbering assumes all seasons and matches the unique episode#.

Understood. I'll do all my basic testing with the "EP" in the filename from now on. Of course I'll still throw the app curve balls to test stability/reliabilty on other formats. I'll try to seperate the two tests/results in my posts from now on.

Originally Posted by roxy99

PS: Weighted relevence for episode matching by various criteria-- may be a bit beyond the scope of the program for now. Lets get the bugs stomped and then get fancy later.

Gotcha. Just feel free to ignore me if I provide anything more than you need. I'm largely thinking out loud and making notes for possible future features, but I'll do my best to avoid it. Please don't think i'm trying to be demanding. However, I'm not known for my short posts.


Simple Report:
The new version worked great on several test dirs. Even on those "problem" episodes of 2 and 14(of Zoids Fuzors). As long as there is the "EP" is in the filename it seems to work nearly perfectly(.my created and correct episode info). As you'll see if you look at the logs I used many VERY weird filename formats(tried 2 episodes of each format). So as far as the recommended format this new build worked great. No crashes that I could force on the tests I ran.

More Expanded Report(Just ignore if you wish) :
The app also worked very well on episode filenames that didn't have the "EP", as long as I provided a showid.txt. I took a look at the logs and found out why it was failing on these "bad" filenames and... The logs all seem to point to the series being identified as "Zoids Fuzor" instead of "Zoids Fuzors"**. Since the logs show your app correctly identified the episode # for each file(even these "bad", no "EP" files). This resulted in a near perfect creation of .my files. The only errors(wrong episode data in the .my file) were on episodes 12 and 25. And these errors weren't necessarily the apps fault, depending on your desired behavior for the app.
Ep 12 was caused by a wrong episode title in my files, so your app matched on the wrong ep title and used info from TV.com's episode 13(http://www.tv.com/zoids-fuzors/three...list;title;12). Still the logs do show you app ID'd it correctly as episode 12 but used the titlename match(of 13) instead of the explicitly specified episode# from the file name*.
EP 25 is basically the same as above, except my file had the right title: "Zoids Fuzors-EP25-Showdown.avi" In this case episode 18 also started out with the word "Showdown". "Showdown at High Noon" is the full TV.com title for 18. So the app title matched to episode 18's info. Once again using title match over the explicitly specified episode# from the file name*.


*I ran a test on 12 and 25 using the ShowTitle-EP##-EpisodeTile.ext format and got the same results. It's in the logs under the name of "Episodes 12 and 25 with recommended format and showid-videolist.txt". I don't know whether you wish the app to use title matching first, as it appears to do now. Or if you want it to use the specified episode# to get the info, when the file uses a recommended format. I hope this test used a format you consider valid. Either this is either a bug or a feature depending on your design desires. Personally I'd like it to use the specified episode# if it's in the right format and then fallback to title matching, but that's your call.

**I just tested several other shows/dirs. It seems like it's the " " in between "Zoids" and "Fuzors" that is causing a failure to find the Show Title. I say this since the other Shows/dirs had their Show Title ID'd by the app(even without a showid.txt) as long as the Show Title was a single word. Just to be clear this issue only seems to arise when 'EP' is missing from the filename format, and the Show Title has a space in it. I can of course get around this with a showid.txt in the missed dir/show, especially since lacking 'EP' is not a reccomended format. Just thought I'd post this additional info in case you wished to look into this, since the app is now working so well in other areas(appearantly even with 'bad' formats). If not, no big worries as I do have a workaround. Don't won't to break the app again just to fix this issue.

EDIT>>OK this time you added something to your post. I quoted it below and it explains most of the behavior I noted above so you can ignore most(maybe look at episode 25 to see why it used 18's info) of that. At least I correctly guessed this was a possible feature, and didn't totally call it a bug. .<<EDIT

The reason for using episode titles as a matching method is that if a match occurs, the program can be sure that its a good match. Whereas people often make mistakes on the episode count the title is very specific. An exact match is the most reliable because failing an exact match, the program looks at teh episode# anyway.

Makes sense, I'll be fine.

[roxy99]

Originally Posted by justme

Done.


Simple Report:
The new version worked great on several test dirs. Even on those "problem" episodes of 2 and 14(of Zoids Fuzors). As long as there is the "EP" is in the filename it seems to work nearly perfectly(.my created and correct episode info). As you'll see if you look at the logs I used many VERY weird filename formats(tried 2 episodes of each format). So as far as the recommended format this new build worked great. No crashes that I could force on the tests I ran.

If your files are in properly named title folders (no funny business in the path name), then showid.txt should'nt be needed. Just be sure to use -t 1 to force the app to not look at the base filename for the title.

Regarding the 's' missing from the title detection- its because 'S' is also a season designator (along with SE/Seas/Season) . If you put Zoid Fuzors - EP## , hyphenated with spaces, then it should work properly. If that still doesn't work then its a simple fix except I'll throw away the 'S' for season designator so that the Regex doesn't make a mistake.

[justme]

If your files are in properly named title folders (no funny business in the path name), then showid.txt should'nt be needed. Just be sure to use -t 1 to force the app to not look at the base filename for the title.

That works great, I just tested it and added it to my batch. One thing I do have in a good name structure is the show title of the directory containing a show. Some of the wierder ones you've run into in my logs were for testting your app to breaking.

Regarding the 's' missing from the title detection- its because 'S' is also a season designator (along with SE/Seas/Season) . If you put Zoid Fuzors - EP## , hyphenated with spaces, then it should work properly. If that still doesn't work then its a simple fix except I'll throw away the 'S' for season designator so that the Regex doesn't make a mistake.

That makes perfect sense. I already know that using the space like you suggest works,as long as the filename has a EP. Actually I believe my posted logs showed that even filenames without spaces worked fine, as long as there is the EP in the name.

So given these two things I wouldn't change that. I'm fine with using -t and besides it's only an issue with shows that have a "s" at the end. Even then the use of 'EP' ,easily done thanx to your renamer, seems to work perfect as is. Even without the -t, in most cases.

All that leaves is the wierd issue of episode 25 of Zoids Fuzors using 18's episode info. Keep in mind my episode title was correct according to TV.com. It's just that both began with Showdown.

I honestly can't properly say, how happy I am with this newest version(all were good but this seems close to perfect). Especially since I'm now using the -t parameter. Thank you for all your feedback, changes and most importantly advice on proper use. I can't promise I'll really use the app the original way you intended but... Thanks to all these recent back and forths, I now have 2 simple bacth files to handle almost any situation that could occur in my 'badly' formatted library.

I actually plan(if time allows) on releasing these batches(using this version, unless you post a new one before tonight) to process my actual library(not just snipped out test sections, like so far). That should really put it to the test considering there are 1,995(just checked the properties for the tree) video files in my anime dir alone. Of course this is just the stuff on my hard drive, not whats archived to DVD. Then that number comes close to 6,000, including non-anime TV shows. I'm just saying all this to give you an idea of how much I trust your app.

/peace and gratitude

[roxy99]

Originally Posted by justme

...
I actually plan(if time allows) on releasing these batches(using this version, unless you post a new one before tonight) to process my actual library(not just snipped out test sections, like so far). That should really put it to the test considering there are 1,995(just checked the properties for the tree) video files in my anime dir alone. Of course this is just the stuff on my hard drive, not whats archived to DVD. Then that number comes close to 6,000, including non-anime TV shows. I'm just saying all this to give you an idea of how much I trust your app.

/peace and gratitude[/COLOR]

Thanks I really appeciate that. Just remmebr if the epsiode# is more than 3 digit long and no EP preface, the utlity will splice the 3 digits : digit 1: season. Digit 2-3 : episode. That's not what you want so be sure to use the EP preface.

[roxy99]

Some minor tweaking on the new version: 1.2.1.10

[dvd_maniac]

Hi Roxy,

I've been noticing my Imported Videos dissappearing and showing up under Music instead.

I have narrowed it down to a metadata problem.
I am not sure if it is a problem with the Beta or the Scraper.

I would bet on it being a Beta issue though.

Just wondering if anybody else here can check their Music section for Unknown Albums or Artists and see if any Imported Videos ended up there?

If so, are you using the latest Beta or not?

[roxy99]

Originally Posted by dvd_maniac

Hi Roxy,

I've been noticing my Imported Videos dissappearing and showing up under Music instead.

I have narrowed it down to a metadata problem.
I am not sure if it is a problem with the Beta or the Scraper.

I would bet on it being a Beta issue though.

Just wondering if anybody else here can check their Music section for Unknown Albums or Artists and see if any Imported Videos ended up there?

If so, are you using the latest Beta or not?

It can only be the .properties files since the .my files do nothing with the placement in the library.

[dvd_maniac]

I have about 2 dozen of them in my Music section that only have .my files...no .properties files created for them at all. I have Jerkiwi's STVI that takes the .my file and imports it into the Wiz.bin database when the video file is first scanned by Sage.
So Sage is seeing the .my file and somehow seeing it as a Music file.

[roxy99]

Originally Posted by dvd_maniac

I have about 2 dozen of them in my Music section that only have .my files...no .properties files created for them at all. I have Jerkiwi's STVI that takes the .my file and imports it into the Wiz.bin database when the video file is first scanned by Sage.
So Sage is seeing the .my file and somehow seeing it as a Music file.

Weird. I don't suppose you have a virgin version of Sage 6 that you can test on so as to isolate either Sage or the plugin or the .my file. The .my file is only a text file so is innocuous by itself unless somehow the .my file is corrupted in some way. But then you'd see that if you tried to open the .my file in a text editor. It has to be something inside of the Sage engine that is not treating the data properly.

Is there any pattern to the types of imports that are messed up? Keep me posted as you narrow done the exact cause. Maybe a tag field is missing in the .my file but I doubt it.