TV.com automatic scraper

[roxy99]

Originally Posted by dvd_maniac

This is working out great so far.
I am finally starting to organize my Video Import library with a proper naming convention so this program will work wonders.
I hope I get finished by the time you add .properties support...


Thanx for all your effort Roxy.

I am hoping you don't have to reorganize too much. Try it out on your collection and send me your videolog.txt for any failed conversions. Yo may be pleasantly surprised.

[dvd_maniac]

I kinda wanted to reorganize everything anyways. This just gave me a good enough reason to.

I have tried it on several different file/folder structures and have only run into a few issues. Mostly just weird filenames.

BUT, I do have a question...
I am naming things like so:
t:\tv1\tv\Adventure\Lost\03\3x01 A Tale of Two Cities.mp4
But, Lost season 3 started with a special which I named 3x00 A Tale of Survival.mp4
Lost Season 3 tv.com
If you look at the tv.com site it shows the specials below, how does the scraper and tv.com handle stuff like this?

[roxy99]

Originally Posted by dvd_maniac

I kinda wanted to reorganize everything anyways. This just gave me a good enough reason to.

I have tried it on several different file/folder structures and have only run into a few issues. Mostly just weird filenames.

BUT, I do have a question...
I am naming things like so:
t:\tv1\tv\Adventure\Lost\03\3x01 A Tale of Two Cities.mp4
But, Lost season 3 started with a special which I named 3x00 A Tale of Survival.mp4
Lost Season 3 tv.com
If you look at the tv.com site it shows the specials below, how does the scraper and tv.com handle stuff like this?

You have to follow the order of this list
http://www.tv.com/lost/show/24313/ep...&tag=nav_bar;3

So episode 49- is se3ep1
to
episode 74 -is se3ep27

So you need to count from the list down. Ignore any 'Sepcial' heading. My scraper simply counts the episode list. Ignore the name 'special' for these filler episodes. Anyway, TV.com doesn't even have a notion of SE##EP##. Only episode # which spans all seasons. (See jutme's comments). I'd rather incorporate justme's suggestion first. For your 'Special episode' consideration. Use (special) after se03ep25. IE name it-- Lost SE03EP27(special) - the answers.avi

the episode name identifier

[dvd_maniac]

I ran it on my Lost folder and found season 2 all messed up. I simply opened up the .my files to check the titles and renamed them appropriately.
I think the cause of the mess up is the specials. Now if they are all treated as continuous episode #s then as soon as it messes up shouldn't the rest of the episodes from that point on be off as well?
But my season 3 was perfect.

[dvd_maniac]

Another interesting one I can't figure out...
24 created all the .my files perfectly except for the last one of every season. ie: 1x24, 2x24, 3x24 etc...
I checked the website to make sure that it didn't treat the last 2 episodes as 1 for the times they had a 2 hour finale, which they didn't.

Scratching my head on this one.

[dvd_maniac]

Happened on Prison Break as well. The last episode of every season seems to be returning this:
VIDEO :
t:\tv1\tv\action adventure\prison break\1\1x23 Fight\prisonbreak-fight.mp4

PRISON BREAK(SE1.EP23)==> Episode does not exist
-------TITLE----PRISON BREAK
-------SEASON --1
-------EPISODE--23

But this is the site:
1x23 Flight on TV.com

Is anybody else having any issue with the last episode of a season not getting a .my file created?

[roxy99]

Originally Posted by dvd_maniac

Happened on Prison Break as well. The last episode of every season seems to be returning this:
VIDEO :
t:\tv1\tv\action adventure\prison break\1\1x23 Fight\prisonbreak-fight.mp4

PRISON BREAK(SE1.EP23)==> Episode does not exist
-------TITLE----PRISON BREAK
-------SEASON --1
-------EPISODE--23

But this is the site:
1x23 Flight on TV.com

Is anybody else having any issue with the last episode of a season not getting a .my file created?

Get Version 1.0.1.2- just uploaded now

Bug fixed--Last episode missing

Lost 'Special' episodes screw up the count. Still working on that bug. Blame the studid Lost filler episodes. These 'Sepcial' episodes (aka - useless) end up not being counted as episodes. I smell another exception rule coming. Season 3 worked because the specials were at the end of the season and not in the middle therefor did not screw up the count.

[dvd_maniac]

I will d/l and test it out tommorow. Thanx

BTW, Does anybody actually watch those stupid specials?
I watched one of them and said Duh! tell me something new already...

[roxy99]

Originally Posted by dvd_maniac

I will d/l and test it out tommorow. Thanx

BTW, Does anybody actually watch those stupid specials?
I watched one of them and said Duh! tell me something new already...

LOL no kidding

[sg010]

The program is working great, thanks, but... I've generated .my files for a couple of series in my imported videos directory that I have from when I switched over from BTV. the problem is cannot get the new program data to appear

what am I doing wrong?

TIA

[dvd_maniac]

Do you have the stvi that allows you to read and import them?

If not, Get it here

[sg010]

no I did not have it, thanks!

[dvd_maniac]

Hi Roxy,

I have noticed that when scanning a large enough folder I get alot of episodes skipped.
They are scanned but display a "FAILED TO GENERATE DATA" message.
I can rerun the scan again and it's hit or miss. Sometimes I have to run a scan 3 or 4 times to complete it. Is this a timeout issue that might be fixed by a timeout parameter?
If not is there a way for you to incorporate a paste feature in the cmd box like the windows default does?

[roxy99]

Originally Posted by dvd_maniac

Hi Roxy,

I have noticed that when scanning a large enough folder I get alot of episodes skipped.
They are scanned but display a "FAILED TO GENERATE DATA" message.
I can rerun the scan again and it's hit or miss. Sometimes I have to run a scan 3 or 4 times to complete it. Is this a timeout issue that might be fixed by a timeout parameter?
If not is there a way for you to incorporate a paste feature in the cmd box like the windows default does?

It sound intermittent then. Let's start off by you sending me a sample videolog.txt file. Does it happen less often on smaller folders or is it mainly these large folders? That will help me diagnose the problem.

[justme]

Hi sorry for taking soo long to get back to you with an report, but real life is busy ATM. I just did some simple tests(3 total) of your app. I used the default shortcut that it creates with it's setup. I did the first test with no showid.txt file in the test dir(Escaflowne) and the other 2 with a showid.txt file present. After each test I moved the successfully created .my files out of the dir before the next test along with the log file. I've attached a zip file that has 3 dirs that contain the logs for all 3 tests, my files created, and a dir listing that shows the results of that run. The dir listing for each test was made using the DOS dir command in bare mode and piped to a text file. ie "Dir /b >1stTry.txt" I've also included a sfv file of the test dir, created before any tests were run. It's got all the same info as the dir listings but also has the files' sizes, dates and times, not to mention the CRC(not that it does you any good, it's just I already had a sfv file of this dir so I used it in the zip to provide a more detailed dir listing). Don't worry about the .sfv extension it's formatted as a text file and came be read by any text editor.

Basic results of a quick tryout:
1) I think I've found the same issue as, dvd_maniac, with what appears to be a timeout issue. ie It will miss an episode one run and find it the next, or vice versa. The log shows what appears to be the right episode title but says "FAILED TO GENERATE TV DATA".
2) I noticed that while the app named the .my files after the video files, it failed to follow the Capitalization used by the video file.
ie "Escaflowne-01-Fateful Confession.ogm" creates "escaflowne-01-fateful confession.my"
It's a small issue but I'd like it to exactly follow my naming format, including Caps, if at all possible.
3) Basically works GREAT, except for the 2 issues above and I love it. Good Job and much thanx.

Things that I'd like(ignore them if you wish since it's your hard work and effort) to see added.
1) In the GUI wrapper/interface it would be nice to be able to choose an example video file from a directory about to be processed. This file could then be used by the GUI to show you possible Show Title matches. You could then select the correct one if it's in the list(maybe a more expanded search option if it isn't?) and the GUI would autocreate the proper showid. The sample file could then be processed against that showid to list possible Episode Title matches. This would hopefully allow the GUI to be able to auto pick(with more difficult situations also using some user selecting of settings needed) the correct mask(basically autoset the CLI switches) for proper episode title parsing. This should make the GUI wrapper much more user friendly without removing the power/versatility of the CLI app it's calling to do the heavy lifting.
2) I'd like a simple .my viewer in the GUI. This ties into the one above(but I see other uses too), as it lets us quickly check(without using Sage) that the returned episode title data is the correct one for that video file.

Thanks for your time and effort on this, and most especially your generosity in sharing this.

[roxy99]

I think I've tackled the timeout issue in 1.0.1.3 just uploaded today:

Version 1.0.1.3

Bug Fixes:

Fixed timeout issue
Fixed episode count wrong with 'Special' episodes (eg ' Lost')
Fixed Capitalization of .my files

[roxy99]

Originally Posted by justme

Hi sorry for taking soo long to get back to you with an report, but real life is busy ATM. I just did some simple tests(3 total) of your app. I used the default shortcut that it creates with it's setup. I did the first test with no showid.txt file in the test dir(Escaflowne) and the other 2 with a showid.txt file present. After each test I moved the successfully created .my files out of the dir before the next test along with the log file. I've attached a zip file that has 3 dirs that contain the logs for all 3 tests, my files created, and a dir listing that shows the results of that run. The dir listing for each test was made using the DOS dir command in bare mode and piped to a text file. ie "Dir /b >1stTry.txt" I've also included a sfv file of the test dir, created before any tests were run. It's got all the same info as the dir listings but also has the files' sizes, dates and times, not to mention the CRC(not that it does you any good, it's just I already had a sfv file of this dir so I used it in the zip to provide a more detailed dir listing). Don't worry about the .sfv extension it's formatted as a text file and came be read by any text editor.

Basic results of a quick tryout:
1) I think I've found the same issue as, dvd_maniac, with what appears to be a timeout issue. ie It will miss an episode one run and find it the next, or vice versa. The log shows what appears to be the right episode title but says "FAILED TO GENERATE TV DATA".
2) I noticed that while the app named the .my files after the video files, it failed to follow the Capitalization used by the video file.
ie "Escaflowne-01-Fateful Confession.ogm" creates "escaflowne-01-fateful confession.my"
It's a small issue but I'd like it to exactly follow my naming format, including Caps, if at all possible.
3) Basically works GREAT, except for the 2 issues above and I love it. Good Job and much thanx.

Things that I'd like(ignore them if you wish since it's your hard work and effort) to see added.
1) In the GUI wrapper/interface it would be nice to be able to choose an example video file from a directory about to be processed. This file could then be used by the GUI to show you possible Show Title matches. You could then select the correct one if it's in the list(maybe a more expanded search option if it isn't?) and the GUI would autocreate the proper showid. The sample file could then be processed against that showid to list possible Episode Title matches. This would hopefully allow the GUI to be able to auto pick(with more difficult situations also using some user selecting of settings needed) the correct mask(basically autoset the CLI switches) for proper episode title parsing. This should make the GUI wrapper much more user friendly without removing the power/versatility of the CLI app it's calling to do the heavy lifting.
2) I'd like a simple .my viewer in the GUI. This ties into the one above(but I see other uses too), as it lets us quickly check(without using Sage) that the returned episode title data is the correct one for that video file.

Thanks for your time and effort on this, and most especially your generosity in sharing this.

Thanks for the feedback. I noticed you're not using S__E__ syntax but only the episode # syntax. For now, the utility requires a season# to work. All of your episodes return season=0.

I like you gui wrapper idea. The only thing is that remember the utlity will actually convert an entire folder tree including all subfolders. So If I have a parent folder for SciFi, then all shows within the parent (say u:\tvshows\SciFi\...) will be converted. So I don't want the user to drill down to the file level. The user will only click 'SciFi' and by default all subfolders are included (until I add a CLI option to surpress). If you are choosing one show folder at a time, then you're missing half the power of the app.

Instead I propose that when the user selects a folder, the app pops up confirmation dialog with the question 'Are your videos stored in folders by title'
If user answers yes, then another question box 'Does each title folder have subfolders by Season'
The answers to these questions is all that's needed to get the -t -s settings.

[Armondole]

Roxy,

You might not realize it, but you are slowly recreating the XBMC scraping program one step at a time... very nice job!

[roxy99]

Originally Posted by Armondole

Roxy,

You might not realize it, but you are slowly recreating the XBMC scraping program one step at a time... very nice job!

I don't mind. It probably explains the low download rate of my utility since I've only re-invented the wheel. The upside is I'm learning a lot and if the utlity can be useful for only a few people then that's just a bonus.

[Armondole]

Originally Posted by roxy99

I don't mind. It probably explains the low download rate of my utility since I've only re-invented the wheel.

I'm sure most people just don't understand the potential behind it. Wait until you release the GUI version... anyone who knows how to use .my files and downloads TV shows will surely put this to good use.

Hopefully I can figure out the .properties file creation... i'm a pure newb with this stuff and I haven't programmed in 10 years, so it might take me a while just to get started.