Utility: Batch Metadata Importer 3.x

[lash]

Originally Posted by stuckless

Are these DVDs? I sometimes see this with DVDs.

Kind of yes. They are Blueray images.

Originally Posted by stuckless

That should work, although, it basically will find S01E19 and use the metadata for that. ie, the E20 is just dropped. BMT has no concept of a single TV show that has 2 episodes.

Great, thanks!

Thank you yet again!

Edit: Nevermind, I discovered the java commandline and tested it till I found Season 0 works for specials. I must've missed that commandline in the tutorial.

[razrsharpe]

Originally Posted by lash

Kind of yes. They are Blueray images.

BDMV folder structure images or iso images? Not sure if it matters but thought i would ask for clarification

Edit: Nevermind, I discovered the java commandline and tested it till I found Season 0 works for specials. I must've missed that commandline in the tutorial.

ive always been curious about this too... so what file naming convention works for specials?

[stuckless]

Originally Posted by ranger

Thanks for the response, but if it should work, then I need help figuring out why it isn't working.
Using the web interface for BMT and performing a scan, the metadata isn't populated. The show title, episode title, season and episode numbers do not populate correctly and so no metadata is found.
An example of how the Display Title populates is "Hogan's Heroes/Hogan's Heroes - 1x12 - The Scientist"
Any ideas?

Here's a tv filename scraper that should work with your #x# format. Just extract it to the scrapers/xbmc/tvfilesnames/ directory and restart the sagetv server for the scraper to be picked up.

I'll include this scraper in the next full build.

[stuckless]

Originally Posted by razrsharpe

ive always been curious about this too... so what file naming convention works for specials?

The best way to get a match on a special is use the Aired Date in the filename. For example, for Heroes special episode, "Destiny - Let Us Pray - Part 1", you would name the file something like...

Heroes - 2008-11-11.avi
Heroes - 2008-11-11 - Destiny - Let Us Pray - Part 1.avi

The Aired Date is the Aired date from the thetvdb.com.

[ranger]

Originally Posted by stuckless

Here's a tv filename scraper that should work with your #x# format. Just extract it to the scrapers/xbmc/tvfilesnames/ directory and restart the sagetv server for the scraper to be picked up.
I'll include this scraper in the next full build.

It is working its' way through my collection now and seems to be working great! Thanks.


Edit -- I seem to be having problems with a few series. Sorry to be a pain.....

examples
Stargate SG-1/Stargate SG-1 - 1x7 - Cold Lazarus
Andy Barker, P.I/Andy Barker, P.I. - 1x1 - Pilot
Doctor Who/Doctor Who - 18x9 - Full Circle (1)

[can3gxw]

Edit -- I seem to be having problems with a few series. Sorry to be a pain.....

examples
Stargate SG-1/Stargate SG-1 - 1x7 - Cold Lazarus
Andy Barker, P.I/Andy Barker, P.I. - 1x1 - Pilot
Doctor Who/Doctor Who - 18x9 - Full Circle (1)

I have found that BMT works pretty exact (providing there is a match for the episode on THETVDB.COM) if you use the file naming convention:

Stargate SG-1/Stargate SG-1 - S01E07 - Cold Lazarus
Andy Barker, P.I/Andy Barker, P.I. - S01E01 - Pilot

(note the "S" for season and "E" for episode)

[can3gxw]

Originally Posted by MeInMaui

I'm on build 17. I'll try to get it out tonight.

Aloha,
Mike

Hey Mike... any good word on your latest release? I am anxious to give it a whirl... I'd love to have BMT and SageMC be friends again!

[stuckless]

Originally Posted by ranger

It is working its' way through my collection now and seems to be working great! Thanks.


Edit -- I seem to be having problems with a few series. Sorry to be a pain.....

examples
Stargate SG-1/Stargate SG-1 - 1x7 - Cold Lazarus
Andy Barker, P.I/Andy Barker, P.I. - 1x1 - Pilot
Doctor Who/Doctor Who - 18x9 - Full Circle (1)

Currently bmt has a few quirks, and given the size of your collection, you are going to find them all...

Stargate and Andy Barker fall into the category of having important characters stripped out of their name (the dash and .) which changes how thetvdb finds them ( or in this case doesn't find them). ie, Stargate SG-1, becomes Stargate SG 1 and unfortunately, it doesn't find a match when the - is missing.

[razrsharpe]

Originally Posted by stuckless

The best way to get a match on a special is use the Aired Date in the filename. For example, for Heroes special episode, "Destiny - Let Us Pray - Part 1", you would name the file something like...

Heroes - 2008-11-11.avi
Heroes - 2008-11-11 - Destiny - Let Us Pray - Part 1.avi

The Aired Date is the Aired date from the thetvdb.com.

thanks that makes sense...

[razrsharpe]

Originally Posted by can3gxw

Hey Mike... any good word on your latest release? I am anxious to give it a whirl... I'd love to have BMT and SageMC be friends again!

http://forums.sagetv.com/forums/show...2&postcount=15 released last week....

[can3gxw]

Thanks, razr... strange that his post didn't come up in my subscribed threads as "new unread post".

[lash]

Originally Posted by razrsharpe

BDMV folder structure images or iso images? Not sure if it matters but thought i would ask for clarification

BDMV, I believe. Thats the name of one of the folders. Sorry, I am not up on all the terminology but its not an iso image.

ive always been curious about this too... so what file naming convention works for specials?

Well, I built myself a regex xbmc importer to fit the naming scheme of my files, but using the default one "S00Exy" works for specials. So, for example, Heroes - S00E06 - Destiny 01 Let Us Pray works. For my setup Heroes\Season 0\H - Episode 06 - Destiny 01 Let Us Pray is what I am using.

I've got yet another question:

How do I exclude certain folders? Is there a way to put in a text file that tells BMT not to check this folder or something like that?

I saw the regex exclusion option, but regex will not work for this situation as there are just a few sub folders I wish to exclude. Unless I am doing something wrong, putting in the exact name of the folders doesn't seem to work, even if I add a little regex code to it.

Thanks!

Edit: I just saw a suggestion to post my other thread about naming specials and keeping the cronological order here. Instead of reposting it, maybe I can link to it here? Is that ok?

http://forums.sagetv.com/forums/showthread.php?t=45098

Sorry!

[ranger]

Originally Posted by stuckless

Currently bmt has a few quirks, and given the size of your collection, you are going to find them all...

Stargate and Andy Barker fall into the category of having important characters stripped out of their name (the dash and .) which changes how thetvdb finds them ( or in this case doesn't find them). ie, Stargate SG-1, becomes Stargate SG 1 and unfortunately, it doesn't find a match when the - is missing.

Yeah, I think I have found most of them...

The funny part is that I can use the BMT manual process to get the metadata. If I browse to an SG-1 episode, I can select "TV" as the media type, fill in the lines "Show Title", "Season #", "Episode #", and then click "Find Metadata" BMT will retrieve all the data. A little painful since I have all 10 seasons.....

Even afterwards the automated scans don't seem to find the data. There is definitely something different in the way the manual "Find MetaData" function works.

[stuckless]

Originally Posted by lash

How do I exclude certain folders? Is there a way to put in a text file that tells BMT not to check this folder or something like that?

I saw the regex exclusion option, but regex will not work for this situation as there are just a few sub folders I wish to exclude. Unless I am doing something wrong, putting in the exact name of the folders doesn't seem to work, even if I add a little regex code to it.

you can use the regex to exclude multiple folders, but it's tricky. There are some key things to note about the regex stuff. First, your filenames will be in a cross platform uri format, so using a windows backslash in the filename will not work. Second, if you want to ignore multiple forlders, use the regex "or" operator "|", and 3rd, the less information in your regex the better....

MySubFolder1|MySubFolder2|MySubFolder3

or

/dir1/MySubFolder1/|/dir1/MySubFolder2/|/dir2/MySubFolder3/

Originally Posted by ranger

Yeah, I think I have found most of them...

It's a work in progess.... with very little progress lately

Even afterwards the automated scans don't seem to find the data. There is definitely something different in the way the manual "Find MetaData" function works.

The manual search assumes that you know what you want to search for, and it does not "clean" the text. The automatic search actually parses the Series name and then it "cleans" it, by removing non alphanumeric characters, such as "." and "-". One of the upcomming changes to bmt will actually do the search using the parsed series name, as is, without "cleaning", and then do a second search with a cleaned version, so this will catch those cases like SG-1.

There is a lot of work needed to make bmt a polished product, but with with only a single developer and a few minutes a night.... it's going to take a long time

[popechild]

Wow, awesome tool. Just getting around to updating Sage and SageMC after months of being too busy to do anything except, you know, watch tv and videos and stuff, and it looks like an amazing amount of work has been done in that time. Especially in all the Fanart stuff, which is clearly much more tightly integrated now, and presumably easier to use as well, but I'm having trouble figuring out how to use all the new fancy features. If I'm missing a usage manual somewhere, someone let me know...

Using 3.1, and all the latest Sage, SageMC, phoenix...

I've got the auto-tool imported into SageMC, and it seems to do what I expect it to, grabbing new Fanart and metadata for newly imported dvds and such. I've also played around with the web UI and *think* I have a handle on at least how to start using it, but I'm stumped on a few things.

Q: Is there a way to update the metadata/fanart on a single piece of media (after it's already in the library)? On the web UI, it looks like you can do a scan and limit it to certain categories and in this mode it will "automatically" grab all the metadata and fanart for everything, or I can use "browse" and skim through the results until I find the one I want to update, but then when I click "Find Metadata" instead of it going and automatically looking for it, it asks me to basically fill out a bunch of the info myself.

Can I somehow just pick one DVD, say, or one folder containing a handful of DVDs, and tell it to go grab it itself, the same way it does all of them if I do a "scan", or do I manually have to type in for each individual one what to look for with a browse?

For example, let's say I have TV DVD rips in the format:
d:\dvds\curb your enthusiasm\season 1\curb your enthusiasm s01e01\.vob
d:\dvds\curb your enthusiasm\season 1\curb your enthusiasm s01e02\.vob
d:\dvds\curb your enthusiasm\season 2\curb your enthusiasm s02e01\.vob

1. Is that the proper way to name them first of all? (I have them ripped as individual episodes rather than as discs, but they're still in .vob format)

2. If I've got six different season folder, and each season has 10 individual episode folders, is there a way for me to have the tool go grab all the metadata and fanart for all of my curb your enthusiasm episodes, or a single season folder, or even a single episode without me having to manually type in the show title and episode information for all 60 one at a time, AND not have it go through my hundreds of other DVDs that I've yet to organize properly at the same time?

Basically, can I just somehow point it to one file, or one folder, and say, Update Just These?

EDIT: One more question I just stumbled onto:

If I do use the manually entry method from the web UI to type in info one episode at a time and grab the info that way, it returns all the correct info, but leaves me with two more questions:

1. As soon as I've "saved" the metadata that the web UI search finds, the episode has been renamed to just the episode title (ie. "Episode Title") in Sage. However, after the library has been refreshed, it changes again to "Show Title - SXXEYY - Episode Title" in Sage. So to truly see the updated metadata in Sage after using the web UI, you also have to refresh the library?

2. Any way to customize the final display title for the episodes? Since I typically organize everything by file system view, I already have a Show Name\Season #\ hierarchy that I'm navigating. I don't need to see "Show Title - SXXEYY - Episode Title" for every episode, it'd be great if I could just see "YY - Episode Title" for each episode. Is there a way for me to set it up to display this way?

Thanks for the awesome tool, and all the help figuring out how to use it!

[stuckless]

Originally Posted by popechild

Q: Is there a way to update the metadata/fanart on a single piece of media (after it's already in the library)? On the web UI, it looks like you can do a scan and limit it to certain categories and in this mode it will "automatically" grab all the metadata and fanart for everything, or I can use "browse" and skim through the results until I find the one I want to update, but then when I click "Find Metadata" instead of it going and automatically looking for it, it asks me to basically fill out a bunch of the info myself.

The web UI can update the fanart for a particular item, but you do need to fill in the search criteria (ie, season, episode, etc). It can also be done from the commandline.

Can I somehow just pick one DVD, say, or one folder containing a handful of DVDs, and tell it to go grab it itself, the same way it does all of them if I do a "scan", or do I manually have to type in for each individual one what to look for with a browse?

The webui is no where near complete. Currently you can't select a folder and scan it. You can do this with the commandline, by passing the directory as the filename arg.

For example, let's say I have TV DVD rips in the format:
d:\dvds\curb your enthusiasm\season 1\curb your enthusiasm s01e01\.vob
d:\dvds\curb your enthusiasm\season 1\curb your enthusiasm s01e02\.vob
d:\dvds\curb your enthusiasm\season 2\curb your enthusiasm s02e01\.vob

1. Is that the proper way to name them first of all? (I have them ripped as individual episodes rather than as discs, but they're still in .vob format)

The default tv filename scrapers in bmt ONLY scrape the filename and not the directory. So, if the files were named, to include the series name and s01E01 info, then bmt would have picked it up automatically.

But, you can add your own scrapers to the scrapers/xbmc/tvfilenames/ directory. Basically a tvfilename scrapers is passed a URI (ie, file:/d:/dvds/curb your enthusiasm/season/curb your enthusiasm s02e01/file.vob) and you provide 3 regexs that parse out the series, season and episode. The default Titles-s00e00.xml just parses the filename, but you could add a new one that utilizes the entire folder structure. Unfortunately, if you don't know regex, it can be a little overwhelming.

(EDIT: Oops, I guess not. When I ran the scan on a single import folder to test, using the structure: d:\dvds\arrested development\season 3\arrested development s03e13\.vob, it didn't grab the episode info for s03e13, it just grabbed generic show info and named the file "arrested development", which seems to indicate I would end up with 13 identically named episodes if I ran the "Scan" on the entire season... How should I be naming them in order for BMT to find and return the individual episode metadata?)

2. If I've got six different season folder, and each season has 10 individual episode folders, is there a way for me to have the tool go grab all the metadata and fanart for all of my curb your enthusiasm episodes, or a single season folder, or even a single episode without me having to manually type in the show title and episode information for all 60 one at a time, AND not have it go through my hundreds of other DVDs that I've yet to organize properly at the same time?

You need a better filename scraper to match your files.

Basically, can I just somehow point it to one file, or one folder, and say, Update Just These?

Commandline is your best option, but you'll still need a better filename scraper.

If you can build the scraper, then I think it will solve some of your issues. If you can't let me know, and I'll see about build the scraper for you.

[MeInMaui]

Originally Posted by can3gxw

Hey Mike... any good word on your latest release? I am anxious to give it a whirl... I'd love to have BMT and SageMC be friends again!

I posted SageMC 6.3.9b pre-release 17 about a week ago. Give it whirl.

Aloha,
Mike

[lash]

Thanks Stuckless! After a lot of trial and error, I got folder exclusion working with the pipes.

After all of this I think I got everything exactly right. I started a full scan of my entire library. Is the process a bit slow? I started it about 10 hours ago and its only at 10%. There are upwards of 6000 files but at this rate its going to take a week to finish. Is that the way it is or is there something wrong?

Thanks!

[stuckless]

Originally Posted by lash

Thanks Stuckless! After a lot of trial and error, I got folder exclusion working with the pipes.

After all of this I think I got everything exactly right. I started a full scan of my entire library. Is the process a bit slow? I started it about 10 hours ago and its only at 10%. There are upwards of 6000 files but at this rate its going to take a week to finish. Is that the way it is or is there something wrong?

Thanks!

That seems excessively slow. I typically have around 1500 files and i've scanned those in about an hour.... so for 6000 files, it will take time, but not days. If it's stuck at 10%, then I'm thinking that there is a problem. Either the scan crapped out, and the progress indicator is not notified, or it's stuck on a file. To do an initial scan of 6000 files, you might want to consider using the command line.

[ChaOConnor]

is there a way to get a list of my total failed? Only shows a few, but says thousands have failed. Thanks!