Utility: Batch Metadata Importer 3.x

[jphipps]

Originally Posted by stuckless

Jeff, try the beta8 with the "workaround" enabled (see the release notes in the releases thread). I think this is related. This exception happens outside of the bmt apis, but it's definately related to what bmt is passing back. The work around, basically returns null instead of the metadata, and then sage reads the metadata directly from the .properties, so it's somewhat transparent from a user perspective.

I tried that, and it had the same result. When I looked through the debug log for BMT, I noticed the last message in the log was that it was loading the existing properties file. I deleted the existing properties file and re-ran the import and it worked successfully. There must be something in the existing properties file that was causing it to throw an exception instead of returning. I am attaching the file incase you want to take a look at it...

Thanks for your help,
Jeff

[gplasky]

I think something broke in beta 8. I'm getting a bunch of errors. And I was trying to redo my TV Recordings (SageTV) metadata and fanart and it just keeps creating 0kb .jpg files for ALL the fanart. Backgrounds, posters and banners. Here's part of the log file:

2009-07-12 13:48:58,187 ERROR [Thread-24] org.jdna.media.util.AutomaticUpdateMetadataVisitor (AutomaticUpdateMetadataVisitor.java:54) - Failed to find/update metadata for resource: file:////MAIN/Videos/MainMovie/Indiana%20Jones%20and%20the%20Kingdom%20of%20the%20Crystal%20Skull/VIDEO_TS/
java.lang.IllegalArgumentException: im == null!
	at javax.imageio.ImageIO.write(Unknown Source)
	at javax.imageio.ImageIO.write(Unknown Source)
	at org.jdna.media.metadata.MediaMetadataUtils.writeImageFromUrl(MediaMetadataUtils.java:37)
	at org.jdna.media.metadata.impl.sage.FanartStorage.downloadAndSaveFanart(FanartStorage.java:113)
	at org.jdna.media.metadata.impl.sage.FanartStorage.downloadAndSageCentralFanart(FanartStorage.java:204)
	at org.jdna.media.metadata.impl.sage.FanartStorage.saveCentralFanart(FanartStorage.java:192)
	at org.jdna.media.metadata.impl.sage.FanartStorage.saveFanart(FanartStorage.java:45)
	at org.jdna.media.metadata.impl.sage.FanartStorage.downloadFanart(FanartStorage.java:289)
	at org.jdna.media.metadata.impl.sage.CentralFanartPersistence.storeMetaData(CentralFanartPersistence.java:38)
	at org.jdna.media.metadata.CompositeMediaMetadataPersistence.storeMetaData(CompositeMediaMetadataPersistence.java:55)
	at org.jdna.media.util.AutomaticUpdateMetadataVisitor.fetchMetaData(AutomaticUpdateMetadataVisitor.java:118)
	at org.jdna.media.util.AutomaticUpdateMetadataVisitor.fetchMetaData(AutomaticUpdateMetadataVisitor.java:69)
	at org.jdna.media.util.AutomaticUpdateMetadataVisitor.visit(AutomaticUpdateMetadataVisitor.java:49)
	at org.jdna.media.util.FilteredResourceVisitor.visit(FilteredResourceVisitor.java:23)
	at org.jdna.media.AbstractMediaResource.accept(AbstractMediaResource.java:49)
	at org.jdna.media.AbstractMediaFolder.accept(AbstractMediaFolder.java:28)
	at org.jdna.media.AbstractMediaFolder.accept(AbstractMediaFolder.java:20)
	at org.jdna.bmt.web.server.BrowserServicesImpl$Scanner.run(BrowserServicesImpl.java:99)
	at org.jdna.bmt.web.server.BrowserServicesImpl$Scanner.run(BrowserServicesImpl.java:1)
	at org.jdna.util.ProgressTrackerManager$1.run(ProgressTrackerManager.java:45)
2009-07-12 13:48:58,187 ERROR [Thread-24] org.jdna.media.util.AutomaticUpdateMetadataVisitor (AutomaticUpdateMetadataVisitor.java:56) - Failed to visit MediaResource: file:////MAIN/Videos/MainMovie/Indiana%20Jones%20and%20the%20Kingdom%20of%20the%20Crystal%20Skull/VIDEO_TS/
java.lang.IllegalArgumentException: im == null!
	at javax.imageio.ImageIO.write(Unknown Source)
	at javax.imageio.ImageIO.write(Unknown Source)
	at org.jdna.media.metadata.MediaMetadataUtils.writeImageFromUrl(MediaMetadataUtils.java:37)
	at org.jdna.media.metadata.impl.sage.FanartStorage.downloadAndSaveFanart(FanartStorage.java:113)
	at org.jdna.media.metadata.impl.sage.FanartStorage.downloadAndSageCentralFanart(FanartStorage.java:204)
	at org.jdna.media.metadata.impl.sage.FanartStorage.saveCentralFanart(FanartStorage.java:192)
	at org.jdna.media.metadata.impl.sage.FanartStorage.saveFanart(FanartStorage.java:45)
	at org.jdna.media.metadata.impl.sage.FanartStorage.downloadFanart(FanartStorage.java:289)
	at org.jdna.media.metadata.impl.sage.CentralFanartPersistence.storeMetaData(CentralFanartPersistence.java:38)
	at org.jdna.media.metadata.CompositeMediaMetadataPersistence.storeMetaData(CompositeMediaMetadataPersistence.java:55)
	at org.jdna.media.util.AutomaticUpdateMetadataVisitor.fetchMetaData(AutomaticUpdateMetadataVisitor.java:118)
	at org.jdna.media.util.AutomaticUpdateMetadataVisitor.fetchMetaData(AutomaticUpdateMetadataVisitor.java:69)
	at org.jdna.media.util.AutomaticUpdateMetadataVisitor.visit(AutomaticUpdateMetadataVisitor.java:49)
	at org.jdna.media.util.FilteredResourceVisitor.visit(FilteredResourceVisitor.java:23)
	at org.jdna.media.AbstractMediaResource.accept(AbstractMediaResource.java:49)
	at org.jdna.media.AbstractMediaFolder.accept(AbstractMediaFolder.java:28)
	at org.jdna.media.AbstractMediaFolder.accept(AbstractMediaFolder.java:20)
	at org.jdna.bmt.web.server.BrowserServicesImpl$Scanner.run(BrowserServicesImpl.java:99)
	at org.jdna.bmt.web.server.BrowserServicesImpl$Scanner.run(BrowserServicesImpl.java:1)
	at org.jdna.util.ProgressTrackerManager$1.run(ProgressTrackerManager.java:45)
2009-07-12 13:48:58,437 ERROR [Thread-24] org.jdna.media.util.AutomaticUpdateMetadataVisitor (AutomaticUpdateMetadataVisitor.java:54) - Failed to find/update metadata for resource: file:////MAIN/Videos/MainMovie/Infamous/VIDEO_TS/
java.lang.IllegalArgumentException: im == null!
	at javax.imageio.ImageIO.write(Unknown Source)
	at javax.imageio.ImageIO.write(Unknown Source)
	at org.jdna.media.metadata.MediaMetadataUtils.writeImageFromUrl(MediaMetadataUtils.java:37)
	at org.jdna.media.metadata.impl.sage.FanartStorage.downloadAndSaveFanart(FanartStorage.java:113)
	at org.jdna.media.metadata.impl.sage.FanartStorage.downloadAndSageCentralFanart(FanartStorage.java:204)
	at org.jdna.media.metadata.impl.sage.FanartStorage.saveCentralFanart(FanartStorage.java:192)
	at org.jdna.media.metadata.impl.sage.FanartStorage.saveFanart(FanartStorage.java:45)
	at org.jdna.media.metadata.impl.sage.FanartStorage.downloadFanart(FanartStorage.java:289)
	at org.jdna.media.metadata.impl.sage.CentralFanartPersistence.storeMetaData(CentralFanartPersistence.java:38)
	at org.jdna.media.metadata.CompositeMediaMetadataPersistence.storeMetaData(CompositeMediaMetadataPersistence.java:55)
	at org.jdna.media.util.AutomaticUpdateMetadataVisitor.fetchMetaData(AutomaticUpdateMetadataVisitor.java:118)
	at org.jdna.media.util.AutomaticUpdateMetadataVisitor.fetchMetaData(AutomaticUpdateMetadataVisitor.java:69)
	at org.jdna.media.util.AutomaticUpdateMetadataVisitor.visit(AutomaticUpdateMetadataVisitor.java:49)
	at org.jdna.media.util.FilteredResourceVisitor.visit(FilteredResourceVisitor.java:23)
	at org.jdna.media.AbstractMediaResource.accept(AbstractMediaResource.java:49)
	at org.jdna.media.AbstractMediaFolder.accept(AbstractMediaFolder.java:28)
	at org.jdna.media.AbstractMediaFolder.accept(AbstractMediaFolder.java:20)
	at org.jdna.bmt.web.server.BrowserServicesImpl$Scanner.run(BrowserServicesImpl.java:99)
	at org.jdna.bmt.web.server.BrowserServicesImpl$Scanner.run(BrowserServicesImpl.java:1)
	at org.jdna.util.ProgressTrackerManager$1.run(ProgressTrackerManager.java:45)

I've enabled debug logging. Let me know if you need more info. Also in Scanning from web browser I'm seeing entries for al the video capture tuners. Scan date never changed for me either. Scanning actually looked like it was going to work better and I saw twice as many items to scan than what I had seen previously. I'm going to roll back to beta 6 and compare the two.

Gerry

[stuckless]

Jeff, I missed an exit point from the plugin... Basically if the metadata plugin has existing properties then they are returned... It will now check the "work around" flag instead of just returning the props.

Gerry, I think that the image url must be invalid, but I'm going to add some code around the image download code to better check for errors and to ensure that 0bytes files are not created.

As for the web scanning.... I use the MediaFileAPI.GetMediaFiles(mask) to return the list of media files. For some reason, I am seeing tuners showing up as media files as well, but I just thought this was normal. Also, note that when you are doing a scan from the web UI, then only results that are returned in the list, are those that CANNOT be automatically updated. All auto updated results are not returned in the list. My thinking here is that you'd only want to edit those results that cannot be automatically updated.

I'll try to get a build out later today.

[gplasky]

Originally Posted by stuckless

Gerry, I think that the image url must be invalid, but I'm going to add some code around the image download code to better check for errors and to ensure that 0bytes files are not created.

As for the web scanning.... I use the MediaFileAPI.GetMediaFiles(mask) to return the list of media files. For some reason, I am seeing tuners showing up as media files as well, but I just thought this was normal. Also, note that when you are doing a scan from the web UI, then only results that are returned in the list, are those that CANNOT be automatically updated. All auto updated results are not returned in the list. My thinking here is that you'd only want to edit those results that cannot be automatically updated.

I'll try to get a build out later today.

I went back to beta 6. If I let the automatic plugin update it or do a scan from the web UI I got a 0KB background file. But if I deleted that directory and performed the scan from the SageMC UI and the metaupdater STVI then I received valid background files. (And also banner and posters.)

I went to beta 8 again and deleted everything in the fanart\tv directory. I performed a manual metaupdate from within the SageTV MC UI and the stvi and got valid background, posters and banner files. But I think (switching back and forth is very confusing) if I scanned from the web UI then I only got a background directory and a 0KB background jpg. (No banner or poster directories or downloads.

Defintely beta 6 did NOT show the tuners from the webUI when you performed a scan. Beta 8 does. I'm also seeing a lot of blank listings in addition to the media files and the media tuners being listed. They are legitimate media files with no titles because if I click on them and look at the details there are descriptions and other data.

At this point I'm manually updating the TV files thru the mataupdater stvi and not doing a webUI scan.

Looking forward to your next build whenever you can get ti done. You've been working pretty hard on updating this and supporting it in the thread and I appreciate the hard work you're putting into it. Part time or not it is still a lot of work.

Gerry

[jphipps]

Originally Posted by stuckless

Jeff, I missed an exit point from the plugin... Basically if the metadata plugin has existing properties then they are returned... It will now check the "work around" flag instead of just returning the props.

Thanks...

[MitchSchaft]

Has anyone had problems with imdb failure?

[stuckless]

Originally Posted by MitchSchaft

Has anyone had problems with imdb failure?

Seems to be working for me. You can test imdb from the commandline, using, the --offline arg.

# java -jar MetadataTool.jar --provider=imdb --offline --auto=false --metadataOnly "Transformers.avi"

This will create a 0byte "Transformers.avi" file in the current directory and then do the lookup.

EDIT: Added --provider arg to specify imdb

[rsagetv99]

Originally Posted by stuckless

It actually uses the Title now. The problem is that bmt doesn't know that a recorded movie is a "movie" and it treats it as a recorded TV show. As a result when bmt does the lookup, only tvdb handles the query, since that is the only provider that can handle tv. BMT doesn't bother passing it to imdb or tvdb, since those providers don't handle tv.

I made a recent change in the phoenix api to set the media type of Movie for recorded movies. This was done so that fanart could be assigned correctly to recorded movies. I need to pull that logic into the core bmt search factory as well.

For the autoscan, would it be better to check check imdb (or other movie sources) if any file does not find a 90% match?

If you follow my logic, I am thinking that your fix to the phoenix api would only find metadata/fanart for movies that are in the sage DB, but not for movies that are in your imports directory - since they may not have the Movie category set.

For instance, I have a lot of movies in import directories with the sage filename format due to a corrupted wiz.bin a while back. These movies do not correctly pull the fanart/metadata either.

Also, is there a way to parse the filename for titles if a distinct title does not exist? For example, the same movies that I am speaking of, some just have the default title that matches the filename. I was thinking that maybe the filename could be parsed to return the correct title. The format seems pretty standard.

E.G. TheNightListener-123456-2.mpg
Parse could find the first "-" character, and if the characters that follow are "numbers-number.ext" remove everything after and including "-" . You would end up with "TheNightListener".

Then put a space before every capital letter (except the first one) and voila, you have the correct title. "The Night Listener"

As easily as I have explained it, I know it is much more difficult to code, and I am sure there are some exceptions that would have to be coded around.

Thanks for all your hard work.

[stuckless]

rsagetv99 - Those are very good points. The curent model starts at the beginning of the list and searches all providers until there is a match of at least 90%. I don't cross search, which is what you are asking. ie, Do a TV search, and then do a Movie search if the match fails using the TV search. I'll look into this, but I have no idea how easy it will be to do it... also, I don't know if I should make this an option, or just do it always.

I'm sure it will be pandoras box if I start it, so I'm a little reluctant to start it

[rmac321]

Originally Posted by stuckless

rsagetv99 - Those are very good points. The curent model starts at the beginning of the list and searches all providers until there is a match of at least 90%. I don't cross search, which is what you are asking. ie, Do a TV search, and then do a Movie search if the match fails using the TV search. I'll look into this, but I have no idea how easy it will be to do it... also, I don't know if I should make this an option, or just do it always.

I'm sure it will be pandoras box if I start it, so I'm a little reluctant to start it

I haven't got a clue how hard it would be to code, so take this for what its worth. If I understand correctly, you are first checking the mediatype metadata field to see if it is a movie or tv. Would it be possible to add a notification of which it was looking for if it didn't get a good match? IOW, if the field is TV and it looks in thetvdb and doesn't get a good match, it would notify the user that it failed to find a tv match so the user would know that they need to reset the field to movie. To get really intricate, you could even offer an option to reset the field.

[rsagetv99]

Originally Posted by stuckless

rsagetv99 - Those are very good points. The curent model starts at the beginning of the list and searches all providers until there is a match of at least 90%.

Wait, that's kind of the same thing. If you are searching all providers (movie and TV) against a title until you get a 90% match, then it should pick up my movies - not sure why it's not, unless you don't have this model ported over to the auto-plugin yet.

Not sure how much longer it would take to search all providers for every title, but it would be nice (and maybe more accurate) for the tool to check all providers, and choose the highest match above 90%. Personally, I have had a few Movies that matched with TV shows, and maybe this method would reduce the occurrence of this.

Either way, these are just suggestions. It's easy to raise your hand and make a suggestion, it's a lot harder to make them work.

[PiX64]

stuckless -

I was wondering if there is a way to get your tool to traverse into a directory-subdirectory when running a full re-index?

I have folders such as

The-Lord-Of-The-Rings-Trilogy -> The Fellowship-Of-The-Ring -> Fellowship Disc 1 & then another directory for fellowship disc 2.

The .properties file sits (i believe as it should) one directory above the VIDEO_TS.

The issue is that when i run a full re-index All properties files are updated as expected EXCEPT for the properties files in the directory structure above.

Any thoughts?

*EDIT*
I also forgot to ask -- with the new tool when i run the scan to view my current collection, it finds all of my movies, but will only display 4 of them. Any ideas on that too?

[razrsharpe]

Originally Posted by PiX64

I was wondering if there is a way to get your tool to traverse into a directory-subdirectory when running a full re-index?

if running from the commandline pass the --recurse=true command line option

[PiX64]

I was not actually running from command line, i was running it through the webpage.

I can certainly try running it from the commandline, but the last time i tried to use the commandline, i couldn't get it to update the actualy properties file on my NAS. It would update the fanart and such, but not on the NAS....so typing that all out i think i figured out why..Freenas file permissions..ha

ill try it and let ya know.

[stuckless]

Originally Posted by rsagetv

Wait, that's kind of the same thing. If you are searching all providers (movie and TV) against a title until you get a 90% match, then it should pick up my movies - not sure why it's not, unless you don't have this model ported over to the auto-plugin yet.

Sort of the same thing.... yet different Each provider in BMT has a flag that identifies what type of queries it will support (TV, Movies, Music, etc). When a filename is scraped, it is assigned a media type (TV, Movies) and then the scanner will poll each provider to see if it supports a search for the given media type, and if does, then the title is passed into the provider. This is done this way, so that eventually when music is handled, we are not wasting time passing music files into tv providers, and tv files into music providers, etc. Part of the solution is to make the imdb provider accept TV querries, which it currently does not do. The other part of the problem is unmunging the Title into something useful.

Originally Posted by Pix64

I was wondering if there is a way to get your tool to traverse into a directory-subdirectory when running a full re-index?

I have folders such as

The-Lord-Of-The-Rings-Trilogy -> The Fellowship-Of-The-Ring -> Fellowship Disc 1 & then another directory for fellowship disc 2.

The .properties file sits (i believe as it should) one directory above the VIDEO_TS.

The issue is that when i run a full re-index All properties files are updated as expected EXCEPT for the properties files in the directory structure above.

2 things... The first is that the WebUI doesn't actually traverse the filesystem, it traverses your collection. This is subtle, but the bottom line, is that there is no need to have a recurse option in the web ui, because unlike the commandline that starts at a folder, the web ui simply gets all of your media files using the sage api GetMediaFiles() and then traverses them.

The second thing is that it's probably not able find a match because of the folder name "Fellowship Disc 2". The search will be for "Fellowship" which may not match anything with any confidence (ie > 90%). Maybe if the disc directories were called, "The Fellowship-Of-The-Ring Disc 2" then it may match. It would be very difficult to have BMT know about a multiple directory hierarchy when trying to scrape titles.

Originally Posted by Pix64

I also forgot to ask -- with the new tool when i run the scan to view my current collection, it finds all of my movies, but will only display 4 of them. Any ideas on that too?

The scanner is constantly being changed... and I keep changing my mind But in the current beta, if you see any files after doing a scan, it's because those are the files that could not be automatically updated. (ie, missing metadata, fanart, or just plain errored out). Ideally, if you did a scan, then the output would be empty, because it got 100% of the movies, etc I figured that while doing the scan, if the scanner returned the failed items, then it would give you a chance to manually update/scan the item, while the background scanner is carrying on. That's why you get the edit panel when you click on the item, but it does not yet allow you do anything useful (working on it )

[MeInMaui]

Hi Sean,

I'm just going to think out loud here.

Instead of the current search method, would it be possible to allow configuration of providers independently for each media type, and then remove the restrictions on supported types for the providers? Further, could the providers in the list be tagged as to whether they will be used to provide metadata, fanart/images, or both?

With this approach, there would be a prioritized list of providers for TV (say thetvdb.org then imdb.com), a separate list for Movies ( maybe DVDProfiler, themoviedb.org, then imdb.com), a third list for Music, etc. BMI would then search the list of providers for the specified media type until a 90% match is found. I'm thinking that this would allow a high degree of specificity and customization while keeping configuration fairly simple. Appropriate default provider lists for each type would allow for a zero-configuration install.

Aloha,
Mike

[LehighBri]

Originally Posted by MeInMaui

Hi Sean,

I'm just going to think out loud here.

Instead of the current search method, would it be possible to allow configuration of providers independently for each media type, and then remove the restrictions on supported types for the providers? Further, could the providers in the list be tagged as to whether they will be used to provide metadata, fanart/images, or both?

With this approach, there would be a prioritized list of providers for TV (say thetvdb.org then imdb.com), a separate list for Movies ( maybe DVDProfiler, themoviedb.org, then imdb.com), a third list for Music, etc. BMI would then search the list of providers for the specified media type until a 90% match is found. I'm thinking that this would allow a high degree of specificity and customization while keeping configuration fairly simple. Appropriate default provider lists for each type would allow for a zero-configuration install.

Aloha,
Mike

This is exactly how XBMC does it and it's definitely a great feature. XBMC has a list of scrapers per content type (TV, Video, Music, etc) and you can choose what the preferred scraper is per. Something XBMC doesn't seem to have is what "capability" each scraper has (e.g. fanart only, metadata only, both, etc).

More here: http://www.xbmc.org/wiki/?title=Set_...(Video_Library)

[stuckless]

Originally Posted by MeInMaui

Hi Sean,

I'm just going to think out loud here.

Instead of the current search method, would it be possible to allow configuration of providers independently for each media type, and then remove the restrictions on supported types for the providers? Further, could the providers in the list be tagged as to whether they will be used to provide metadata, fanart/images, or both?

With this approach, there would be a prioritized list of providers for TV (say thetvdb.org then imdb.com), a separate list for Movies ( maybe DVDProfiler, themoviedb.org, then imdb.com), a third list for Music, etc. BMI would then search the list of providers for the specified media type until a 90% match is found. I'm thinking that this would allow a high degree of specificity and customization while keeping configuration fairly simple. Appropriate default provider lists for each type would allow for a zero-configuration install.

Aloha,
Mike

What are you doing??? I thought you'd be be last person to try and create more work for me

Great suggestion (although I think that someone, maybe sflamm, may have posed something similar a few months back).

It's not quite trivial and I have looked at doing it. I may still do it... but it will require some extra thought on how I would go about configuring everything, especially with the fanart stuff.

[MeInMaui]

Originally Posted by stuckless

What are you doing??? I thought you'd be be last person to try and create more work for me

I know, I know....

It's more work now, but if you can get the success rate up then you'll spend less time on support later.

Aloha,
Mike

[Taddeusz]

I'm not sure if this has been fixed in later releases. I'm currently using 2.3.1.

The problem I'm having is that files which have a colon in the name like "Kill Bill: Volume 1" end up with an empty file with the name of the text before the colon. Like the movie in my example has an empty file "Kill Bill" created with it.

Edit: Never Mind, it appears that SageMC is doing this when the MKV has a title with a colon in it. This also prevents SageMC from properly importing the cover art from DVD Profiler but is for some reason able to get everything else. Sorry, this doesn't appear to be a problem with BMI.