Utility: Batch Metadata Importer 3.x

[S_M_E]

Originally Posted by stuckless

Currently the search works by removing all non alphanumeric characters, and then it removes certain words (such as TS, CAM, DVDRip, etc), so a filename that has a year in it should still search using the year. The --agressive switch is actually used so that if a match is NOT found, then it will then attempt to remove the year from the title and search again. So by default, if a filename has a Year in the filename, then it will become part of the search criteria. (hope that clears it up)

Merry Christmas...

I understand what you're saying but I'm not sure it's working that way using all of the providers. I usually try nielm_imdb because it has more complete information, perhaps that's why I'm having other issues too.

Here are some of the issues I'm having:
1) The Maxx (1995).mp4 (and VIDEO_TS from the dvd)- If I use the imdb provider it get's picked up as Mad Max. If I use the nielm_imdb provider it lists options but not the right one. If I type in The Maxx it returns the correct option as 01. If I type 01 it fails.

56813 INFO [main] org.jdna.metadataupdater.MetadataUpdater - Processing Media File: file:////Abit/videos/Movies/The%20Maxx%20Collection/The%20Maxx%20(1995)%20DVD/
56828 DEBUG [main] org.jdna.media.metadata.impl.sage.SageVideoMetaDataPersistence - Loading Sage Video Metadata properties: \\Abit\videos\Movies\The Maxx Collection\The Maxx (1995) DVD.properties
56844 DEBUG [main] org.jdna.media.metadata.impl.sage.SageVideoMetaDataPersistence - Loading Sage Video Metadata properties: \\Abit\videos\Movies\The Maxx Collection\The Maxx (1995) DVD.properties
56844 WARN [main] org.jdna.media.metadata.impl.sage.SageVideoMetaDataPersistence - Failed to parse cast member from string:
56844 WARN [main] org.jdna.media.metadata.impl.sage.SageVideoMetaDataPersistence - Failed to parse cast member from string:
56859 DEBUG [main] org.jdna.metadataupdater.MetadataUpdater - Refreshing MetaData for: file:////Abit/videos/Movies/The%20Maxx%20Collection/The%20Maxx%20(1995)%20DVD/
57281 ERROR [main] org.jdna.media.metadata.impl.nielm.NielmIMDBMetaDataProvider - IMDB Lookup Failed:http://www.imdb.com/title/tt0112065/
java.lang.NumberFormatException: For input string: "UK:23"

So I end up using the imdb provider, it thinks it's Mad Max, then I edit the .properties file with the correct tt# from the imdb site and run an --update using the imdb provider. (I'm not having much luck with themoviedb.com provider either.)

2) Often (with many files), trying to --update imdb metadata with nielm_imdb metadata fails with various errors:

Video: file:////Abit/videos/Movies/The%20Maxx%20Collection/The%20Maxx%20(1995)%20DVD/
Error: java.lang.NumberFormatException: For input string: "UK:23"

(and more commonly)
Video: file:////Abit/videos/Movies/Ocean's%20%23%20Collection/Ocean's%2011%20(2001).avi
Error: net.sf.sageplugins.sageimdb.DbFailureException: java.io.FileNotFoundException: http://www.imdb.comhttp://www.imdb.c...mbinedcombined

3) I tried to use tt0112065 to specify which movie but I must have the wrong syntax, how am I supposed to do it?

Speaking of syntax, here are some of the commands I (try to) use, maybe I'm doing something wrong:

java -jar MetadataTool.jar --recurse --prompt=true --provider=nielm_imdb --setProperty=/imdb/forcedIMDBImageSize:1920 \\Abit\videos\Movies

(the imagesize property doesn't seem to help, I still get low quality, small thumbnails so I manually dl many covers which is a major pita)

java -jar MetadataTool.jar --recurse --prompt=true --provider=imdb --update --setProperty=/imdb/forcedIMDBImageSize:1920 "\\Abit\videos\Movies\The Maxx Collection\The Maxx (1995) DVD"

java -jar MetadataTool.jar --recurse --update \\Abit\videos\Movies\

Please advise...

[Karen0302]

How do you run Java in a command line window. Vista keeps telling me it can't find Java.

Karen

[S_M_E]

You probably have to open the command prompt as administrator but I haven't tried it in Vista...

[Karen0302]

I tried that but Vista still tells me that 'java' is not recognized as an internal or external command.

Karen

[S_M_E]

Originally Posted by Karen0302

I tried that but Vista still tells me that 'java' is not recognized as an internal or external command.

Karen

I just ran "java -version" on one of my vista boxes and it worked, perhaps you should re-install/update your JRE...

[stuckless]

I understand what you're saying but I'm not sure it's working that way using all of the providers.

Actually the search processor is independent from the search providers, so the process of searching with the date, and then without, will happen no matter which provider is used. This can be seen from the output when using the nielm_imdb provider...

IMDB getting URL:http://www.imdb.com/find?s=tt&mx=200&q=The+Maxx++1995
IMDB getting URL:http://www.imdb.com/find?s=tt&mx=200&q=The+Maxx++

If you pass --agressive=false then it will only perform the first search, with the date, and never attempt the second search if the first one doesn't yield anything useful.

I tried to use tt0112065 to specify which movie but I must have the wrong syntax, how am I supposed to do it

I tried it using the nielm_imdb provider, and it fails. It works with the default imdb provider. I don't how neil is actually performing searches, so I can't offer any explanation as to why it would fail. When you enter the search into the web it works as well.

the imagesize property doesn't seem to help

This is more of a hint to the imdb service. If there are larger images available then you will get it, otherwise, you'll get the default thumb size. For you particular case, there wasn't a larger one available. I did try it a couple of times, and to be honest, I don't think that IMDB has many larger sized images.

2) Often (with many files), trying to --update imdb metadata with nielm_imdb metadata fails with various errors:

Video: file:////Abit/videos/Movies/The%20Maxx%20Collection/The%20Maxx%20(1995)%20DVD/
Error: java.lang.NumberFormatException: For input string: "UK:23"

The nielm apis are throwing an exception when getRuntime() is called. I can trap the error so it doesn't cause the whole thing to fail.

(and more commonly)
Video: file:////Abit/videos/Movies/Ocean's%20%23%20Collection/Ocean's%2011%20(2001).avi
Error: net.sf.sageplugins.sageimdb.DbFailureException: java.io.FileNotFoundException: http://www.imdb.comhttp://www.imdb.c...mbinedcombined

That I can't fix. The nielm api seems to somehow calculate/use an incorrect url.

The Maxx (1995).mp4 (and VIDEO_TS from the dvd)- If I use the imdb provider it get's picked up as Mad Max.

The seach interpeter assumes the any "popular" or "exact" hits are good to use as an exact match. Sometimes, as in this case, it's wrong. Not all search providers provider, such as the nielm_imdb provider, can provide information about the quality of the search result. For those providers will always prompt you. For providers that do return information about the quailty of the search result, it usually is the "popular" or "exact" match. It's usually right more than it's wrong. So the basic premise that it's better to have a few bad matches than to be prompted for everyone.

That being said, I'm looking at putting back the --auto=false switch so that you can be prompted.

The reason the nielm_imdb doesn't show the correct results for the query, is because it's only showing your results of the first search (with the year). When you search without the year, (just enter The Maxx at the prompt), then your movie is the second choice.

I'm not having much luck with themoviedb.com provider either

That provider is a little more pickier than the imdb provider in how it searches. It seems that themoviedb.com will result in a larger number of failures if the titles aren't really clean. IMDB is a little forgiving. Although, I hope it stays around... It's the only open provider that has an external api, which makes it easy to implement

Good luck and Merry Christmas!

[S_M_E]

Very thorough explanations, thank you.

Since the main reason that I use the nielm_imdb provider is for the more complete data, perhaps you could add a --switch to the default imdb provider (and maybe themoviedb.com provider) to fetch more complete data?

Sometimes bigger images are available; I, Robot for example.
The .properties file lists media-imdb.com/images/M/MV5BMTI0NjI0NTYyOV5BMl5BanBnXkFtZTcwNTQ0MjcyMQ@@._V1._SX100_SY140_.jpg
but manually searching imdb.com yields the default:
http://ia.media-imdb.com/images/M/MV...285_SY400_.jpg
and if I change the url:
http://ia.media-imdb.com/images/M/MV...20_SY1920_.jpg
so, despite the the imagesize property, it's actually fetching a smaller image than what's available in most cases.

Perhaps there could be a way to specify image fetch urls in the metadata.properties file:
/media/imageFetchURLorder=nicecovers.com|imdb.com|themoviedb.com|dvdprofiler|etc...
(nicecovers.com has nice covers but the selection is lacking.)


Also, this has nothing to do with BMI but perhaps another STVi could fix the Media Center - "Search" behavior to better use your data. If, for example, I have all of the Rocky movies in a folder named "Rocky Collection" I can easily select that folder (in folder view) and either play all or add all to the playlist. However, if I "search" by actor in Videos or TV I don't get an option to play and/or queue all of the results so I can't easily queue up a complete Stallone marathon. Does that make sense?

[stuckless]

Since the main reason that I use the nielm_imdb provider is for the more complete data, perhaps you could add a --switch to the default imdb provider (and maybe themoviedb.com provider) to fetch more complete data?

I'll look into it... but which type of more information are you looking for. Just better "plot" or more actors, etc. The reason I include the nielm provider is for people that just want more

so, despite the the imagesize property, it's actually fetching a smaller image than what's available in most cases.

Again, the size parameter is really just a hint to IMDB to fetch an image with a Max size of the one the specify. In the url link that provided, but the 1920 image, I actually get an image that 356px × 500px, which is still much larger than the default thumbnail. When you are updating existing entries, are you using the --updateThumbnail, because if not, then the property file will get updated, but it will not overwrite an exsiting thumbnail unless the --forceThumbnail is used.

Perhaps there could be a way to specify image fetch urls in the metadata.properties file

You can specify a url to a thumbnail in the properties file, and then use the --update --forceThumbnail, and it will download the thumbnail that you specify. I know this is not what you are asking... but just letting you know.

As for coverart... I'm working on it, but coverart is hard to do in a command line application. Also, after wrting the screen scraper for imdb and dvd profiler (remote urls), I'm no fan of screen scraping. It's tedious and prone to errors. Also, many sites are moving to javascript rendering, so that screen scrapers are becomming harder to write. So, currently I use google images to find the image that I want, then I past that in the properties file. The other thing is that you can, since the CompositeProvider, search using imdb, and then use themoviedb.com for images/data.

FYI... if I do another screen scraper, it will probably be for tv.com... I have alot of TV shows that need metadata

[S_M_E]

I'm looking for everything; plot, cast, trivia, quotes, etc. I'm not against using the nielm provider if it worked but even you said there's nothing you can do to fix some of the issues like the broken URL. If the default imdb provider had a --verbosity option to pull down everything nielm's provider fetches then we wouldn't need it, we could just set --verbosity=full or add it to the main metadata.properties file...or whatever.

It only pulls down the small thumbnails on the first run. I don't use --forceThumbnail or --updateThumbnail in batch mode because it would overwrite the ones I've manually downloaded too. I could use that on individual files that may be stubborn. After I get all of the images right I'll probably make them all read-only then, hopefully, they wont get overwritten.

I'd try the composite provider but my file names aren't exact enough for themoviedb.com and you warned against it if we don't know what we're doing...and I don't. Luckily, I only have a few hundred, not thousands.

[Kroc]

Thank you for creating such a wonderful tool.

I just read through the entire Wiki and Thread. I started out testing the Manual Meta Data Tool, which works great. Now I am testing the sage plugin. I am unable to get the plugin to work automatically, so I tried testing it on the command line using java –jar –metaupdata-updater-1.13.jar. It gives me an error Failed to load Main-Class manifest attribute from metadata-updater-1.13.jar. Do I have to point it to the location of the MetadataTool.jar? I also need to use the –recurse switch to go through all my sub directories. Can I specify/define –recurse in a metadata.properties or sage.properties.metadataupdate?

Thanks again for a great utility. I have been using the TV scrapper and IMDB manually in Sage. This will save so much time.

[stuckless]

I also need to use the –recurse switch to go through all my sub directories. Can I specify/define –recurse in a metadata.properties or sage.properties.metadataupdate?

Currently, no. I think that most people create a simple batch script with the most commonly used options for their environments.

I am unable to get the plugin to work automatically, so I tried testing it on the command line using java –jar –metaupdata-updater-1.13.jar

The command line tool and the plugin are packaged separately because they are used differently. You cannot "execute" the plugin by using a java -jar command.

The "plugin" mode basically will detect when new files are added to your library, and then automatically try find metadata for the new entries. All of this happens without any user interaction.

If it's not working, then there may be 2 causes, provided it's installed correctly. The first, is that maybe it didn't find any metadata for your movie. I try to name my movies the same name as the title so the metadata tools will have a better chance of finding it online.

The second reason, may be that Sage hasn't detected the new movie entry, and as such, the metadata tools is not being called.

The best place to start, to see if the tool is being called, is the sagetv_0.txt log file. If the tools is being called, you will see the following lines...

Thu 12/25 20:09:00.156 ** Batch Metadata Plugin; Using ProviderId: imdb
Thu 12/25 20:09:00.156 ** Configuration for Metadata Plugin: /opt/sagetv/server/metadata.properties

If this doesn't show up anywhere in the log file, then check that the plugin is installed in the Sage.properties file. If you edited the file while sage was still up and running the changes will not be persistent. If everything looks OK, then tell Sage to refresh your media. If Sage hasn't picked up you new movie, then it should when you force it to refresh.

If all seems to be correct, and you still can't figure it out, then feel free to PM with your sagetv_0.txt log file. I can take a look and see if I see anything obvious.

good luck.

[Kroc]

Thanks for the fast response.

The plugin is working perfectly. I misunderstood the application of the utility. I did not know that it would only worked on new media being added to the library and not some existing media that had no cover art, cast, or summary. I added some new media performed a rescan and it worked nicely.

I have been waiting for this utility for a long time. Great work.

[stuckless]

That's a good point. The plugin does not scan the existing media. It's only good for media as it is added to the library.

Originally Posted by Kroc

Thanks for the fast response.

The plugin is working perfectly. I misunderstood the application of the utility. I did not know that it would only worked on new media being added to the library and not some existing media that had no cover art, cast, or summary. I added some new media performed a rescan and it worked nicely.

I have been waiting for this utility for a long time. Great work.

[electronics4lif]

When adding a new movie and then refreshing, I am getting an
"Could not automatically update:" file:////the name of my movie in my log file. I do see the Batch Metadata Plugin in the log. The command line tool works just fine though. Any thoughts on why it could not automatically update? Thank you.

[stuckless]

Originally Posted by electronics4lif

When adding a new movie and then refreshing, I am getting an
"Could not automatically update:" file:////the name of my movie in my log file. I do see the Batch Metadata Plugin in the log. The command line tool works just fine though. Any thoughts on why it could not automatically update? Thank you.

If there was a java error, then there should be an "Exception" in the log file as well. Otherwise it appears that maybe a good match could not be found, and as a result, the file is not automatically updated.

Also, check to make sure that you only have a single version of the metadata-updater jar and the sagex-api jar in the JARs directory.

[stuckless]

You can download new released of the BMT and the BMT Plugin.

New new features include...
Fanart (backdrops) Downloading
Only themoviedb.com supports backdrops, so if use this provider, it will grab the fanart as well.

Backdrops are stored in the same directory are the movie, thumbnail, and properties. I do not have suppprt, nor will I add it, for the option to store fanart in a separate location. Since this tool runs outside of Sage, I have no idea how I would support storing metadata to another location.

2 New Composite Providers
2 new composite providers are now provided as standard providers. The are imdb-2 and themoviedb-2.

imdb-2 is a provider that searches using the default imdb provider but supplements it's details using Nielm's imdb provider.

themoviedb-2 is a provider that searches using the default imdb provider but supplements it's metadata from themoviedb.com. This this provider will allow you to download backdrops WHILE fetching metadata for a movie.

Disabling Backdrops
Use the --backdropsNever to prevent backdrops from being downloaded. Or set this property in your metadata.properties

/sageMetadata/ignoreBackdrop=true

Downloading ONLY backdrops
If you want to download ONLY backdrops, then set the provider as themoviedb.com and pass the --backdropsOnly command.

The command to download only backdrops is a simple command. It will do a search on the title, not the filename, and download if there is a hit. If a hit cannot be found, then it will skip. There is no manual searching.

Misc Options
--auto is back. You can set it to false, and it will prompt for every file.
--displaySize is fixed. You can change displaySize to show more results in manual searching. (default is 10)
--thumbnailMaxWidth can be used to limit the thumbnail size. For example if you never want a thumbnail to be larger than 400pixels, then set this to 400. It will not upscale, it will simply downscale, if the thumbnail is larger than the value you pass. This can be handy if the thumbnail content is really large.

Any provider can be used with a Composite provider
Initially only IMDB providers could be used in the composition. Now you can use any provider. For example you could create a composite provider that searches using dvd profiler, but then supplements it's details using themoviedb.com. This would add fanart support to a dvd profiler provider.

[bcjenkins]

Sean,

This is great. How would one add DVD local as a composite provider?

TIA

B

[stuckless]

Originally Posted by bcjenkins

Sean,

This is great. How would one add DVD local as a composite provider?

TIA

B

In your local metadata.properties, add the following, for starters...

/compositeMetadataProviders/dvdprof2/name=DVD Profiler / Themoviedb.com Provider
/compositeMetadataProviders/dvdprof2/description=Composite provider that uses DVD Profiler for searching and supplements with themoviedb.com

/compositeMetadataProviders/dvdprof2/iconUrl=
/compositeMetadataProviders/dvdprof2/searchProviderId=dvdprofiler_local
/compositeMetadataProviders/dvdprof2/detailProviderId=themoviedb.com
/compositeMetadataProviders/dvdprof2/fieldsFromSearchProvider=

Now, if there are fields, that you absolutely MUST have comming from dvd profiler, then add those fields as a semi-colon separated list of values to the /compositeMetadataProviders/dvdprof2/fieldsFromSearchProvider field.

For example,

/compositeMetadataProviders/dvdprof2/fieldsFromSearchProvider=GenreList;PosterArt

That will force the provide to pull the Genre information and the main cover art from dvd profiler, if it exists.

The full list of supported metadata fields can be seen using the --showSupportedMetadata command.

here's the list.

               Album : 
         AlbumArtist : 
              Artist : 
         AspectRatio : 
       BackgroundArt : 
      CastMemberList : 
             Comment : 
             Company : 
            Composer : 
         Description : 
            Duration : 
             Episode : 
           GenreList : 
            Language : 
          MPAARating : 
MPAARatingDescription : 
        MediaArtList : 
           PosterArt : 
     ProviderDataUrl : 
          ProviderId : 
         ReleaseDate : 
         RunningTime : 
              Season : 
               Title : 
         TotalTracks : 
               Track : 
          UserRating : 
                Year :

Once you've added the provider to you properties, then you can test it using --provider=dvdprof2

Hope that helps.

[S_M_E]

That does sound great.

I'd run updates trying the imdb-2 provider but sometimes IMDB defaults to a foreign title rather than the US title and then the metadata has to be manually edited. That's not BMI's fault it's IMDB's fault.

I just tried a --backdropsOnly test on one file and it failed, I assume because I use file names with dates. Is there any way to get themoviedb.com searches to ignore dates in the " (xxxx)" format in order to get better hits from that provider? I just ran the same test with the --auto=false switch and it prompted and fetched the properties but not the backdrop. I'm confused, I thought the --backdropsOnly switch only grabbed an image?

[stuckless]

I just tried a --backdropsOnly test on one file and it failed, I assume because I use file names with dates

backdropsOnly only works if there is existing metadata for a file. It uses the Title as the search and not the filename. --backdropsOnly was added because I already have metadata properties for all my movies, and I wanted to way to just update the backdrops without have to research for the movies. So, to clarify, --backdropsOnly does not use the filename, it uses the movie Title from an existing movie properties file.

where --backdropsOnly is passed, pretty much all other command line properties are ignored. So using --auto=false with --backdropsOnly doesn't work. Mainly because backdropsOnly is a simple non interactive process that fetches backdrops for movies if it can find an exact match title.

Is there any way to get themoviedb.com searches to ignore dates in the " (xxxx)" format in order to get better hits from that provider?

It should do that now already. Most of my movies have dates in the filename and I don't have an issue. Or maybe I'm not understanding the issue?

I just ran the same test with the --auto=false switch and it prompted and fetched the properties but not the backdrop

This should prompt and it will only download a backdrops, if one exists in themoviedb.com database.

Perhaps you can paste some of your movie titles that you are having issues with, and the commandlines that you are using. I can then test them using the --offline switch.