Utility: Batch Metadata Importer 3.x

[electronics4lif]

Hello,

Can you take a look at my log file and see if you see anything wrong. I am still getting the "could not automatically update error" when using the Sage plugin. I checked to make sure that I do not have duplicate JARs. I just updated to version 1.16 this morning hoping it would fix my problem. The command line tool finds the metadata just fine.

I had to zip the file due to it being too large to attach.

Thank you,

[stuckless]

I've fixed a number of bugs. Thanks to everyone that found issues and provided the log files.

Several null pointer exceptions were found and fixed. Also, I was not handling DVD folders correctly int the plugin, so hopefully this is now fixed.

As for enhancements, this release includes an implementation for the Xbmc Scraper file format. So, you can now use any scraper file from the Xmbc project as a metadata provider. I've included their imdb.xml scraper implementation in the scrapers/xbmc/video directory. You can use this provider by passing --provider=imdb.xml on the command line.

You can download and test other [http://xbmc.svn.sourceforge.net/view...crapers/video/ scrapers from their site]. Just put the scraper in the scrapers/xbmc/videos directory and it will be automatically picked by the tool (retart required if you are running as a plugin). Keep in mind, that scraping is tedious, and as a result, many scrapers are broken, especially if their maintainer is retired. For example, I tried the amazonus.xml scraper it was broken. So you mileage may vary.

You can read the wiki for more information on using the Xbmc scrapers.

In time, as the the Xbmc scraper becomes more stable, I'll probably deprecate my implementations of the Imdb scrapers in favour of theirs.

Even though some of the Xbmc scrapers are for TV, BMT does not yet support processing TV shows.

[S_M_E]

I get the same problem where it prompts for a handful of media that already has metadata but now when I try to get backdrops it fails completely.

java -jar MetadataTool.jar --recurse --prompt=true --provider=themoviedb
.com --backdropsOnly --setProperty=/imdb/forcedIMDBImageSize:1920 \\Abit\videos\Movies\
Batch MetaData Tools (1.17)
Processing Failed, see log for details.

Log attached...

[stuckless]

I'm really sorry I forgot to mention that the provider 'themoviedb.com' was renamed to 'themoviedb.org'. So, if you are using themoviedb provider, you'll need to use the new ID.

Sorry about this, but their is no such site as themoviedb.com. I noticed a comment on the wiki about it, and I quickly made the change, but I forgot to relay the information.

[S_M_E]

That would explain the failure.

I'm still getting the fatal errors while fetching backdrops but you said that it was due to their api. I'm also still seeing a lot of skipped but I think that was low on your priority scale.

I still don't know why it won't skip some files that already have metadata. Any suggestions? I've ran BMT from my workstation and from the server, same results.

[evilpenguin]

Originally Posted by stuckless

As for enhancements, this release includes an implementation for the Xbmc Scraper file format.

Wow, good job, that must have been a pain in the ass. Those XMBC scrapers are soooo complicated.

[stuckless]

Originally Posted by evilpenguin

Wow, good job, that must have been a pain in the ass. Those XMBC scrapers are soooo complicated.

I admit, it was a challenge at first, and I doubt that I have it 100% yet Once I got started, it turned out not to be that bad. They only have <regexp> and <expression> elements, so once you figure it out once, it's just repeating the calls from there on in. I think their biggest twist is what they "Functions". The site is fairly well documented, so that helped a lot.

It's interesting because I was working on a scraper tool, then I saw a post that you wrote about the xbmc scrapers, so I decided to take a look. I didn't realize that they used a scraper language until then, so thanks for pointing it out

I know you ended up doing your own, and there was a part of me that wanted to do the same as well But in the end, I figured that by implementing the Xbmc scrapers, I could leverage the work/maintenance that the xbmc team would provide. (That means I spend less time maintaining this )

[stuckless]

Originally Posted by S_M_E

That would explain the failure.

I'm still getting the fatal errors while fetching backdrops but you said that it was due to their api. I'm also still seeing a lot of skipped but I think that was low on your priority scale.

I still don't know why it won't skip some files that already have metadata. Any suggestions? I've ran BMT from my workstation and from the server, same results.

It's not so much that it's low on my priority list.... it's just hard to troubleshoot when I can't reproduce. Maybe you can post up a list of filenames that you are having issues with, and I can create those locally and give it a try. The logs that you've sent haven't shown anything that pops out at me.

As for the fatal errors... i've opened a bug, but they haven't acknowledged it yet. If they don't fix it, then maybe I'll move the themoviedb provider into an xbmc scraper. They use regular expressions instead of xml, so it won't care that the xml is not wel formed.

BTW, the imdb.xml providerId actually uses imdb, but it fetched backdrops from themoviedb.org, so that provider may work well for you, as an alternative themoviedb.org one.

[S_M_E]

It's actually 2 different issues.

I get a *lot* of skipped backdrops but I've gone to themoviedb.org website and manually downloaded some of those that were skipped. To figure out which don't have backdrops and which are getting skipped for other reasons would take some time. You said you were going to make that search less strict (I think) but that I shouldn't hold my breath so I assumed it was a low priority item.

The other issue is the one where it keeps prompting for a handful of movies (no matter which provider I use, including imdb.xml) even though metadata and imdb info exists. I gave you logs for one of them but here are more titles:
Kung Fu Killer (2008).avi
Hate 2 O (2006).avi
Alone in the Dark II (2008).avi
The Great Global Warming Swindle (2007).avi
Stolen Honor Wounds That Never Heal (2004).mpg

If you'd like, I can run those manually and attach a log for each one, like I did for Alone in the Dark II (2008).avi

I see that the timestamps get touched so it's not read only and metadata does exist. I don't think it did this using 1.12 but from 1.13-1.15 I had forgotten to remove the older libs so I don't know when this behavior started. I guess I could try 1.12-1.14 to see which work. It's the same server and the files existed before. I did recently upgrade my workstation but I've tried running BMT from the server too so I doubt it's the workstation either.

[stuckless]

SME, Based on the names the you gave me.... none of those entries have coverart on imdb. Since there is no cover art for this media file, then the BMT will continue to prompt you.

I can change the tool so that it only checks for the existence of the properties, because currently, it checks if the file exists, and that it has a Title and _thumbnailUrl. In your case, since it does not have a thumbnail, then it continues to prompt.


I received a response from themoviedb.org. They will not fix the underlying xml issues, but they did fix the movie entry for Wall-E that I sent in. But it looks like either I'll have to stop using an xml parser for their xml, or I'll have to file a bug for each movie that generates an xml issue.

[S_M_E]

Oh, coverart. I did download coverart for those at other sites. I forgot that some titles didn't have art at imdb. You could also check to see if the filename.jpg/.png exists for your skipping function. That way it will keep trying to grab coverart until somebody (like me) finds it elsewhere or scans it to make their own. Now it makes sense. Did that change after 1.12?

Filing a bug for every movie that is broke sounds like a lot of work.

[stuckless]

Originally Posted by S_M_E

Oh, coverart. I did download coverart for those at other sites. I forgot that some titles didn't have art at imdb. You could also check to see if the filename.jpg/.png exists for your skipping function. That way it will keep trying to grab coverart until somebody (like me) finds it elsewhere or scans it to make their own. Now it makes sense. Did that change after 1.12?

Filing a bug for every movie that is broke sounds like a lot of work.

Good idea on the filename checking... I'll add that in with the next release.

[S_M_E]

Awesome, I really appreciate all of your work on this utility. You might also have it say why it's prompting "coverart does not exist," "title does not exist," "metadata does not exist," etc...

[electronics4lif]

Originally Posted by electronics4lif

Hello,

Can you take a look at my log file and see if you see anything wrong. I am still getting the "could not automatically update error" when using the Sage plugin. I checked to make sure that I do not have duplicate JARs. I just updated to version 1.16 this morning hoping it would fix my problem. The command line tool finds the metadata just fine.

I had to zip the file due to it being too large to attach.

Thank you,

Updating to 1.17 now yields the following error "Fri 1/23 15:00:56.122 Error instantiating metadata parser plugin of:java.lang.RuntimeException: No Provider: themoviedb.org"

Any thoughts?

[stuckless]

electrons4lif, could you zip and post your metadataupdater.log file.
Thanks,

[electronics4lif]

Originally Posted by stuckless

electrons4lif, could you zip and post your metadataupdater.log file.
Thanks,

Stuckless,

My metadataupdater.log zipped is attached. Thank you for taking a look at it.

[stuckless]

electronics4lif, could you check your libs directory for this tool and see if you have duplicate metadata-updater jar files. If so, then remove all duplicates except for the one with the largest version number. You don't appear to be running the latest version, based on what I see in the log file, and this could be because there are multiple metadatar-updater jar files in the tool libs directory.

[heffe2001]

I was running just fine, and I noticed that some of my artwork was still from IMDB (I'm switching everything over to the artwork supplied from DVD Profiler, using the dvdprofiler_local setting), so I decided to use the --forceThumbnail and --force (just in case the metadata wasn't what was supplied from DVDPro), and get a Java crash:

C:\metadata\Batch>java -jar MetadataTool.jar --forceThumbnail --provider=dvdprof
iler_local --force --recurse m:\action
Batch MetaData Tools (1.17)
Error Launching Class: org.jdna.metadataupdater.MetadataUpdater
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at MetadataTool.main(MetadataTool.java:134)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.awt.image.DataBufferByte.<init>(Unknown Source)
        at java.awt.image.ComponentSampleModel.createDataBuffer(Unknown Source)
        at java.awt.image.Raster.createWritableRaster(Unknown Source)
        at javax.imageio.ImageTypeSpecifier.createBufferedImage(Unknown Source)
        at javax.imageio.ImageReader.getDestination(Unknown Source)
        at com.sun.imageio.plugins.jpeg.JPEGImageReader.readInternal(Unknown Sou
rce)
        at com.sun.imageio.plugins.jpeg.JPEGImageReader.read(Unknown Source)
        at javax.imageio.ImageIO.read(Unknown Source)
        at javax.imageio.ImageIO.read(Unknown Source)
        at org.jdna.media.metadata.MediaMetadataUtils.writeImageFromUrl(MediaMet
adataUtils.java:33)
        at org.jdna.media.metadata.impl.sage.SageVideoMetaDataPersistence.save(S
ageVideoMetaDataPersistence.java:277)
        at org.jdna.media.metadata.impl.sage.SageVideoMetaDataPersistence.save(S
ageVideoMetaDataPersistence.java:236)
        at org.jdna.media.metadata.impl.sage.SageVideoMetaDataPersistence.storeM
etaData(SageVideoMetaDataPersistence.java:170)
        at org.jdna.media.impl.AbstractMediaResource.updateMetadata(AbstractMedi
aResource.java:164)
        at org.jdna.media.util.AutomaticUpdateMetadataVisitor.fetchMetaData(Auto
maticUpdateMetadataVisitor.java:83)
        at org.jdna.media.util.AutomaticUpdateMetadataVisitor.fetchMetaData(Auto
maticUpdateMetadataVisitor.java:50)
        at org.jdna.media.util.AutomaticUpdateMetadataVisitor.visit(AutomaticUpd
ateMetadataVisitor.java:40)
        at org.jdna.media.impl.AbstractMediaResource.accept(AbstractMediaResourc
e.java:124)
        at org.jdna.media.impl.MediaFolder.accept(MediaFolder.java:174)
        at org.jdna.media.impl.MediaFolder.accept(MediaFolder.java:166)
        at org.jdna.media.impl.MediaFolder.accept(MediaFolder.java:174)
        at org.jdna.metadataupdater.MetadataUpdater.process(MetadataUpdater.java
:272)
        at org.jdna.metadataupdater.MetadataUpdater.main(MetadataUpdater.java:92
)
        ... 5 more

Also attached is my .log file. As long as I run it without the --force or the --forceThumbnail, it runs fine (but doesn't update some of the .properties files).

[stuckless]

Originally Posted by heffe2001

I was running just fine, and I noticed that some of my artwork was still from IMDB....

I'm guessing you have a fairly large dvd profiler collection or you have some very hi-res artwork. The Java virtual machine is running out of memory, so you can try forcing the jvm memory using -Xmx###m

java -Xmx512m -jar MetadataTool.jar --forceThumbnail --provider=dvdprof
iler_local --force --recurse m:\action

if 512 still gives the same error, then try upping the size to 768, then 1024...

[theringnebula]

Hello,

I'm noticing a few things on this new version and I want to make sure I am doing things correctly. I switched to using the imdb.xml profile as I like the information it pulls, but there are a number of movies that don't seem to work and it prompts for every single movie, it can never detect the exact or most popular match. Examples of movies that don't work are An Inconvenient Truth and The Constant Gardiner. By "don't work" I mean that it doesn't generate any files (properties/artwork/etc.) even though it shows a list of search results from IMDB and I tell it the one to use. Am I missing something?

BTW this is a great tool!