Utility: Batch Metadata Importer 3.x

[stuckless]

Originally Posted by theringnebula

Hello,

I'm noticing a few things on this new version and I want to make sure I am doing things correctly. I switched to using the imdb.xml profile as I like the information it pulls, but there are a number of movies that don't seem to work and it prompts for every single movie, it can never detect the exact or most popular match. Examples of movies that don't work are An Inconvenient Truth and The Constant Gardiner. By "don't work" I mean that it doesn't generate any files (properties/artwork/etc.) even though it shows a list of search results from IMDB and I tell it the one to use. Am I missing something?

BTW this is a great tool!

Thanks for trying the new XBMC scrapers and providing feedback. One of the problems with the XBMC Scrapers (imdb.xml) is that the results don't return any information as to whether or not the search result is an exact match, partial match, popular match, etc. Because of the that the results are marked as "unknown" and the tool won't automatically pick a result and it prompts you to select one, every time. This is same problem that I had with the nielm_imdb provider as well, and it's why I wrote my own.

To solve this problem i'll be implementing a results check that will "test" the filename against a result for cases where the results do not include information on the type of result. This will hopefully enable automatic selection for providers that don't provide that information (ie, xbmc scrapers and the nielm_imdb provider)


Your second problem is most likely a bug. The Xbmc scrapers are a new addition and no doubt include some bugs. If you could re-run the tool on a movie that doesn't work, then then post a zipped metadataupdater.log file, then I can take a look for you. Or you can simply reply with the full filename of the movies that don't work, and I can test using the --offline switch.

[theringnebula]

Originally Posted by stuckless

Thanks for trying the new XBMC scrapers and providing feedback. One of the problems with the XBMC Scrapers (imdb.xml) is that the results don't return any information as to whether or not the search result is an exact match, partial match, popular match, etc. Because of the that the results are marked as "unknown" and the tool won't automatically pick a result and it prompts you to select one, every time. This is same problem that I had with the nielm_imdb provider as well, and it's why I wrote my own.

To solve this problem i'll be implementing a results check that will "test" the filename against a result for cases where the results do not include information on the type of result. This will hopefully enable automatic selection for providers that don't provide that information (ie, xbmc scrapers and the nielm_imdb provider)


Your second problem is most likely a bug. The Xbmc scrapers are a new addition and no doubt include some bugs. If you could re-run the tool on a movie that doesn't work, then then post a zipped metadataupdater.log file, then I can take a look for you. Or you can simply reply with the full filename of the movies that don't work, and I can test using the --offline switch.

I will get you some logs when I get home tonight. But here are the paths of a couple that don't work. Usually I drag and drop the "Movies" folder right onto the tool but I also tried to drag and drop the specific movie folder directly as well.

P:\Movies\Inconvenient Truth\VIDEO_TS
P:\Movies\Constant Gardener\VIDEO_TS

All together I think there are 5 or so out of about 80 movies that don't work. I though it might be because I left the article off the title but i have other movies where that is the case and it didn't matter. Also the search does fine a match and I tell it to use the match it never says "Updated..." or produces files.

Thanks for you help and this great tool.

[heffe2001]

I've got about 600 DVD's listed in the DVD Profiler database (and I'd also have to guess the artwork is pretty high-res), I'll try running it with the 512m setting when I get home this evening and see how it fares.


UPDATE: Yep, 512m fixed it. Have to remember that next time .

Originally Posted by stuckless

I'm guessing you have a fairly large dvd profiler collection or you have some very hi-res artwork. The Java virtual machine is running out of memory, so you can try forcing the jvm memory using -Xmx###m

java -Xmx512m -jar MetadataTool.jar --forceThumbnail --provider=dvdprof
iler_local --force --recurse m:\action

if 512 still gives the same error, then try upping the size to 768, then 1024...

[stuckless]

Originally Posted by theringnebula

I will get you some logs when I get home tonight. But here are the paths of a couple that don't work. Usually I drag and drop the "Movies" folder right onto the tool but I also tried to drag and drop the specific movie folder directly as well.

P:\Movies\Inconvenient Truth\VIDEO_TS
P:\Movies\Constant Gardener\VIDEO_TS

All together I think there are 5 or so out of about 80 movies that don't work. I though it might be because I left the article off the title but i have other movies where that is the case and it didn't matter. Also the search does fine a match and I tell it to use the match it never says "Updated..." or produces files.

Thanks for you help and this great tool.

I tested those files locally and I was able to fetch the metadata for each of them. So I guess I have to wait for your log files to see why it didn't create the necessary output for you.

[theringnebula]

Originally Posted by stuckless

I tested those files locally and I was able to fetch the metadata for each of them. So I guess I have to wait for your log files to see why it didn't create the necessary output for you.

Log is attached. Thanks for your help!

[stuckless]

Originally Posted by theringnebula

Log is attached. Thanks for your help!

It appears that you xml that is being created from the xbmc scraper is invalid. The interesting thing is that I don't get an error on my system (linux) for the same search.

I'm going to try some work arounds.

[mp328]

is it possible to stop creating these mailcheck.log files or make them all the same name, right now i keep deleting them manually but it's kind of annoying lol

this is from using the plugin off course

[stuckless]

Originally Posted by mp328

is it possible to stop creating these mailcheck.log files or make them all the same name, right now i keep deleting them manually but it's kind of annoying lol

this is from using the plugin off course

Well.... it's not that simple. I don't create a mailcheck log file. The Plugin sends all errors (and logging) to log4j. It would appear that another plugin is "hijacking" the log4j configuration and sending it's output to maillog.txt. When I configure log4j, I intentionally don't configure a root logger, so that I don't hijack someone else's logging.

I'll look into the logging some more to see if there is anything that I can do on my end.

[theringnebula]

I just wanted to update you with a list of all the titles I have that don't generate a properties file using imdb.xml if this is of help to you. All of them do work using the default imdb provider you created. If you want any more logs let me know.

MOVIEDIR\Constant Gardiner\VIDEO_TS
MOVIEDIR\Inconvenient Truth\VIDEO_TS
MOVIEDIR\Fabuleux destin d'Amélie Poulain\VIDEO_TS
MOVIEDIR\Cidade de Deus\VIDEO_TS
MOVIEDIR\Harold & Kumar Escape from Guantanamo Bay\VIDEO_TS
MOVIEDIR\Antz\VIDEO_TS

[S_M_E]

Is there a --switch to get BMT to just touch the timestamps of all the files in a folder (and subfolders) without forcing updates of the metadata, media or .jpgs? I had a drive failure and I want to refresh my media library to get rid of some ghost entries.

I've found a couple of win32 ports of the unix touch command but either they don't support network shares or they don't do subfolders depending on which one I tried.

[stuckless]

Originally Posted by S_M_E

Is there a --switch to get BMT to just touch the timestamps of all the files in a folder (and subfolders) without forcing updates of the metadata, media or .jpgs? I had a drive failure and I want to refresh my media library to get rid of some ghost entries.

I've found a couple of win32 ports of the unix touch command but either they don't support network shares or they don't do subfolders depending on which one I tried.

bcjenkins posted this Sage.properties entry that forces a full rescan.

force_full_content_reindex=true

[S_M_E]

Yeah, I just installed cygwin and used the touch command, it does the same thing but without having to stop and start the service.

Thanks though...

[stuckless]

Originally Posted by S_M_E

Yeah, I just installed cygwin and used the touch command, it does the same thing but without having to stop and start the service.

Thanks though...

I don't spend a lot time in windows... but when I do... cygwin is my best friend

[S_M_E]

Originally Posted by stuckless

I don't spend a lot time in windows... but when I do... cygwin is my best friend

I know what you mean, I like FreeBSD and Solaris; cygwin is nice to have at times. So is BMT.

[stuckless]

Download

Fixes/Enhancements

• Potential fix for why imdb.xml sometimes fails
• Missing Metadata will also check for the missing .jpg and not only rely the empy _thumbnailUrl field when determing if a media file requires updating
• Added Basic TV Support (using Xmbc tvdb.xml scraper) This scaper is slow, but it does the trick. (See Notes a the end of this post)
• Added scoring system for providers that don't provider thier own score on a result. This means that any providers can now be used for automatically selecting a "good" result. (See Notes)
• Updated log4j so that it does not use maillog log file
• Can accept multiple providers. ie, --provider=themoviedb.org,imdb.xml,imdb,tvdb.xml (it will use each provider, in order, until it finds a "good" search) (See Notes)
• Added option to re-write titles (The Mummy becomes Mummy, The) (See Notes)
• Added ${_fileName} variable, so that Title substitution can use ${_fileName}
• Added TV Specific Title Mask, so that TV Titles can be formatted differently than Movie Titles. TV Titles currently format as eg, "Dexter - s01e01 - Sins of the Father" (See Notes)

Multiple Providers
You can now pass/set multiple providers to use while scanning. Just comma separate the provider ids, no spaces.

If you use multiple providers, it will use them in order for the type of media that it is scanning. For example, if you set imdb.xml,themoviedb.org,imdb,tvdb.xml as the provider string, and it's movie file, then it will use imdb.xml and check for a good match, if there is no match, then it will move onto themoviedb.org, etc, until it finds a good match. If the file happens to be a tv file, then any provider that doesn't support TV metadata will be skipped, so in this case, all of them except for tvdb.xml will be skipped.

TV Support
This is very basic TV Support (mainly for my needs when using it this as a plugin). To enable tv support, either add tvdb.xml to the providers on the command line or set it in the metadata.properties

/metadata/defaultProviderId=imdb.xml,themoviedb.org,imdb,tvdb.xml

Currently the only TV provider is the tvdb.xml provilder from the Xbmc scraper project.

TV Support only checks one kind of TV title, in the format "Show SxxExx", as defined the in the title scraper file scrapers/xbmc/tvfilenames/Title-s00e00.xml. You can define your own TV Title scraper format by editing this file, or creating a new .xml file in the same directory.

TV Title scrapers get passed a filepath uri, and you are required to provider 3 Functions. A GetShowName Function, GetEpisode Function and GetSeason function. These are just reqular expressions in the xbmc format for parsing out that information from a show's filename uri.

The current title scraper looks like...

<?xml version="1.0" encoding="UTF-8"?>
<!-- 
   TV Filename Scraper.  Used to Scrape Show Name, Season and Episode from a given file URI
   $$1 is always the complete file uri
-->
<scraper name="title-s00e00" content="filename" thumb="thumb.png">
	<!-- 	input: 		$1=complete file uri -->
	<!-- 	returns: 	show name -->
	<GetShowName dest="3">
		<RegExp input="$$1" output="\1" dest="3">
			<expression>.*/(.*)s([0-9]{1,2})[ex]([[0-9]]{1,2})</expression>
		</RegExp>
	</GetShowName>

	<!-- 	input: 		$1=complete file uri -->
	<!-- 	returns: 	Season #-->
	<GetSeason dest="3">
		<RegExp input="$$1" output="\2" dest="3">
			<expression>[/]*(.*)s([0-9]{1,2})[ex]([[0-9]]{1,2})</expression>
		</RegExp>
	</GetSeason>

	<!-- 	input: 		$1=complete file uri -->
	<!-- 	returns: 	Episode #-->
	<GetEpisode dest="3">
		<RegExp input="$$1" output="\3" dest="3">
			<expression>[/]*(.*)s([0-9]{1,2})[ex]([[0-9]]{1,2})</expression>
		</RegExp>
	</GetEpisode>
</scraper>

TV Title Mask
The TV Title mask can be set using the following property

/sageMetadata/tvTitleMask=${Title} - s${_season}e${_episode} - ${_showTitle}

By default it will write TV Titles as, "Dexter - s01e01 - Sins of the Father"

Movie Title Re-Writing
Movie titles that start with A,The, etc, can be rewritten by setting the following property

/sageMetadata/rewriteTitle=true

You can control how it determines to rewrite the title by adjusting this reqular expression

/sageMetadata/rewriteTitleRegex=^(in\\s+the|in\\s+a|i\\s+am|in|the|a|an|i|am)\\s+(.*)

If you don't understand what the reqular epression does, then I'd suggest not touching it

Some examples...

• The Mummy -> Mummy, The
• I am Legend -> Legend, I am
• I Robot -> Robot, I

Search Result Scoring
I added a scoring system that will score search results for providers that do not provide information about the quality of the search result. This was added so that any provider, such as the nielm_imdb provider, can be used for automatically searching and selecting a good result.


This will probably be my last release of this tool (barring some real important bugs) for some time. I have a few other projects on which I need to focus for the time being. I will fix bugs, but I think that this tool is all that it can be (for now). I think that between this tool and EP's metadata tool, that the batch metadata field is quite well covered.

Feel free to post suggestions on how to improve this... I will eventually get back to working on this again... but for now... I hope this will suffice.

Enjoy/

[S_M_E]

Looks like some good bug fixes there, I'll be testing shortly. I hate to see this tool going away though...

Can --displaySize be set in the .properties file or only in the commandline?

"Failed while applying arg: displaySize (20) to method: setDisplaySize"

EDIT:
Oh I see, it's been changed to --setDisplaySize but the --help still shows --displaySize and it's ignoring the =20 parameter

EDIT2:

I'm also seeing this for the not-skipped titles:
"Search Results: Stolen Honor Wounds That Never Heal 2004
00 (/) - null [null]
01 (/) - Stolen Honor: Wounds That Never Heal [2004]
LEGEND: * Exact Match; / Partial Match; + Popular Match; ? Unknown;"

[stuckless]

Originally Posted by S_M_E

Looks like some good bug fixes there, I'll be testing shortly. I hate to see this tool going away though...

Can --displaySize be set in the .properties file or only in the commandline?

"Failed while applying arg: displaySize (20) to method: setDisplaySize"

EDIT:
Oh I see, it's been changed to --setDisplaySize but the --help still shows --displaySize and it's ignoring the =20 parameter

EDIT2:

I'm also seeing this for the not-skipped titles:
"Search Results: Stolen Honor Wounds That Never Heal 2004
00 (/) - null [null]
01 (/) - Stolen Honor: Wounds That Never Heal [2004]
LEGEND: * Exact Match; / Partial Match; + Popular Match; ? Unknown;"

Anything that is passed on the command line, can be set in the properties files (i think ) ... Do a --showProperties and grep/search for size... it should be in there.

I'll check the app tonight though... to see if i've accidentally introduced a bug with this release, related to setting the display size.

What provider are you using?

the command line parameter is --displaySize... but it get's mapped to a method call setDisplaySize.... so --displaySize is correct, but It appears that maybe there is a bug... i'll check tonight.

[S_M_E]

I used this command:
java -jar MetadataTool.jar --recurse --setDisplaySize=20 --prompt=true --auto=false --provider=nielm_imdb,themoviedb.org,imdb,imdb.xml,tvdb.xml --setProperty=/imdb/forcedIMDBImageSize:1920 \\Abit\videos\Movies\

Using "--displaySize=20" resulted in the error that I quoted yesterday.

I found the setting for the .properties file:
/metadataUpdater/searchResultDisplaySize=20
but I still only get 10 results. Must be a bug but I don't know if it's related to the "00 (/) - null [null]" that I see on most files.

Also, it's still prompting for those movies (listed a couple of pages back) that that have an empy _thumbnailUrl field.

I ran BMT against one movie that's still giving problems so I could give you a smaller log:
java -jar MetadataTool.jar --recurse --prompt=true --auto=false --provider=nielm_imdb,themoviedb.org,imdb,imdb.xml,tvdb.xml --setProperty=/imdb/forcedIMDBImageSize:1920 "\\Abit\videos\Movies\Documentaries\Stolen Honor Wounds That Never Heal (2004).mpg"
Batch MetaData Tools (1.18)
** Automatic Updating Disabled ***
IMDB getting URL:http://www.imdb.com/find?s=tt&mx=200...ver+Heal++2004

Search Results: Stolen Honor Wounds That Never Heal 2004
00 (/) - null [null]
01 (/) - Stolen Honor: Wounds That Never Heal [2004]
LEGEND: * Exact Match; / Partial Match; + Popular Match; ? Unknown;

[q=quit, n=next (default), ##=use result ##, TITLE=Search TITLE]
> 01
Updated: Stolen Honor: Wounds That Never Heal; file:////Abit/videos/Movies/Documentaries/Stolen%20Honor%20Wounds%20That%20Never%20He
al%20(2004).mpg

MetaData Stats...
Auto Updated: 0; Auto Skipped: 0; Manual Updated:1; Manual Skipped: 0;
(log attached)

[stuckless]

Download

I guess i was in too much of a rush to get this out last night... some things slipped though the cracks.... sorry about this..

• Missing Metadata will not check for the missing the actual .jpg and not only rely the empy _thumbnailUrl field (really fixed this time)
• Fixed --displaySize bug
• Fixed null[null] when using themoviedb.org provider

[theringnebula]

Hi,

I've tried the new version 1.20 but I am still having issues with certain movies. I now get a "fatal error". I have attached a log for the movie Pleasantville.

Thanks