Utility: Batch Metadata Importer 3.x

[stuckless]

Documentation | Download | Fanart FAQ | Releases Thread | sflamm's Install Guide | Configure Logging



Overview
Batch Metadata Tools (BMT) is a tool (command line) and a Sage plugin that scans a media collection and attempts to automatically search/download metadata for each item.

• Multi-Platform (Windows/Linux/Mac)
• Multple Metadata Sources (imdb, tvdb, themoviedb, dvd profiler, mymovies)
• Full TV Support for Sage Recordings or Archived TV shows (tvdb)
• Downloads Posters and Backdrops (support for central fanart using phoenix apis)
• Scrubs filenames to better search hits (works on Sage Recordings, TV Files (sXX eXX), and Movies)
• Optionally Prompts for entries that cannot be located (and stores selected result for future reference) (command line)
• Many configuration options (use --showProperties)
• Includes STVi for Default STV and SageMC for doing on demand fanart and metadata for Movies, TV, and EPG items.
• WebUI for configuration (since 3.0)

NOTE: Plugin mode runs metadata searches automatically when new files are added to the Sage Library. If metadata cannot be found the the file is skipped.

BMT Supports the Following Metadata Locations

• IMDB (imdb, niel's imdb, xbmc's imdb using imdb.xml scraper file)
• TheMovieDB.org
• DVD Profiler (using DVD Profiler's Xml and images)
• Xbmc Sources Using Xbmc's Scraper Language (imdb.xml)
• thetvdb
• MyMovies (Uinsg MyMovies collection xml)

How should my files be named, to ensure best results?
Movie Files
Finding Nemo.avi
Finding Nemo (2005).avi
Finding Nemo cd1.avi
Finding Nemo cd2.avi

DVDs
Finding Nemo/VIDEO_TS/
Finding Nemo/*.vob
Finding Nemo (2005)/*.vob

Sage Recordings (Movie)
FindingNemo-000000-0.ts

Sage Recordings (TV)
Futurama-MothersDay-0000000-0.ts

Archived TV Shows
Futurama - S01E02 - Mother's Day.avi
Futurama - S01E02.avi
Lie To Me - S01E01.avi

Ripped TV DVDs
Lie To Me S01D01/*.vob


Avoid using dots in spaces, ie, Lie.To.Me-S01E01.avi.
Avoid using too many tokens in your title, like, DVDRip, etc.

BMT does attempt to clean up the filenames, but, the cleaner they are to to start with, the better your chances of getting a good hit.

Known Issues
In Windows, after a server restart, BMT web will fail to show the UI. Instead it will show only the directory listing or generate an error. This post explains the problem and the solution.



Hint's Tips from Users
How do I force Sage to reindex my entire collection?
Add this to your Sage.properties to force it to completely rebuild your metadata. Useful to clean up Genres.

Originally Posted by bcjenkins

force_full_content_reindex=true

Note: As of SageTV 6.5.18, you can now tell sage to do a Full Reindex from withint the STV
Note: As fo bmt 3.0 you can force sage to do a full reindex from the web ui

Windows Batch File
razrsharpe provided this windows .bat file that some windows users may find useful. You still need to configure the bat file with your default options, but once you do, you can then drag and drop a file/folder onto the bat file and it will invoke the scan/update.

How to update the timestamp of a Media File so sagetv will re-read the .properties (Windows)
babgvant posted this tip
sflamm posted this tip
And since 3.0-beta8, you can "touch" a file using the --touch commandline option.

# java -jar MetadataTool.jar --touch YOUR_FILE|YOUR_DIR

Do not make your files read-only

Originally Posted by ChuckSchick

Before running the script, make sure none of your movies are set to "Read Only".
Otherwise, the file dates won't update and Sage won't import the .properties info. Somewhere in the history of my library I was setting the files to Read Only in an effort to protect them. About half of my collection still had this property.

Submitting Bugs/Requesting Help
Because of the complexities of how BMT can be used, I need to know some information about your environment before I can help your. Use the following a starting point for information.

What bmt version 2.x (stable) or 3.x (beta)?
What SageTV version?
SageMC or default STV?
SageMC version?
Java version?
Phoenix version?

SageTV setup:
Is central FanArt folder enabled?
Is central FanArt folder local or a network share?
Do you have SageTV set up to do automatic scans?
If yes, how often?
Do the scans import new videos?

BMT setup:
Command Line Tool:
Have you successfully used the command line tool?

On-demand Tool:
Have you imported the on-demand plugin (stvi of xml)?
Do you have the on-demand plug-in enabled?

Automatic tool:
Do you have these 2 properties set in sage.properties?

	
	
mediafile_metadata_parser_plugins=org.jdna.sage.MetadataUpdaterPlugin
custom_metadata_properties=UserRating;EpisodeNumber;EpisodeTitle;MediaTitle;MediaType;OriginalAirDate;SeasonNumber;MediaProviderDataID;DiscNumber
Please give a description of the steps to reproduce your issue:

[PGPfan]

Thanks Sean,

I'm definately going to give this a spin!

-PGPfan

[jusjoken]

Excellent tool.

So far my results look good and it processed 36 movies very quickly.

1 file I needed to manually provide a title for "Serenity (Aka.Firefly).avi", I provided Firefly and that worked.

1 movie it got wrong "Terminator 1.avi", it found Terminator II. I then ran the command for --force with the filename and then manually selected "The Terminator" and that worked, except that it left the jpg for Terminator II that it had previously downloaded. I then had to create a metadataupdater.properties file and adjust the overwriteThumbnail value to true. I would have assumed if I was using --force then the thumbnail would have been overwritten by default, but others might think otherwise as potentially I might have had my own thumbnail collection that I didn't want overwritten. Either way it worked well with just a little effort.

I have not uploaded into sage yet, so I will let you know the results then.

[bcjenkins]

From Java 1.5

sagetv@sagetv-server:~/scripts/imdb_importer$ PATH=/opt/jre1.5.0_15/bin:$PATH
sagetv@sagetv-server:~/scripts/imdb_importer$ export PATH
sagetv@sagetv-server:~/scripts/imdb_importer$ java -jar MetadataTool.jar
Exception in thread "main" java.lang.UnsupportedClassVersionError: Bad version number in .class file
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(Unknown Source)
	at java.security.SecureClassLoader.defineClass(Unknown Source)
	at java.net.URLClassLoader.defineClass(Unknown Source)
	at java.net.URLClassLoader.access$100(Unknown Source)
	at java.net.URLClassLoader$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(Unknown Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.loadClassInternal(Unknown Source)

Will use 1.6, but FYI

B

[bcjenkins]

Is there support for VIDEO_TS?

/var/media/videos/$DVD_NAME/VIDEO_TS/(vob files)

TIA

B

[stuckless]

From Java 1.5

I mentioned the release info that I suspected that java 5 would not work, simple because I use java 6... But I will attempt to recompile using Java 5 next time... I don't think i'm using any Java 6 extensions yet

[stuckless]

1 movie it got wrong "Terminator 1.avi", it found Terminator II. I then ran the command for --force with the filename and then manually selected "The Terminator" and that worked, except that it left the jpg for Terminator II that it had previously downloaded. I then had to create a metadataupdater.properties file and adjust the overwriteThumbnail value to true. I would have assumed if I was using --force then the thumbnail would have been overwritten by default, but others might think otherwise as potentially I might have had my own thumbnail collection that I didn't want overwritten. Either way it worked well with just a little effort.

I wrestled with the idea of overwriting the thumbnails or not. In the end, I opted to NOT overwrite ever, but that functionality can be overridded using a custom metadataupdater.properties and setting the

org.jdna.media.metadata.impl.sage.SageVideoMetaData.overwriteThumbnail=true

I could potentialy use another command line arg to set that value as well.

[stuckless]

Is there support for VIDEO_TS?

Not yet... it is on my todo list... and it's not hard to implement (i think). I actually check for VIDEO_TS directories and currently skip them

FYI.... there tool does create a log file in your user's home directory called metadataupdater.log... you'll see messages in there about "Skipping DVD Folders for now"

[bcjenkins]

It isn't giving me that error, I am getting:

73 ERROR [main] org.jdna.metadataupdater.MetadataUpdater - Unknown Media Resourse for: file:/var/media/videos/MovieName/;org.jdna.media.impl.MediaFolder

It doesn't seem to be finding VIDEO_TS

B

[stuckless]

It doesn't seem to be finding VIDEO_TS

I'll look into that for you. I'll just go ahead and implement dvd folder support

Update: I can see the problem now... I'm only checking for DVD Folders when --recurse is used. I'll try to add in dvd folder support sometime today (later )

[jusjoken]

My import into Sage worked well. The only issue I see is that the Categories (Genre) seem to import as a complete string including the ; seperators. I would have thought that the first one would be the Category and the second would be the SubCategory and the rest would be ignored. This is likely an issue with the import in Sage and not your tool unless the formating in the .properties is not what Sage is expecting.

The other thing I noticed is that I thought there was a propertiy for a Rating and for an Extended Rating. A Rating might be PG where and extended rating might be "Rated PG for ....". Not sure IMDB offers both nor if the properties file support both, I just remember getting a simpler Rating (only the PG) when using the imports in SageMC (now using the Default STV).

Otherwise, great so far.

[bcjenkins]

Originally Posted by stuckless

I'll look into that for you. I'll just go ahead and implement dvd folder support

Update: I can see the problem now... I'm only checking for DVD Folders when --recurse is used. I'll try to add in dvd folder support sometime today (later )

Thanks! All of my videos are in DVD format. Oddly the properties file for DVD folders needs to go at the same level as the $DVD_NAME directory. Instead of in the VIDEO_TS folder like the folder.jpg.

B

[JREkiwi]

Originally Posted by bcjenkins

Oddly the properties file for DVD folders needs to go at the same level as the $DVD_NAME directory. Instead of in the VIDEO_TS folder like the folder.jpg.

You can have a .jpg in the same level as the $DVD_NAME directory with the .properties file. It just needs to be called $DVD_NAME.jpg

John

[bcjenkins]

I would prefer a folder.properties at the same level as the folder.jpg. It makes for a cleaner videos directory.



B

[stuckless]

So if I understand this, for dvd folders, the property name is the same name as the dvd folder, and the thumbnail can either be the same as video folder name or VIDEO_TS/folder.jpg?

eg, if the DVD Folder is called Terminator,
Terminator/Terminator.properties
Terminator/Terminator.jpg
or
Terminator/Terminator.properties
Terminator/VIDEO_TS/folder.jpg

[bcjenkins]

./Terminator/VIDEO_TS/folder.jpg
./Terminator.properties

B

[JREkiwi]

With 6.4 the requirement for a VIDEO_TS was removed for DVD's and support for .properties file for DVD's was introduced and defined as being located in the parent folder. So for DVD's

In the parent folder DVDs
DVDs\<DVDName>\dvd files
DVDs\<DVDName>.properties
DVDs\<DVDName>.jpg
DVDs\<DVDName>\folder.jpg does not work
and
DVDs\<DVDName>\VIDEO_TS\dvd files
DVDs\<DVDName>.properties
DVDs\<DVDName>.jpg
or
DVDs\<DVDName>\VIDEO_TS\folder.jpg

John

[Brent]

Originally Posted by JREkiwi

With 6.4 the requirement for a VIDEO_TS was removed for DVD's and support for .properties file for DVD's was introduced and defined as being located in the parent folder. So for DVD's

In the parent folder DVDs
DVDs\<DVDName>\dvd files
DVDs\<DVDName>.properties
DVDs\<DVDName>.jpg
DVDs\<DVDName>\folder.jpg does not work
and
DVDs\<DVDName>\VIDEO_TS\dvd files
DVDs\<DVDName>.properties
DVDs\<DVDName>.jpg
or
DVDs\<DVDName>\VIDEO_TS\folder.jpg

John

True, but since SageMC (with DVDProfiler) hasn't caught up with that requirement many still have the "video_ts" folder.

[JREkiwi]

Originally Posted by Brent

True, but since SageMC (with DVDProfiler) hasn't caught up with that requirement many still have the "video_ts" folder.

I suppose the point I was really trying to make is that folder.jpg only works if you are using VIDEO_TS folders. <DVDNAME>.jpg works in both configurations, and there is only 1 valid location for the .properties file

John

[Brent]

Originally Posted by JREkiwi

I suppose the point I was really trying to make is that folder.jpg only works if you are using VIDEO_TS folders. <DVDNAME>.jpg works in both configurations, and there is only 1 valid location for the .properties file

John

Okay, understood.