Utility: Batch Metadata Importer 3.x

[stuckless]

Download | Issues

Issues/Enhancements
14 Add Support for Different Size Fanart
added

/metadata/posterImageWidth=200
/metadata/bannerImageWidth=-1
/metadata/backgroundImageWidth=-1

default is that posters are scaled down to 200px, banners and backgrounds are not
scaled, but they can be. Setting a value of -1 disables scaling.


32 Implement the image download tracking file
prevents images from being re-downloaded, if you physically remove them from the fanart folder

36 DVD Fanart broken when used in plugin mode?
This is most likely configuration, ie, the fanart central folder is not set. I've
also updated the plugin code to overwrite existing metadata/fanart always, just in case.

37 Phoenix: Add better tv scraping
For TV Filename that contain S## E## (or D##) information, then the fanart apis will use that information if it's not already set in the metadata properties.

40 Movie titles that start with Dr. are getting the same imdb lookup for Dr.
For DVD Folders. Fixed the filename scraper that was treating everything afer the dot as a file extension.

43 Update composite provider to allow for a separate "fanart" provider
Reworked the composite provider, it is now a little simpler. simply provide 2
providers and indicate which mode of copy you want to use. there are 2 modes,
1 - Use the search details and supplement using the second detail provider
2.- Use the second detail provider as the main details, and supplement from the
searcher details.

So, to create a dvdprofiler that uses themoviedb for fanart...

/compositeMetadataProviders/dvdprof2/name=DVD Profiler + TheMovieDB
/compositeMetadataProviders/dvdprof2/description=DVD Profiler, but get fanart and
missing details from TheMovieDB
/compositeMetadataProviders/dvdprof2/iconUrl=
/compositeMetadataProviders/dvdprof2/searchProviderIdd=dvdprofiler
/compositeMetadataProviders/dvdprof2/detailProviderId=themoviedb.org
/compositeMetadataProviders/dvdprof2/compositeMode=1

45 Handle movie files that have 1080p or 720p in filename
Fixed

49 /sageMetadata/rewriteTitle is missing, need an alternative
Fixed

50 add separate overwrite for metadata and fanart
Added separate overwrite options for fanart and metadata.

--overwriteFanart
--overwriteMetadata

Where
--overwrite
would simple be the same as calling --overwriteFanart and --overwriteMetadata

51 add separate process for fanart and metadata
You can now specify --fanartOnly or --metadataOnly when doing a scan.

52 Allow Filters to be used on a file as well as filter
updated the MovieFilter to look for the directory filter on each file as well as
directory.

/media/excludeVideoDirsRegex is a regex, and multiple directories can be specified as long as
it uses the regex "OR" character "|", ie,

/test2/|/test3/

would ignore test2 and test3 directories

Also note that bmt files use uri syntax, so you must specify directory separators in
the URI forwardslash.

55 Bug in scoring when titles almost match
Added a check to the scoring algorithm to ensure that 2 unequal titles didn't return
a perfect score.

56 not updating on re-index-bmt
If this is related to the plugin, then the plugin has been updated to replace and
overwrite metadata at all times. In the previous releases, the plugin would skip
over files if the existing .properties existed for a mediafile.

57 Add limit property to # of images being downloaded
added
/metadata/maxDownloadableImages=5
as the default value, people can change to -1 for all images.

59 Url Caching Fixes / Enhancements
themoviedb now caches
cache files now use a hash instead of the url paramters;

60 Add configuration to disable writing of backward compatible thumbnails
added the following property
/metadata/enableDefaultSTVPosterCompatibility=false

By default default stv poster compatibility will be disabled, you can enable it by
setting it true.

It is sometimes useful to provide default stv compatibility if you have a mix of
default stv and sagemc clients.

61 Enabled plugin to support ShowTitle-airingid.mpg titles
Not fully tested, but it should try to find an airing for the file, and if it's
there, then it will use the airing series title for the lookup... otherwise, it will
continue on with the normal plugin operation.

It uses the following regex to identify a filename as a sage airing and parse the airing id
/metadata/airingIdRegex=([0-9]+)-[0-9]{1,2}\.

62 Fix log4j so that it will use a common log4j.properties, if it exists.
phoenix will now check for log4j.properties in the sagetv homedir, and if it exists,
it will use that logger. If it does not exist, then it use the logger information
bundled with each respective jar file.

46 Make supporting properties files compatible with Windows line breaks
After doing some research and testing on a windows platform I could not reproduce the
problem. Under windows xp, if I write the properties file using bmt, then it will
write it so that it opens fine in notepad (ie, contains crlf and not just lf)

According the Java this is the normal behaviour.

For now i'm closing the bug, since I can't reproduce it, but I'll happy re-open if
there more information on how to reproduce.

39 Downloaded images affecting performace of extenders
This was marked as a duplicate of 14 and should be resolved using the resize fanart options

47 TV Scan only downloads 1 Background FanArt Image
This is an issue with the tvdb.xml scraper. For some reason they only grab a single
backgound and pass it back.

I decided not to do anything at this time, since eventually EP will have his scrapers
integrated with bmt, and his scraper will download all backgrounds.

The Xbmc scrapers are complex enough that I can't really make it do this without a
lot of effort.


Good luck.

[stuckless]

Originally Posted by appelm

The new "Date Added' sort menu for video library is a great add-on to SageMC. But BMT run in batch mode "touches" video files and thus the "date added" functionality is reset. Not sure if there's a way around "touching" the file but it is unfortunate since I was quite pleased to get the "date added" sort. Do files get "touched" also when run from the STV?

The plugin and stv operations do not "touch" the file as of the 2.1 build. But the files are still "touched" if you run from the commandline. The touching stems from the fact that sage won't reload the metadata for an item unless it has changed. Once sage adds in some more metadata handling functions, we'll be able able to get around that... but for now... using the commandline... it has to touch.

[jaminben]

Great stuff Sean

Does this mean that metadata should now be updated when using the plugin?

Originally Posted by stuckless

56 not updating on re-index-bmt
If this is related to the plugin, then the plugin has been updated to replace and overwrite metadata at all times. In the previous releases, the plugin would skip over files if the existing .properties existed for a mediafile.

Just wanted to clarify before I start pulling my install about as its not working at the moment.

Cheers

Ben

[stuckless]

#56 means that the plugin, if given a file, will now do a scan and download the metadata/fanart even if there is an existing properties file. Before the 2.1 build it would not do a metadata scan if there was an existing .properties file.

[stuckless]

Just a request, that if you submit an issue here for something not working, could you provider some information, such as...

BMT Operation: Plugin or Commandline
File Name: Sample filenames for testing
File Type: DVD Folder, Reqular video file, Recorder TV File
OS: Windows, Linux, Mac

For me to reproduce an issue, I need to know those things, so that I'm not chasing down the wrong thing.

Thanks.

[stuckless]

As of 2.1, bmt ships with practically no logging enabled (since there were complaints of the log files getting to large).

If you are having issues and you need to configure logging, then edit log4j.properties, in the sagetv home dir.

Here's the information for configure debug logging in bmt, just add this to the end of the existing log4j.properties.

# for excessive debug logging in bmt
log4j.logger.org.jdna=debug
log4j.logger.bmt=debug

# for excessive debug loggging in phoenix
log4j.logger.sagex=debug

# remove excessive logging in the imdb parsers
log4j.logger.org.jdna.url=WARN
log4j.logger.org.jdna.url.CachedUrl=WARN
log4j.logger.org.jdna.url.URLSaxParser=WARN
log4j.logger.org.jdna.media.metadata.impl.imdb.IMDBSearchResultParser=WARN
log4j.logger.org.jdna.media.metadata.MediaMetadataUtils=WARN

# removes excessive logging for stacking
log4j.logger.org.jdna.media.impl.CDStackingModel=WARN

# remove excessive logging for xbmc scrapers
log4j.logger.org.jdna.media.metadata.impl.xbmc.XbmcScraperProcessor=WARN

As a side note, if you have an existing log4j.properties, then it's probably comes from the mailcheck program. If you don't have an existing log4j.properties in the SageTV home (ie, where MetadataTools.jar is located), then you can create one like this...

# set log levels for loggers (categories) debug < info < warn < error < fatal
log4j.rootCategory=error, log


# ipps log is set to be a RollingFileAppender, 12 hours per file, 1 week, using PatternLayout.
log4j.appender.log=org.apache.log4j.DailyRollingFileAppender
log4j.appender.log.DatePattern='.'yyyy-MM-dd-a
log4j.appender.log.File=mailcheck.log
log4j.appender.log.layout=org.apache.log4j.PatternLayout
log4j.appender.log.layout.ConversionPattern=%d{ISO8601} %5p [%t] %C (%F:%L) - %m%n

For example, my logj4.properties file from my server looks like....

# enable the internal log4j logging (usefull for debuging log4j)
#log4j.debug=true

# set log levels for loggers (categories) debug < info < warn < error < fatal
log4j.rootCategory=debug, log

# ipps log is set to be a RollingFileAppender, 12 hours per file, 1 week, using PatternLayout.
log4j.appender.log=org.apache.log4j.DailyRollingFileAppender
log4j.appender.log.DatePattern='.'yyyy-MM-dd-a
log4j.appender.log.File=mailcheck.log
log4j.appender.log.layout=org.apache.log4j.PatternLayout
log4j.appender.log.layout.ConversionPattern=%d{ISO8601} %5p [%t] %C - %m%n

# phoenix/bmt logging
log4j.logger.org.jdna=debug
log4j.logger.bmt=debug
log4j.logger.phoenix=debug
log4j.logger.sagex.phoenix=debug

log4j.logger.org.jdna.url=warn
log4j.logger.org.jdna.url.CachedUrl=warn
log4j.logger.org.jdna.media.impl.CDStackingModel=warn
log4j.logger.org.jdna.url.URLSaxParser=warn
log4j.logger.org.jdna.media.metadata.impl.imdb.IMDBSearchResultParser=warn
log4j.logger.org.jdna.media.metadata.MediaMetadataUtils=warn
log4j.logger.org.jdna.media.metadata.impl.xbmc.XbmcScraperProcessor=warn

And then add the log4j lines that I listed above to turn on logging for bmt.


NOTE: If you configure nothing, then by default all logging will end up in the mailcheck.log file because SageMC installs a log4j.properties, but you'll only get Errors in the log.

[Wade]

Great addon, thanks!

Just have one issue, titles that have an apostrophe. IE "She's the One"

If I manually search for "shes the one" it finds it correctly.

Thanks,
Wade

[toricred]

Does 61 mean that automatic metadata/fanart should work for TV recordings if the episode title is not included in the file name?

[sflamm]

Does this handle BDMV folder structures now?

[stuckless]

Originally Posted by toricred

Does 61 mean that automatic metadata/fanart should work for TV recordings if the episode title is not included in the file name?

I'm not exactly sure what it will do to be honest. The idea is that if you are doing a rescan by setting the reindex property to true, then any recorded shows that get passed to the plugin, will scrape the AiringId from the filename, and then lookup the corresponding Sage Airing for that Id and then download fanart based on the recorded show series name. But, I haven't been able to fully test it, so I have no idea if it's really going to work

The format for the airing filename should be something like, ShowName-Title-AiringId-#.ts, but as long as the filename contains the AiringId-#, then it should be able to pull out the airing id.

[stuckless]

Originally Posted by sflamm

Does this handle BDMV folder structures now?

Yes that should have been in the release notes.. I added BDMV support so that BDMV folder will behave much like VIDEO_TS folders, ie, it will treat the parent dir as the movie title and search on that.

[talbot3]

Stuckless,

Thanks for the great update. I am only having one issue:

With the rewriteTitle set to true using the command line tool, the titles are being rewritten correctly but the fanart is being downloaded to a folder using the original title. To give you an example:
The title of "The bourne identity" is correctly rewritten to "bourne identity, the" in the properties file, but the fanart is downloaded to a folder called "c:\fanart\the bourne identity". The requires a manual rename of the fanart folder to make the fanart show up in SageMC.

Thanks for the all the hard work in making this possible.

[LehighBri]

Originally Posted by stuckless

I'm not exactly sure what it will do to be honest. The idea is that if you are doing a rescan by setting the reindex property to true, then any recorded shows that get passed to the plugin, will scrape the AiringId from the filename, and then lookup the corresponding Sage Airing for that Id and then download fanart based on the recorded show series name. But, I haven't been able to fully test it, so I have no idea if it's really going to work

The format for the airing filename should be something like, ShowName-Title-AiringId-#.ts, but as long as the filename contains the AiringId-#, then it should be able to pull out the airing id.

Think you might need to re-open this. It finds the fanart for my recordings just fine, but it puts the Fanart in my Fanart\Movies folder instead of the Fanart\TV folder.

[stuckless]

Originally Posted by LehighBri

Think you might need to re-open this. It finds the fanart for my recordings just fine, but it puts the Fanart in my Fanart\Movies folder instead of the Fanart\TV folder.

Are your recordings recognized by sage as being TV recordings? I've checked with a number of my mine, and they get picked up as TV for me... I'll look a little deeper and see what I can find.

[LehighBri]

Originally Posted by stuckless

Are your recordings recognized by sage as being TV recordings? I've checked with a number of my mine, and they get picked up as TV for me... I'll look a little deeper and see what I can find.

Yes, my recordings are in my SageTV Recordings area.

[PLUCKYHD]

Couple of issues

The rescan works great now and rescans files already scanned.

Using the sage plugin when I have it set to autodownload cover art new dvd's or videos do not show up in sage at all it only shows the movies that have cover art. If I uncheck auto fetch and restart sage they show up?

Also when you have a tv dvd named:

the office s01d01

It puts the metadate in a tv folder but it doesn't show up in recorded tv should it?

[stuckless]

Originally Posted by talbot3

Stuckless,

Thanks for the great update. I am only having one issue:

With the rewriteTitle set to true using the command line tool, the titles are being rewritten correctly but the fanart is being downloaded to a folder using the original title. To give you an example:
The title of "The bourne identity" is correctly rewritten to "bourne identity, the" in the properties file, but the fanart is downloaded to a folder called "c:\fanart\the bourne identity". The requires a manual rename of the fanart folder to make the fanart show up in SageMC.

Thanks for the all the hard work in making this possible.

Are you using the latest beta of sage that supports GetMediaFileMetadata()? And have you added the extra metadata fields for MediaTitle, MediaType, etc? Phoenix uses the MediaTitle field,if set, and then falls back to the DisplayTitle if MediaTitle does not exist.

I think the only fix for this right now would be to use the latest beta and ensure that you have the extra metadata fields configured, or disable title rewriting.

I'll look to see if there something I can do in the phoenix library to help with this as well.

[stuckless]

Originally Posted by LehighBri

Yes, my recordings are in my SageTV Recordings area.

Ok, i'll reopen the bug, and do some further investigating.

[stuckless]

Originally Posted by PLUCKYHD

Couple of issues

Using the sage plugin when I have it set to autodownload cover art new dvd's or videos do not show up in sage at all it only shows the movies that have cover art. If I uncheck auto fetch and restart sage they show up?

Is this a sagemc setting? or the enable metadata plugin setting that I added to sagemc and the default stv?

The metadata plugin should not prevent movies from showing up in the media center. When new movies are added, sage have to be told to "rescan" and then they get picked up, and after Sage imports the video, it calls my plugin to fetch the coverart or other missing metadata.

Also when you have a tv dvd named:

the office s01d01

It puts the metadate in a tv folder but it doesn't show up in recorded tv should it?

The TV fanart folder is for all media that is TV, not just Sage Record TV. so, in that case, The Office would be under TV but I don't import the vidoes into the SageTV recorded shows. I'm working on that, but it's not there yet.

[onlydarksets]

I tried to rescan my entire library, and I got a boatload of errors. It seems to be rescanning properly, but it fails on the 5th or 6th movie, and then on every movie after it. I tried moving the failing movie to the end (by prepending a "z"), but it still craps out.

Any thoughts? Thanks...

Updated: 8 Mile; file:/g:/Media/Movies/8%20Mile/
Updated: A Christmas Story; file:/g:/Media/Movies/A%20Christmas%20Story.m2ts
Updated: A Knight's Tale; file:/g:/Media/Movies/A%20Knight's%20Tale/
Updated: American Pie; file:/g:/Media/Movies/American%20Pie/
Updated: American Pie 2; file:/g:/Media/Movies/American%20Pie%202/
Updated: Antz; file:/g:/Media/Movies/Antz/
Updated: Star Wars: Episode II - Attack of the Clones ; file:/g:/Media/Movies/Attack%20of%20the%20Clones/
[Fatal Error] :1:1: Premature end of file.
[Fatal Error] :1:1: Premature end of file.
[Fatal Error] :1:1: Premature end of file.
[Fatal Error] :1:1: Premature end of file.
[Fatal Error] :1:1: Premature end of file.
[Fatal Error] :1:1: Premature end of file.
[Fatal Error] :1:1: Premature end of file.
[Fatal Error] :1:1: Premature end of file.
[Fatal Error] :1:1: Premature end of file.
Updated: Boogie Nights ; file:/g:/Media/Movies/Boogie%20Nights/
[Fatal Error] :1:1: Premature end of file.
[Fatal Error] :1:1: Premature end of file.
[Fatal Error] :1:1: Premature end of file.