Utility: Batch Metadata Importer 3.x

[stuckless]

Download

Issues/Enhancements
13 Add Banner Support to BMT
24 Remove HTML charaters from titles
38 Remove ... from titles
67 URL Cache bug
71 Backgrounds not downloading
72 Add better support for TheTVDB
73 Various issues surrounding dvd/bluray folders
78 Update Plugin to use the .properties instead of forcing an overwrite
81 In Plugin Mode, Allow plugin to use dvd profiler configuration in sagemc, if set

A little more info
13, 72 - Means that tvdb is now natively supported. Before I was relying on the xbmc scraper. The new tvdb provider will download all series and season fanart (banners, posters, backgrounds)

24, 81 - DVD Profiler related. Html characters are now removed from the title and description. And, you can use the SageMC dvd profiler configuration for dvd profiler, if you are running bmt as a plugin. ONE NOTE: since the bmt configuration requires an Image folder, then it will look at the SageMC configuration for DVD Profiler Root and add on "Images", and use that at the image folder. This is ONLY done if you have not configured dvd profiler information in your metadata.properties, AND you have added dvdprofiler as a metadata provider.

38 - . is no longer a valid character for a fanart title... Any dots in the title will be removed.

67 - Added some extra checking around the url caching. This should prevent the "premature end of file" error.

71 - Fixed issue with the tiles that contain "Mr." or more specifically tiles that contain .

73 - Added better checks for BDMV folder support. Also added m2ts as file ext.

78 - Default operation of the plugin is to use the properties if they exist. Previously, it would ignore the properties and for a lookup anyways. So if you are planning on doing a full rescan of your collection, you better delete all .properties, so that bmt will actually do the lookup.


Word of Caution - Upgrade Notice
In order for some of the new features to work, you will need to remove your existing metadata.properties, and then reconfigure.

or keep your existing metadata.properties, but remove the following lines.

/metadata/persistenceClass=org.jdna.media.metadata.impl.sage.SageTVPropertiesWithCentralFanartPersistence
/metadata/videoMetadataProviders=org.jdna.media.metadata.impl.tvdb.TVDBMetadataProvider,org.jdna.media.metadata.impl.imdb.IMDBMetaDataProvider,org.jdna.media.metadata.impl.nielm.NielmIMDBMetaDataProvider,,org.jdna.media.metadata.impl.dvdproflocal.LocalDVDProfMetaDataProvider,org.jdna.media.metadata.impl.themoviedb.TheMovieDBMetadataProvider,
/metadata/wordsToClean=1080p,720p,480p,1080i,720i,480i,dvd,dvdrip,cam,ts,tc,scr,screener,dvdscr,xvid,divx,avi,vrs,repack,mallat,proper,dmt,dmd,stv,HDTV,x264

And make sure that you add remove tvdb.xml and add tvdb

/metadata/defaultProviderId=tvdb,themoviedb.org,themoviedb.org-2,imdb.xml,imdb

[sflamm]

Updated document attached for review.

More info. Refers to BMT v2.2 now.

[stuckless]

good instructions... but I do have a question... You mention that you need to install dvdprop2sage, but then you disable it in the configuration. BMT doesn't require or use dvdprof2sage. BMT has it's own dvd profiler profiler that is independent of dvdprof2sage, and it's included in the build by default.

[sflamm]

Agreed.

If folks want to use DVDProfiler they would need to install it (so it can serve as a provider) Then still export the collections XML. It is possible to use BMT, but not for Videos by only configuring say a TV or Music provider in BMT... yes? Is there a problem if you use both DVDProfiler plug-in and BMT plug-in? Or can you use only 1 or the other?

Since BMT also now uses the DVDProfiler path settings when BMT is configured to use DVDProfiler as a provider - it is necessary to enter that information even when you are not using DVDProfiler Auto Meta Data if you dont enter it in the metadata.properites file.

I see that one or the other should really be configured according to your release notes (I should update documentation):

This is ONLY done if you have not configured dvd profiler information in your metadata.properties, AND you have added dvdprofiler as a metadata provider.

(I should probably have mentioned the DVDProfiler install - but I assumed that was obvious if you want to use it) In the example we use BMT for everything, but that doesnt have to be true.

The instructions disable DVDProfiler Auto Meta Data updating. They dont disable DVDPro2Sage.

And I think you will need DVDPro2Sage installed for the collections to show up as collections, no? (pretty sure) Unless you have implemented the same functionality...how else would the collections show up correctly under the videos section? (collections are a big deal to folks so this is pretty important IMO)

[Comitizer]

Originally Posted by stuckless

bmt has limited support (out of the box) for how it finds tv shows. The format that will work.... is something like,
<show>/<season>/<show> S<seasonE<episode>.avi

That is the out of the box functionality. You can configure something more specific to your setup, by adding a new filename scraper to the scrapers/xbmc/tvfilenames/ folder. (the filename scrapers use regex to pull out the series, season and episode for a given file path)



I think that I need to add some sort category/genre map so that we can have consistent genres. ie, if we see "Horror Film", then just use "Horror", etc.

Hmm, that's pretty much how my files look. XBMC has no issue with them and you're using the same scrapers right? Are these supposed to show in my "Videos" section (SageMC) or under TV?

The category map is a great idea. It would be much easier if everything wound up under the correct category. Love the plug-in, thanks for the response.

[sflamm]

Updated doc attached.

[Wheemer]

Does BMT require something to be in the metadata.properties file? Mine is blank and I'm not even sure what I should put in there...

[Nick_W]

Hi,

Just a couple of questions:

Does the following code still need to be manually added to Sage.properties? And if so, is it Sage.properties or SageClient.properties if running in service mode?

# This property needs to be added to your Sage.properties
# in order for the Metadata tools to be used as a metadata plugin source for SageTV
mediafile_metadata_parser_plugins=org.jdna.sage.MetadataUpdaterPlugin
custom_metadata_properties=MediaProviderDataID;MediaTitle;MediaType;OriginalAirDate;EpisodeTitle;EpisodeNumber;SeasonNumber;DiscNumber;UserRating

Here is my metadata.properties file, does it look OK? I just want to import metadata and fanart automatically when new stuff is scanned in by Sage.

#Configuration Properties
#Tue Apr 21 08:03:00 BST 2009
/media/useTSFolderForThumbnail=false
/metadata/airingIdRegex=([0-9]+)-[0-9]{1,2}.
/metadata/backgroundImageWidth=1280
/metadata/bannerImageWidth=-1
/metadata/defaultProviderId=tvdb,themoviedb.org,themoviedb.org-2,imdb.xml,imdb
/metadata/enableDefaultSTVPosterCompatibility=false
/metadata/goodScoreThreshold=1.0
/metadata/maxDownloadableImages=10
/metadata/posterImageWidth=350
/metadata/scoreAlternateTitles=true
/metadataUpdater/automaticUpdate=false
/metadataUpdater/fanartCentralFolder=//NAS1/FanArt
/metadataUpdater/overwrite=true

Lastly, should the regex be:

/metadata/airingIdRegex=([0-9]+)-[0-9]{1,2}.

Or:

/metadata/airingIdRegex=([0-9]+)-[0-9]{1,2}\.

Or:

/metadata/airingIdRegex=([0-9]+)-[0-9]{1,2}\\.

I should say it seems to be working OK.

TIA,

Nick.

[jaminben]

Originally Posted by sflamm

Updated doc attached.

Nice job, this will come in very handy. There is one small error which I may have misunderstood and that's:

A. ‘Automatic Metadata/Fanart may need to be “disabled”. Some folks are experiencing trouble with it.
If disabled, BMT will have to be run by command line to update metadata.
Fan Art will still be downloaded but metadata will not be automatically updated.

As far as I'm aware it will get the metadata and fanart images as long as Sage initiates the scan i.e you set Sage to auto scan your directories or you use the Sage rescan import folders option. If you try to scan via the SageMC BMT STVi import tool then it doesn't grab the metadata because Sage hasn't asked for the data.

[stuckless]

Originally Posted by Wheemer

Does BMT require something to be in the metadata.properties file? Mine is blank and I'm not even sure what I should put in there...

BMT does not require anything in the metadata.properties. The only time there will be something in there, is if you are using the UI to change the order of metadata providers. So, if you have nothing in there, then it means you are using the defaults, which is fine as well, for most people.

[stuckless]

Originally Posted by Nick_W

Hi,

Just a couple of questions:

Does the following code still need to be manually added to Sage.properties? And if so, is it Sage.properties or SageClient.properties if running in service mode?

# This property needs to be added to your Sage.properties
# in order for the Metadata tools to be used as a metadata plugin source for SageTV
mediafile_metadata_parser_plugins=org.jdna.sage.MetadataUpdaterPlugin
custom_metadata_properties=MediaProviderDataID;MediaTitle;MediaType;OriginalAirDate;EpisodeTitle;EpisodeNumber;SeasonNumber;DiscNumber;UserRating

This needs to be added to the server and not the client. So, if it's not added to the server, then I would add it. Not quite sure what would happen if it's added to the client.

Here is my metadata.properties file, does it look OK? I just want to import metadata and fanart automatically when new stuff is scanned in by Sage.

#Configuration Properties
#Tue Apr 21 08:03:00 BST 2009
/media/useTSFolderForThumbnail=false
/metadata/airingIdRegex=([0-9]+)-[0-9]{1,2}.
/metadata/backgroundImageWidth=1280
/metadata/bannerImageWidth=-1
/metadata/defaultProviderId=tvdb,themoviedb.org,themoviedb.org-2,imdb.xml,imdb
/metadata/enableDefaultSTVPosterCompatibility=false
/metadata/goodScoreThreshold=1.0
/metadata/maxDownloadableImages=10
/metadata/posterImageWidth=350
/metadata/scoreAlternateTitles=true
/metadataUpdater/automaticUpdate=false
/metadataUpdater/fanartCentralFolder=//NAS1/FanArt
/metadataUpdater/overwrite=true

Looks ok to me.

Lastly, should the regex be:

/metadata/airingIdRegex=([0-9]+)-[0-9]{1,2}.

Or:

/metadata/airingIdRegex=([0-9]+)-[0-9]{1,2}\.

Or:

/metadata/airingIdRegex=([0-9]+)-[0-9]{1,2}\\.

Probably the last one... i'll fix that in the next build... although the period is somewhat optional. If it did catch a file that wasn't an ariing, it wouldn't find it in the sage db, and it would just carry on as a normal lookup.

[Nick_W]

Thanks Sean,

In your previous post you said the guy didn't need anything in metadata.properties, so can I just get rid of all that? I notice that id I don't have it that the fanart does no get saved to the central folder?

TIA,

Nick.

[stuckless]

Technically... you don't need anything in there... If you are running in a plugin mode, and you have set your fanart folder in sagemc, then the plugin will use that value. But if you are running it from the command line, then you'll either have to pass the --fanartFolder paramter, or set it in the properties.

The most important thing that gets saved in the properties is the order of the providers. SO, if you are using the defaults, and you are using it as a plugin, and you've configured the SageMC central fanart folder location, then you don't need anything in the main metadata.properties.

[sflamm]

stuckless:

Attaching updated document with new information from thread (please review and check).

Can you please answer the following remaining questions...?

1. jaminben suggested:

As far as I'm aware it will get the metadata and fanart images as long as Sage initiates the scan i.e you set Sage to auto scan your directories or you use the Sage rescan import folders option. If you try to scan via the SageMC BMT STVi import tool then it doesn't grab the metadata because Sage hasn't asked for the data.

Is this true? Can you clarify what happens and how it happens when ‘Automatic Metadata/Fanart' is disabled?

2. Is there any issue with having both 'Automatic Metadata/Fanart' enabled and 'DVDProfiler Auto Meta Data' enabled?

3. Has the display of collections using the DVDPro2Sage plug-in been tested? Does it work correctly?

I will update document with your answers.

[Nick_W]

Originally Posted by stuckless

Technically... you don't need anything in there... If you are running in a plugin mode, and you have set your fanart folder in sagemc, then the plugin will use that value. But if you are running it from the command line, then you'll either have to pass the --fanartFolder paramter, or set it in the properties.

The most important thing that gets saved in the properties is the order of the providers. SO, if you are using the defaults, and you are using it as a plugin, and you've configured the SageMC central fanart folder location, then you don't need anything in the main metadata.properties.

Thanks Sean, it all seems OK without any properties. I'll let you know if I have any problems

Cheers,

Nick.

[stuckless]

Originally Posted by sflamm

stuckless:
1. Can you clarify what happens and how it happens when ‘Automatic Metadata/Fanart' is disabled?

When the Automatic Fanart/Metadata is enabled or disabled, it requires that the SageTV server be restart to take effect. Enabling and disabling "Automatic Metadata/Fanart" basically adds/removes the following Server properties.

mediafile_metadata_parser_plugins=org.jdna.sage.MetadataUpdaterPlugin

So, if this is enabled, and the server has been restarted, then when media is added to the libarary, this plugin will automatically search and download all metadata and fanart for the item. If this is disabled, then no automatic metadata will be fetched when new items are added to the libarary. This has no effect on any other aspect of metadata/fanart. ie, Manual Fanart searches still work.

2. Is there any issue with having both 'Automatic Metadata/Fanart' enabled and 'DVDProfiler Auto Meta Data' enabled?

I doubt there's any real issue with having both installed, but one will clobber the other, in terms of metadata... but I guess since they are getting it from the same source, there's no real issue. The important thing to realize is that work differently. The SageMC dvd profiler in no way works with bmt plugin.

3. Has the display of collections using the DVDPro2Sage plug-in been tested? Does it work correctly?

I don't use dvd profiler, and the dvdpro2sage is not my plug-in, so I have no knowledge of that plugin at all.

[razrsharpe]

Originally Posted by sflamm

2. Is there any issue with having both 'Automatic Metadata/Fanart' enabled and 'DVDProfiler Auto Meta Data' enabled?

I don't know if it is an issue or not but i see the two as completely separate entities that may adversely compete with each other to accomplish the same goal (provide metadata and posters to sage). There should be no reason to use the 'DVDProfiler Auto Meta Data' plugin if you are using BMT for your metadata.

On a side note: The DVD Profiler plugin places the coverart at ..\MOVIENAME\folder.jpg. On the extenders this seems to really slow down displaying coverart. (i know there are some property items to supposedly make this faster). I upgraded to the fanart system this past weekend (and deleted all the .jpg's in the movie folders). Browsing my collection is now just about instantenous (no delay in displayin coverart).

Side note #2: Now that everything seems to moving to fanart/posters in a central directory provided by BMT or mediascraper maybe the dvd profiler code for SageMC should be taken out (or not part of the default install or made into a seperate plugin) to remove confusion on the subject.

[sflamm]

razrsharpe: have you been able to test BMT with the DVDPro2Sage plugin to see that collections still are displayed correctly in 'My Videos'?


Attached is latest installation document.

[razrsharpe]

Originally Posted by sflamm

razrsharpe: have you been able to test BMT with the DVDPro2Sage plugin to see that collections still are displayed correctly in 'My Videos'?

Are you referring to multi disc dvd sets when you say collections? I guess im a little confused when you refer to collections. Is the DVDPro2Sage plugin responsible for something more than gathering meta data?

I believe that multidisc dvd sets (TITLE - Disc # format) gather all the correct metadata/posters from within BMT but will confirm tonight when i get home.

[sflamm]

Yes, by collections I mean multi-disc dvds sets.

As you said, I believe the DVDPro2Sage gathers the metadata information from the DVDProfiler collection.xml export and then associates it with the hierarchy of directories (allows SageTV to store the right meta data).

I just want to check that when BMT looks at the DVDProfiler collection.xml that it creates metadata for SageTV that also reflects the multi-discs correctly in the directory hierarchy.

Here's an excerpt/example of a Bob Dylan 2-disc set represented by DVDProfiler. Does BMT know how to parse and create the right meta data for the multi-discs highlighted?

<DVD>
<ProfileTimestamp>2008-12-03T08:31:05.000Z</ProfileTimestamp>
<ID>097360310542</ID>
<MediaTypes>
<DVD>True</DVD>
<HDDVD>False</HDDVD>
<BluRay>False</BluRay>
</MediaTypes>
<UPC>097360-310542</UPC>
<CollectionNumber>62</CollectionNumber>
<CollectionType>Owned</CollectionType>
<Title>Bob Dylan No Direction Home</Title>
<DistTrait>Full Screen Collection</DistTrait>
<Discs>

<Disc>
<DescriptionSideA>Disc 1</DescriptionSideA>
<DescriptionSideB/>
<DiscIDSideA>A6224B24FFDB2B55</DiscIDSideA>
<DiscIDSideB/>
<LabelSideA>NO_DIRECTION_HOME_D1</LabelSideA>
<LabelSideB/>
<DualLayeredSideA>True</DualLayeredSideA>
<DualLayeredSideB>False</DualLayeredSideB>
<DualSided>False</DualSided>
<Location/>
<Slot/>
</Disc>
<Disc>
<DescriptionSideA>Disc 2</DescriptionSideA>
<DescriptionSideB/>
<DiscIDSideA>FE82E86263553D3E</DiscIDSideA>
<DiscIDSideB/>
<LabelSideA>NO_DIRECTION_HOME_D2</LabelSideA>
<LabelSideB/>
<DualLayeredSideA>True</DualLayeredSideA>
<DualLayeredSideB>False</DualLayeredSideB>
<DualSided>False</DualSided>
<Location/>
<Slot/>
</Disc>

</Discs>
</DVD>