Batch Metadata Tools new releases subscription thread

[stuckless]

Download | Discussions

This is a thread that I'll use to announce new releases to the Batch Metadata Tools (BMT). Please don't post comments or questions in this thread, but rather use the general discussion thread for comments/questions.

I Probably should have started this thread a long time ago to keep the release notes together in a more readable manner.

Users can subscribe to this thread for new releases.

[stuckless]

Download

Requirements: SageTV 6.5.14 beta

48 Add MyMovies xml support
MyMovies support is added in the same way that DVDProfiler support is added. You provider BMT with the location of your MyMovies xml file and it index it and use it as a metadata source. BMT will re-index the xml file automatically as the file changes. MyMovies doesn't require you to set an images folder, since it stores the complete path to its images in its xml file. Set the following property in your metadata.properties file to identify the MyMovies xml file.

/mymovies/xmlFile=

61 Enabled plugin to support ShowTitle-airingid.mpg titles
85 Parse recorded TV shows
88 Added Episode Title lookups to tvdb provider
These are all similarly related. The bottom line is that the tool/plugin will now support fetching metadata/fanart for Sage Recordings. If you are running bmt as a command line tool and see a warning about Sage Remote APIs, then you can ignore it. If the remote apis are installed, then it will be used to find the Series Name and Episode Title based on the AiringId in the filename. Without it, it will still work, it will just have to work a little harder since it will pull the Series Name and Episode Title from the filename and try to find matches using the compressed names.

84 Some Manual Titles not being stored in metadata-titles.properties (mainly dvdprofiler)
Fixed a bug in the commandline tool where it was not recording a match when the user selected a search result that be created from the dvd profiler search results.

86 Use new SetMediaFileMetadata API call found in 6.5.13 beta
SageTV 6.5.14 added the SetMediaFileMetadata api. This is now used by the On Demand Manual Fanart/Metadata Search. When you do a Metadata/Fanart search for a media item, it's metadata and fanart will be downloaded. (Previously only it's fanart was downloaded). Even though metadata is fetched for a given media item, it's metadata may not be refreshed until the next sagetv refresh cycle. Fanart should show up immediately.

89 Added composite fanart providers for all standard providers
By default, the list of known metadata providers will include 3 composite providers that are automatically created a registered. Their ID will be the same as the parent id and include a -2 suffix. The purpose of these providers is to add Fanart support to dvdprofiler, mymovies and imdb. The 3 new ids are, dvdprofiler-2, mymovies-2, and imdb-2. You should see these providers show up in the list of available provider in the STV or when you use the --listProviders command line arg


One final note is that this release also includes an experimental feature than can be used to import TV media items into your sage recordings. By default this feature is turned off. If enabled, when you do a Metadata/Fanart search on a TV show from your video library, it will be imported into the SageTV recordings. One media is imported into the sage recordings, it's very hard to move it out of the sage recordings. So make sure this is something that you want to happen, or don't enabled it. I don't want to hear about how BMT messed up your wiz.bin

To enable this experimental feature for on demand searches from within the stv, then set this property....

/metadata/importTVAsRecordedShows=true

Good luck and report all issues to the BMT main thread.

[stuckless]

Download

Issues Fixed
90 Ensure RunningTime is a number in the metadata properties
The RunningTime field should now be stored as a number (milliseconds) for all providers.

91 DVD Profiler Actor/Cast is broken
Fixed a bug that prevented the Actors/Cast from being parsed in the dvdprofiler xml.

In order for sage to reimport the metadata, you'll need to remove your existing .properties for each mediafile. Otherwise, the tool/plugin will simply re-read whats stored in the properties file.

[johnm160]

Thank you !!!!!

Now I can finally settle on a solution.

[stuckless]

Here's beta of the upcomming 3.0 release.

This beta is pretty much a rewrite of the 2.x release. The configuration format has changed. It now uses the phoenix configuration metadata for all configuration operations. That means that there is NO LONGER a metadata.properties, and all configuration options are stores in the Sage.properties. Because of this, all the configuration keys have changed. They are pretty much the same as before, except they now are prefixed with "bmt/".

The 3.0 release now has a web based configuration application. You will need to install the Jetty plugin, if you don't have it installed already. Once jetty is installed, you should be able to access the BMT configuration by using the following url

http://YOUR_JETTY_SERVER:YOUR_PORT/bmt

I've included 2 screen shots of the new bmt web based application. Currently it shows a status screen and you can access the configuration using the "Configuration" link. Ignore the "Scan" link.

The "scan" link, when 3.0 is released, will enabled you to perform batch scans on your collection from the web ui. It will also allow you to perfom single scans, and allow you edit the metadata for an existing media item. Currently it doesn nothing interesting.

As far as the configuration is concerned, browse the configuration to get a feel for what you can configure. Not much is new, but you might be surprised by some of the configuration options you may have missed. The configuration screen also allows you to search the configuration, so you use that if you are having probleming finding something specific.

Given this is more beta than my normal beta code, expect to find some issues. Post the issues NOT in this them, since this is a notification thread. Post them in the BMT forum, or create a new thread, but NOT HERE.

[stuckless]

This beta includes fixes for the dvd scanning that broke in moving from 2.x to 3.x. I've also fixed some issues with the status panels.

I've been running this build, and it appears to be working fine in my environment.

I've started to implement the "Scan" dialog. The dialog will now perform a scan on your collection, but it will NOT update anything. But you can use it browse your current metadata values, etc. I've attached a screen shot to show what the scan results look like. Perhaps in the next build, I'll include the ability to edit/sage metadata values and hopefully the ability to perform metadata/fanart scans on the collections (tv, dvds, videos, etc).

I've also added a "Refresh Library" command to the top nav menu. This will open a dialog and allow you to do a full reindex (ie, set the force reindex property in sagetv), or a normal, just find/import new media scan.

[stuckless]

This new beta fixes 2 issues reported by babgvant.
* Some Files were not being picked by by the automatic plugin
* Bug in the web UI, provider list changes were not saving.

BTW... I just confirmed, that under linux, when a new show is recorded, it does call the metadata plugin. I'm attempting to verify under windows as well.

[stuckless]

A new beta that fixes a bunch of null pointer exceptions.

Here's brief rundown....
* Automatic Plugin will now overwrite metadata if Overwrite metadata is enabled in the plugin section of the configuration
* Fixed NPE when doing searches using providers that don't exist. (i saw this in several log files that were submitted to me)
* Fixed status tracking bug... counts were off.

Web Updates
* Fixed Status counts for Scanned, Success and Fail.
* Fixed Status display for the last 5 failed
* Added new statuses for JVM Memory
* Fixed bug in how web ui determined if the plugin as active
* Fixed status update error bug. Sometimes the web ui would return an error when updating the SageTV statatus panel.
* Scanning is getting closer. You can scan, but it still won't actually update anything. But still useful if you want to look at your metadata for an item.
* Scanning now uses an async progress monitor so that scan results are incrementatally updated as scan results as available.


Keep in mind, that I haven't started to tackle the known issues in the Issues database yet. Once I get a final build of 3.x, then I'll start to tackle those.

Thanks to everyone that is testing the beta releases. Much appreciated.

[stuckless]

Another beta for 3.0

This beta doesn't add much, but it should fix the Web problem that rmac321 and robogeek have reported. In the even that it does not fix the problem it will log information that should help me track down the real issue.

Also, the scanning from the web ui is somewhat working for real, but I haven't been able to do much testing. I'm rushing out this release because of the exception that robogeek posted in relation to the web ui.

[stuckless]

Download

This beta adds Alternate title support to the DVDProfiler provider. Previously, only the <Title> element was ever indexed and searched. Now both the <Title> and <OriginalTitle> elements are indexed and searched. I never noticed the <OriginalTitle> element before, but I noticed in a user's collection.xml where the <Title> was Vacation and the <OriginalTitle> was National Lampoon's Vacation.

Something that I may not have mentioned is that somewhere in the 2.x builds, the Composite Provider, themoviedb-2 was renamed to imdb-2. This was done because I autocreated a bunch of -2 providers that used a core provider for searching and themoviedb.org for fanart. I've noticed in several people's logs that they are still referencing themoviedb-2 instead of imdb-2. So, I decided to add back themoviedb-2 as an alias for imdb-2.

[stuckless]

Download


This build fixes a bug in the DVD timestamping... when bmt would run on a dvd, the timestamp would only update to 1969.... It now should update correctly.

Added a --touch command line option to be used if you want to simply touch a dvd or media file, so that sage can pull in the updated .properties. (I see that babgvant posted a "touch" utility as well, thx)

The domain for IMDB is not configurable in the IMDB Configuration panel.

Added XMBC IMDB configuration panel

Added configuration option in the Plugin configuration where you can enable a "Missing Media Items Workaround". When this is enabled, the plugin will fetch metadata/fanart, create the properties, but it will return NULL to sage. Sage will then read the .properties and import he metadata. For some reason, there is something in the Map that BMT returns that is causing an issue to Sage/BMT pair, and I still have to resolve the root cause. This is a work around proposed/tested by babgvant.

[stuckless]

Download

Fixed a bug where the "work around" flag was not being read when the automatic plugin was reusing existing properties.

Added more error checking around the image download. It now will check for 0byte files and remove them.

Tweaked the web ui scanner to prune out capture devices, etc, from the video list.

[stuckless]

Download



Here is the final build for 3.0. I have to apologize because not all features that I had hoped to put into this build made it, but it's fairly complete. The major focus of the 3.0 build was a rewrite of the VFS engine and a new Web UI for configuration and management.

Here's the complete list of bugs/enhancements for 3.0 (that I tracked)
ID Type Status Summary
122 Defect Duplicate Season tv series on disc do not write season information to properties file or disk information
41 Defect Fixed Movies that match on the aka title, puts it's fanart in the original titles directory
63 Defect Fixed Exclude 0k (zero) images from phoenix_api_GetFanartBackgrounds()
95 Defect Fixed For Recorded Movies, check GetShowCategory to check if a Movie
98 Defect Fixed String Index out of bounds in FanartUtil
99 Defect Fixed Composite Providers not being loaded via the Plugin
100 Defect Fixed Plugin reports exception for some cases
101 Defect Fixed metadata-titles not being used for on-demand searched
102 Defect Fixed Add option to force Fanart Lookups even if properties do exist (for plugin)
103 Defect Fixed Plugin Fails to update Fanart
104 Defect Fixed Ensure that Plugin cannot throw an Exception
106 Defect Fixed Prevent 0 byte image files from being created
113 Defect Fixed Error in the Similary Scoring Algorithm that generates NaN when only #s are in a Title
117 Defect Fixed & in titles getting removed
119 Defect Fixed Season and Disc not being stored for TV Discs
124 Defect Fixed Ignore Filter not working
125 Defect Fixed GetMetadataScanComplete not working
18 Enhancement Fixed Allow optional direct TV import for TV media
53 Enhancement Fixed Refactor: Replace BMT config with phoenix config
65 Enhancement Fixed Add Phoenix api: GetFanartActor
109 Enhancement Fixed Use Specific Title masks for TV, Recorded TV, Movies, and Music
29 Defect Invalid Central Fanart Directory prepends current working directory in some cases
93 Defect Fixed More Issues with . and & in the titles
96 Defect Invalid Rating does not appear to be parsed from some providers
97 Defect Invalid MetadataPlugin not being called when new Shows are Recorded
105 Defect Invalid Posters not written to the default location when Fanart Disabled
111 Defect Invalid Jetty 1.6 not working with current BMT

In terms of the Web UI, you can scan and edit metadata. You can import as TV recordings, you can remove a mediafile from being a TV recording. As always, in doing any metadata operations, you risk making a mess of your media file metadata. With a Remove from TV recording operation, you may risk losing your media file, since the only way to remove a sagetv recording is to rename it (which is what bmt does). When you save the metadata, if you have the "Sage Recording" checked then the media file and metadata will be imported as a Sage Recording. If it is a Sage Recording and you uncheck it, then the media file will be removed from the Sage Recordings.

The Web UI added some extra help notes in the configuration panels.

A special not about enhancement 109... Check the tv masks and you'll see that their now a tv title mask specifically for Sage Recordings. This is the Title mask that will be used when you import a TV file as a Sage Recording. There is still a title mask for a TV file that is NOT a Sage Recording.

[stuckless]

This release fixed a small issue where on-demand searching, the automatic plugin, and web ui was broken for TV media file types.

I've testing the on-demand searches and the web ui for TV shows, and it now works. I haven't tested the automatic plugin, but it calls the same code and uses the same process, so it should work as well.

[stuckless]

This is small untested change to the metadata plugin that hopefully should prevent it from going into an endless loop and crashing the sagetv service.

My apologies if this doesn't do anything... since I had no time to test it... and I can't even log in after this to see if this worked or not

My recommendation, if this doesn't work, is to disable the automatic plugin until I release a permanant fix. WebUI, on demand scanning from the stv, and the commandline should still function normally.

[stuckless]

Here is release 3.0.3.

Fixes...

76	Enhan	Add rename option for renaming media to a consistent format
114	Enhan	WEBUI: Enable Save Metadata
121	Defect	WEBUI: Fix File Selector
130	Defect	TVDB provider is trying to find matches using Season 0 episode 0
133	Defect	Sage Object Wrapper appears broken for ALL TV
139	Defect	Plugin Causes the LibraryImport thread to die on DVDs
142	Defect	WebUI: Add "Browse" link to the menu that bypasses the search dialog
143	Defect	SageShowPersistence is not saving titles

There are some noteworthy fixes in the this release.

I'll start with #139. This is a bug in the SageTV LibraryImporter thread where if a metadata plugin passes Null data back to the SageTV Importer, then the Importer will stop scanning media. I've filed a bug with SageTV, and in the meantime, i've added extra checks to ensure that BMT does not pass NULL data values back from the Automatic Plugin. For some reason, this only affected DVDs. As a side note, I've removed the "Work around hack" since it wasn't reliably in my my testing.

#114,#143 are are related. In previous versions of the WebUI, when you saved Metadata, it would only save to the properties and the custom metadata fields. Then a "refresh" is required for sage to pick up the values. This resulted in a timing issue where the return page would display the old data, but eventually sage would pick up the new data. Now, the web ui will write directly to the Sage Show/Airing/MediaFile objects.

Not listed here, is a fix for another potential issue. I did find that the on-demand scanner EPG items was calling SetMediaFileMetadata() and it should not do that, since there isn't a real media file in those cases. Not sure what the side effects that are to be honest, but I'm sure there are some.

In terms of the UI... I've updated the File Selector (#121) so that it will list all of your root filesystems (ie, c:, d:, etc) and there are buttons to quickly jump you to the file system roots or the sagetv home dir.

In the UI, the "Find Metadata" will not show a dialog with information that you populate when doing a search. It pre-fills it from the existing metadata, but this should be easier for people to fill in querries. Also, you can select a specific metadata provider when doing that search, or use the default system configuration.

Also, in the UI, I've separated the browse and scan functions into 2 separate dialogs for ease of use. I've also added help labels (denoted by a ? icon). If you click on the ?, a simply help/description will appear.

And finally, in the UI, I've added a configuration panel under the SageTV section for "Video Sources". This will allow you to add/remove video sources (ie, SageTV Video Import Diectories). I added this because in my testing, I'm constantly adding/removing video folders for various tests, etc, and I find doing this in the web to be faster than doing this via the STV interface.

[stuckless]

I just noticed some title issues in 3.0.3.... i'd hold off downloading and installing this release until version 3.0.4.

[stuckless]

Download

You can read the release notes for 3.0.3 (a few posts back) since nothing new was added/fixed in this build.

You might be wonder why I opted to call it a 3.1 build, if nothing new was added/fixed since 3.0.3. Well, the answer is that lots of things were fixed in this build, but nothing from the change list. In 3.1, I decided to do a full regression test on the complete suite. I tested the web ui, command line, automatic plugin, on demand plugin, tv importing, tv exporting, etc, etc, etc. What I was after was basically consistency. ie, each aspect of BMT that imports metadata should be consistent, such that titles that are created when you use the web ui should be consistent with titles that are created from the automatic plugin, or on demand plugin, etc.

While nothing new was added in terms of functionality, ALOT of changes were made to each subsystem to ensure that things remained the same no matter which system you used to import data (web, commandline, automatic plugin, on demand, etc).

While there are still bugs in the application (no application is without bugs), I'm hoping that they'll be consistent in each sub system

This is the first build, perhaps since the first 1.0 build that has undergone such extensive regression testing and unit testing. I'm hoping it will pay off. I've wiped my wiz.bin several times, imported full TV sets as sage recordings, imported TV as non sage recordings, import movies, imported DVDs, multi-disc movies, etc.

Good luck, and remember, this application messes with your metadata... there is no undo, so tread lightly. No one should blindly just use a tool that can have such destructive potential. If you have never used this tool, then I suggest your start with the commandline, on-demand plugin, or web ui, since you'll have better control over what happens. And if you are happy with those results, then move to the Automatic Plugin.

One final note about importing TV shows as Sage Recordings. The commandline and Automatic Plugin CANNOT import shows as sage recordings (there are technical reasons). So, if you want to import TV shows as Sage Recordings, then you'll need to use the Web UI or on demand plugin. Personally I'd use the Web UI.

[stuckless]

This is just an initial notice to let people know that the beta for 4.0 has started. I won't continue to post updates here, but you can follow the beta progress in this thread.