Utility: Batch Metadata Importer 3.x

[stuckless]

Originally Posted by mkanet

Actually, I dont have any Jetty duplicate/old JARs since this install is a virgin BMT/Jetty setup (I reverted to a clean non-BMT sageTV setup after giving up on BMT 2.x. I noticed you told plucky to check for duplicates and other JARs. Maybe he doesnt have duplicates either.. and probably something to do with other jar files. Im pretty sure I have all the latest jars needed for other plugins (most of them shared). When I get home from work, I will see if/how jetty is failing from starting up via the sageTV log.

I tell people to check for duplicate jar files because it's a easy test to rule out. ie, either you have them or you do not. Since you do not, we move past that, and move on to misconfiguration.

This may sound really naive; especially since I dont write code.... but can't you just make a large jar (integrating pheonix, jetty, etc)? This way there's less dependencies to worry about. I'm thinking, there's gotta be a way to make BMT more dependable. I think in the past MauiMike offered a suggestion with code that would make things more dependable (maybe I misunderstood his coding suggestion).

Actually what you are suggesting would be a nightmare to support. Imagine if every plugin that used phoenix, sagex-api, etc, all decided to deploy their own embedded versions of that phoenix jars. How would be know which version of the class files the java application is loading. jars are really like .dll files. They are libraries that can be shared by multiple applications. I really don't think it's too much to ask that people inspect their installations after they extract a plugin.... personally, I extract a plugin to a tmp area first, inspect the files that are being added, so that I know what is being added and what needs to be removed when it doesn't work

Since you've ruled out the duplicate jars, and I can confirm that the application works (since I use it daily), then really all we are left with is configuration.

personally i would ask for jetty support in the jetty forum... and once that is working, then apply the bmt plugin. Plugins should be installed, configured and verified, one at a time. While bmt does not require jetty to run, it's experience can be greatly enhanced via the web ui, which does require jetty (only if you want to use it).

[stuckless]

Originally Posted by Ikarius

Hey there- I got this loaded (funny, it was only 2 nights ago I jumped back into Sage fiddling, and had installed the last beta of it), and it looks great. Thanks for the impressive work on this!

First, if you haven't done so, I'd update the non-beta 3.0, since there are usability enhancements and bug fixes in that release.

1. When I tell it to scan DVDs- it appears to work with a couple of niggles; it took all of my Battlestar Galactica DVDs (previously titled Battlestar Galactica S1D1 etc), and reset the titles to "Battlestar Galactica"- so now under iMovies they are indistinguishable from one another. Is iMovies simply using the wrong fields, and needing an update, or is there something to adjust under BMT?

I don't use iMovies, so I don't know what it does... but bmt uses a specific title mask for multi-dvd sets. Search the configuration (in the web ui) for "mask", and make sure the tv multi disc mask is set for your needs. (I'll also verify that the multi-dvd mask is working, it may be a bug, so i've submitted an issue)

2. When I have it scan DVDs, the only ones it shows me are ones which it didn't feel it could match Metadata for. Any possibility it could show *everything* it scanned so that if we'd like to touch-up metadata, or if it's matched against something with a very similiar title, and we need to point it at something different, we have that opportunity?

Scanning only returns the items for which there wasn't complete metadata/fanart. I do this, because in my library I have over 1100 items and i don't want to see all 1100 items, if only a couple have failed. The failed list is returned so that you can immediately start fixing the metadata while the background scanner is still running. But, after the scan is complete, you can select scan again, and check the "do nothing" box, and it will return all media files in your collection (or you narrow down the list using a filter).

3. It appeared unable to match metadata for the movie "Léon". A straight search on imdb comes up with it, so I'm going to guess it's got issues with the special character. Also, when I adjusted the title to "The Professional" in the BMT interface and re-ran the search, it still was unable to find metadata for it. I suspect it's pulling the title from the existing metadata, instead of from the web form, leaving me unable to adjust what it's searching for.

Was this a sage recording movie, or a non sage recording? I'll try to do a test with this movie. btw... bmt will search using the MediaTitle not the display title.

4. When I have it scan recordings- it simply appears to fail at looking up metadata. My TV shows all have metadata already in them from SageMC + neilm, but it appears unable to locate results for any of them. From the jetty interface, I select one of my TV recordings- an episode of Breaking Bad for example- I choose "Find Metadata", and it comes back with 0 results. This is consistent across all the shows I have recorded. tvdb is enabled as a provider, and searching for "Breaking Bad" on tvdb immediately comes back with the various languages it is available in. Any guesses what might be going on here?

I'm not sure about this one... but it may be similar to the fact that Sage Recordings don't appear to work using the stv plugin. In my environment, though, Sage Recording do search fine via the web ui.

Last- I recall there were a lot of discussions about how the fanart API would work, and flagging where fanart images came from; what if I have an image I've personally touched up or created. Can I simply copy it into the fanart directory structure and have it work?

Yes, you can. Just copy any image to the posters, backgrounds, etc, and it will be used. The web UI will show you the default folder locations for a media item.

[jaminben]

Hiya Sean,

Thought I'd just send a congrats and thankyou for the new release, can't wait to try it.....just gotta wait for the system to become free now.

[PLUCKYHD]

I can't seem to get the latest to work now when I add new media it never appears in Sagemc althought the properties are created.

[Ikarius]

Sean,
1. I did already update to the full release version of BMT.

2. "Léon" is a ripped DVD from my movie collection. The folder it resides in is "The Professional", and it's current Display Title is "Léon". I've done IMDB searches for "Leon", "Léon", and "The Professional", and all of them come up with the movie in question; when BMT searches, it comes up with a large list (~25) of potential results, but none of them are the movie in question. In the jetty interface, BMT does not show me the Media Title, so I cannot verify what it's actually using to search.

3. How do we go forward on troubleshooting the failure at scanning the Sage recordings? I can look at debug logging and whatnot. One thing I would like to validate is what it's actually using to search based on.

4. Last- you didnt answer my question about UI elements to pick "preferred" Fanart. Is there no UI available for this functionality yet?


Cheers
Ikarius

[PLUCKYHD]

Originally Posted by PLUCKYHD

I can't seem to get the latest to work now when I add new media it never appears in Sagemc althought the properties are created.

Fixed I just cleaned wiped everything sense I was have issues with Jetty as well

Great work so far!!!

[stuckless]

Originally Posted by Ikarius

Sean,
1. I did already update to the full release version of BMT.

Good... just making sure.

2. "Léon" is a ripped DVD from my movie collection. The folder it resides in is "The Professional", and it's current Display Title is "Léon". I've done IMDB searches for "Leon", "Léon", and "The Professional", and all of them come up with the movie in question; when BMT searches, it comes up with a large list (~25) of potential results, but none of them are the movie in question. In the jetty interface, BMT does not show me the Media Title, so I cannot verify what it's actually using to search.

Even though I said "Media Title", the web ui labels it as "Show Title". I just verified that when you do a lookup from the web UI, it will use the "Show Title" as the search criteria.

3. How do we go forward on troubleshooting the failure at scanning the Sage recordings? I can look at debug logging and whatnot. One thing I would like to validate is what it's actually using to search based on.

For now, the log files is the only way. Eventually I want to update the "Find Metadata" search panel to include which search providers to use, etc... time, if only I have more time....

4. Last- you didnt answer my question about UI elements to pick "preferred" Fanart. Is there no UI available for this functionality yet?

I think i mentioned in the release notes that not all of the features that I had hoped for made it it into the 3.0 release . Actually just being able to quickly popup the fanart was something was just added last night night. I haven't yet added functions for removing fanart, renaming, or setting the defaults... it's comming... in time... (maybe a 3.01 release in a month or so)

[razrsharpe]

Originally Posted by Ikarius

4. Last- you didnt answer my question about UI elements to pick "preferred" Fanart. Is there no UI available for this functionality yet?

When random Fanart is disabled, SageMC (and i think most (all?) other imports that use fanart and the phoenix api) will pick the first one when sorted lexically (at least this is what it appears to do for me and my testing). So you could open up windows explorer and pick the fanart you want then rename it so its first in the list (sorted by filename).

[jaminben]

Originally Posted by razrsharpe

When random Fanart is disabled, SageMC (and i think most (all?) other imports that use fanart and the phoenix api) will pick the first one when sorted lexically (at least this is what it appears to do for me and my testing). So you could open up windows explorer and pick the fanart you want then rename it so its first in the list (sorted by filename).

Or you could use Amis and I think SMW, these allows you to pick which image you want......the only downside is that it requires a restart of the client or extender. It will sometimes change on its own but this only happens when the cache gets too full and needs to refresh (normally with large images).

[Ikarius]

Sean- thanks for the answers.

Here's an example of a failed lookup of a Sage recording from my log files:

Originally Posted by code

55239275 [14040275@qtp-14628397-5] DEBUG org.jdna.media.metadata.impl.composite.MetadataProviderContainer - Not a good search using provider: tvdb; Query: SearchQuery; Type: TV; TITLE:BurnNotice WantedMan 2474116 0;; will try another one.
55239275 [14040275@qtp-14628397-5] DEBUG org.jdna.media.metadata.impl.composite.MetadataProviderContainer - Provider: imdb-2 cannot handle query: SearchQuery; Type: TV; TITLE:BurnNotice WantedMan 2474116 0;
55239275 [14040275@qtp-14628397-5] DEBUG org.jdna.media.metadata.impl.composite.MetadataProviderContainer - Provider: themoviedb.org cannot handle query: SearchQuery; Type: TV; TITLE:BurnNotice WantedMan 2474116 0;
55239275 [14040275@qtp-14628397-5] DEBUG org.jdna.media.metadata.impl.composite.MetadataProviderContainer - Provider: imdb.xml cannot handle query: SearchQuery; Type: TV; TITLE:BurnNotice WantedMan 2474116 0

I was able to update Leon- "Show Title" was empty, so I filled that in, saved it, and was able to locate a match. Note- with "Léon" for Show Title, it came up with a seemingly random lot of results, none of which actually matched the movie, so looks like support for extended characters may have some issues.


Back on UI questions; there's one file in each Phoenix directory called "images", which has the list of filenames. If I add or rename an image, should I touch up that file, or leave it as-is? I forget the exact purpose of it.

Cheers
Ikarius

[toricred]

Originally Posted by stuckless

Maybe you can post the filenames... I'll try it in my environment. Also, when you are doing lookups from the web ui for tv shows... you have to fill in the Media Title, Season and Episode fields and then click find metadata. For TV shows the search will just return the matches for the show name... and when you select it, it will then pull out the specific season/episode information.

The filename is accordingtojim-thebaby-2849411-0.mpg. I've tried it both from the webui (although I didn't put the season and episode in) and inside SageMC on demand. In SageMC nothing is found.

[toricred]

Verified that I am running with Phoenix 1.42 and 3.0 for the metadata tools. Here is one of the error messages I see in the log file:

2009-08-18 21:59:18,338 ERROR [btpool0-7 - /bmt/bmtweb/browser] org.jdna.media.metadata.impl.tvdb.TVDBItemParser (TVDBItemParser.java:214) - Failed to get season/episode specific information for 73545; Season: 0; episode: 0
java.io.FileNotFoundException: http://www.thetvdb.com/api/5645B594A...45/default/0/0
at sun.net.http://www.protocol.http.HttpURLConn...Stream(Unknown Source)
at org.jdna.url.CachedUrl.cache(CachedUrl.java:135)
at org.jdna.url.CachedUrl.getUrl(CachedUrl.java:118)
at org.jdna.url.CachedUrl.getInputStream(CachedUrl.java:178)
at org.jdna.media.metadata.impl.tvdb.TVDBItemParser.updateMetadataFromUrl(TVDBItemParser.java:116)
at org.jdna.media.metadata.impl.tvdb.TVDBItemParser.addSeasonEpisodeInfo(TVDBItemParser.java:212)
at org.jdna.media.metadata.impl.tvdb.TVDBItemParser.getMetadata(TVDBItemParser.java:90)
at org.jdna.media.metadata.impl.tvdb.TVDBMetadataProvider.getMetaDataByUrl(TVDBMetadataProvider.java:57)
at org.jdna.bmt.web.server.BrowserServicesImpl.getMetadata(BrowserServicesImpl.java:317)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:527)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:166)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.doPost(RemoteServiceServlet.java:86)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:362)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:729)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:320)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:843)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:205)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)
2009-08-18 23:20:56,510 ERROR [VideoFrame-001e40b5a062] org.jdna.sage.MetadataUpdaterPlugin (MetadataUpdaterPlugin.java:178) - Failed to Fetch Metadata for Media: E:\vids\Motives-2906879-0.ts

[skiingwiz]

Since upgrading from beta9 to 3.0 final, the first two things that were recorded failed when scanned. These are both series that had been successfully scanned in beta9. Looking through the log, I see the following message

2009-08-19 22:24:16,226  WARN [24440346@qtp-3096409-6] org.jdna.media.metadata.impl.tvdb.TVDBSearchParser (TVDBSearchParser.java:67) - Could not find any results for: CachedUrl: http://www.thetvdb.com/api/GetSeries.php?seriesname=Leverage+TheTopHatJob+4836543+0; UrlId: d615172fcac4ebc44bbaa24063a20137

Comparing this log to previous logs (from before the upgrade) it appears that the entire file name is now being used in the TVDB getSeries call. Previously, only the portion up to the "-" in the name was used. (i.e. the series title)

EDIT: A little more info: the file name being scanned in the message above is "Leverage-TheTopHatJob-4836543-0.ts" and it is a sage recording from this evening.

[toricred]

I think this might be a clue as well.

2009-08-19 18:02:00,193 ERROR [MiniUIServer] sagex.phoenix.configuration.XmlMetadataProvider (XmlMetadataProvider.java:20) - Missing Configuration File: C:\Program Files\SageTV\SageTV\.\STVs\Phoenix\Configuration\Phoenix.xml

I don't think the \.\ should be there and there is no such file.

[Ikarius]

Okay- from what others have posted, this looks like an issue with the mask value found under "Configuration->Metadata Tools->Sage Persistence->Sage Recording TV Title Mask".

In my 3.0 install, the value is simply "${Media Title}".

Looking at the names of recordings, the filenames look like this:
ShowName-EpisodeTitle-UnkownNumber-0.ts

I'm guessing that the field I labelled UnknownNumber is the Sage internal MediaID, but I'm not sure. I also don't know what the trailing "0" is- my guess is if the same show is recorded multiple times it would increment, but it's only a guess.

I fiddled with the values for Sage Recording TV Title Mask, but without documentation (pardon me if this is documented, but if it is, I couldn't find it), I didn't get come up with a working value.

I suspect this is a simple matter of Sean letting us know what the correct value should be.

Cheers
Ikarius

[bcjenkins]

Sean,

When I import into the default STV I notice -

Thu 8/20 7:29:19.532 PluginManager.displayMessage: Message: Importing plugin
MetadataFanartTools-DefaultSTV.stvi
Thu 8/20 7:29:20.040 Module processing STVImported Hook
Thu 8/20 7:29:20.041 PATCH ERROR Missing Widget - Symbol=BASE-72617 "Metadata/Fanart" could not add as child; parent symbol=STUCKLESS-5871 "FocusLost"

Anything to be concerned about?

B

[stuckless]

Originally Posted by Ikarius

I was able to update Leon- "Show Title" was empty, so I filled that in, saved it, and was able to locate a match.

I added an issue for titles with special characters

Back on UI questions; there's one file in each Phoenix directory called "images", which has the list of filenames. If I add or rename an image, should I touch up that file, or leave it as-is? I forget the exact purpose of it.

the images file tracks the images that have been downloaded to the fanart directory, so that they are never downloaded again. (unless overwriteMetadata is specified)

[stuckless]

Originally Posted by toricred

The filename is accordingtojim-thebaby-2849411-0.mpg. I've tried it both from the webui (although I didn't put the season and episode in) and inside SageMC on demand. In SageMC nothing is found.

I think that on demand tv searching from the stv, is broken. I have to investigate that further next week, once I'm back from vacation.

as for your exception, i'm wondering if it's related to this or not... i did start to notice to that season 0 and episode 0 values were showing up. I thought that I found all cases where they were being used... typically the code checked if the season was empty... but i need to check for empty or 0 (and skip a season/episode lookup in those cases)

I'v added a bug for this.

[stuckless]

Originally Posted by skiingwiz

Since upgrading from beta9 to 3.0 final, the first two things that were recorded failed when scanned. These are both series that had been successfully scanned in beta9. Looking through the log, I see the following message

2009-08-19 22:24:16,226  WARN [24440346@qtp-3096409-6] org.jdna.media.metadata.impl.tvdb.TVDBSearchParser (TVDBSearchParser.java:67) - Could not find any results for: CachedUrl: http://www.thetvdb.com/api/GetSeries.php?seriesname=Leverage+TheTopHatJob+4836543+0; UrlId: d615172fcac4ebc44bbaa24063a20137

Comparing this log to previous logs (from before the upgrade) it appears that the entire file name is now being used in the TVDB getSeries call. Previously, only the portion up to the "-" in the name was used. (i.e. the series title)

EDIT: A little more info: the file name being scanned in the message above is "Leverage-TheTopHatJob-4836543-0.ts" and it is a sage recording from this evening.

I've added a bug for the TV scanning issues... This is very odd, but at least you've given me something very specific to focus on. Thanks.

[stuckless]

Originally Posted by toricred

I think this might be a clue as well.

2009-08-19 18:02:00,193 ERROR [MiniUIServer] sagex.phoenix.configuration.XmlMetadataProvider (XmlMetadataProvider.java:20) - Missing Configuration File: C:\Program Files\SageTV\SageTV\.\STVs\Phoenix\Configuration\Phoenix.xml

I don't think the \.\ should be there and there is no such file.

This particular error is not important (yet). BMT uses the configuration management apis from phoenix. Since phoenix hasn't been released, then it's configuration metadata (Phoenix.xml) is missing, which currently is more like a warning, than an error.