BMT 4.1 - The Changes (Please read before or shortly after upgrading)

[stuckless]

4.1 represents a huge amount of changes to the metadata tools plugin. As you may be aware, but I'm involved in BMT, sagex-apis, phoenix, and a couple of projects. In 4.1, I decided to move the metadata lookup code to phoenix to provide better integration with the Phoenix project. This means that "bmt" was basically stripped of many features/options.

When SageTV moved to version 7 they introduced some new apis and events that had a better fit with what bmt was trying to do, and so this release leverages those new APIs. The 4.0 release in Sage7 was basically a Sage6 release that simply installed installed a Sage7 environment.

Since all metadata operations are moved to the phoenix apis, then the core bmt plugin became a command-line shell. ie, it doesn't provide any real functionality other than enabling some basic lookup functionality via the command-line.

The bmt web ui was changed to depend on the phoenix core services that now provide the metadata lookups.

In moving the lookups from bmt to phoenix, mnay options were removed. Some were removed because they no longer made sense, and other options were removed simply to make the configuration easier. And, in some cases, the defaults for some options were changed. For example, when you install/upgrade the 4.1 release, automatic lookups are disabled by default. System messages are disabled by default as well. You can configure the metadata options by configuring the phoenix-core plugin, or by configuration the "Phoenix" node in the web ui configuration.

If you upgrade from 4.0 to 4.1, then there are some "upgrade" tasks that take place, since some file locations are moved. For example, in the root of the SageTV home directory, there used to be a "scrapers" directory. That directory is now located under STV/Phoenix/scrapers area. The scrapers are not moved, since the install process will install the new/updated scrapers to the new location. SO, if you have added any scrapers yourself, you will need to copy them to the new scrapers location. The upgrade process will attempt to migrate your MediaTitles.xml to the new scrapers area, but not the filename scrapers.

As I mentioned, 4.1 (or more correctly, the phoenix core metadata services) represents a native Sage7 plugin. .properties files are no longer required to update metadata. File touching (ie, updating timestamps) is no longer required to update metadata. So, when metadata is updated, it is written directly to the sagetv database using the new Sage7 apis.

One thing that is new, is that the metadata services, by default, will not overwrite epg data for recordings, but it will still update the season/episode information.

If you upgrade to 4.1 and you see a VFS error related to ext/bmt.xml, then somehow, the upgrade process did not do it's job, and that file will need to be manually removed, since it is no longer valid.

The current release notes for the phoenix core metadata services includes the following...
2.0
* Phoenix version now bumped to 2.0 to reflect lots of small api changes and the addition of metadata and fanart core code.
* Phoenix now owns all scrapers and Metadata Parsers
* Removed MyMovies since it was not being used
* Updated MetadataAPIs to support searches, lookups, updates, etc
* Added a default 10 second timeout for all URL connections (ie, for fanart downloading, etc, so that it would not block forever)
* Added Phoenix Configuration to the SageTV plugin configuration
* Now uses Sage7 MetadataAPIs for reading/writing metadata
* Now uses Sage7 events for automatic lookups (mediafile imported and recording completed)
* Will not attempt to find metadata simply by tuning to a channel
* Removed many configuration options as it relates to overwriting metadata fanart
- Will never overwrite metadata when doing an automatic lookup
* Property files are no longer created since Sage7 provides direct APIs for setting metadata values
* Fixed bugs in the imdb scraper for writers, directors, etc
* Upgraded tmdb movies to the 2.1 api
* Fixed year issue when looking for best match against a year
* TV guest stars are stored in "Guests" not "Actors"
* Added flag to prevent updating metadata for recordings

Good luck with your upgrade/install, and feel free to ask questions. I apologize in advance for those that are upset about missing features, or if you experience issues. This is a large project and I have many test cases that are run before each build, but sometimes, things just don't work as they should

[aflat]

Does phoenix have a command line scraper like Metadatatools.jar used to?

[davidk21770]

Will it still allow me to replace metadata from the web ui?
Especially, will it allow me to bring in my own shows from other sources and add them as sage native shows that will mix properly in the Sage Recordings view in SageTV?

[gplasky]

Originally Posted by stuckless

The current release notes for the phoenix core metadata services includes the following...
2.0
* Removed MyMovies since it was not being used

Except by me. Bummer.

Gerry

[stuckless]

Originally Posted by gplasky

Except by me. Bummer.

Gerry

Really.... you used the mymovies provider? I honestly didn't think anyone used it. It's an easy thing to add in, so I'll put it back in the next release. I was just looking to strip features that wasn't being used, to save on the amount of code that I needed to support

[stuckless]

Originally Posted by aflat

Does phoenix have a command line scraper like Metadatatools.jar used to?

BMT still has it's own commandline, it's just been stripped of some features. But the bmt commandline should still to metadata/fanart lookups and it will create .properties for files.

Originally Posted by davidk21770

Will it still allow me to replace metadata from the web ui?
Especially, will it allow me to bring in my own shows from other sources and add them as sage native shows that will mix properly in the Sage Recordings view in SageTV?

Yes, the web ui hasn't actually changed that much. The main changes to web ui is that the option panels for saving/searching have been removed, or stipped down. Also, web editor will not allow you to edit some fields in recordings, if you have option enabled that prevents recording metadata from being overwritten.

[gplasky]

Originally Posted by stuckless

Really.... you used the mymovies provider? I honestly didn't think anyone used it. It's an easy thing to add in, so I'll put it back in the next release. I was just looking to strip features that wasn't being used, to save on the amount of code that I needed to support

And BMT does a great job all on it's own. But with My Movie providing the details and moviedb.org providing the fanart it was a great combo. Sometimes it seemed to have troubles grabbing the fanart using the mymovies-2 so I would grab the fanart first from moviedb.org and then grab the mymovie metadata.

I haven't upgraded yet but the only other issue was getting the running time or duration to show accurately in Sage. Sometimes it would and sometimes it wouldn't. Not sure what was going on there.

I feel bad making more work for you to code and maintain because I know it takes a lot of work and time to create and maintain this great tool. So I want to let you know all of your hard work is greatly appreciated.

Gerry

[stuckless]

Not to worry Gerr, the MyMovies plugin is 99% similar to the dvdprofiler plugin, since they both work the same way. I just removed it because I had a large # of issues of people not getting it working (or it wasn't working as they expected), so i assumed that perhaps it was fundamentally broken, especially since I've never run MyMovies

There are a number of 4.1 issues that have come up so, I'm going to try and get them resolved this weekend, and I'll probably get the MyMovies plugin back in there as well.

As for the running time/duration, bmt has always showed both, but in 4.1, I'm only showing running time, since duration is a non-editable field (although I did allow you to edit that in 4.0). Once it is set, I pass that Sage, and it appears, that for some media types, such as DVDs, it does not like it when you set the Running Time. (Although I haven't really confirmed that in Sage7)

[stanger89]

Sean, I haven't been paying attention, but did I remember you commenting about optomizing the DVD Profiler reader? Converting it to a SAX parser?

I ask because I'd love to run BMT, but frankly I'm affraid to run it on my server because the DOM DVD Profiler parser had memory problems with my rather large DVD Pro database.

Oh, and two questions/comments/bugs in 4.1:

1) Is it possible to change the order providers are used in the config UI?

2) It's seems not possible to select anything in the DVD Profiler XML selection Options Menu, enter doesn't work to select anything in there but cancel.

[stuckless]

Originally Posted by stanger89

Sean, I haven't been paying attention, but did I remember you commenting about optomizing the DVD Profiler reader? Converting it to a SAX parser?

I ask because I'd love to run BMT, but frankly I'm affraid to run it on my server because the DOM DVD Profiler parser had memory problems with my rather large DVD Pro database.

Oh, and two questions/comments/bugs in 4.1:

1) Is it possible to change the order providers are used in the config UI?

2) It's seems not possible to select anything in the DVD Profiler XML selection Options Menu, enter doesn't work to select anything in there but cancel.

There is a open bug about converting from a DOM to Sax parser for memory reasons, but that has not happened, yet. Maybe what i'll do, is bite the bullet and simply comvert the plugin to use jdom (provides a nice dom/sax hybrid) and use that as the basis for the mymovies as well, since it works in a very similar way.

Unfortunately, the Sage7 config UI does not have an option for moving items around. That isn't my config screen, I'm just leveraging the plugin configuration that Jeff has added. If I think about it, I'll submit an enhancement request to have them modify their OPTION LIST to include a mechanism to reorder the list. Currently you can only toggle options on or off.

I'll check that out as well, I didn't test selecting a file, since that is all handled by the core, but maybe I'm doing something in the plugin code that prevents the value from being set. (i'm wondering if i've set the type to be a "Dir" instead of a "File" )

[stanger89]

Also, any way to confirm if BMT is actually finding/using the xml file? I set 4.1 up on my test VM and it can't seem to find any movies in the XML even though I verified they're there.

i didn't see any errors in the log for openning the XML or anything, but I'm not sure I've got enough logging enabled.

-edit

I take that back, there are errors now, maybe they took a bit to show up:

MetadataException [query=SearchQuery; Type: MOVIE; QUERY:Black Lagoon: Volume 001;YEAR:2006;, ]
	at sagex.phoenix.metadata.MetadataManager.search(MetadataManager.java:350)
	at sagex.phoenix.metadata.MetadataManager.search(MetadataManager.java:289)
	at org.jdna.bmt.web.server.MetadataServicesImpl.searchForMetadata(MetadataServicesImpl.java:175)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:562)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:188)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:224)
	at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
	at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:230)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
	at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:879)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)

And another:

2010-06-26 13:51:09,570 [4641203@qtp-19436885-12] ERROR org.jdna.bmt.web.server.MetadataServicesImpl - Metadata Retreival Failed!
java.lang.NullPointerException
	at sagex.phoenix.metadata.provider.dvdprofiler.DVDProfilerParser.addCastMembers(DVDProfilerParser.java:95)
	at sagex.phoenix.metadata.provider.dvdprofiler.DVDProfilerParser.getMetaData(DVDProfilerParser.java:47)
	at sagex.phoenix.metadata.provider.dvdprofiler.DVDProfilerMetadataProvider.getMetaData(DVDProfilerMetadataProvider.java:124)
	at sagex.phoenix.metadata.MetadataManager.getMetdata(MetadataManager.java:382)
	at org.jdna.bmt.web.server.MetadataServicesImpl.getMetadata(MetadataServicesImpl.java:199)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:562)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:188)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:224)
	at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
	at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:230)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
	at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:879)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)

[stuckless]

In the log file, before the error, i should dump out what it is searching for, and the provider that it is using. ie, there is nothing in the exception that tells me that it failed using dvdprofiler

btw, the key, if you wanted to set the xmlFile manually is
phoenix/metadata/dvdprofiler/xmlFile

I just tested the File Chooser, and it's not working for me either. I need to do some more testing to see if this a bug in Phoenix or Sage.

EDIT: You should be able to set the xml file via the web ui as well.

[stanger89]

Crap, thought I copied that part, it was using dvdprofiler provider. And yeah, setting it via the web gui is how I got the XML specified. And how I verified it was checking providers in the right order.

[stuckless]

Actually, now that I look at your second excpetion, I can see that it is using DVDProfiler, but due to some sloppy coding there is a Null Pointer Exception Fortunately those as easy to fix.

[stanger89]

And looking more closely, it did find a lot of stuff, so between that and your comment, it sounds like there was just an entry in the XML it couldn't get past.

[Fuzzy]

Originally Posted by davidk21770

Especially, will it allow me to bring in my own shows from other sources and add them as sage native shows that will mix properly in the Sage Recordings view in SageTV?

I actually haven't seen this capability in 4.1 Is it hidden somewhere now? I'd really like the ability to have it automatically move items scraped as TV shows to the Recordings database, but I don't even see the option to do so manually anymore (I believe it was part of the SCAN section that has since been removed).

[stuckless]

Originally Posted by Fuzzy

I actually haven't seen this capability in 4.1 Is it hidden somewhere now? I'd really like the ability to have it automatically move items scraped as TV shows to the Recordings database, but I don't even see the option to do so manually anymore (I believe it was part of the SCAN section that has since been removed).

I'm unsure how to proceed with this particular piece of functionality. It's actually quite trivial in Sage7 to import a show as a recording, but, I'm not sure I want to be the person to do it. When it works, it's great, but when it fails (much like other stuff in bmt) people are not happy

Maybe I'll consider adding an advanced option (ie, normally hidden) to automatically import tv shows as recordings.

[Fuzzy]

Originally Posted by stuckless

I'm unsure how to proceed with this particular piece of functionality. It's actually quite trivial in Sage7 to import a show as a recording, but, I'm not sure I want to be the person to do it. When it works, it's great, but when it fails (much like other stuff in bmt) people are not happy

Maybe I'll consider adding an advanced option (ie, normally hidden) to automatically import tv shows as recordings.

How trivial? I've looked through the Sage7 API's, and nothing stood out that made it any easier. There's the MoveTVFileOutOfLibrary API call, but that won't work on imported media. Is there something else that has to be done to 'trick' sage into making the show a TV file? If so, any pointers? I know this is possible, as it works in both the web interface, and older BMT versions.

I realize your reluctance to doing this in BMT, and that's fine, I'll work something out, but it seems you've at least got some idea of the tasks that need to be performed.

[davidk21770]

Originally Posted by Fuzzy

I actually haven't seen this capability in 4.1 Is it hidden somewhere now? I'd really like the ability to have it automatically move items scraped as TV shows to the Recordings database, but I don't even see the option to do so manually anymore (I believe it was part of the SCAN section that has since been removed).

In the WebUI, call it a sage recording, call it a tv show, give it the correct name and screen name (same as in the sage recordings -- case is significant), fill in the rest of the info that you want filled-in, save tell it to overwrite the metadata and it'll integrate. If it shows up in the wrong place, go back to the web UI and fix your typo.

[damian.flynn]

I have been using metabrowser to tag my tv and movies for the last few years, and they are now launching a new version, which currently has no TV and no support for the sage .properties files or the new API to embed the tags.

So - its time to dump this tool and move over to the new generation. I have all my media on a large multi TB disk array (single Drive/UNC).

The format i have been using is along the lines:

For TV
\Video\TV\Series Name\Season 0X\Series Name - S0X-E0X - Title.MKV

For Movies
\Video\Movies\Movie Name (Year)\Random File Name.MKV

All the new media is dropped in
\Drop box

Here the file name could be the name of the Movie, or the TV Series with Season/Episode information, and the utility then Moved the file to the correct destination. I have .properties files, episode image, banners and fanart currently, but dont mind deleting it all.

So - my question - as a DUMMY to BMT, is there a start guide.
I have not even installed the tool yet, so high level is fine, and Ill do a full document and post it for everyone else that decides to adopt this engine.

cheers
Damian