Utility: Batch Metadata Importer 3.x

[BigDragon]

It is not from command line.

I change this line in sage.properties:
epg/force_ask_to_copy_lineup=false
to
epg/force_ask_to_copy_lineup=true

but nothing happens.

What do you mean by "automatic plugin"?

[BigDragon]

Update:
When I run the tool form cmd using
"java -jar MetadataTool.jar -recurse d:\media\Movies" command,
the BMT run!

So, why can't I run this tool from sageMC?

[stuckless]

did you import the xml plugin? does on-demand searching work? ie, if you select a media item and then use the metadata/fanart option to lookup metadata.

BTW, i have no idea what epg/force_ask_to_copy_lineup=true is used for... but bmt doesn't use it.

[BigDragon]

Yes, I import the xml plugin. (I see the fanart options)

I had a "copy" mistake. I meant this line:
force_full_content_reindex=false to force_full_content_reindex=true.

When I use "java -jar MetadataTool.jar --demand d:\media\Movies"
I get this lines:

MetaData Stats...
Auto Updated: 0; Auto Skipped: 0; Manual Updated:0; Manual Skipped: 0;

???

[ghostlobster]

OK...I'm jumping in to do the install today. However, I've hit snag #1... The instructions say "Install the latest DVDPro2Sage found here: "
I don't use DVD Profiler, nor do I plan to. Is DVD Profiler a requirement?

[tmiranda]

No, DVD profiler is not a requirement. Did you finally sober up?

[BigDragon]

Originally Posted by BigDragon

Yes, I import the xml plugin. (I see the fanart options)

I had a "copy" mistake. I meant this line:
force_full_content_reindex=false to force_full_content_reindex=true.

When I use "java -jar MetadataTool.jar --demand d:\media\Movies"
I get this lines:

MetaData Stats...
Auto Updated: 0; Auto Skipped: 0; Manual Updated:0; Manual Skipped: 0;

???

Any idea why it is not working?

[ghostlobster]

Ok, I installed 6.5.15 on my WHS box, installed the latest MC and the .41 update, installed the phoenix APIs 30, then put BMT in place. went through the instructions, edited my sageclient.properties file, set it to force the reindexing, and started the client (all this was done on the server. if I can get the server displaying fanart, then I'll move to one of my clients.) After reinitializing a media scan within the UI, i wated a while, and no fanart was showing up in my fanart repository (\\server\sagetv\fanart) which has completely open permissions for any user.
So, as a test, from a comman line I ran java -jar MetadataTool.jar x:\ where x:\ is my recorded TV drive. I get the following error on all files attempted:

... 22 more
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeExcepti
on: Timed out waiting for a Sage Server Reply.
at sagex.SageAPI.getRemoteProvider(SageAPI.java:54)
at sagex.SageAPI.getProvider(SageAPI.java:23)
at sagex.SageAPI.call(SageAPI.java:71)
at sagex.api.MediaFileAPI.IsDVD(MediaFileAPI.java:155)
at sagex.phoenix.fanart.SageFanartUtil.IsDVDFile(SageFanartUtil.java:27)

at sagex.phoenix.fanart.FanartUtil.getLocalFanartForFile(FanartUtil.java
:192)
at org.jdna.media.metadata.impl.sage.FanartStorage.saveLocalFanart(Fanar
tStorage.java:88)
at org.jdna.media.metadata.impl.sage.FanartStorage.saveLocalFanartForTyp
es(FanartStorage.java:75)
at org.jdna.media.metadata.impl.sage.FanartStorage.saveFanart(FanartStor
age.java:57)
at org.jdna.media.metadata.impl.sage.FanartStorage.downloadFanart(Fanart
Storage.java:289)
at org.jdna.media.metadata.impl.sage.CentralFanartPersistence.storeMetaD
ata(CentralFanartPersistence.java:37)
at org.jdna.media.metadata.impl.sage.SageTVPropertiesWithCentralFanartPe
rsistence.storeMetaData(SageTVPropertiesWithCentralFanartPersistence.java:39)
at org.jdna.media.util.AutomaticUpdateMetadataVisitor.fetchMetaData(Auto
maticUpdateMetadataVisitor.java:111)
at org.jdna.media.util.AutomaticUpdateMetadataVisitor.fetchMetaData(Auto
maticUpdateMetadataVisitor.java:62)
at org.jdna.media.util.AutomaticUpdateMetadataVisitor.visit(AutomaticUpd
ateMetadataVisitor.java:47)
at org.jdna.media.util.MissingMetadataVisitor.visit(MissingMetadataVisit
or.java:47)
at org.jdna.media.impl.AbstractMediaResource.accept(AbstractMediaResourc
e.java:121)
at org.jdna.media.impl.MediaFolder.accept(MediaFolder.java:174)
at org.jdna.metadataupdater.MetadataUpdater.process(MetadataUpdater.java
:327)
at org.jdna.metadataupdater.MetadataUpdater.main(MetadataUpdater.java:10
8)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Timed out wai
ting for a Sage Server Reply.
at sagex.remote.server.SimpleDatagramClient$2.onFailure(SimpleDatagramCl
ient.java:78)
at sagex.remote.server.SimpleDatagramClient.send(SimpleDatagramClient.ja
va:54)
at sagex.remote.server.SimpleDatagramClient.findRemoteServer(SimpleDatag
ramClient.java:67)
at sagex.SageAPI.getRemoteProvider(SageAPI.java:38)
... 19 more
Caused by: java.lang.RuntimeException: Timed out waiting for a Sage Server Reply
.
... 22 more
Updated: Fringe; file:/x:/Fringe-TheresMoreThanOneofEverything-2813337-0.mpg
Skipping: file:/x:/Fringe-TheTransformation-1682962-0.mpg
SageAPI Provider is not set, will try to find the server...

I'm thinking the key item here is "SageAPI Provider is not set, will try to find the server..."
The Sage service is running just fine locally and I can launch the local client without a hitch. Ideas?

Another update here...did a little searching and decided to try putting the Sage ermote APIs in place, so I extrated those, had it overwrite existing files, and per an earlier post here, i tried explicitly telling the command what to attach to and got the following:
C:\Program Files\SageTV\SageTV>java -Dsagex.SageAPI.remoteURL=rmi//192.168.2.161
:1098 -jar MetadataTool.jar --report=quick
Batch MetaData Tools (2.3.1)
Requesting a Fanart report: quick on the SageTV Server.
Embedded SageAPI is not functional. We are most likely running remotely.
SageAPI Provider is not set, will try to find the server...
java.lang.RuntimeException: java.lang.RuntimeException: Timed out waiting for a

followed by more java errors.
Also, netstat reports nothing listening on port 1098

Ideas?

[stuckless]

Originally Posted by BigDragon

Any idea why it is not working?

--demand is not a command line option, so it is ignored. the result is that without --recurse, the command will simply look for videos in the dir you specified, and not any subdirs.

As for why the automatic plugin isn't working, I can't really say, but I'll just re-iterate a few things...
1. The plugin must be installed on the server
2. verify that the server properties contains the bmt metadata plugin in sage's mediafile metadata plugin (see manual install instructions)
3. Check the sagetv_0.txt... If the plugin is being called by sage then you'll see log statements show up in there indicating that BMT is handling a file.

[stuckless]

@ghostlobster - The automatic plugin works with sage's metadata plugin, and I think that it has to be installed on the sage server. ie, installing it on a sage client and then setting the reindex property will do nothing (as far as i know)

As for the command line, that's interesting... bmt should not have an explicit dependency on a remote server, so this is really a bug. I've uploaded a new version of the sagex remote apis, so try downloading extracting that build and try the command line again.

As for the installing of the remote apis as a server (which is not required for normal bmt usage).... you'll need to either follow the instructions on setting them up using nielm's server, or if you run the jetty server plugin, then a simple restart of the sage server (after you extract the sagex bundle) will start the apis. Also, if you are running a single sage server, then you do not need to specify the server, since the sagex apis can automaticly discover your server.

[BigDragon]

The BMT is indeed being loaded in sage but there is an BMT error in the log.
I attach part of the log:

PHP Code:

ב 5/25 14:48:01.196 BMT: ========= BEGIN BATCH METADATA TOOLS ENVIRONMENT ==============
ב 5/25 14:48:01.197 BMT:     BMT Version:  2.3.1
ב 5/25 14:48:01.198 Loading Phoenix APIs: 1.24
ב 5/25 14:48:01.846 Sage Provider Implementation: sagex.remote.EmbeddedSageAPIProvider; sage://embedded
ב 5/25 14:48:01.979 BMT: Phoenix Version:  1.24
ב 5/25 14:48:01.980 BMT:   Sagex Version:  6.5.9-8
ב 5/25 14:48:01.980 BMT:    Java Version:  1.6.0_10
ב 5/25 14:48:01.980 BMT:  Java Classpath:  C:\Program Files\SageTV\SageTV\Sage.jar;C:\Program Files\SageTV\SageTV\xerces.jar;C:\Program Files\SageTV\SageTV\plugin.jar;C:\Program Files\SageTV\SageTV\;;JARs\activation.jar;JARs\commons-codec-1.3.jar;JARs\commons-io-1.4.jar;JARs\commons-jxpath-1.1.jar;JARs\commons-lang-2.4.jar;JARs\commons-logging-api.jar;JARs\commons-logging.jar;JARs\commons-vfs-20070823.jar;JARs\df_sageutils.jar;JARs\htmlparser.jar;JARs\imageutils.jar;JARs\jcifs-1.1.6.jar;JARs\jdic.jar;JARs\log4j-1.2.15.jar;JARs\log4j.jar;JARs\lucene-core-2.3.2.jar;JARs\mail.jar;JARs\mailcheck.jar;JARs\metadata-updater.jar;JARs\miglayout-3.6.3-swing.jar;JARs\mx4j-impl.jar;JARs\mx4j-jmx.jar;JARs\mx4j-remote.jar;JARs\mx4j-tools.jar;JARs\nielm_getrssnews.jar;JARs\nielm_sageimdb.jar;JARs\nielm_sagemenureader.jar;JARs\nielm_sageutls.jar;JARs\phoenix.jar;JARs\sagemc.jar;JARs\sagex.api.jar;JARs\sbbi-jmx-1.0.jar;JARs\sbbi-upnplib-1.0.3.jar
ב 5/25 14:48:01.981 BMT: ========= END BATCH METADATA TOOLS ENVIRONMENT ==============
ב 5/25 14:48:02.097 BMT: ** Batch Metadata Plugin; Using ProviderId: dvdprofiler,tvdb,themoviedb.org,imdb.xml,imdb
ב 5/25 14:48:02.097 BMT: ** Configuration for Metadata Plugin: C:\Program Files\SageTV\SageTV\metadata.properties
ב 5/25 14:48:02.204 BMT: BMT: ERROR: Failed while performing Automatic Metadata/Fanart lookup
ב 5/25 14:48:02.205 BMT: =============== BEGIN BMT EXCEPTION ================
ב 5/25 14:48:02.206 BMT: =============== END BMT EXCEPTION ================
ב 5/25 14:48:02.206 Doing pre-emptive file format detection on recording MediaFile....D:\Media\Recordings\SageTVDVBSEnhancer1on1270016975DigitalTVTuner-0.mpgbuf
ב 5/25 14:48:02.206 Using internal format detector first...
ב 5/25 14:48:02.207 Now using external format detector...
ב 5/25 14:48:02.468 FORMATERROR UNABLE TO PARSE FILE TYPE FOR FILE-1: D:\Media\Recordings\SageTVDVBSEnhancer1on1270016975DigitalTVTuner-0.mpgbuf
ב 5/25 14:48:02.469 BMT: BMT: ERROR: Failed while performing Automatic Metadata/Fanart lookup
ב 5/25 14:48:02.469 BMT: =============== BEGIN BMT EXCEPTION ================
ב 5/25 14:48:02.470 BMT: =============== END BMT EXCEPTION ================ 


[ghostlobster]

Originally Posted by stuckless

@ghostlobster - The automatic plugin works with sage's metadata plugin, and I think that it has to be installed on the sage server. ie, installing it on a sage client and then setting the reindex property will do nothing (as far as i know) .

All of this is being done on the server. As I said I did not bring any clients into play at all. I was referring to the sageclient.properties file on the server.

Originally Posted by stuckless

As for the command line, that's interesting... bmt should not have an explicit dependency on a remote server, so this is really a bug. I've uploaded a new version of the sagex remote apis, so try downloading extracting that build and try the command line again.

I'll give it a whirl

Originally Posted by stuckless

As for the installing of the remote apis as a server (which is not required for normal bmt usage).... you'll need to either follow the instructions on setting them up using nielm's server, or if you run the jetty server plugin, then a simple restart of the sage server (after you extract the sagex bundle) will start the apis. Also, if you are running a single sage server, then you do not need to specify the server, since the sagex apis can automaticly discover your server.

I'm not certain I have a clue what you're talking about, but I'll see if I can make heads or tails of it once reading through stuff.

[ghostlobster]

OK, I added the line to the servlets.properties file on Nielm's webserver, recycled everything and it kind of worked. The report still barfs, but I can run the command line tool against my library. However, unless I explicitly state the path to my library and fanart in the command, it drops all the images in the same directory as the media. I thinnk I'll have to hack at some .properties file, but I'm not positive of the syntax. I'll mess with it and see what I can come up with.
The I don't think that the automated plugin is working though. Again, I went into the client (on the server box) and told it to refresh imported media, and nothing was added. Any pointers as to what I could have bungled in the install/config?

[sflamm]

I believe Sean is looking into the UI plugin as we speak. Here are the issues I've pointed out to him (please add to the list if I am missing anything):

1. When the plug-in was enabled in the UI, it did not update the fanart/metadata

Fan Art “ enabled”
Automatic Metadata/Fanart “enabled”

2. When "Automatic Metatadata/Fanart" was set to disabled (and "Fan Art" was left enabled), then new source movies etc. would not be seen at all when import directories were manually 'rescanned' from within SageTV. Disabling the plug-in and manually rescanning imported the movies correctly.

3. Manually added providers (i.e. dvdprof2) were not showing in the UI list (the new dvdprofiler-2 provides a workaround for that specific combination).

In a nutshell, the only way I could use BMT to get good results was by using the command line (no problems with the command line interface)

There were also a number of plug-in UI bugs which I believe were going to be addressed with a new release of SageMC (buttons would not appear etc)

[redman]

Originally Posted by BigDragon

The BMT is indeed being loaded in sage but there is an BMT error in the log.
I attach part of the log:

PHP Code:

ב 5/25 14:48:01.196 BMT: ========= BEGIN BATCH METADATA TOOLS ENVIRONMENT ==============
ב 5/25 14:48:01.197 BMT:     BMT Version:  2.3.1
ב 5/25 14:48:01.198 Loading Phoenix APIs: 1.24
ב 5/25 14:48:01.846 Sage Provider Implementation: sagex.remote.EmbeddedSageAPIProvider; sage://embedded
ב 5/25 14:48:01.979 BMT: Phoenix Version:  1.24
ב 5/25 14:48:01.980 BMT:   Sagex Version:  6.5.9-8
ב 5/25 14:48:01.980 BMT:    Java Version:  1.6.0_10
ב 5/25 14:48:01.980 BMT:  Java Classpath:  C:\Program Files\SageTV\SageTV\Sage.jar;C:\Program Files\SageTV\SageTV\xerces.jar;C:\Program Files\SageTV\SageTV\plugin.jar;C:\Program Files\SageTV\SageTV\;;JARs\activation.jar;JARs\commons-codec-1.3.jar;JARs\commons-io-1.4.jar;JARs\commons-jxpath-1.1.jar;JARs\commons-lang-2.4.jar;JARs\commons-logging-api.jar;JARs\commons-logging.jar;JARs\commons-vfs-20070823.jar;JARs\df_sageutils.jar;JARs\htmlparser.jar;JARs\imageutils.jar;JARs\jcifs-1.1.6.jar;JARs\jdic.jar;JARs\log4j-1.2.15.jar;JARs\log4j.jar;JARs\lucene-core-2.3.2.jar;JARs\mail.jar;JARs\mailcheck.jar;JARs\metadata-updater.jar;JARs\miglayout-3.6.3-swing.jar;JARs\mx4j-impl.jar;JARs\mx4j-jmx.jar;JARs\mx4j-remote.jar;JARs\mx4j-tools.jar;JARs\nielm_getrssnews.jar;JARs\nielm_sageimdb.jar;JARs\nielm_sagemenureader.jar;JARs\nielm_sageutls.jar;JARs\phoenix.jar;JARs\sagemc.jar;JARs\sagex.api.jar;JARs\sbbi-jmx-1.0.jar;JARs\sbbi-upnplib-1.0.3.jar
ב 5/25 14:48:01.981 BMT: ========= END BATCH METADATA TOOLS ENVIRONMENT ==============
ב 5/25 14:48:02.097 BMT: ** Batch Metadata Plugin; Using ProviderId: dvdprofiler,tvdb,themoviedb.org,imdb.xml,imdb
ב 5/25 14:48:02.097 BMT: ** Configuration for Metadata Plugin: C:\Program Files\SageTV\SageTV\metadata.properties
ב 5/25 14:48:02.204 BMT: BMT: ERROR: Failed while performing Automatic Metadata/Fanart lookup
ב 5/25 14:48:02.205 BMT: =============== BEGIN BMT EXCEPTION ================
ב 5/25 14:48:02.206 BMT: =============== END BMT EXCEPTION ================
ב 5/25 14:48:02.206 Doing pre-emptive file format detection on recording MediaFile....D:\Media\Recordings\SageTVDVBSEnhancer1on1270016975DigitalTVTuner-0.mpgbuf
ב 5/25 14:48:02.206 Using internal format detector first...
ב 5/25 14:48:02.207 Now using external format detector...
ב 5/25 14:48:02.468 FORMATERROR UNABLE TO PARSE FILE TYPE FOR FILE-1: D:\Media\Recordings\SageTVDVBSEnhancer1on1270016975DigitalTVTuner-0.mpgbuf
ב 5/25 14:48:02.469 BMT: BMT: ERROR: Failed while performing Automatic Metadata/Fanart lookup
ב 5/25 14:48:02.469 BMT: =============== BEGIN BMT EXCEPTION ================
ב 5/25 14:48:02.470 BMT: =============== END BMT EXCEPTION ================ 


Hi,
I have got fanart and metadata successfully downloading from the command line for recorded TV, and imported videos.

I have the same error as above. I have updated to the new sagex api as suggested, with no change.

I noticed that if i manually try to update in SageMC, the sage log shows the following:

PHP Code:

Tue 5/26 2:12:12.203 EXCEPTION in getValueAsObject:sage.a.e: Error in method reflection of phoenix_api_GetMetadataSearchResults of java.lang.reflect.InvocationTargetException for:phoenix_api_GetMetadataSearchResults(Airing) 


In a seperate matter:
Some shows appear in the epg as C.S.I. Miami which does the lookup for C S I Miami and finds no match (due to spaces).
Any help on regex or settings for this?

I also noticed that if i have the titles in the Metadata-Title.properties file it created the correct metadata property files for each show, but these do not appear in the SageMC gui.

Thanks for all your hard work with this.

Cheers
Chris

[rmccoy]

Hey guys, so I've just gotten started messing around with Sage TV, SageMC and the like. I've got things acting how I like for the most part, with one small exception.

I really like the idea of having all of my TV shows showing up in the "Recorded TV" list, and was able to accomplish this using MediaScraper. I have it set up with DirMon2 to automate this process (whenever a tv show file is added to one of my media directories, it gets added to the "Recorded TV" list.)

I want to use BMI to get all the metadata and fanart, the problem is that when MediaScraper runs, it leaves has to get metadata, and therefore creates .properties files. When BMI sees these .properties files, it skips over these files, even though I want it to get the fanart and metadata for them. Any suggestions on how to make this work? Or a more elegant way of integrating this?

I would use MediaScraper for everything, but I really like the way that BMI handles the fanart, and it does a MUCH better job with movies than MediaScraper.

Thanks in advance!

[stuckless]

@BigDragon, @redman - That Exception would have made much more sense to me had I ACTUALLY printed out the exception Instead I have nothing between the BEGIN and END exception statements.

@BigDragon - Nothing I can do about the C S I title for now. I will check to see why metadata-titles is not being used from within sagemc.

@rmccoy - I'm not sure what to do about your issue. Maybe I can look at adding another option to tell the plugin to ALWAYS fetch fanart, even if the properties exist. BMT does have experimental TV import support, but it is turned off by default.


Just a FYI on BMT... I know there are several outstanding issues and I would love to get a build out to fix them... but, unfortunately, I rewrote a large portion of the code to utilize the phoenix configuration/metadata system. In doing so, much of the code has changed. I didn't branch the original code, so I'm sort of committed to going down this path. The good news, is that I'm ready to release a new build of BMT, and I'll try to get some of these fixes in there, but the side effects of changing so much code is that there will be other bugs. I've started to add more unit tests to the code and that's been helping a lot. But the next build will be more like a 3.0 build rather than a minor 2.4 build. I may try to get a couple of beta build out there within the next couple of weeks with a full release to follow.

The move to the phoenix configuration/metadata system is being done because i want to offer a web based (jetty plugin) UI for configuration. BMT has so many configuration options that most people are getting lost in the configuration. I'm hoping that I can offer a web UI to configure bmt and make it easier to troubleshoot, etc. Extending the web UI, i'd like offer some services via the web ui, such as full collection scanning, partial collection scanning, and single file metadata editing.

[razrsharpe]

Originally Posted by rmccoy

I really like the idea of having all of my TV shows showing up in the "Recorded TV" list, and was able to accomplish this using MediaScraper. I have it set up with DirMon2 to automate this process (whenever a tv show file is added to one of my media directories, it gets added to the "Recorded TV" list.)

I want to use BMI to get all the metadata and fanart, the problem is that when MediaScraper runs, it leaves has to get metadata, and therefore creates .properties files. When BMI sees these .properties files, it skips over these files, even though I want it to get the fanart and metadata for them. Any suggestions on how to make this work? Or a more elegant way of integrating this?

you could run BMT from the command line... set up dirmon2 to run against your imported media the same you have it run mediascaper against it.

or the latest version of BMT supports adding tv to the sage database... yes its expermintal... yes its a pain to fix if the metadata is wrong... but it does work

From the command line you could use the option --overwriteMetadata=true to ignore the .properties file and create a new one. I haven't personally tested this but i believe that is how it is supposed to work

I would use MediaScraper for everything, but I really like the way that BMI handles the fanart, and it does a MUCH better job with movies than MediaScraper.

using the command line to execute either BMT or mediascraper you should be able to use mediascraper for tv metadata and BMT for movie metadata and fanart via three separate .bat files with specific options for what you want to accomplish

[BigDragon]

stuckless,
Is there anything more to do?
or just should I wait to the next version?

[stuckless]

Originally Posted by BigDragon

stuckless,
Is there anything more to do?
or just should I wait to the next version?

Unfortunately, I can't reproduce the issue... I use the plugin and I've never encountered the error. That being the said the only other option for me is to provide a build with more/better logging so that I can see what is happening in your case. But, as i noted above, that can't happen until the next major release.

So yeah, for now, i don't think there is much more you can do.