Utility: Batch Metadata Importer 3.x

[mkanet]

Hey Sean, I upgraded from 2.3 to 3.0 beta 3 and reimported the stvi again from scratch. I wish I had better news, but the automatic feature is still not working. This is on a pretty stock Windows Vista32/SageTV/SageMC setup using the latest full release of Java.

If you think I can help provide any useful information for you using 3.0 debugging/logging, just let me know what you would like for me to do. I'm curious to know how it worked out for you under Windows.

Originally Posted by stuckless

BTW... I just confirmed, that under linux, when a new show is recorded, it does call the metadata plugin. I'm attempting to verify under windows as well.

* moved *

[MitchSchaft]

beta3 works perfect for me!

* moved *

[stuckless]

Originally Posted by mkanet

Hey Sean, I upgraded from 2.3 to 3.0 beta 3 and reimported the stvi again from scratch. I wish I had better news, but the automatic feature is still not working. This is on a pretty stock Windows Vista32/SageTV/SageMC setup using the latest full release of Java.

If you think I can help provide any useful information for you using 3.0 debugging/logging, just let me know what you would like for me to do. I'm curious to know how it worked out for you under Windows.

mkanet - I posted my reply here, since the other thread is a notification thread.

I just spent 2 hrs in windows and i'll probably be grumpier than normal as a result. But, I found no issues with the automatic plugin, once it was configured.

sflamm - I used your image, and it was not configured to start. So I had to edit the metadata provider list to use the dvdprofiler-2 so that it could get metadata from dvdprofiler and fanart from themoviedb. Your image also never had mediafile_metadata_parser_plugins=org.jdna.sage.MetadataUpdaterPlugin set in the Sage.properties. If that isn't set, then then Sage will never call bmt for metadata.

Once I had everything configured, I removed the videos from the library, and then added them back in, told sage to refresh the library, and the automatic plugin kicked in for each item.

I used both the 2.x and 3.x plugins, and both worked, although I found and issue with the 3.x plugin, in that my context.xml is still invalid. For some reason the jetty plugin under linux doesn't seem to care, but on windows, all I got when i went to the bmt url was directory listing. I had to update the context.xml to set the extract webapp value to true. And then I had to delete the bmt web dir from the jetty webapps directory, and restart sage.

Bottom line... no issue with windows as it relates to bmt and sagetv, and I'm thankful that I'll not have to venture back into windows for some time

So, if the automatic plugin is not working, then I would recheck your Sage.properties for the property that I listed above, and make sure that your provider list is set correctly (which is will be, if you are just using the defaults)

[stuckless]

Originally Posted by toricred

I know this is an old issue, but I found more information on it thanks to version 3.0.

On the shows where command-line and/or the automatic plugin weren't updating the season and episode numbers it turns out the Episode Title is missing. When it is there or I manually do a metadata search in Sage everything works right.

That would make sense (to me)... BMT relies on the episode title to do a match. If the episode title is missing from the filename, then the filename matcher will not match it as tv file, but if it does happen to match it has a tv title, it will still fail because it will need an episode title.

[babgvant]

Originally Posted by stuckless

Bottom line... no issue with windows as it relates to bmt and sagetv, and I'm thankful that I'll not have to venture back into windows for some time

Always with the windows hating

BTW, 3.0 is working great. Thanks for the crazy fast turn around.

[stuckless]

Originally Posted by babgvant

Always with the windows hating

It's not so much with the windows hating... It's just not what I'm used to, so I find it frustrating to do the simplest tasks... probably like most people when they are forced to use linux/mac... and for the record... i did have osx installed for a weekend, and i was equally frustrated by it for same reasons

BTW, 3.0 is working great. Thanks for the crazy fast turn around.

It helps that you provided me with good information

[mkanet]

Sean, I'm not sure if I should post while youre grumpy , but I think I might have some good news.. at least, what I think might be a huge clue:

It looks like 3.0 IS downloading fanart on my setup now; but putting it in again in the TV Recordings folder (just like it did one time before). The odd thing is the fanart jpg files get mixed in with the TV recording mpgs and also the BMT properties files (without creating the TV/movie folders). The last time this happened, I dismissed it because I couldnt get it to do that again.

Now, here's the really odd thing, and; the clue... if I right click a TV show and select metadata/fanart in either 'My Videos' or 'Recorded TV' the fan art will be saved to the central fanart folder I specified under SageMC/Extras/BMT Options. I'm not sure if it's as simple as assuming it's a BMT configuration issue; as, there's only one way (that I know) of to set the central fan art directory path. Edit: I remember you mentioned about the possibility for needing to edit the metadata.properties... but, then, if I understood you correctly, you said that if its saving to the fanart folder with the on demand manual method, its working. If you think I need to still do this, could you please let me know what to put in the metadata.properties file if my central fanart folder path is: E:\Fanart?

I know you said the automatic and On Demand are two completely different ways of doing things (or something like that). Sorry, if this sounds completely dumb, but.. can you get the automatic method to use the same method as On Demand when saving the fanart in the correct paths?

PS: One of the first things I ever did was make sure I had the below in the properties file (just like the installation instructions say to do).
mediafile_metadata_parser_plugins=org.jdna.sage.MetadataUpdaterPlugin

-Thanks,
Michael

Originally Posted by stuckless

mkanet - I moved your post here, since the other thread is a notification thread.

I just spent 2 hrs in windows and i'll probably be grumpier than normal as a result. But, I found no issues with the automatic plugin, once it was configured.

sflamm - I used your image, and it was not configured to start. So I had to edit the metadata provider list to use the dvdprofiler-2 so that it could get metadata from dvdprofiler and fanart from themoviedb. Your image also never had mediafile_metadata_parser_plugins=org.jdna.sage.MetadataUpdaterPlugin set in the Sage.properties. If that isn't set, then then Sage will never call bmt for metadata.

Once I had everything configured, I removed the videos from the library, and then added them back in, told sage to refresh the library, and the automatic plugin kicked in for each item.

I used both the 2.x and 3.x plugins, and both worked, although I found and issue with the 3.x plugin, in that my context.xml is still invalid. For some reason the jetty plugin under linux doesn't seem to care, but on windows, all I got when i went to the bmt url was directory listing. I had to update the context.xml to set the extract webapp value to true. And then I had to delete the bmt web dir from the jetty webapps directory, and restart sage.

Bottom line... no issue with windows as it relates to bmt and sagetv, and I'm thankful that I'll not have to venture back into windows for some time

So, if the automatic plugin is not working, then I would recheck your Sage.properties for the property that I listed above, and make sure that your provider list is set correctly (which is will be, if you are just using the defaults)

[stuckless]

3.0 uses a different configuration setup than the 2.x, which is why I also include the web ui for configuring it as well. If you have 3.0 plugin installed, then use the web ui, and click on Configure / Metadata Tools / Application Configuration and set your fanart folder there. Based on your description, it sounds as if the fanart folder is not set.

As for the plugin/on-deman, they are different entry points in to the same system. They both use the same configuration, etc, but "how" they invoked and the data they are passed is what makes them different. Once a search happens in either, they are sharing the same code, etc.

You may be seeing a difference in the results, if, you are running a client connected to the server, and you've set the fanart folder on the client ui, but it's not set on the server. In that case, the on-demand will work fine since the client will have it's fanart folder set, but the automatic plugin, which is invoked on the server, will not have its fanart folder set, and will download fanart to the local directory.

The Sage.properties property for the central fanart folder is phoenix/mediametadata/fanartCentralFolder

EDIT:
The 3.x release no longer uses the metadata.properties file, it stores it configuration in the Sage.properties file.

Originally Posted by mkanet

Sean, I'm not sure if I should post while youre grumpy , but I think I might have some good news.. at least, what I think might be a huge clue:

It looks like 3.0 IS downloading fanart on my setup now; but putting it in again in the TV Recordings folder (just like it did one time before). The odd thing is the fanart jpg files get mixed in with the TV recording mpgs and also the BMT properties files (without creating the TV/movie folders). The last time this happened, I dismissed it because I couldnt get it to do that again.

Now, here's the really odd thing, and; the clue... if I right click a TV show and select metadata/fanart in either 'My Videos' or 'Recorded TV' the fan art will be saved to the central fanart folder I specified under SageMC/Extras/BMT Options. I'm not sure if it's as simple as assuming it's a BMT configuration issue; as, there's only one way (that I know) of to set the central fan art directory path. Edit: I remember you mentioned about the possibility for needing to edit the metadata.properties... but, then, if I understood you correctly, you said that if its saving to the fanart folder with the on demand manual method, its working. If you think I need to still do this, could you please let me know what to put in the metadata.properties file if my central fanart folder path is: E:\Fanart?

I know you said the automatic and On Demand are two completely different ways of doing things (or something like that). Sorry, if this sounds completely dumb, but.. can you get the automatic method to use the same method as On Demand when saving the fanart in the correct paths?

PS: One of the first things I ever did was make sure I had the below in the properties file (just like the installation instructions say to do).
mediafile_metadata_parser_plugins=org.jdna.sage.MetadataUpdaterPlugin

-Thanks,
Michael

[mkanet]

I wasn't able to find any documentation for 3.0 configuration, if there is any; so, forgive me, if these questions can be answered in a document. But, I have a couple of quick questions.. is the "Web UI" the only way to configure BMT without having to manually edit the properties file? I dont have phoenix/mediametadata/fanartCentralFolder set in my properties file; yet, the on demand fanart download feature still seems to know where to save the files (from sageMC/extra/BMT options?). It looks like the Web UI requires a webserver for sageTV. I already use the Web User Interface for SageTV for sageTV scheduling, maintenance, etc. Could I use that to access the config screen for BMT? Is so, what URL would I use?

PS: I am currenly using the same machine for client/server.

I'll add the phoenix/mediametadata/fanartCentralFolder and hopefully thats all thats left to do!

Thanks! Crossing my fingers..

Originally Posted by stuckless

3.0 uses a different configuration setup than the 2.x, which is why I also include the web ui for configuring it as well. If you have 3.0 plugin installed, then use the web ui, and click on Configure / Metadata Tools / Application Configuration and set your fanart folder there. Based on your description, it sounds as if the fanart folder is not set.

As for the plugin/on-deman, they are different entry points in to the same system. They both use the same configuration, etc, but "how" they invoked and the data they are passed is what makes them different. Once a search happens in either, they are sharing the same code, etc.

You may be seeing a difference in the results, if, you are running a client connected to the server, and you've set the fanart folder on the client ui, but it's not set on the server. In that case, the on-demand will work fine since the client will have it's fanart folder set, but the automatic plugin, which is invoked on the server, will not have its fanart folder set, and will download fanart to the local directory.

The Sage.properties property for the central fanart folder is phoenix/mediametadata/fanartCentralFolder

EDIT:
The 3.x release no longer uses the metadata.properties file, it stores it configuration in the Sage.properties file.

[babgvant]

Originally Posted by mkanet

I wasn't able to find any documentation for 3.0 configuration, if there is any; so, forgive me, if these questions can be answered in a document.

I don't think there's separate document yet but Sean documented the requirements in the notification thread.

Relevant bits below

Originally Posted by stuckless

The 3.0 release now has a web based configuration application. You will need to install the Jetty plugin, if you don't have it installed already. Once jetty is installed, you should be able to access the BMT configuration by using the following url

http://YOUR_JETTY_SERVER:YOUR_PORT/bmt

The jetty plugin is pretty straight forward to install with the added bonus that you can install the mobile interface.

[mkanet]

Thanks babgvant. Something threw me off maybe you could answer or Sean. In the notification thread it says, "all the configuration keys have changed. They are pretty much the same as before, except they now are prefixed with "bmt/"". However, Sean just said to look for "phoenix/mediametadata/fanartCentralFolder", which doesnt have a prefix "bmt". Would it be, "bmt/phoenix/mediametadata/fanartCentralFolder"? Also, I'm still not sure if the sageMC/extras/BMT Options update the correct settings in the properties file or not. I defined my centeral fanart folder there; and it seems the on demand fanart download does recognize it; but not the automatic; also, I dont have phoenix/mediametadata/fanartCentralFolder in my properties file at all.

Sean: There are a lot of people already using Neilm's webserver (using it for remote scheduling and many other sageTV maintainence tasks and streaming. It would be great if that could be used instead of jetty. Is there a way to do it? If, so how? Actually I also have IIS running as my primary webserver separate from sageTV on the same machine.

Originally Posted by babgvant

I don't think there's separate document yet but Sean documented the requirements in the notification thread.

Relevant bits below



The jetty plugin is pretty straight forward to install with the added bonus that you can install the mobile interface.

[toricred]

Originally Posted by stuckless

That would make sense (to me)... BMT relies on the episode title to do a match. If the episode title is missing from the filename, then the filename matcher will not match it as tv file, but if it does happen to match it has a tv title, it will still fail because it will need an episode title.

OK, any idea why it works when it is requested from within Sage? I'll start looking at the problem files for matching the episode title that should be in the filename. Thanks for the clue.

[Peter_h]

Originally Posted by mkanet

Sean: There are a lot of people already using Neilm's webserver (using it for remote scheduling and many other sageTV maintainence tasks and streaming. It would be great if that could be used instead of jetty. Is there a way to do it? If, so how? Actually I also have IIS running as my primary webserver separate from sageTV on the same machine.

I'll give this a try. There are probably a few reasons that Sean decided to go with Jetty but if I were in his shoes, this would be a list of my reasons.

1.) Jetty is very flexible and easy to work with from a developer standpoint
2.) The Web server is not his code. So, he either has to rely on another person to add in the functionality he needs or he starts modding it and releasing his own versions which opens a whole other can of worms. IE, he starts maintaining that as well.
3.) Doing it this way creates a dedicated web server specifically for this. IE a dedicated installer.
4.) Jetty allows on the fly updates. You don't have to stop the sage and the sage service to apply a new version of BMT.

There are actually a lot of other plugins that also use Jetty. You will benefit from setting it up. Slugger has 4 or 5 that use the Jetty so by configuring Jetty for BMT, it will open the doors for a lot of other things. Plus it's not super difficult to setup.

[loomdog32]

Im not sure what I did wrong or if I did anything wrong, but I cant get BMT to come up with Jetty...

I installed jetty per instructions (copied files and made changes to sage.properties file)
I copied files for BMT v3b3 as usual. I didnt load the stvi as I do not want the automatic metadata/fanart (is this my problem??)

when i goto https://localhost:808/bmt i get a 404
when i goto https://localhost:8080 i get a 403

webserver works fine at https://localhost:8070/sage

I am looking to use this to get metadata for tv shows and import them from "videos" to "recorded tv".. If I can use custom (manually typed) metadata, Id like to do this for music videos as well

[sflamm]

Sean,

1st off, many thanks for spending the time sorting through the windows VM. Much appreciated.

I took a quick look and just as you said, the mediafile_metadata_parser_plugins property was not set! Not sure how that happened (it has been in my install instruction as step #4 forever). I wonder if I somehow overwrote the sage.properties file...

In any case, glad to know what the issue was in the end.

I am confused by your comment:

I used your image, and it was not configured to start. So I had to edit the metadata provider list to use the dvdprofiler-2 so that it could get metadata from dvdprofiler and fanart from themoviedb.

What do you mean by "it was not figured to start"? Do you mean that SageTV did not automatically start (SageTV was not configured as a running process on boot). If so that was on purpose since the VM was for experimenting only.

Is there a reason you edited the provider list to use dvdprofiler-2 instead of the custom dvdprofiler?

[sflamm]

I've updated the documentation to include BMT 3.x. (still includes 2.x instructions as well)

The documentation can be found here:

http://docs.google.com/Doc?id=dff28jvm_13f4rf57f6

Please let me know of any correction/additions that need to be made.

[stuckless]

Originally Posted by mkanet

I wasn't able to find any documentation for 3.0 configuration, if there is any; so, forgive me, if these questions can be answered in a document. But, I have a couple of quick questions.. is the "Web UI" the only way to configure BMT without having to manually edit the properties file? I dont have phoenix/mediametadata/fanartCentralFolder set in my properties file; yet, the on demand fanart download feature still seems to know where to save the files (from sageMC/extra/BMT options?). It looks like the Web UI requires a webserver for sageTV. I already use the Web User Interface for SageTV for sageTV scheduling, maintenance, etc. Could I use that to access the config screen for BMT? Is so, what URL would I use?

PS: I am currenly using the same machine for client/server.

I'll add the phoenix/mediametadata/fanartCentralFolder and hopefully thats all thats left to do!

Thanks! Crossing my fingers..

I haven't updated any documentation related to 3.x because it's not done yet. When I release I will try to update the documentation soon after. For now, I've been putting important notes in the release notifications thread.

Manual configuration and the web ui are the only 2 ways to configure the 3.x release. But, some fields, such as the fanart central folder, can be configured using the sagemc configuration, since it's not a specific bmt property. In bmt 2.x the fanart folder had to be configured in 2 locations, but bmt 3.x, because it stores it configuration in the Sage.properties, simply uses the "phoenix" property for fanart.

As for the web ui. It is not required to in order to run bmt. It is simply there to help you configure and troubleshoot. Jetty was chosen because it's a very good web container, and it makes it easy deploy web application as a standard war file. I also use nielms server, but I also run jetty. I have no intentions to figuring out how to deploy a war file to neilm's server, given that installing jetty and deploying webapps to it, is very trivial.

Regarding my statement about how the 3.x configuration is for the most part the same as the 2.x configuration, with the exception of the bmt/ prefix... that is, for the most part, true. I sort of answered this above, but 3.x now uses the phoenix fanart property, and no longer uses its own duplicate property, so there is no bmt prefix.


@toricred - when you fetch metadata for a tv file in sage (on demand), then the handler for that has access to existing metadata for that item, including the seriesname and episode title (if it exists), so it can do a match. When a file is handled outside of sage as a filesystem file, and it doesn't match the fairly strict naming convention for sagetv recordings, then there is no way to get the series name and episode title, so it fails. That what I think is happening.

@loomdog32 - if you've posted your urls correctly, then the first one, is using an incorrect port. I would try, https://localhost:8080/bmt , or http://localhost:8080/bmt

@sflamm - My statement is just bad english. My intention for the statement is that basically, for starters, the environment was not configured, and I had to configure both the metadata plugin and the list of providers. I removed the custom composite provider and used the built-in one (dvdprofiler-2) since there was no need for the other composite one. In the 2.3 build, I think I now ship with a default composite provider for dvd profiler and mymovies, so that you don't need to configure one. If you just use the "dvdprofiler" profiler as is, then it will not get fanart.

So, just to re-iterate a theme here.... Even though the stv may provider a way to configure some things, and the web ui allows you to configure some things, don't blindly trust those tools when you are having issues. If you have having issues, ie, automatic plugin is not working, but you've configured it... then validate that by checking the configuration file as well. BMT 3.x will do some validation for you when you fire up the web ui. In sflamm's case, it immediately showed me that the automatic plugin was not installed on the server, but again, I validated that by checking the configuration.

sorry for such a long post, but there was alot of activity after I went to bed last night... on the forum, that is

[sflamm]

Sean,

Now I understand -- the reason the automatic plugin was not enabled in the VM is that I disabled it (via the STV) because it wasnt working.... not because it wasnt enabled originally.

That may explain how the mediafile_metadata_parser_plugins property got reset. If you disable the 2.x plugin using the STV does that reset the property?


If it does not, then I had an user error - and that accounts for my plugin not working. If it does, then there must have been something else wrong. You said that the only thing else you changed was removing the customer provider, yes?

Can you please send me the metadata.properties file and the sagetv.properties files from the working VM so that I can compare what you changed/corrected with what was in the original VM? I can also use this to make sure the documentation is clearer if necessary.

Can you also please read through the updated documentation I posted last night for accuracy?

Thx again for all the hard work.

[stuckless]

Sure... I'm out of town for a few days... so I'll try to remember to do this when I get back.

Originally Posted by sflamm

Sean,

Now I understand -- the reason the automatic plugin was not enabled in the VM is that I disabled it (via the STV) because it wasnt working.... not because it wasnt enabled originally.

That may explain how the mediafile_metadata_parser_plugins property got reset. If you disable the 2.x plugin using the STV does that reset the property?


If it does not, then I had an user error - and that accounts for my plugin not working. If it does, then there must have been something else wrong. You said that the only thing else you changed was removing the customer provider, yes?

Can you please send me the metadata.properties file and the sagetv.properties files from the working VM so that I can compare what you changed/corrected with what was in the original VM? I can also use this to make sure the documentation is clearer if necessary.

Can you also please read through the updated documentation I posted last night for accuracy?

Thx again for all the hard work.

[tmiranda]

Originally Posted by stuckless

Once I had everything configured, I removed the videos from the library, and then added them back in, told sage to refresh the library, and the automatic plugin kicked in for each item.

Sean, Did you confirm that that plugin is called when new TV shows are recorded? Your statement above seem to indicate that you are referring to imported video (which has always worked fine for me) but not new recorded TV.

Thanks,

Tom