Utility: Batch Metadata Importer 3.x

[frontlinegeek]

OK... I am either losing my mind or the instructions for installing BMT have changed a LOT...

I cannot get BMT (Or accessing BMT via JETTY) to work at all...

I have had it installed once already but I reinstalled WHS and SageTV and now I cannot get it to install. The last time I was down this road, there seemed to be longer instructions than what there is now...

[Ikarius]

Okay, so I got 3.0.2 installed, and fired it up. It's faring a lot better than 3.0- definitely, but I'm still having some problems. I fired up the web UI, kicked off some scanning, it picked up most stuff reasonably. One notable exception is that all my Battlestar Galactica recordings got automatically tied to the 70's era Battlestar Galactica, and lost all their Episode info.

Next thing I noticed, was a lack of a lot of significant metadata bits- year, director, actors, MPAA rating, etc. I did a "just scan" run through movies, picked Army of Darkness, and asked it to find the Metadata. After finding the metadata, all those missing items were filled- when I looked at "Cast Members", Year, MPAA Rating... all there. I then hit the "save" button, and once it finished saving, the fields which weren't there before finding metadata empty out again. More on this later.

Lastly, processing files seems to take an inordinate amount of time. Maybe 15 seconds to process an item. I let BMT do a full reindex, and it took several hours to complete. I'm not sure why it's taking so long; I didn't expect it to be re-downloading fanart, and the logs didn't look like it was doing that. I did see an error in the log indicating it was getting a Java NullPointerException instantiating the metadata parser plugin. I don't know if that's harmless or not. Below is a clipping from my logfile.

Sat 8/22 18:08:17.711 [main@329f3d] Error instantiating metadata parser plugin of:java.lang.NullPointerException
Sat 8/22 18:08:17.751 [main@329f3d] Redetecting format for TV file because it changed:MediaFile[id=305568 A[216931,167829,"Battlestar Galactica",58623@0213.10:01,60] host=zion encodedBy=video0 Great-H.264 format=MPEG2-TS 0:59:50 8008 kbps [#0 Video[H.264 1920x1088 interlaced]#1 Audio[AAC 48000 Hz 2 channels MAIN idx=1 id=1100 at=ADTS-MPEG2]{MediaType=TV, UserRating=7, EpisodeNumber=, MediaProviderDataID=tvdb:71173, SeasonNumber=, MediaTitle=Battlestar Galactica, OriginalAirDate=1978-09-17}] /data/recording/BattlestarGalactica-SineQuaNon-216931-0.ts, Seg0[Fri 2/13 10:01:02.522-Fri 2/13 11:01:00.001]]
Sat 8/22 18:08:17.766 [main@329f3d] Using internal format detector first...
Sat 8/22 18:08:17.852 [main@329f3d] File Format Parsed-2 /data/recording/BattlestarGalactica-SineQuaNon-216931-0.ts=MPEG2-TS 0:59:50 8008 kbps [#0 Video[H.264 1920x1088 interlaced]#1 Audio[AAC 48000 Hz 2 channels MAIN idx=1 id=1100 at=ADTS-MPEG2]]
Sat 8/22 18:08:29.794 [main@329f3d] BMT: Processed File: /data/recording/BattlestarGalactica-SineQuaNon-216931-0.ts

Back on the missing metadata topic. I poked around, and saw it had written out ".properties" files for my media which *do* include the information. I'm going to go poke around in studio with Phoenix metadata calls and see if I can pull up these info bits on any of my movies. If not, I'm going to poke at the Sage.properties custom_metadata_properties field, and see if adding them into that list will get them saved. I'll report back with what I find, but if I get it to work by addding them there, my next question will be why those fields aren't included in the extra metadata by default.

...

Okay, after fiddling around a while, I have verified that only the metadata fields listed under "custom_metadata_properties" are available.
They're all available through GetMediaFileMetadata(). I tried using phoenix_api_GetMetaDataValue()- passing it exactly the same Airing object and "UserRating" as I passed to the GetMediaFileMetadata call, and got "null" instead of the metadata returned via the sage API. Looking at things, I'm still a bit nervous about adding additional fields to custom_metadata_properties.

...

Okay- I think I've reached the end of the rope with BMT 3.0.2. In the config, under Automatic Plugin, I had "Overwrite Metadata" and "Overwrite Fanart" enabled. I dropped down to the most recent Sage default STV, and kicked off a full re-index. What I found was, it would successfully go through all videos *except* DVDs (DVDs are the vast majority of my collection). It would get to the first DVD it scanned, import full information for it, and then bomb out the LibraryImport thread with an exception. Then I enabled "Miissing Media Items Workaround", and re-attempted. After that change, it would successfully process all my DVDs, but without pulling in any actors, years, etc. I think that's as far as I can take this until Sean gets back and pokes around.

I also found a major omission in the metadata downloaded by the scraper- There's no "Writer" info in the metadata. I'd expect this to go into the "PeopleInShow" metadata the same way "Director" does. No idea if EPs scraper gets this or not, but it's pretty important. Any chance we can see this go in?


Cheers
Ikarius

[mkanet]

Hi Guys,

I am very happy to say everything is working for me with BMT 3.0.2 and Jetty 1.6. On Demand feature works like a charm.

However, as some of you may know, I have never been able to get the automatic feature to work; even on BMT 2.x.

I dont think the problem is with Java, Windows, or issues with wrong/missing/duplicate/mismatched JARs; as, I have never had issues with any other sageTV plugin, sageTV itself, or java apps (outside of sageTV). Also, in previous versions of BMT (where On Demand worked and Automatic didnt), when I enabled logging, BMT automatic didnt have any errors; because it never started. I also wasnt getting errors for On Demand; because it was working correctly.

It could be as simple as a sageTV.properties / sageTVclient.properties missing or changed from the default value (the function/feature necessary to tell BMT automatic to process a new media file that was just added to sageTV recordings. Could someone please tell me what sageTV properties settings are definitely needed to initiate automatic fanart when a file is added?

Also, now that BMT/sageTV has better logging features since the last time I tried to get this to work; maybe someone can tell me exactly what logging settings to enable (and what/where to look... and in which log file).

Thanks,
Michael

[razrsharpe]

Originally Posted by Ikarius

Back on the missing metadata topic. I poked around, and saw it had written out ".properties" files for my media which *do* include the information. I'm going to go poke around in studio with Phoenix metadata calls and see if I can pull up these info bits on any of my movies. If not, I'm going to poke at the Sage.properties custom_metadata_properties field, and see if adding them into that list will get them saved. I'll report back with what I find, but if I get it to work by addding them there, my next question will be why those fields aren't included in the extra metadata by default.

...

Okay, after fiddling around a while, I have verified that only the metadata fields listed under "custom_metadata_properties" are available.
They're all available through GetMediaFileMetadata(). I tried using phoenix_api_GetMetaDataValue()- passing it exactly the same Airing object and "UserRating" as I passed to the GetMediaFileMetadata call, and got "null" instead of the metadata returned via the sage API. Looking at things, I'm still a bit nervous about adding additional fields to custom_metadata_properties.

You are correct that the custom_metadata_properties property item controls which "extra" property items are available to the user (regardless of what is in the .properties file.

Here is what my line looks like (and i believe is what stuckless recommends):

custom_metadata_properties=MediaProviderDataID;MediaTitle;MediaType;OriginalAirDate;EpisodeTitle;EpisodeNumber;SeasonNumber;DiscNumber;UserRating

Getting the .property file is half the battle. After the property file is created sage must refresh its metadata so it can read the .property file and populate the extra metadata fields in its database. I believe the webui has a means to refresh the sage db... but i do all my scanning via commandline so im not much help there...

however, what i do:

1. Use BMT to make the .property file
2. Refresh video import libraries in sagetv.

What i would reccommend (assuming the .property files exist and are correct):

1. Make sure the custom_metadata_properties line looks like above.
2. Remove your video import directories
3. Refresh your media from the Sage UI
4. Add back your video import directories
5. Refresh your media from the Sage UI.

An alternative to 2-4 is to "touch" the video files so that they appear new to sage and will have there metadata reimported. Deleting the .properties files and rerunning BMT should create new .propterty files and update all the timestamps.

RE - phoenix_api_GetMetaDataValue vs GetMediaFileMetadata - I am not sure of the difference between these calls but i believe the phoenix_api_GetMetaDataValue is not intended for looking up metadata inside the Sage DB. GetMediaFileMetadata is the correct call.

[PLUCKYHD]

Originally Posted by razrsharpe

RE - phoenix_api_GetMetaDataValue vs GetMediaFileMetadata - I am not sure of the difference between these calls but i believe the phoenix_api_GetMetaDataValue is not intended for looking up metadata inside the Sage DB. GetMediaFileMetadata is the correct call.

Man you guys need to use autoscanning

GetMetaDataValue = Given an item, return a piece of metadata by name. An item can be a search result or Map of properties

GetMediaFileMetaData = Returns a string for the corresponding metadata property in the MediaFile's format. These are set during format detection/import.
Only names set in the property "custom_metadata_properties" (which is a semicolon/comma delimited list) will be available.

those are straight from the phoenix api if you don't understand above I can explain further.

[can3gxw]

I have noticed over the past several days that when metadata is imported for my videos (movies, tv series, etc...) and I subsequently DELETE this video once I am done with it, the .properties file remains...

Has anyone else noticed this? Does BMT have an auto cleanup of orphaned .properties files, or am I just looking at the directory too soon after I delete it?

BTW - I am deleting these files as I watch them through the SageMC interface... I don't always want to keep ALL of my videos...

Anyone? Thanks!

[Taddeusz]

I use dirmon2 to clean up the old files.

[mkanet]

For those of you interested in knowing the issue with BMT.

For whatever reason, defining the Central Fanart folder in SageMC Options/Extras will correctly work for On Demand (putting images in the central fanart folder). However, the automatic plugin will not use this setting at all; and download fanart without any folders into the same folder as the new media that was just added. Even when I would define the central fanart folder in the web interface using early 3.0 BMT betas, it would still behave the same way. EVEN, when manually adding the folder in my sageTV properties file.

Whatever Sean did in BMT 3.0 final using the web interface; when I set the central fanart folder for Phoenix, fanarts get downloaded every time a new recording completes in sageTV; and dynamically updates my background!

So, are there two separate Central fanart folder settings that need to be defined? If so, where are they? If there is only one; it looks like the only way for me to update that setting for automatic to work is through the web interface of BMT 3.x.

I am finally part of the club!!

tmiranda (and others who had a problem with automatic), I hope this will fix it for you!

[skiingwiz]

Originally Posted by Ikarius

Okay, so I got 3.0.2 installed, and fired it up. It's faring a lot better than 3.0- definitely

I just put 3.0.2 into full use. It has fixed all of my issues. No more crashing of the service and season/episode info is all there.

[can3gxw]

Originally Posted by Taddeusz

I use dirmon2 to clean up the old files.

Thanks... Checked it out... looks like exactly what I need.

[can3gxw]

OK new problem, folks... actually, it's been happening for a few days now and I can't figure it out nor find the answer. I am not sure if it is a BMT issue, or a SageMC issue, so I'm posting here since I am using BMT.

When I choose the scan library functions, be it from the SageMC Setup screen on the client/server, on a placeshifter client, from the Sage MC WebUI or from the BMT WebUI, the system tells me that it is kicking off a scan... it will run for xx minutes and "complete".

However, when I go back into my folders in My Video, some new files may show up, but old files that I have deleted are still there, and files that I have moved are still showing in their old spot.

I moved my PCs around a couple of weeks ago, and everything has been working great - it almost seems like it all broke when I updated to BMT 3.0 - I don't think I had any problems with the beta. Is there maybe a set of log files that are too big now and I need to change a setting? I have MORE than enough drive space (10s of GBs).... anyone have any ideas what will help?

Let me know if anyone needs to see any log files, and where to dig them up.

My install of Sage on this machine is less than a month old, and like I mentioned above - everything appeared to be working great until about a week ago (maybe less).

Thanks folks!

PS - I have not tried in the past day (nothing new to add...), but I think the FanArt is still actually downloading for my new stuff...

[tmiranda]

mkanet - What exactly did you do to get it working? Sorry, but your post is unclear to me. I think you are saying the issue was solved by installing 3.0.2 and setting the central fanart folder in the web interface?

[mkanet]

Hi tmianda, yeah, now that I look at what I wrote, I didnt make much sense. Yes, that's exactly what I did.

After I set the fanart folder in sageMC; I went into the web interface under, Configure-Metadata Tools/Central Fanart Folder... I realized that that section was completely blank; so, I selected the same folder I selected in sageMC and hit the Save button. I got out of the interface, restarted sageTV/sageTVservice. Then went back into the webinterface under that section, the entry was there. Be very careful you verify that the correct path is saved (you can check on the main BMT status webpage). The first time I selected the path, it somehow had an extra dot somewhere in the path.

There are actually several people who have this issue; as I'm getting PM's from other people about this. Hopefully, this is the fix for the problem.

Originally Posted by tmiranda

mkanet - What exactly did you do to get it working? Sorry, but your post is unclear to me. I think you are saying the issue was solved by installing 3.0.2 and setting the central fanart folder in the web interface?

[tmiranda]

Thanks. I'm still running the 3.0 beta because it looked as if people were having difficulties with the 3.0 final and stuckless is away visiting Mickey Mouse. Sean is vacationing only 30 minutes from where I live so if I have trouble with the upgrade I'll just go and track him down!

[mkanet]

Yeah, I'm sure he would love that Anyway, me me know if this finally fixes it for you. It worked for me one person already. Good luck!

Originally Posted by tmiranda

Thanks. I'm still running the 3.0 beta because it looked as if people were having difficulties with the 3.0 final and stuckless is away visiting Mickey Mouse. Sean is vacationing only 30 minutes from where I live so if I have trouble with the upgrade I'll just go and track him down!

[Peter_h]

Originally Posted by can3gxw

OK new problem, folks... actually, it's been happening for a few days now and I can't figure it out nor find the answer. I am not sure if it is a BMT issue, or a SageMC issue, so I'm posting here since I am using BMT.

When I choose the scan library functions, be it from the SageMC Setup screen on the client/server, on a placeshifter client, from the Sage MC WebUI or from the BMT WebUI, the system tells me that it is kicking off a scan... it will run for xx minutes and "complete".

However, when I go back into my folders in My Video, some new files may show up, but old files that I have deleted are still there, and files that I have moved are still showing in their old spot.

I moved my PCs around a couple of weeks ago, and everything has been working great - it almost seems like it all broke when I updated to BMT 3.0 - I don't think I had any problems with the beta. Is there maybe a set of log files that are too big now and I need to change a setting? I have MORE than enough drive space (10s of GBs).... anyone have any ideas what will help?

Let me know if anyone needs to see any log files, and where to dig them up.

My install of Sage on this machine is less than a month old, and like I mentioned above - everything appeared to be working great until about a week ago (maybe less).

Thanks folks!

PS - I have not tried in the past day (nothing new to add...), but I think the FanArt is still actually downloading for my new stuff...

It's a bug with BMT. I think it is fixed in the latest version. I had to do this last night to get a couple of newly added movies to show up.

1.) Shut down sage and sage service
2.) Edit properties file and set BMT/Plugin = False
3.) Start up Sage in the default STV
4.) Do a library scan
5.) Shut down sage, sage service
6.) edit properties file and enable BMT/Plugin
7.) Start up sage like normal - should be all fixed now.

Let me know how it goes.

[can3gxw]

Thanks for the info, Peter... FWIW - I am already running 3.0.2, but I WAS running 3.0, so perhaps that was the initial cause.

I'll try it out tonight and post my results.

Thanks!

[redman]

dang.
I had all the 3.0 betas working great after a new sage install.

upgraded to final and now i get no updates to metadata, and sage cant see the files unless i manually change the properties to bmt enabled=false and redo it.

This only happens for non sagetv recorded files.

I tried regressing back to beta 9 (last one that worked) but now i get errors from the web ui when i try to scan.

Any ideas?

[mkanet]

In case you havent checked, can you make sure you only have the JAR files from the latest BMT 3.x and no duplicates/mismatched versions of the same file?

-Michael

Originally Posted by redman

dang.
I had all the 3.0 betas working great after a new sage install.

upgraded to final and now i get no updates to metadata, and sage cant see the files unless i manually change the properties to bmt enabled=false and redo it.

This only happens for non sagetv recorded files.

I tried regressing back to beta 9 (last one that worked) but now i get errors from the web ui when i try to scan.

Any ideas?

[redman]

I just 'found' a box tick that says 'tick here if you are missing files'.
Guess what? Ticked it, and stuff showed up.

typical. always happens just after you post.

Whatever this workaround does, it worked.