Utility: Batch Metadata Importer 3.x

[stuckless]

@ccourtney - Thanks for the descriptive problem. I'm wondering why you think that it should have found "National Lampoon's Vacation". From the log that you posted, it logs out the movie title that is being searched against the movie title in the dvd profiler collection, and the best hit it found was at 86%. The default automatic plugin will reject any hits below 90% match.

Similarity Score: [National Lampoon's Vacation][National Lampoon's European Vacation]=[0.8627451]
Similarity Score: [National Lampoon's Vacation][National Lampoon's European Vacation]=[0.8627451]
Similarity Score: [National Lampoon's Vacation][National Lampoon's Christmas Vacation]=[0.84615386]
Similarity Score: [National Lampoon's Vacation][National Lampoon's Christmas Vacation]=[0.84615386]
Similarity Score: [National Lampoon's Vacation][Vegas Vacation]=[0.42424244]
Similarity Score: [National Lampoon's Vacation][Vegas Vacation]=[0.42424244]

Given that your movie title is "National Lampoon's Vacation" there is no exact match. Try renaming your folder to "National Lampoon's European Vacation" (or appropriate title) and try again.

When you ran from the commandline, did you use the --auto=false, because that will prompt. The other reason an item will prompt is if it cannot find a good title match, or if the search/write throws an error.

[MacDaddy]

Grandpa lived on a farm far away from the big city and used an old wood stove to keep warm during the winters. When I visited last, he was working hard downing trees with his old cross-cut saw and axe. When I got back to the city, I sent him a brand new Husky 255 chainsaw to make his life a little easier. I received the first call from him thanking me for the gift, but wanting to give it back to me because it was too hard to use. I walked him through a few techniques on cutting the trees and promised to send him an article on how to sharpen the teeth to make it cut better. I made him promise to give it good try, suspecting that he was just adverse to change. After several more complaints, I decided to visit him and see what I could do to help. We walked out to the timber line with the Husky and selected a small red oak. I showed Grandpa what I thought would be the best cut to fell the tree in the best place for us to section it. I then told him to stand back and I started the Husky up. Grandpa stepped back a few feet and yelled out "What's that noise?".

PS - No this isn't a real story, but it seemed time for a little comic relief. Sean - I'm one of the non-vocal (hopefully) majority that is using your tool. I will never understand exactly how you do it, but the thing approaches PFM. I'm happy to put up with the occasional hiccups (never knew 2001 was not a Space Odyssey, but rather an art film) and thankful to be in a better place now that before I met you. Keep the faith.

[stuckless]

Actually after re-reading I can see that it did index National Lampoon's Vaction, but it did not show up in the search results. If you could send me your collection.xml, then I can retest using your collection and see why the indexer failed to index and search the title.

[mkanet]

No doubt youre obviously an extremely busy person stuckless. I certainly appreciate all the incredible work that you do (as do the other people who can't get the automatic plugin to work). I know your time is very valuable to you. I'm hoping when you do have time, you'll get around to figuring out whats causing the automatic plugin to not work for several people; whether it's a bug in BMT or a common system configuration that some people have that prevents it from not working.

Originally Posted by stuckless

Actually....

[ccourtney]

Originally Posted by stuckless

When you ran from the commandline, did you use the --auto=false, because that will prompt. The other reason an item will prompt is if it cannot find a good title match, or if the search/write throws an error.

I ran the following command
java -jar MetadataTool \\sageserver\DVDs

It prompted to select the right result for National Lampoon's Vacation which it did have one on the list with a score of 1.000.

While I'm interested in why this particular one failed, I'm really interested in why such a failure would cause the rest of the scan to stop. That could be a culprit in things like mkanet's problem where he is only seeing half of his collection imported during a scan.

I have also attached my Collection.xml file.

Thanks,
Craig

[stuckless]

@McDaddy - thanks for the chuckle
@mkanet, et al - I know that even while you guys are frustrated, that you do appreciate the effort being put forth. Sometimes I get cranky (as MeInMaui pointed out )

@ccourtney - You've given me something tangible that i can look at, I'll take a look at this over the next couple of days. (I neglected to mention in my prrevious post that wednesday night is a no code night - I golf )

While I'm interested in why this particular one failed, I'm really interested in why such a failure would cause the rest of the scan to stop. That could be a culprit in things like mkanet's problem where he is only seeing half of his collection imported during a scan.

This is the part that I hard time getting my head around. When bmt is used as a plugin, it does not "scan" in the sense that people think. ie, when you do a folder scan using bmt, then it will scan a folder and process each item, but when used as an automatic plugin, SageTV, does the scan, and simply requests information from the plugin as needed. So BMT doesn not have the power to "stop" the scan or prevent media from being imported.

ie, an automatic plugin scan dialog would be something like this...

SageTV: For each item in the Collection (or just a newly added item)
   SageTV: Request Metadata from Metadata Plugins for 1 item
   BMT: Hey, I've been given a file, let me do a lookup
   BMT: Parse title and search for matches using the configured providers
   BMT: Return Properties or Null back to SageTV
   SageTV: Hey, thanks for BMT for that work, User's will be happy to have metadata for thier movie (ok, maybe, there's no thanks, but it sort of works like that :) )
   SageTV: BMT gave me some metadata, let me apply it to the item
   SageTV: if more items waiting to be scanned, then repeat, otherwise, complete
SageTV: All Done Scanning

Now as of bmt 2.3x and 3.x and especially sagetv 6.5.18, there is no way for bmt to prevent items from being added to the collection. As of 6.5.18, sagetv implemented features that detect if a metadata plugin failed and not allow it to interrupt the scanning process. There are 3 things that I can think of that may prevent a collection from being 100% scanned.
1. BMT creates an infinate loop and as such, when sage calls BMT for metadata, then it never returns.
2. BMT causes an out of memory error, which is not typically trapped, and a JVM has a very hard time recovering from OOM errors.
3. Scanning is still happening, but taking a long time to complete. So people are only seeing a fraction of their movies because Sage is still scanning. (bmt is the bottleneck here, since it takes times to search/find/download metadata/fanart) Usually when i do a full rescan of my collection, it can take a couple of hours to complete when the plugin is installed, but it can take only minutes when the plugin is not isntalled. That's a big difference.

Sometimes when people say items are not being added, they mean that items are being added but without metadata, and other times, they mean, they are physically not being added. So I always have to decipher that as well.

[ccourtney]

Originally Posted by stuckless

This is the part that I hard time getting my head around. When bmt is used as a plugin, it does not "scan" in the sense that people think. ie, when you do a folder scan using bmt, then it will scan a folder and process each item, but when used as an automatic plugin, SageTV, does the scan, and simply requests information from the plugin as needed. So BMT doesn not have the power to "stop" the scan or prevent media from being imported.

ie, an automatic plugin scan dialog would be something like this...

SageTV: For each item in the Collection (or just a newly added item)
   SageTV: Request Metadata from Metadata Plugins for 1 item
   BMT: Hey, I've been given a file, let me do a lookup
   BMT: Parse title and search for matches using the configured providers
   BMT: Return Properties or Null back to SageTV
   SageTV: Hey, thanks for BMT for that work, User's will be happy to have metadata for thier movie (ok, maybe, there's no thanks, but it sort of works like that :) )
   SageTV: BMT gave me some metadata, let me apply it to the item
   SageTV: if more items waiting to be scanned, then repeat, otherwise, complete
SageTV: All Done Scanning

Now as of bmt 2.3x and 3.x and especially sagetv 6.5.18, there is no way for bmt to prevent items from being added to the collection. As of 6.5.18, sagetv implemented features that detect if a metadata plugin failed and not allow it to interrupt the scanning process. There are 3 things that I can think of that may prevent a collection from being 100% scanned.
1. BMT creates an infinate loop and as such, when sage calls BMT for metadata, then it never returns.
2. BMT causes an out of memory error, which is not typically trapped, and a JVM has a very hard time recovering from OOM errors.
3. Scanning is still happening, but taking a long time to complete. So people are only seeing a fraction of their movies because Sage is still scanning. (bmt is the bottleneck here, since it takes times to search/find/download metadata/fanart) Usually when i do a full rescan of my collection, it can take a couple of hours to complete when the plugin is installed, but it can take only minutes when the plugin is not isntalled. That's a big difference.

I am seeing NullPointer exceptions. See the following error messages I also found taking another look through.

This one which I don't think is causing the process to stop on recent configurations. (Troubleshooting poster art issue as well.)

Wed 7/8 12:24:24.762 [LibraryImporter@f1e6e4] Error instantiating metadata parser plugin of:java.lang.NullPointerException

This one seems to inidicate the problem as now I'm encountering this on a different file. (Renamed National Lampoon's Vacation to Vacation for troubleshooting purposes).

Wed 7/8 11:43:44.168 [LibraryImporter@1f25d83] BMT: BatchMetadataTools 2.3.1; Handling File: \\sageserver\DVDs\Star Wars I The Phantom Menace\VIDEO_TS; arg: 
Wed 7/8 11:43:44.196 [LibraryImporter@1f25d83] BMT: Reusing existing metadata for MediaFile: \\sageserver\DVDs\Star Wars I The Phantom Menace\VIDEO_TS
Wed 7/8 11:43:44.198 [LibraryImporter@1f25d83] Seeker has finished the library import scan.
Wed 7/8 11:43:44.198 [LibraryImporter@1f25d83] PooledThread ended w/ an exception: java.lang.NullPointerException

This seems to indicate that a NullPointer exception is happening processing this file which is ending the scan. There are a few more files in that directory which are not scanned. I have attached the properties file from Star Wars indicated above.

Have fun playing golf!

[rsagetv99]

Hi Sean,

Great plugin.

I am new to the plugin. I followed the instructions and successfully installed version 3 beta 4 and configured it for automatic scanning.

Everything was working, including the web interface, and manual scanning from SageMC. I'm pretty sure it auto-scanned a few shows as well.

Somehow, after I refreshed the library, BMT broke and I have not been able to fix it.

Whenever I go to the jetty page, I get the errors "Failed to get status for: SageTV", then "...:Metadata Tools", then "...:Phoenix".

Someone else mentioned in the thread to reindex the entire media library, but when I choose that option, it says "Failed to start scan."

Is there a way to reset everything, and start from scratch? I don't mind if I lose stuff that was already processed.

Thanks in advance.

[rmac321]

Originally Posted by stuckless

This is the part that I hard time getting my head around. When bmt is used as a plugin, it does not "scan" in the sense that people think. ie, when you do a folder scan using bmt, then it will scan a folder and process each item, but when used as an automatic plugin, SageTV, does the scan, and simply requests information from the plugin as needed. So BMT doesn not have the power to "stop" the scan or prevent media from being imported.

Is there any difference in the way SageTV scans are implemented based on how they are triggered? IOW, is a scan any different if it as an automatic hourly scan, a scan triggered from the options menu in SageMC or a scan triggered from a menu in the stock STV? It would seem logical that they all trigger the same calls in SageTV which would then trigger the same calls in the MetadataUpdaterPlugin, but my assumption could be incorrect.

[dbone1026]

Originally Posted by rsagetv99

Hi Sean,

Great plugin.

I am new to the plugin. I followed the instructions and successfully installed version 3 beta 4 and configured it for automatic scanning.

Everything was working, including the web interface, and manual scanning from SageMC. I'm pretty sure it auto-scanned a few shows as well.

Somehow, after I refreshed the library, BMT broke and I have not been able to fix it.

Whenever I go to the jetty page, I get the errors "Failed to get status for: SageTV", then "...:Metadata Tools", then "...:Phoenix".

Someone else mentioned in the thread to reindex the entire media library, but when I choose that option, it says "Failed to start scan."

Is there a way to reset everything, and start from scratch? I don't mind if I lose stuff that was already processed.

Thanks in advance.

I think this may be a known problem (correct me if I am wrong Sean). I have just been using Razrsharpes command line bat file to handle for the time being.

[stuckless]

Originally Posted by ccourtney

I am seeing NullPointer exceptions. See the following error messages I also found taking another look through.

This one which I don't think is causing the process to stop on recent configurations. (Troubleshooting poster art issue as well.)

Wed 7/8 12:24:24.762 [LibraryImporter@f1e6e4] Error instantiating metadata parser plugin of:java.lang.NullPointerException

This one seems to inidicate the problem as now I'm encountering this on a different file. (Renamed National Lampoon's Vacation to Vacation for troubleshooting purposes).

Wed 7/8 11:43:44.168 [LibraryImporter@1f25d83] BMT: BatchMetadataTools 2.3.1; Handling File: \\sageserver\DVDs\Star Wars I The Phantom Menace\VIDEO_TS; arg: 
Wed 7/8 11:43:44.196 [LibraryImporter@1f25d83] BMT: Reusing existing metadata for MediaFile: \\sageserver\DVDs\Star Wars I The Phantom Menace\VIDEO_TS
Wed 7/8 11:43:44.198 [LibraryImporter@1f25d83] Seeker has finished the library import scan.
Wed 7/8 11:43:44.198 [LibraryImporter@1f25d83] PooledThread ended w/ an exception: java.lang.NullPointerException

This seems to indicate that a NullPointer exception is happening processing this file which is ending the scan. There are a few more files in that directory which are not scanned. I have attached the properties file from Star Wars indicated above.

Have fun playing golf!

Could you email me the entire sagetv_0.txt and mailcheck.log files. And just to clarify, you are running the SageTV 6.5.19?

Thanks.

[stuckless]

Originally Posted by rsagetv99

Hi Sean,

Great plugin.

I am new to the plugin. I followed the instructions and successfully installed version 3 beta 4 and configured it for automatic scanning.

Everything was working, including the web interface, and manual scanning from SageMC. I'm pretty sure it auto-scanned a few shows as well.

Somehow, after I refreshed the library, BMT broke and I have not been able to fix it.

Whenever I go to the jetty page, I get the errors "Failed to get status for: SageTV", then "...:Metadata Tools", then "...:Phoenix".

Someone else mentioned in the thread to reindex the entire media library, but when I choose that option, it says "Failed to start scan."

Is there a way to reset everything, and start from scratch? I don't mind if I lose stuff that was already processed.

Thanks in advance.

I have had this happen to me... Normally I restart the SageTV server and all is well again. When it was happening to me, I noticed that my SageTV JVM memory size was set to 256mb, which is not enough if bmt is going to be running as well.

@dbone - Just noticed your comment. beta-4 added a new check to where I had noticed the problem before. So the problem is known, but it is also considered fixed. But this may be a slightly different problem.

[stuckless]

Originally Posted by rmac321

Is there any difference in the way SageTV scans are implemented based on how they are triggered? IOW, is a scan any different if it as an automatic hourly scan, a scan triggered from the options menu in SageMC or a scan triggered from a menu in the stock STV? It would seem logical that they all trigger the same calls in SageTV which would then trigger the same calls in the MetadataUpdaterPlugin, but my assumption could be incorrect.

You are correct, they are essentially the same. Certainly from a bmt perspective, they are exactly the same and they end up utilizing the same bmt apis to do the lookups, etc. Scans that are triggered from the UI end up calling RunLibraryImportScan. i don't know for sure if sage internally calls the exact same API hourly or not, but i suspect that they do.

On on-demand scan of a single item from the UI is handled slightly different but does end up calling the same bmt apis.

[rmac321]

Originally Posted by stuckless

Sometimes when people say items are not being added, they mean that items are being added but without metadata, and other times, they mean, they are physically not being added. So I always have to decipher that as well.

That is one of many great points you have made regarding supplying you with enough info to help make a diagnosis. A good analogy is trying to diagnose a problem over the phone. Customer: "My car/toaster/computer just died, what's wrong with it?" How about calling the vet?: "My cat is way up in a tree and I can't see her moving. Is she dead?"

In my case, I have an issue with the automatic updater, but since I haven't even had time to do minimal tinkering to see what is happening, I haven't bothered to waste your time asking for all the various possible things that could be causing it to not work. It's too bad some people don't recognize that so many of these tools started as someone trying to simplify something for themselves and then sharing their work.

If I'm not stepping on toes, I'd suggest a form/list of info you need to aid in diagnosis. If it was linked from post#1, then you or anyone else reading the thread could point to it as a standard response when people ask questions without supplying enough info. It would have the benefit of anybody on the thread being able to point people at the info needed and you could comfortably ignore posts that don't contain enough info for you to help in diagnosis. I'll start to put together a list/form unless someone beats me to it.

Oh, yeah - Thanks for a cool tool! The on-demand tool does a great job for me.

[rmac321]

Originally Posted by stuckless

You are correct, they are essentially the same. Certainly from a bmt perspective, they are exactly the same and they end up utilizing the same bmt apis to do the lookups, etc. Scans that are triggered from the UI end up calling RunLibraryImportScan. i don't know for sure if sage internally calls the exact same API hourly or not, but i suspect that they do.

Does SageTV do a separate scan for recordings, or is that part of the same process? I seem to recall seeing something about SageTV doing very frequent scans of the recording directory regardless of whether automatic scanning is enabled.

[rmac321]

Here's a start of a list of items to aid in diagnosis. What else should be on here?

What bmt version 2.x (stable) or 3.x (beta)?
What SageTV version?
SageMC or default STV?
SageMC version?
Java version?
Phoenix version?

SageTV setup:
Is central FanArt folder enabled?
Is central FanArt folder local or a network share?
Do you have SageTV set up to do automatic scans?
If yes, how often?
Do the scans import new videos?

BMT setup:
Command Line Tool:
Have you successfully used the command line tool?

On-demand Tool:
Have you imported the on-demand plugin (stvi of xml)?
Do you have the on-demand plug-in enabled?

Automatic tool:
Do you have these 2 properties set in sage.properties?
  mediafile_metadata_parser_plugins=org.jdna.sage.MetadataUpdaterPlugin
  custom_metadata_properties=UserRating;EpisodeNumber;EpisodeTitle;MediaTitle;MediaType;OriginalAirDate;SeasonNumber;MediaProviderDataID;DiscNumber

Please give a description of the steps to reproduce your issue:

[stuckless]

rmac321 - great idea on the support form. I'll take your list and make it a part of the first post. It's also something that I had hoped to incorporate into the web ui. ie, a bug submission form that would collect various pieces of information, including the problem, and allow the user to copy and past that into thread post or bug submission.

[rsagetv99]

Originally Posted by stuckless

I have had this happen to me... Normally I restart the SageTV server and all is well again. When it was happening to me, I noticed that my SageTV JVM memory size was set to 256mb, which is not enough if bmt is going to be running as well.

After rebooting I still have the error.

I tried removing everything after

mediafile_metadata_parser_plugins=

in sage.properties and restarting Sage, thinking that the automatic plugin was causing the problem, but I still get the same error.

Another symptom is that when I go to http://<servername>:<port>/bmt, the default page is not loaded. Instead, I am presented with links to the contents of the /bmt folder (instead of launching the default page), and I have to manually choose BMTWeb.html. Once I do, it goes to the page, but then gives me the status errors.

[ccourtney]

Originally Posted by stuckless

Could you email me the entire sagetv_0.txt and mailcheck.log files. And just to clarify, you are running the SageTV 6.5.19?

Thanks.

Yes, I am using 6.5.19. I'll reset the config and get you a clean set of log files with debug turned on for the importer. You should have the e-mail tonight.

[dbone1026]

Originally Posted by rsagetv99

After rebooting I still have the error.

I tried removing everything after

mediafile_metadata_parser_plugins=

in sage.properties and restarting Sage, thinking that the automatic plugin was causing the problem, but I still get the same error.

Another symptom is that when I go to http://<servername>:<port>/bmt, the default page is not loaded. Instead, I am presented with links to the contents of the /bmt folder (instead of launching the default page), and I have to manually choose BMTWeb.html. Once I do, it goes to the page, but then gives me the status errors.

That is exactly where I am at. When I initially set up BMI it scanned through much of my collection before crapping out. Have not been able to get back to the WebUI since then.