Utility: Batch Metadata Importer 3.x

[panteragstk]

Yeah, I've been redoing all of my metadata today and thetvdb has been very slow since it came back on line. I finally figured out why I've been having so many problems though. The naming convention that bmt likes is the name.s01e01 and it works very well. Any others create massive problems. It has been a huge pain getting all of my files renamed, but it will be worth it.

BTW, does anyone know how to get sage to stop scanning when you choose the "rescan existing library" option? It won't stop and I've been changing file names so much today that I just need to make it stop until I'm finished.

Any advice?

[Fuzzy]

Originally Posted by panteragstk

BTW, does anyone know how to get sage to stop scanning when you choose the "rescan existing library" option? It won't stop and I've been changing file names so much today that I just need to make it stop until I'm finished.

I think the only way to stop sage mid scan is to shutdown the service.

[craigap]

What bmt version: 3.1
What SageTV version: 6.6
SageMC or default STV: SageMC
SageMC version? 6.3.9b_2
Java version? 1.6 update 14
Phoenix version? 1.44

SageTV setup:
Is central FanArt folder enabled: Yes
Is central FanArt folder local or a network share: Local
Do you have SageTV set up to do automatic scans: Yes
If yes, how often: 1hr
Do the scans import new videos: yes

BMT setup:
Command Line Tool:
Have you successfully used the command line tool: Haven't tried

On-demand Tool:
Have you imported the on-demand plugin (stvi of xml): I did now.
Do you have the on-demand plug-in enabled: I do now and it works fine.

Automatic tool:
Do you have these 2 properties set in sage.properties: Yes

  mediafile_metadata_parser_plugins=org.jdna.sage.MetadataUpdaterPlugin
  custom_metadata_properties=UserRating;EpisodeNumber;EpisodeTitle;MediaTitle;MediaType;OriginalAirDate;SeasonNumber;MediaProviderDataID;DiscNumber

Please give a description of the steps to reproduce your issue: The BMT Web UI will always work just fine the first time I install it and go to it, but after some time something happens and I can no longer access it. I've attached a pic of what I get when it doesn't work. I've tried restarting the entire HTPC and that won't fix it.

[Taddeusz]

I don't believe I've seen this mentioned anywhere else. I don't know if it's a SageMC or BMT problem. Maybe it's just me but I ran into something very odd. If on a library scan I have auto metadata enabled it will revert any manual changes I've made to some of my imports. Basically the ones that came up incorrect to begin with. Example, my Fantastic 4.mkv file automatically gets detected as "Fantastico 4". Yesterday I had manually changed it in the web interface but the next media scan I did in SageMC reverted it back to the way it was before. Is this a bug or a setting I need to change?

[razrsharpe]

Sean,
mkanet asked me if would take a look at his debug logs. I figured you were busy so thought i would take a look to see if i find anything

Short answer
It appears that when the automatic plugin looks up tv recorded movies (non series) it tries searching with type = TV and returns null. Any thoughts on why this would be?

Long answer
looking through the 2 sage logs i dont see anything obvious. Nobody seems to be throwing any IO exceptions.

looking through the mailcheck.log i see these lines (among other ERROR lines for other movies)

2009-09-14 11:05:04,785 DEBUG [Seeker] org.jdna.media.util.AutomaticUpdateMetadataVisitor (AutomaticUpdateMetadataVisitor.java:126) - Nothing Found for query: SearchQuery; Type: TV; TITLE:Analyze That;EPISODE_TITLE:;; File: file:/E:/TV%20Recordings/AnalyzeThat-5074192-0.mpg
2009-09-14 11:05:04,786 ERROR [Seeker] org.jdna.sage.MetadataUpdaterPlugin (MetadataUpdaterPlugin.java:195) - Failed to Fetch Metadata for Media: E:\TV Recordings\AnalyzeThat-5074192-0.mpg
2009-09-14 11:05:04,788 DEBUG [Seeker] org.jdna.sage.MetadataUpdaterPlugin (MetadataUpdaterPlugin.java:208) - Returning NULL Properties for File: E:\TV Recordings\AnalyzeThat-5074192-0.mpg

I think the problem is that the automatic plugin is trying to search for "Analyze That" using type = TV. My limited understanding of BMT and the automatic plugin is that the automatic plugin limits its search providers (aka TV = thetvdb.com, movies = imdb, dvdprofiler, themoviedb, mymovies, etc). My theory is that when BMT does an automatic lookup for a TV recorded movie it says type = tv and looks only at the thetvdb... naturally it will not find any movies there ("Analyze That" in this example). This would explain why the search works for the on demand lookup (because my understanding is this will look through all the providers until a match is found) but not when done automatically. mkanet has no properties or fanart files for movies scanned only with the automatic plugin.

Anyway, hope you have some insight, and thanks for all your hard work

Originally Posted by mkanet

Sean, thanks! Restoring an older wiz.bin allowed BMT/sageTV to automatically add TV shows and their respective fanart pictures and metadata.

However, there's an issue with movies. For some reason, none of the TV movies I'm recording have fanart (over 15 movies I've tried). The metadata is there when viewing it using the "brose" feature , but fanart is not downloaded. This is for movies that have never been recorded before. In my other video imports directories that I have manual simple names such as "The Pursuit of Happyness.ts" do get fanart.

A big possible clue is that the On Demand feature, as always, can download the fanart for any of these movies (that will not work with automatic). This is true for even movies that only have one simple entry to select from in the On Demand list.

Is there anything I can do so the movies get fanart too? Unless I'm mistaken, I dont have too much control over movie names. Here are some examples that dont work with automatic, but work with On Demand. Bare in mind, that TV show work pretty good.

Here's an example of TV Movies that do not work:
AnalyzeThat-5074192-0.mpg
NurseryUniversity-5046198-0.mpg

Here's an example of a TV show that works just fine:
KeepingUpWiththeKardashians-LeavingtheNest-5077520-0.mpg

PS: Here's another clue, although I can see metadata information for these movies (like in the screenshot below), I dont get any properties files for these movies, just TV shows).



EDIT: I included a screenshot of the web configuration interface browsing the files. It will metadata for a movie, but no fanart pictures... like the way they are for TV series such as the Kardashians. I also included a link to full debugging on while recording a some movies.

screenshot

debug

Thanks,
Michael

[razrsharpe]

BMT 3.1

I have a question about Browse and Finding Metadata.

Last nite I (for the first time) used BMT to search for and update metadata for a newly added movie (I do not use the automatic plugin). Here is what i did (from the BMT homepage)

1. Browse Metadata, Movie name ("Wanted"), checkbox for movies (or dvds i forget what it is called... working from memory).
2. It found the new movie, and then i clicked on the find metadata button on the upper right corner.

Now here is my question: Should the title field be automatically filled out with the file name or folder name? In my case i needed to fill it out manually. Not really a big deal just a question because i would think the default behavior would be for the title field to default to the file or folder name (similar to passing the file/folder to bmt using the commandline).

[nyplayer]

BMT 3.1,

Why is BMT unable to determine the episodes when it is in the file and SAGE DB. The episodes on all my twilight zones is The Twilight Zone and they all have the same descriptions. If I delete the property file everything is ok.

ex
TheTwilightZone-TheLineman-3726971-0.mp4

this is the property below.

#Generator: Batch Metadata Tools (3.1); MediaFile: file:////192.168.2.240/t$/sageunct/TheTwilightZone-TheLineman-3726971-0.mp4
#Fri Sep 18 07:27:02 PDT 2009
Actor=Rod Serling -- ;
Description=The series was a collection of various tales that range from the tragic to the comedic. They may be scary or just thought-provoking. Most episodes have unexpected endings and a moral lesson. But, no matter what, it's "a journey into a wondrous land, whose boundaries are that of the imagination."
Genre=Science-Fiction
MediaProviderDataID=tvdb\:73587
MediaTitle=The Twilight Zone
MediaType=TV
OriginalAirDate=1959-10-01
RunningTime=1800000
Title=The Twilight Zone
UserRating=8
Year=1959
x-MetadataProviderDataUrl=73587;;;;EPISODE_TITLE\=The+Twilight+Zone
x-MetadataProviderId=tvdb
x-ser-Cast=0|Rod Serling||null
x-ser-Genres=Science-Fiction

[stuckless]

Originally Posted by craigap

The BMT Web UI will always work just fine the first time I go to it, but after some time something happens and I can no longer access it. I've attached a pic of what I get when it doesn't work. I've tried restarting the entire HTPC and that won't fix it.

I'm not sure what would cause it to do this... but some things that I would try are....
1. Use Shift+Reload on the browser to force a reload of the webpage.
2. Use another browser to see if it is consistently broken in different browsers
3. Blow away the bmtweb dir (not the .war file) and restart the sagetv server, and try again.

Originally Posted by Taddeusz

I don't believe I've seen this mentioned anywhere else. I don't know if it's a SageMC or BMT problem. Maybe it's just me but I ran into something very odd. If on a library scan I have auto metadata enabled it will revert any manual changes I've made to some of my imports. Basically the ones that came up incorrect to begin with. Example, my Fantastic 4.mkv file automatically gets detected as "Fantastico 4". Yesterday I had manually changed it in the web interface but the next media scan I did in SageMC reverted it back to the way it was before. Is this a bug or a setting I need to change?

It sounds like it could be a bug. Seems like it's rescanning your file for some reason (even though it has metadata). HOw are you doing a media scan in SageMC?

Originally Posted by razrsharpe

Sean,
It appears that when the automatic plugin looks up tv recorded movies (non series) it tries searching with type = TV and returns null. Any thoughts on why this would be?[B]

You re correct in that BMT will detect the media type as being either TV or a Movie and then use an appropriate provider based on the Type. One thing that BMT will do, is attempt to use the movie category, if it's a a recorded movie, then set the type of Movie. I added code for this in the fanart apis and bmt, but I have not Idea if it's working, since I don't have any recorded movies. So, it may be a bug in BMT, or may be that for some reason, this recording is not considered a "movie" by sage.

Originally Posted by razrsharpe

Should the title field be automatically filled out with the file name or folder name? In my case i needed to fill it out manually. Not really a big deal just a question because i would think the default behavior would be for the title field to default to the file or folder name (similar to passing the file/folder to bmt using the commandline).

The title field should be filled in with the Show TItle field from the metadata screen. I think that what you are saying makes sense, when the SHow Title is blank (i currently don't check that), so I'll add an issue to auto fill the title field with something useful when the show title is blank.

[Taddeusz]

Originally Posted by stuckless

It sounds like it could be a bug. Seems like it's rescanning your file for some reason (even though it has metadata). HOw are you doing a media scan in SageMC?

Within My Vidoes selecting Options and Rescan Import Folders.

[stuckless]

Originally Posted by nyplayer

BMT 3.1,

Why is BMT unable to determine the episodes when it is in the file and SAGE DB. The episodes on all my twilight zones is The Twilight Zone and they all have the same descriptions. If I delete the property file everything is ok.

ex
TheTwilightZone-TheLineman-3726971-0.mp4

#Generator: Batch Metadata Tools (3.1); MediaFile: file:////192.168.2.240/t$/sageunct/TheTwilightZone-TheLineman-3726971-0.mp4
MediaProviderDataID=tvdb\:73587
Year=1959

The first is that BMT simply selects the wrong show for "The Twilight Zone", but you can change this in the metadata-titles.properties.

When I run the commandline

java -jar MetadataTool.jar --offline --overwrite --auto=false --metadataOnly "TheTwilightZone-TheLineman-3726971-0.mp4"

And then select the 2002 Twilight Zone, it does get the metadata. There is an enhancement put in where bmt would use both the year and title when making selections.

For now, add

The\ Twilight\ Zone=tvdb\:78611

To the metadata-titles.properties.

[stuckless]

Originally Posted by Taddeusz

Within My Vidoes selecting Options and Rescan Import Folders.

So, in your case, sage is calling the bmt plugin with your media item. I wonder if this is part of the same issue that Ben reported a pages back. BMT should not be fetching metadata though, if it already has it, so it's possible it's a bug in bmt.

[Taddeusz]

Originally Posted by stuckless

So, in your case, sage is calling the bmt plugin with your media item. I wonder if this is part of the same issue that Ben reported a pages back. BMT should not be fetching metadata though, if it already has it, so it's possible it's a bug in bmt.

Another thing I noticed was an unusually long scan time as if it were fetching metadata for everything in my library during a regular scan. I don't know how long it took as I gave up and switched the TV over to the Xbox while it completed the scan.

[nyplayer]

Originally Posted by stuckless

The first is that BMT simply selects the wrong
For now, add

The\ Twilight\ Zone=tvdb\:78611

To the metadata-titles.properties.

Does this require a restart of the server?

[stuckless]

Originally Posted by Taddeusz

Another thing I noticed was an unusually long scan time as if it were fetching metadata for everything in my library during a regular scan. I don't know how long it took as I gave up and switched the TV over to the Xbox while it completed the scan.

This sounds like the issue that Ben reported as well. I'm going to try and make some time for bmt this weekend.

Originally Posted by nyplayer

Does this require a restart of the server?

I think so.

[razrsharpe]

Originally Posted by stuckless

You re correct in that BMT will detect the media type as being either TV or a Movie and then use an appropriate provider based on the Type. One thing that BMT will do, is attempt to use the movie category, if it's a a recorded movie, then set the type of Movie. I added code for this in the fanart apis and bmt, but I have not Idea if it's working, since I don't have any recorded movies. So, it may be a bug in BMT, or may be that for some reason, this recording is not considered a "movie" by sage.

That makes sense. I will record some movies and see if I can verify anything. What logic do you use to determine if sage considers the recording to be a movie or not? Also is there any way to determine from the ui (sage or bmt) if a particular recording is a movie (before BMT scans the file)?

EDIT: i just checked one of my recorded movies ("24: Redemption"). It has not been scanned by BMT and the default fanart directories are set to the Movies folder (see the attached jpg). In mkanet's screenshot they are also set to the "movies" fanart directory. Maybe this points to a bug in bmt (with the auto plugin) but not the fanart apis?

The title field should be filled in with the Show TItle field from the metadata screen. I think that what you are saying makes sense, when the SHow Title is blank (i currently don't check that), so I'll add an issue to auto fill the title field with something useful when the show title is blank.

Yes, the Show Title was blank in this case. Thanks for adding it to your full plate

[stuckless]

Originally Posted by razrsharpe

Maybe this points to a bug in bmt (with the auto plugin) but not the fanart apis?

I think you are right. The BMT plugin in 3.0.1 did actually use the "movie" flag, but I had to remove that portion of the code, because it was responsible for the "endless loop" issue that resulted in sage crashing

The problem is that when the plugin is scanning a file, that file is technically NOT in the library, yet. So, you have to be very careful when you access the file as a Sage Media Object, since if you call the wrong method, sage will try to fetch metadata for the object and as a result, call the plugin again (while you are still in it).

I think that ultimately, I may remove the automatic plugin support, and just use the scanner from the web ui.

[jaminben]

Originally Posted by stuckless

I think that ultimately, I may remove the automatic plugin support, and just use the scanner from the web ui.

Does this mean we would always have to manually add metadata from the web UI? This would be disastrous from my point of view.

Just my thoughts.

[babgvant]

Originally Posted by stuckless

I think that ultimately, I may remove the automatic plugin support, and just use the scanner from the web ui.

Please don't do that. The automatic plugin is the best part, and why I euthanized my metadata tool.

[sflamm]

Sean,

Why not just code your plug-in to be re-entrant? You know, like an interrupt handler... then you wont have an endless loop.

[gplasky]

Originally Posted by stuckless

I think that ultimately, I may remove the automatic plugin support, and just use the scanner from the web ui.

NNNNNNNNNNNOOOOOOOOOOOO!!!!!!!!!!!!!

That's what makes this such a joy to use. (I love the WebUI too.)This is what got me to move away from using DVD Profiler. (And I can't use the euthanized metadata tool anymore.)

Gerry