Utility: Batch Metadata Importer 3.x

[PLUCKYHD]

Originally Posted by heffe2001

I bumped mine up to around 700m, I've never noticed it going any higher than about 400m in the past, but the extra headroom gives that piece of mind .

Using my app at times I have seen it get close to 500mb just using one extender.

[heffe2001]

I wondered if that might be part of my speed issue with SMW, I'll keep an eye on it when I load it next time.

[tmiranda]

Sean,

Last night I was manually downloading fanart (using 2.3) and noticed that I had a number of 0kb files. When I looked on theTVdb I saw that banner/backgrounds were available but the tool did not download them.

Any idea why this is happening? I think I saw this issue mentioned somewhere else in this thread but I can't seem to find the reference to it anymore...

I do not want to upgrade to version 3 yet because I do not see any real advantage to me at this point. I'm using the stock STV on Windows XP.

Thanks,

Tom

[stuckless]

Tom, this issue was raised in the 2.x builds, and it has not been addressed in the 3.x builds, so there is no real advantage to upgrading for this particular issue.

I think that I need to simply do better error checking around the image fetching. I think that for now, i'm blindling writing to the file, and I'm not cleaning up if there is an error. For that reason, we are left with a 0k file. I haven't encountered any 0k files myself, but enough people have for me to know that I need to address it at some point in time. Maybe I'll try to spend some time tonight looking at that code, but, if I provider a fix, it will be in the 3.x code.

[PiX64]

Stuckless,

I am finally going to get around to trying the newest Beta you just released. Hopefully it will fix my issues from a few weeks ago.

I was wondering though when using the jar file directly to find metadata/fanart via a batch file, where does BMT drop the .properties file?

do i need to specify this somewhere?

~Mike

[tmiranda]

Originally Posted by stuckless

Tom, this issue was raised in the 2.x builds, and it has not been addressed in the 3.x builds, so there is no real advantage to upgrading for this particular issue.

I think that I need to simply do better error checking around the image fetching. I think that for now, i'm blindling writing to the file, and I'm not cleaning up if there is an error. For that reason, we are left with a 0k file. I haven't encountered any 0k files myself, but enough people have for me to know that I need to address it at some point in time. Maybe I'll try to spend some time tonight looking at that code, but, if I provider a fix, it will be in the 3.x code.

OK, thanks. I'll be watching this thread and will update when it makes the most sense.

For the most part the tool works really well. Please try to get it to download automatically for recorded TV shows (in Windows).

[plnewton]

Hi All,

I just wanted to say that I am struggling with this plugin, can some one please point me in the direction of some detailed instructions on BMT?

I am coming from Windows Media centre and I was using a product called YAMM to do all of the Metadata, it was very simple to Install and configure and just worked.
I just can't believe how complex BMT is to install and configure for such a trivial process; download cover art and movie details.

I honestly think that if it has all these extra features in there that it needs to be broken up into different versions...

Can someone please answer the following for me:
How can I edit the movie that it finds if it gets it wrong, which it has on many?

I still can't get it to show the Cover Art from within SageTV, it shows up on the BMT webpage on Localhost but just will not show in Save

If I set it to do a scan how do I know what it is doing? there is no status on scans and I can't see what it has done or not done or done incorrectly without looking at every single title


Cheers!!

[stuckless]

Originally Posted by plnewton

I just wanted to say that I am struggling with this plugin, can some one please point me in the direction of some detailed instructions on BMT?

I assume you read and digested the first post, and all the links that it has, including the wiki and the document by sflamm??

I am coming from Windows Media centre and I was using a product called YAMM to do all of the Metadata, it was very simple to Install and configure and just worked.
I just can't believe how complex BMT is to install and configure for such a trivial process; download cover art and movie details.

When you write and maintain your own metadata tools, you can call it trivial, until them, how about you just assume, like everyone else, that maybe it's not quite as trivial as your think it is. Besides, telling everyone how trivial it is to write a metadata tool will just inspire everyone to write their own metadata scraping engine, and I don't want that... I want to control the entire metadata spectrum with the illusion that it's a complex task, and your comments, are not helping me maintain the illusion.

I honestly think that if it has all these extra features in there that it needs to be broken up into different versions...

I agree... And I'm almost finished with the phoenix Clone api. Once that's done, I intend to clone myself, so that I can assign a cloned "me" to each of separate versions. I realize it's pretty trivial to write and maintain this stuff, time is still an issue. Cloning will definately help here.

Can someone please answer the following for me:
How can I edit the movie that it finds if it gets it wrong, which it has on many?

I know that you already know this, because you have read the wiki and the other posts here, but I'll reiterate for your benefit. I've always recommened that people use the command line tool, which offers an interactive way for you to be prompted for each and every movie.

I still can't get it to show the Cover Art from within SageTV, it shows up on the BMT webpage on Localhost but just will not show in Save

I assume that since you have read sflamm's document that are in fact running SageMC. (Just for the record, I sometime find it helpful that people actually provide me with information about their environment, sagetv version, sagemc version, bmt version, etc, etc, etc)

If I set it to do a scan how do I know what it is doing? there is no status on scans and I can't see what it has done or not done or done incorrectly without looking at every single title

I get plenty of feedback when I do a command line scan. Are you referring to when sagetv refereshes its media library. In that case, yeah, there's no status on how much progress has been made. Although the web UI will show the last scanned file, and the last 5 failed items. After reading all the posts here, you probably know that the web ui is in beta and it not complete, so there is no scanning available from the UI.

[MeInMaui]

Hey Sean, can you get crackin' on that clone api? I really need it like yesterday.



@plnewton

We really are a friendly and helpful bunch here. But when someone shows up and belittles our efforts, we tend to get a little cranky.

Aloha,
Mike

[babgvant]

Originally Posted by stuckless

When you write and maintain your own metadata tools, you can call it trivial, until them, how about you just assume, like everyone else, that maybe it's not quite as trivial as your think it is. Besides, telling everyone how trivial it is to write a metadata tool will just inspire everyone to write their own metadata scraping engine, and I don't want that... I want to control the entire metadata spectrum with the illusion that it's a complex task, and your comments, are not helping me maintain the illusion.

Maybe start calling

public string MungeBeforeSearching(string MovieTitle, double pctFailRate)

with a random number instead of .5? That should bring the illusion back.

[PiX64]

I am sad to report that after installing the new BMT, removing import directory, re-importing my problems still remains.

When i start fresh and let BMT do its thing, i only get about 101 DVD's which is less than half of my collection.

If i disable the autoplugin as done previously then my collection imports just fine....

As a side note is there a way to save the metadata look up from the bmt web gui or is it only possible when the autoupdate plugin is enabled?

~Mike

[babgvant]

Originally Posted by PiX64

I am sad to report that after installing the new BMT, removing import directory, re-importing my problems still remains.

When i start fresh and let BMT do its thing, i only get about 101 DVD's which is less than half of my collection.

If i disable the autoplugin as done previously then my collection imports just fine....

As a side note is there a way to save the metadata look up from the bmt web gui or is it only possible when the autoupdate plugin is enabled?

~Mike

- do you get properties files for the items that fail
- any errors in the sage log
- any errors in the mailcheck log
- is the # of successes reproducible (does it always import 101)
- any pattern to the items that succeed or to the items that fail

[plnewton]

Sorry guys I didn't mean to ruffle any feathers, it has just thrown me how much is involved in setting this up over what I had running previously.

The main reason I went to Sage was because of its extender support which has been a simple setup.

By the way I did not mean to imply that coding a metadata tool was trivial, I meant the amount of data that is stored for each movie is Trivial (small) so I was stunned to see how much was involved in this.

[stuckless]

Originally Posted by plnewton

Sorry guys I didn't mean to ruffle any feathers, it has just thrown me how much is involved in setting this up over what I had running previously.

The setup, is simply extract it, and optionally, install the configure plugin if you want automatic metadata lookups, or import the stvi/xml if you want on demand lookups. Or use the command line.

By the way I did not mean to imply that coding a metadata tool was trivial, I meant the amount of data that is stored for each movie is Trivial (small) so I was stunned to see how much was involved in this.

Some day, I'd like to start a topic on "how to ask for help" because posts like, "hey, your stuff sucks because it doesn't work the way I want it to.... and btw, can you help me, without me actually providing any details about what i've done or what my problem is...." really isn't going to inspire many people to provide help. (I'm not being sarcastic here)

I'm here a lot... if you are really serious about wanting help, then I'm here. Just send along the basics of....
* What bmt version 2.x (stable) or 3.x (beta)
* What sage version?
* SageMC or default STV?
* Are you using the automatic plugin, on-demand, or command line (I need to know what you are doing when "it's not working"
* a description of steps to reproduce your issue

That's the basics.... from there, I can try to help and request logs, etc.

[mkanet]

Everything works great for me in BMT except for the automatic feature. I know stuckless would rather not hear about this, but it has crossed my mind how many more people will not be able to get the automatic plugin to work (even when configured correctly) before there's more focus on fixing it. It looks like the problem is in both BMT 2.x and 3.0 (at least for me).

Originally Posted by razrsharpe

Glad i could help and you got it working i dont think i ever really tried to get the automatic plugin to work...

Originally Posted by PiX64

razrsharpe - the .bat you provided works great....Thanks a bunch and i will use this until i can figure out with stuckless what is the deal with the autoplugin. ~Mike

Originally Posted by tmiranda

OK, thanks. I'll be watching this thread and will update when it makes the most sense. For the most part the tool works really well. Please try to get it to download automatically for recorded TV shows (in Windows).

[babgvant]

Originally Posted by mkanet

Everything works great for me in BMT except for the automatic feature. I know stuckless would rather not hear about this, but it has crossed my mind how many more people will not be able to get the automatic plugin to work (even when configured correctly) before there's more focus on fixing it. It looks like the problem is in both BMT 2.x and 3.0 (at least for me).

If it's not working, how can you with any validity assert that it's configured correctly?

[babgvant]

Originally Posted by stuckless

Some day, I'd like to start a topic on "how to ask for help" because posts like, "hey, your stuff sucks because it doesn't work the way I want it to.... and btw, can you help me, without me actually providing any details about what i've done or what my problem is...." really isn't going to inspire many people to provide help. (I'm not being sarcastic here)

Chances are if someone isn't going to read the install instructions, they won't read the "how to ask for help" instructions either.

[stuckless]

Originally Posted by mkanet

Everything works great for me in BMT except for the automatic feature. I know stuckless would rather not hear about this, but it has crossed my mind how many more people will not be able to get the automatic plugin to work (even when configured correctly) before there's more focus on fixing it. It looks like the problem is in both BMT 2.x and 3.0 (at least for me).

Actually... just the opposite. I don't have any statistics on how bmt is used. I don't know how many people are having issues with ONLY the automatic plugin, but it's something that I would like to know. I realize that people are frustrated when they try things and it doesn't "just work".

While you may think that "nothing" is being done, I can assure you that the opposite is true. What you see as a lack of progress of one issue, is not a trivial issue. For example, I run the automatic plugin, almost daily, and I've yet to encounter an issue. Some have suggested that it's perhaps because I run a linux setup, and that may be the case. But, I have windows users that run the automatic plugin as well, and have not experienced the issues described here. Given that I have confirmation that it is working on both windows and linux, I can rule out windows as being the culprit (oh how I wish I could blame windows, but I can't). So, it does occur to me, that there are bugs in my software. I'm human, and I while I try to ensure that the software works, by testing, unit testing, etc, I am just single person. But, I am working on the issues and the enhancements, but time is the real factor here.

Let me give you indication of how my day is spent, so that you, and everyone else can see how little time I devote to this....
8:00 - 5:00 - I'm at work. While at work, I spend my day, leading a team of developers, providing support to those developers, architecting solutions, and if i'm lucky, I get to spend some time writing code to implement those solutions. I only mention this, because after spending all day, developing solutions, I sometimes find it hard to get motivated to work on coding projects at night. At various times throughout the day, I check the SageTV forums, but I never write sagetv plugin code when I'm at work.

5:00 - 6:00 - I'm at home. I eat dinner.

6:00 - 9:00 - I spend time with the kids. This usually includes taking them for a long bike ride down by the river trail, where we spend time, making sail boats out of twigs and sailing them in the river. I realy look forward to this part of my day.

9:00 - 9:30 - Putting the kids to bed.
9:30 - 11:00 - I spend my time either watching TV/Movies and/or working on the various projects. I try to watch a couple of movies/shows a week, so all my spare time is not dedicated to the projects.

The window of time is currently spent among the following projects....
1. BMT core fixes / enhancements
2. BMT Web UI
3. Phoenix
4. Private Joint Venture project A (non sagetv related)
5. Private project B (non sagetv related)

So, you can see, if the issue that you have only takes about 8 hours to find/fix/test, then that 8 hours may actually span an entire month or more to find/fix/test.

I hope this post helps clarify why progress on bmt is fairly slow comming. Time is really the barrier, but I'm trying to fit as much in as I can.

I log the complaints, track the issues, and I do even try to fix them, when i can reproduce the issues consistently.

[stuckless]

Originally Posted by babgvant

Chances are if someone isn't going to read the install instructions, they won't read the "how to ask for help" instructions either.

Please... I need hope Don't take away hope... that's all i have left

[ccourtney]

To start off with thanks for all the effort on these tools. I truly appreciate all that you and many others have done to enhance the SageTV experience.

I am also having problems with the automatic lookups. I have it configured to use my DVD Profiler collection as well as IMDB per the instructions linked on the first page of this thread. The plugin seems to work perfectly on the movies I have transcoded to mkv based H264 video files. I recently started doing the math and realized I'm not saving that much space on the H264 (I keep duplicates of the H264 movies since it takes 4-6 hours per movie to transcode). I then started using straight rips of the movies and that's when the problems started.

My Setup:
WHS With two video Import Directories: \\sageserver\Videos;\\sageserver\DVDs

SageTV 6.5.19
HD200 with latest beta firmware
Metadata Importer: 2.3.1
SageMC 6.3.9a pre-release 46

sagemc.jar: 6.3.8
df_sageutils.jar: 2.2
phoenix.jar: 1.24

I notice two odd behaviors on the DVD rip folders which use the %MOVIENAME%/VIDEO_TS structure. First is some movies don't get .properties files created during automatic scan. I cranked up the debug logging on the importer and noticed the following for "National Lampoon's Vacation/VIDEO_TS" which was not properly getting metadata.

2009-07-07 11:48:41,979 DEBUG [LibraryImporter] org.jdna.media.util.FileNameUtils (FileNameUtils.java:107) - Created Query: SearchQuery; Type: MOVIE; 
2009-07-07 11:48:41,979 DEBUG [LibraryImporter] org.jdna.media.metadata.impl.composite.CompositeMetadataProvider (CompositeMetadataProvider.java:90) - Searching using composite provider: dvdprofiler-2
2009-07-07 11:48:42,088 DEBUG [LibraryImporter] org.jdna.media.metadata.impl.dvdproflocal.LocalDVDProfMetaDataProvider (LocalDVDProfMetaDataProvider.java:84) - DVD Profiler Xml: \\sageserver\SageTV\DVD Profiler\Collection.xml
2009-07-07 11:48:42,370 DEBUG [LibraryImporter] org.jdna.media.metadata.impl.dvdproflocal.LocalDVDProfMetaDataProvider (LocalDVDProfMetaDataProvider.java:108) - Rebuilding DVD Profiler Indexes....
2009-07-07 11:48:42,370 DEBUG [LibraryImporter] org.jdna.media.metadata.impl.dvdproflocal.LocalMovieIndex (LocalMovieIndex.java:124) - Deleting All Currently indexed documents.
2009-07-07 11:48:42,370 DEBUG [LibraryImporter] org.jdna.media.metadata.impl.dvdproflocal.LocalMovieIndex (LocalMovieIndex.java:64) - Opening Lucene Index: C:\Program Files\SageTV\SageTV\cache\indexDVDProfLocal
2009-07-07 11:48:42,510 DEBUG [LibraryImporter] org.jdna.media.metadata.impl.dvdproflocal.LocalMovieIndex (LocalMovieIndex.java:134) - Finished Deleting documents.
2009-07-07 11:48:42,541 DEBUG [LibraryImporter] org.jdna.media.metadata.impl.dvdproflocal.LocalMovieIndex (LocalMovieIndex.java:71) - Indexing Movie: Barbie As Rapunzel; date: 2002; id: 012236129509
[...snip...]
2009-07-07 11:48:42,620 DEBUG [LibraryImporter] org.jdna.media.metadata.impl.dvdproflocal.LocalMovieIndex (LocalMovieIndex.java:71) - Indexing Movie: National Lampoon's European Vacation; date: 1985; id: 085391152125
[...snip...]
2009-07-07 11:48:42,620 DEBUG [LibraryImporter] org.jdna.media.metadata.impl.dvdproflocal.LocalMovieIndex (LocalMovieIndex.java:71) - Indexing Movie: National Lampoon's Vacation; date: 1983; id: 085392753529
2009-07-07 11:48:42,620 DEBUG [LibraryImporter] org.jdna.media.metadata.impl.dvdproflocal.LocalMovieIndex (LocalMovieIndex.java:71) - Indexing Movie: National Lampoon's Christmas Vacation; date: 1989; id: 085392753628
[...snip...]
2009-07-07 11:48:42,620 DEBUG [LibraryImporter] org.jdna.media.metadata.impl.dvdproflocal.LocalMovieIndex (LocalMovieIndex.java:71) - Indexing Movie: Vegas Vacation; date: 1997; id: 085392885725
[...snip...]
2009-07-07 11:48:42,713 DEBUG [LibraryImporter] org.jdna.persistence.PropertiesPersistence (PropertiesPersistence.java:248) - Saving: org.jdna.media.metadata.impl.dvdproflocal.DVDProfilerLocalConfiguration
2009-07-07 11:48:42,745 DEBUG [LibraryImporter] org.jdna.util.Similarity (Similarity.java:111) - Similarity Score: [National Lampoon's Vacation][National Lampoon's European Vacation]=[0.8627451]
2009-07-07 11:48:42,760 DEBUG [LibraryImporter] org.jdna.util.Similarity (Similarity.java:111) - Similarity Score: [National Lampoon's Vacation][National Lampoon's European Vacation]=[0.8627451]
2009-07-07 11:48:42,760 DEBUG [LibraryImporter] org.jdna.util.Similarity (Similarity.java:111) - Similarity Score: [National Lampoon's Vacation][National Lampoon's Christmas Vacation]=[0.84615386]
2009-07-07 11:48:42,760 DEBUG [LibraryImporter] org.jdna.util.Similarity (Similarity.java:111) - Similarity Score: [National Lampoon's Vacation][National Lampoon's Christmas Vacation]=[0.84615386]
2009-07-07 11:48:42,760 DEBUG [LibraryImporter] org.jdna.util.Similarity (Similarity.java:111) - Similarity Score: [National Lampoon's Vacation][Vegas Vacation]=[0.42424244]
2009-07-07 11:48:42,760 DEBUG [LibraryImporter] org.jdna.util.Similarity (Similarity.java:111) - Similarity Score: [National Lampoon's Vacation][Vegas Vacation]=[0.42424244]
2009-07-07 11:48:42,760 DEBUG [LibraryImporter] org.jdna.media.metadata.impl.composite.MetadataProviderContainer (MetadataProviderContainer.java:50) - Not a good search using provider: dvdprofiler-2; Query: SearchQuery; Type: MOVIE; TITLE:National Lampoon's Vacation;; will try another one.
2009-07-07 11:48:42,760 DEBUG [LibraryImporter] org.jdna.media.metadata.impl.composite.MetadataProviderContainer (MetadataProviderContainer.java:59) - Provider: tvdb cannot handle query: SearchQuery; Type: MOVIE; TITLE:National Lampoon's Vacation;
2009-07-07 11:48:42,791 DEBUG [LibraryImporter] org.jdna.media.metadata.impl.themoviedb.TheMovieDBSearchParser (TheMovieDBSearchParser.java:63) - TheMovieDB SearchQuery Url: CachedUrl: http://api.themoviedb.org/2.0/Movie.search?title=National+Lampoon%27s+Vacation&api_key=d4ad46ee51d364386b6cf3b580fb5d8c; UrlId: 4e05a428257f3887b18303b7fa5fa122
2009-07-07 11:48:42,791  WARN [LibraryImporter] org.jdna.media.metadata.impl.themoviedb.TheMovieDBSearchParser (TheMovieDBSearchParser.java:91) - TheMovieDB Item didn't contain a title: Your query didn't return any results.
2009-07-07 11:48:42,791 DEBUG [LibraryImporter] org.jdna.media.metadata.impl.composite.MetadataProviderContainer (MetadataProviderContainer.java:50) - Not a good search using provider: themoviedb.org; Query: SearchQuery; Type: MOVIE; TITLE:National Lampoon's Vacation;; will try another one.

Obviously there is something wonky going on since there is an exact match for the title in my DVDProfiler xml file. In addition the imdb search when manually entered on the does return the correct movie as the first popular title. I also noticed that all movies missing properties files in the automatic scan where after this movie alphabetically. It seems this failure causes the scan to stop.

The second odd behavior I notices was if I did a rescan library from inside the SageMC UI the behavior was different. When the automatic updater fired off I would get %MOVIENAME%.properties files in the parent folder of the %MOVIENAME% directory (\\sageserver\DVDs\National Lampoon's Vacation\VIDEO_TS = \\sageserver\DVDs\National Lampoon's Vacation.properties). When you trigger a manual scan you get a VIDEO_TS.properties file inside the movie folder (\\sageserver\DVDs\National Lampoon's Vacation\VIDEO_TS = \\sageserver\DVDs\National Lampoon's Vacataion\VIDEO_TS.properties). It seemed to catch all of these movies even if one was in error.

When I ran the command line tool against this directory then it prompted me for the National Lampoon's Vacation (even though correct option was listed as 1.000) and properly cycled through to the rest of the movies after that.

I hope this helps track down some of the problems that have been articulated in this forum. I realize some of the posts where not the most cordial but I do believe there are some real bugs with the automatic scan at least in 2.3.1.