Utility: Batch Metadata Importer 3.x

[Bacon2002]

Hi,

I just wanted to say thanks, great program.

I got the program working, I moved everything to a new folder C:\Metadata\ and wrote a small batch file, it works great.

Out of interest is there anyway to automatically select "n" when no match is found.

Thanks again,

David

PS
for anyone using Vista 64 you need to declare the path for the main Java files. You do this in Control Panel > System > advanced system setting > advanced > environmential variables > system variables > path > at the end of the existing paths simply add ";" and the path of you main Java folder

[bcjenkins]

Originally Posted by stuckless

You are way too optimistic No, unfortunately, I need to better understand the "rules" around how to name and store fanart before i can create that feature. And, I'll need to install SageMC on at least my dev client, so that I can test if it works. And honestly, I haven't looked into it yet.

Rats! No rush, how's tomorrow? Seriously, thanks for a great plugin. I appreciate the work.

B

[stuckless]

Out of interest is there anyway to automatically select "n" when no match is found.

NOt yet, but I'm currently restructuring the main processor so that all files that need to be manually searched will be queued up and processed at the end. When that change is made, the default will be to simply skip those files, unless --prompt is passed, and in which case, it would work like it does now, in that it would prompt you to enter a new title or select an item from the list.

This change should allow you to run the tool in a non-interactive fashion.

Here are some other things that are on the list, that may make it in between now and xmas depending on the time....

• Modify the Composite provider to support ALL properties in the fieldsFromSearchProvider (currently it only support about 5 or 6 core ones)
• Allow All NON-Imdb providers to take part in a composite provider.
• Allow tool to run for downloading images only. ie, don't update metadata, just update images.
• FanArt - I doubt this would make it before xmas, but I'll put it on the list.

[mp328]

is it possible to add a log file to be created??

i noticed when i did my library some movies were chosen wrong by default and i saw them being wrong in the cmd window while it was going thru, and the cmd window only keeps a certain length of text in the window so i could not go back to select it.

what im trying to do, is once the tool is finished, i would just like to browse the log file and see the obvious ones that were picked wrong and adjust them

[stuckless]

i'll put it on the list. There is a log file created now, metadataupdater.log, but it probably contains a lot of information.

[mrdrbdt]

Originally Posted by mp328

i noticed when i did my library some movies were chosen wrong by default and i saw them being wrong in the cmd window while it was going thru, and the cmd window only keeps a certain length of text in the window so i could not go back to select it.

You can change this. Click on the cmd prompt icon in the top left of the cmd window, then choose properties. On the "Layout" tab, you can change the screen buffers and sizes. I typically use 140,300,140,50 from top to bottom.

stuckless-the tool is working well! In case you were interested, I was able to create meta-data for 1200 movies in just shy of an hour and a half. Sure speeds up the process!

[stuckless]

There is new release of the Batch Metadata Tools 1.13.
Not much new, just restructured a little. One point of interest, is that there is now --prompt option that you can set to false, if you can't want to be asked to select/search for metadata.


Now for some new....
I've also released a SageTV MediFileMetadataParser plugin/implementation. This is basically the same project, just repackaged to run within the SageTV environment as a MediaFileMetadataParser plugin

This is probably my last update for this year.... so Merry xmas... and enjoy!

[h2guy]

Thanks Stuckless, I was going to get my collection up to to date today and you released this right before I started.

[bcjenkins]

Sean,

The tool doesn't like the Wonder Pets!

Here's the entry from my Collection.xml DVD Profiler:

  <DVD>
    <ProfileTimestamp>1899-12-30</ProfileTimestamp>
    <ID>097368923140</ID>
    <MediaTypes>
      <DVD>True</DVD>
      <HDDVD>False</HDDVD>
      <BluRay>False</BluRay>
    </MediaTypes>
    <UPC>097368-923140</UPC>
    <CollectionNumber>392</CollectionNumber>
    <CollectionType>Owned</CollectionType>
    <Title>Wonder Pets - Save the Nutcracker</Title>
    <DistTrait/>
    <OriginalTitle/>
    <CountryOfOrigin/>
    <ProductionYear>0</ProductionYear>
    <RunningTime>0</RunningTime>
    <RatingSystem>Film</RatingSystem>
    <Rating/>
    <RatingAge>0</RatingAge>
    <RatingVariant>0</RatingVariant>
    <CaseType/>
    <Genres/>
    <Regions/>
    <Format>
      <FormatAspectRatio/>
      <FormatVideoStandard>NTSC</FormatVideoStandard>
      <FormatLetterBox>False</FormatLetterBox>
      <FormatPanAndScan>False</FormatPanAndScan>
      <FormatFullFrame>False</FormatFullFrame>
      <Format16X9>False</Format16X9>
      <FormatDualSided>False</FormatDualSided>
      <FormatDualLayered>False</FormatDualLayered>
    </Format>
    <Features>
      <FeatureSceneAccess>False</FeatureSceneAccess>
      <FeatureCommentary>False</FeatureCommentary>
      <FeatureTrailer>False</FeatureTrailer>
      <FeaturePhotoGallery>False</FeaturePhotoGallery>
      <FeatureDeletedScenes>False</FeatureDeletedScenes>
      <FeatureMakingOf>False</FeatureMakingOf>
      <FeatureProductionNotes>False</FeatureProductionNotes>
      <FeatureGame>False</FeatureGame>
      <FeatureDVDROMContent>False</FeatureDVDROMContent>
      <FeatureMultiAngle>False</FeatureMultiAngle>
      <FeatureMusicVideos>False</FeatureMusicVideos>
      <FeatureInterviews>False</FeatureInterviews>
      <FeatureStoryboardComparisons>False</FeatureStoryboardComparisons>
      <FeatureOuttakes>False</FeatureOuttakes>
      <FeatureClosedCaptioned>False</FeatureClosedCaptioned>
      <FeatureTHXCertified>False</FeatureTHXCertified>
      <OtherFeatures/>
    </Features>
    <Studios/>
    <MediaPublisher/>
    <Audio/>
    <Subtitles/>
    <SRP DenominationType="USD" DenominationDesc="United States (Dollar)" FormattedValue="$0.00">0</SRP>
    <Actors/>
    <Credits/>
    <Overview/>
    <EasterEggs/>
    <Discs/>
    <SortTitle>Wonder Pets - Save the Nutcracker</SortTitle>
    <LastEdited>2008-12-22T00:16:27.000Z</LastEdited>
    <WishPriority>0</WishPriority>
    <PurchaseInfo>
      <PurchasePrice DenominationType="USD" DenominationDesc="United States (Dollar)" FormattedValue="$0.00">0</PurchasePrice>
      <PurchasePlace/>
      <PurchasePlaceType/>
      <PurchasePlaceWebsite/>
      <PurchaseDate>2008-12-21</PurchaseDate>
    </PurchaseInfo>
    <Review Film="0" Video="0" Audio="0" Extras="0"/>
    <Events/>
    <BoxSet>
      <Parent/>
      <Contents/>
    </BoxSet>
    <LoanInfo>
      <Loaned>False</Loaned>
    </LoanInfo>
    <Notes/>
    <Tags/>
    <Locks>
      <Entire>False</Entire>
      <Covers>False</Covers>
      <Title>False</Title>
      <MediaType>False</MediaType>
      <Overview>False</Overview>
      <Regions>False</Regions>
      <Genres>False</Genres>
      <SRP>False</SRP>
      <Studios>False</Studios>
      <DiscInformation>False</DiscInformation>
      <Cast>False</Cast>
      <Crew>False</Crew>
      <Features>False</Features>
      <AudioTracks>False</AudioTracks>
      <Subtitles>False</Subtitles>
      <EasterEggs>False</EasterEggs>
      <RunningTime>False</RunningTime>
      <ReleaseDate>False</ReleaseDate>
      <ProductionYear>False</ProductionYear>
      <CaseType>False</CaseType>
      <VideoFormats>False</VideoFormats>
      <Rating>False</Rating>
    </Locks>
  </DVD>

It is a manual add as I just care about the cover art, there isn't any further data. The file exists where it should, but when I run the tool it returns no match for this movie. All others function as needed.

TIA

B

[stuckless]

Originally Posted by bcjenkins

Sean,

The tool doesn't like the Wonder Pets!

It is a manual add as I just care about the cover art, there isn't any further data. The file exists where it should, but when I run the tool it returns no match for this movie. All others function as needed.

TIA

B

I'll try to explain how the magic works... (or in the case, doesn't work )

The tool reads all you dvd profiler data and indexes it using an indexer, Lucene. Once the index is built, then the program will take the filename from the filesystem, clean it, and use that as the search tokens.

If a single entry is returned then it is used. If multiple entries are returned, then one an entry with a "score" of 1.0 is used. (scoring is a lucene term to indicate the weight of the search result, ranging from 0.0f to 1.0f).

In your case, are you saying that the xml fails to parse, and therefore it doesn't get added to the index, or that the filename of the movie isn't found in the index.

Could you send me your filename as well. I suspect that in your case, either you have several "wonder pets" movies and the indexer can't find a one with a score of 1.0.

[bcjenkins]

Actually only one Wonder Pets movie. Wonder Pets - Save the Nutcracker is the dir name.

B

[stuckless]

Originally Posted by bcjenkins

Actually only one Wonder Pets movie. Wonder Pets - Save the Nutcracker is the dir name.

B

I'll try to take a look at this sometime during xmas. If you could zip up your .cache/index dir and send it to me. Not sure if you have my email address or not. If not, then PM me. I can take a look at your lucene index and see why that name is causing issues.

If you are savy enough, you can open your lucene index using the Lucene index toolkit tool (Luke). Using Luke, you can open your index, and perform searches. One thought that I had is that perhaps my Search code is NOT removing the '-', which is a reserved token in lucene... it's a NOT operator, so that my be the issue. But if you use luke, you can test this. Or, send me your index, and I'll test it

[bcjenkins]

As a note, I frequently use - as a separator in my DVD names. Also, your process did display the name as Wonder Pets Save the Nutcracker

Notice the extra spaces. The naming convention isn't unusual, just something about this entry

B

[mrdrbdt]

Originally Posted by stuckless

There is new release of the Batch Metadata Tools 1.13.
Not much new, just restructured a little. One point of interest, is that there is now --prompt option that you can set to false, if you can't want to be asked to select/search for metadata.


Now for some new....
I've also released a SageTV MediFileMetadataParser plugin/implementation. This is basically the same project, just repackaged to run within the SageTV environment as a MediaFileMetadataParser plugin

This is probably my last update for this year.... so Merry xmas... and enjoy!

Sweet! I really like saving the mismatches till the end. All seems to work well for me on this release.

I only have one comment, what happened to the "auto" option? The new "prompt" option is a bit ambiguous. I mean, I figured it out eventually, but only because I read your release in the forum. The bit on google code about the options doesn't clearly state what the option does.

Originally Posted by From GoogleCode Page

--prompt : When a media file cannot be updated, then prompt to enter a title to search on. (default true)

This seemed to be equivalent to "auto=on". And to me, setting prompt=false would be the equivalent of just skipping the file.

No biggie, for sure.

Edit: After playing around some more, prompt=false actually does seem to just skip the files not automatched.

[stuckless]

I only have one comment, what happened to the "auto" option? The new "prompt" option is a bit ambiguous. I mean, I figured it out eventually, but only because I read your release in the forum. The bit on google code about the options doesn't clearly state what the option does.

I agree that the auto and prompt are somewhat ambiguous. Basically after I refactored the main processing function of the program, I realized that I had not accounted for the --auto flag, so I dropped it and hoped that no one would notice

--auto was meant to run so that if auto was false, then it would prompt you for each and every title. If you want, I can add it back in.

--prompt was added in response to an earlier posting where someone asked about whether the program would run and NOT prompt when it did not find a match.

I'm open to better named command options, they are easy to change

[mrdrbdt]

Originally Posted by stuckless

--auto was meant to run so that if auto was false, then it would prompt you for each and every title. If you want, I can add it back in.

--prompt was added in response to an earlier posting where someone asked about whether the program would run and NOT prompt when it did not find a match.

Ah, I see now! I think the prompt option is aptly named. I just mistook it as a replacement for auto. Perhaps the description on the googlecode page for prompt should just read "Set true to enable prompting on questionable matches, or false to skip questionable matches all together".

If it's not too much trouble, I would like having the auto option back. I use auto=false for rescanning my collection after subsequent additions. To be more clear, I ran it once with auto=true in the beginning to get something down for the 1200 movies in the collection. But now, I only use auto=false as I add movies a few at a time. The end result is that it will scan through and skip all the movies that already have upto date data, and ask me about the 4 or 5 that I just added. The automatch you have set up is pretty good, but I have some movies it chokes on. Particularly, remakes. It's impossible to tell old from new without just looking at the movie.

On a separate note, or perhaps a tip, I've started renaming some of the file names to match the imdb index (if that's what it is). I noticed when you goto a movie on imdb you get an address where the last bit looks something like "tt0407965", and if I rename the movie to tt0407965.avi I get a perfect match everytime. This was really helpful on the Family Guy Blue Harvest movie, which btw, you can't even find on imdb. I had to find it using a google search with site:imdb.com.

[S_M_E]

Originally Posted by mrdrbdt

The automatch you have set up is pretty good, but I have some movies it chokes on. Particularly, remakes. It's impossible to tell old from new without just looking at the movie.

On a separate note, or perhaps a tip, I've started renaming some of the file names to match the imdb index (if that's what it is). I noticed when you goto a movie on imdb you get an address where the last bit looks something like "tt0407965", and if I rename the movie to tt0407965.avi I get a perfect match everytime. This was really helpful on the Family Guy Blue Harvest movie, which btw, you can't even find on imdb. I had to find it using a google search with site:imdb.com.

I add the year to the name of the end of the file so I can tell which remake is which. It would be nice if the BMI would append the date to the .properties file name so I can tell in Sage too. Perhaps an optional metadata.properties item telling it to append either "- XXXX", " [XXX]" or " (XXXX)" to the name in the .properties file and/or use that expression from the file name for look ups. It might make more exact matches if it read the date from the file too

I never thought of using the imdb "code" to get a particular match, I wonder if we could get stuckless to program that in the matching query too? Instead of just
"[q=quit, n=next (default), ##=use result ##, TITLE=Search TITLE]" perhaps we could enter the "tt" code? Then there would be no need to rename files either.

[stuckless]

If it's not too much trouble, I would like having the auto option back

ok

I never thought of using the imdb "code" to get a particular match, I wonder if we could get stuckless to program that in the matching query too? Instead of just
"[q=quit, n=next (default), ##=use result ##, TITLE=Search TITLE]" perhaps we could enter the "tt" code? Then there would be no need to rename files either.

If you enter the tt## code as the search string, then that should already work.

I add the year to the name of the end of the file so I can tell which remake is which. It would be nice if the BMI would append the date to the .properties file name so I can tell in Sage too. Perhaps an optional metadata.properties item telling it to append either "- XXXX", " [XXX]" or " (XXXX)" to the name in the .properties file and/or use that expression from the file name for look ups. It might make more exact matches if it read the date from the file too

I you are asking that the the Title field actually show like "Title (Year)" in Sage, then you can already do that, by specifying your own "Title" mask in your metadata.properties file

/sageMetadata/multiCDTitleMask=${Title} (${Year}) Disc ${_disc}
/sageMetadata/titleMask=${Title} (${Year})

[S_M_E]

I had no idea that I could specify the "tt#" at the prompt already. I had problems with one file so I accepted the wrong name then manually edited the .properties files and used --update. I guess I should have tried it, it would have been easier.

Not only title (year) in Sage but perhaps BMI could use the (year) in the file name to do it's initial search too. That way, when it searches for a title with the same name and multiple years (like remakes), it doesn't need to ask because it's in the file name already. Good to know that it'll add the year in Sage already though.

[stuckless]

Not only title (year) in Sage but perhaps BMI could use the (year) in the file name to do it's initial search too. That way, when it searches for a title with the same name and multiple years (like remakes), it doesn't need to ask because it's in the file name already. Good to know that it'll add the year in Sage already though.

Currently the search works by removing all non alphanumeric characters, and then it removes certain words (such as TS, CAM, DVDRip, etc), so a filename that has a year in it should still search using the year. The --agressive switch is actually used so that if a match is NOT found, then it will then attempt to remove the year from the title and search again. So by default, if a filename has a Year in the filename, then it will become part of the search criteria. (hope that clears it up)