Utility: Batch Metadata Importer 3.x

[stuckless]

Originally Posted by r00st3r

I don't know if this makes sense, or helps pin point where the issue is. I think the problem is when the web gui is used to get the metadata, it breaks the File System listings in SAGE MC.

That is the case. BMT has several ways to the metadata, include direct writing to the wiz.bin via the AddShow(), which used for the On Demand and WebUI scanning. The Automatic Plugin doesn't write directly to the Sage wiz.bin, since the plugin just passes back the properties to sage, so there's no need for bmt to call AddShow().

The issue comes from the way that BMT stores the Show Title. I guess when Sage imports a file, it puts the relative path of the media file in the Show Title field. BMT put's the Show's actual title in the Show Title field. (There is no other place for bmt to put the Show Title). So this variance in how BMT creates the new metadata is what is causing the issues, since SageMC relies on the relative path of the file in order to build the file system navigation. MeInMaui and I are working on a solution.

The default stv uses a slightly different approach for building the filesystem view, which is why the issue is not apparent in the default stv.

[Fuzzy]

Originally Posted by stuckless

I didn't test this from within the sage application, but I did quickly test it from the command line using...

java -jar MetadataTool.jar --offline --metadataOnly "SpecialAgentOso-DrOffLicensetoDress-2089020-0.mpg"

and it did lookup the file and find it correctly... the .properties were...

#Generator: Batch Metadata Tools (3.1); MediaFile: file:/home/seans/DevelopmentProjects/workspaces/sage/MovieMetadataUpdater/target/sage/SpecialAgentOso-DrOffLicensetoDress-2089020-0.mpg
#Sun Sep 13 08:38:11 EDT 2009
Description=Dr. Off\: A special alert is called as Oso searches for a hidden tunnel in the city. He teaches a girl, Adison, how to conserve power and water. They must turn off the unused lights, TVs and water faucets before family game night. License to Dress\: While Oso dresses into his scuba gear for a training assignment, he receives a special alert. He helps a boy, Frank, dress for school and Oso learns to dress by himself as well. Can Oso and Frank get dressed properly before Frank leaves for school?
EpisodeNumber=11
EpisodeTitle=Dr. Off; License to Dress
Genre=Animation
MediaProviderDataID=tvdb\:112931
MediaTitle=Special Agent Oso
MediaType=TV
OriginalAirDate=2009-04-18
Rated=TV-Y
RunningTime=1500000
SeasonNumber=1
Title=Special Agent Oso - S01E11 - Dr. Off; License to Dress
Year=2009
x-MetadataProviderDataUrl=112931;;;;EPISODE_TITLE\=Dr+Off+Licenseto+Dress
x-MetadataProviderId=tvdb
x-ser-Genres=Animation;Children

Strange, this is not what I get (or at least, I don't know how to use the cmdline. When I run that same command line, I get:

Batch MetaData Tools (3.1)
Skipping: file:/C:/Program%20Files/SageTV/SageTV/SpecialAgentOso-DrOffLicensetoDress-2089020-0.mpg; Message: Nothing Found for Query: SearchQuery; Type: TV; TITLE:Special Agent Oso;EPISODE_TITLE:Dr Off Licenseto Dress;


Search Results: Special Agent Oso


[q=quit, n=next (default), ##=use result ##, TITLE=Search TITLE]
>

Also, the WebUI does not find it in scan, and even manually searching with the Find Metadata option, selecting TV and entering Special Agent Oso in the title description returns "0 results".

Maybe I should try redownloading BMT? I don't know.. most other shows seem to scan down fine.

[stuckless]

Originally Posted by Fuzzy

Maybe I should try redownloading BMT? I don't know.. most other shows seem to scan down fine.

In the sage home dir, delete the cache/url/ directory and then try this again. I'm thinking that bmt may be using a older cached copy of the xml metadata from tvdb. And, make sure that tvdb is in your list of providers (it is by default, unless you've manually removed it)

[razrsharpe]

Originally Posted by stuckless

I think that's the route that I'd like to take, but instead of having a separate file, i think what I'd do is have a single file (probably xml) that will allow you to set the provider information, and specific properties, such as Title, etc. I'd also provider a web ui for editing that information.

sounds like the most robust, easiest to maintain (from a user perspective) solution...

Originally Posted by stuckless

The web ui provides editor dialogs for certain types of fields. The Description has a "..." button beside it, which pops up a multi-line text box for entry. I could add a "..." to the show title as well, or simply make the default text box length a little larger.

ya i saw that (the "..." to bring up the muliline input box)... is there anyway that multiline text box could be part of the main page and not a popup? I think it would be cleaner... I know I'm being nitpicky

Show title is a little bit of a misnomer... it's the MediaTitle field from the custom metadata fields. I think it would help, if I put a help on each field, explaining where/what each field does and where it's stored.

A help/popup would help out a lot.. then people know what they are modifying and where the information is stored

Actually the web ui doesn't use the properties files as all. It will sage to the properties files, but it actually builds the metadata that you see from the Sage Show/Airing/MediaFile objects and the Custom Metadata fields. Currently the code that reads that metadata doesn't read metadata from the show object that would exist in the custom metadata fields. For that reason, in your case, the Show Title (MediaTitle) and Episode Title (EpisodeTitle) and Media Type (MediaType) fields are blank, because they were never scanned and updated. I've added a bug and I'll fix the web ui to pull that extra metadata from the show object if it exists.

So the first time i read this it made sense, then i reread it and now it doesnt make sense. Let me try to explain again From the screenshot that i posted earlier. BMT has never scanned that file. However, metadata exists for that episode in the show object that sage created for the recording. Specifically GetShowEpisode (Episode Title), and Release Date (GetOriginalAiringDate). This information (I would think) should be displayed with the browse feature. Maybe this is the bug you are referring too and i just confused myself.... it's been known to happen

[tmiranda]

Sean,

Iran a scan on all my files and now some of them have the wrong fanart associated with them. Unfortunately the shows effected are my spouse's so the WAF is dropping.. (Some of her home and garden shows now have SpongeBob fanart, which is particularly amusing because she HATEs SpongeBob and I LOVE spongeBob.)

How do I "unassociate" fanart for a particular show?

[dbone1026]

Originally Posted by tmiranda

Sean,

Iran a scan on all my files and now some of them have the wrong fanart associated with them. Unfortunately the shows effected are my spouse's so the WAF is dropping.. (Some of her home and garden shows now have SpongeBob fanart, which is particularly amusing because she HATEs SpongeBob and I LOVE spongeBob.)

How do I "unassociate" fanart for a particular show?

Spongebob isn't that bad. My home videos (i.e family videos) got Camcorder Capers (i.e. porn movies) pulled in as the covers. Fortunately the wife had a good sense of humor

[stuckless]

Originally Posted by razrsharpe

So the first time i read this it made sense, then i reread it and now it doesnt make sense. Let me try to explain again From the screenshot that i posted earlier. BMT has never scanned that file. However, metadata exists for that episode in the show object that sage created for the recording. Specifically GetShowEpisode (Episode Title), and Release Date (GetOriginalAiringDate). This information (I would think) should be displayed with the browse feature. Maybe this is the bug you are referring too and i just confused myself.... it's been known to happen

That's ok... after I re-read with all the missing words and typos, it didn't make a lot sense to me either Bottom line... OriginalAirDate, EpisodeTitle, actually comes from the custom metadata fields and NOT the sage show object. Why? because there was no way to populate those fields from the .properties, so they were added to the custom metadata fields. Now that I've written code to write directly to the wiz.bin via the sage show object, there is a duplication of fields. BMT uses the custom field, and doesn't look at the duplicated fields in the sage show object (for reading). The change the I need to make for bmt is to read all metadata from the show object and the overlay the metadata from the custom metadata, if they exist. Hope that better explains it.

[razrsharpe]

Originally Posted by stuckless

The change the I need to make for bmt is to read all metadata from the show object and the overlay the metadata from the custom metadata, if they exist. Hope that better explains it.

yup that explains it perfectly and is exactly what i was talking about

[stuckless]

Originally Posted by tmiranda

Sean,

Iran a scan on all my files and now some of them have the wrong fanart associated with them. Unfortunately the shows effected are my spouse's so the WAF is dropping.. (Some of her home and garden shows now have SpongeBob fanart, which is particularly amusing because she HATEs SpongeBob and I LOVE spongeBob.)

How do I "unassociate" fanart for a particular show?

Fanart is tied to the "Show Title" in the web UI. Changing that changes the folder in which the phoenix apis will look for fanart. If the Show Title looks correct in the web ui, then do a manual check to make sure that the fanart folder for the given item (can be found by checking the web ui), doesn't contain the incorrect fanart.

@dbone1026 - your comment made me chuckle... Good thing you noticed that before you decided to pull up sagetv and show your friends/family your home movies

[lovingHDTV]

I just installed and now have a question.

I'm in the Web configuration, setup my central fanart area, then hit scan sage recordings. It never comes back with anything. I also tried DVDs with the same results.

Anyone have any idea how long it should take for the scan to complete?

I have 183 DVDs and about 100 recordings.

thanks,
dave

[KJake]

Originally Posted by stuckless

Could you post a sample filename that I can use for testing. it should have prompted for the first time, but then it should have stored that selection and never prompt again.

example:

\TV\Stargate SG-1\Season 09\Stargate SG-1 - S9E15.avi

metadata.properties has this line in it
bmt/metadataUpdater/rememberSelectedSearches=true

I even tried adding this to my metadata-titles.properties
Stargate\ SG\ 1=tvdb\:72449
Stargate\ SG-1=tvdb\:72449
Stargate\ SG1=tvdb\:72449

I've deleted everything in my /cache/ dir multiple times to make sure it is actually searching. It seems like metadata-titles.properties is ignored entirely. Even having that fixed would be an OK workaround, but it prompts for each file irregardless of what I searched for last time.

Skipping: /tv/Stargate%20SG-1/Season%2009/Stargate%20SG-1%20-%20S9E15.avi; Message: Nothing Found for Query: SearchQuery; Type: TV; DISC:;TITLE:Stargate SG 1;SEASON:09;EPISODE:15;

[stuckless]

Originally Posted by lovingHDTV

I just installed and now have a question.

I'm in the Web configuration, setup my central fanart area, then hit scan sage recordings. It never comes back with anything. I also tried DVDs with the same results.

Anyone have any idea how long it should take for the scan to complete?

I have 183 DVDs and about 100 recordings.

thanks,
dave

Dave, the scan may be counter intuitive to some... basically if you do a scan, and you don't get any videos to show up, then that's a good thing, because it means that bmt scanned the vidoes without issue. If you scan, and you start seeing videos showing up, it means that bmt could not find a match and it returns it, so that you can begin manually scanning/editing it.

[stuckless]

Originally Posted by KJake

Skipping: /tv/Stargate%20SG-1/Season%2009/Stargate%20SG-1%20-%20S9E15.avi; Message: Nothing Found for Query: SearchQuery; Type: TV; DISC:;TITLE:Stargate SG 1;SEASON:09;EPISODE:15;

The good news, it's not your fault. The bad news is that it's really an issue with thetvdb.com/bmt. For example the thetvdb is very picky about it's searches; ie, "stargate sg 1" returns nothing, yet, "stargate sg-1" return 1 hit and "stargate" returns several hits.

The second issue is a misunderstanding of how the metadata-titles works. basically the metadata-titles will only be used in the event that there are multiple hits for a search result and then the metadata-titles is used to norrow the result. BMT does not use the metadata-titles as an override, and if there are no results, then metadata-titles is never used at all. (you could argue that this is a bug, but there is a technical reason why it's not used). It's my intention to redo the metadata-titles, and add the support to make it work as you think it does.

[lovingHDTV]

Originally Posted by stuckless

Dave, the scan may be counter intuitive to some... basically if you do a scan, and you don't get any videos to show up, then that's a good thing, because it means that bmt scanned the vidoes without issue. If you scan, and you start seeing videos showing up, it means that bmt could not find a match and it returns it, so that you can begin manually scanning/editing it.

For me the little swirling icon never goes away. Does that mean it is still scanning or that it completed?

[KJake]

Originally Posted by stuckless

The good news, it's not your fault. The bad news is that it's really an issue with thetvdb.com/bmt. For example the thetvdb is very picky about it's searches; ie, "stargate sg 1" returns nothing, yet, "stargate sg-1" return 1 hit and "stargate" returns several hits.

The second issue is a misunderstanding of how the metadata-titles works. basically the metadata-titles will only be used in the event that there are multiple hits for a search result and then the metadata-titles is used to norrow the result. BMT does not use the metadata-titles as an override, and if there are no results, then metadata-titles is never used at all. (you could argue that this is a bug, but there is a technical reason why it's not used). It's my intention to redo the metadata-titles, and add the support to make it work as you think it does.

okey dokey - any ideas why the command-line isn't saving the search? here's my .properties file.

Sage Provider Implementation: org.jdna.metadataupdater.BMTSageAPIProvider; org.jdna.metadataupdater.BMTSageAPIProvider[BMT Command Line Impl]
Batch MetaData Tools (3.0-beta-3)
# -- Group: null
# -- Group: Application Configuration
# Default Folder to Scan when Doing an interactive scan from the Web UI
bmt/metadataUpdater/guiFolderToScan=null

# Number of results to display in the search results
bmt/metadataUpdater/searchResultDisplaySize=10

# Once Scanning is Complete, notify SageTV of the changes. (commandline only)
bmt/metadataUpdater/refreshSageTV=false

# Automatically update by auto selecting 'best' search result. (commandline only)
bmt/metadataUpdater/automaticUpdate=true

# Recursively process sub folders
bmt/metadataUpdater/recurseFolders=true

# Only process files that is missing metadata. (commandline only)
bmt/metadataUpdater/processMissingMetadataOnly=false

# Enable Fanart downloading
phoenix/mediametadata/fanartEnabled=true

# Location of the central fanart folder
phoenix/mediametadata/fanartCentralFolder=\\SAGESERVER\MediaBackgrounds

# Remember a search result when you select it form a list, for use in later searches.
bmt/metadataUpdater/rememberSelectedSearches=true

# -- Group: Metadata Configuration
# Default class name for storing metadata
bmt/metadata/persistenceClass=org.jdna.media.metadata.impl.sage.SageTVPropertiesWithCentralFanartPersistence

# Comma separated list of known metadata providers (ie, can be used for searching for metadata)
bmt/metadata/videoMetadataProviders=org.jdna.media.metadata.impl.tvdb.TVDBMetadataProvider,org.jdna.media.metadata.impl.imdb.IMDBMetaDataProvider,org.jdna.media.metadata.impl.nielm.NielmIMDBMetaDataProvider,,org.jdna.media.metadata.impl.dvdproflocal.LocalDVDProfMetaDataProvider,org.jdna.media.metadata.impl.themoviedb.TheMovieDBMetadataProvider,org.jdna.media.metadata.impl.mymovies.MyMoviesMetadataProvider

# Comma separated list of words that will be removed from a title when doing a search
bmt/metadata/wordsToClean=1080p,720p,480p,1080i,720i,480i,dvd,dvdrip,cam,ts,tc,scr,screener,dvdscr,xvid,divx,avi,vrs,repack,mallat,proper,dmt,dmd,stv,HDTV,x264

# Default provider id to use (comma separate, if more than 1)
bmt/metadata/defaultProviderId=tvdb,themoviedb.org,themoviedb.org-2,imdb.xml,imdb

# Score which must be exceeded to consider a result a good match
bmt/metadata/goodScoreThreshold=0.9

# If true, then providers will check alternate titles for matches.
bmt/metadata/scoreAlternateTitles=true

# Resize poster to scale using the specified max width
bmt/metadata/posterImageWidth=200

# Resize banner to scale using the specified max width
bmt/metadata/bannerImageWidth=-1

# Resize backgrond to scale using the specified max width
bmt/metadata/backgroundImageWidth=-1

# Maximum # of images within each fanart type to download.
bmt/metadata/maxDownloadableImages=5

# When writing fanart, if this is enabled, an additional poster file will be written that is compatible with the default stv.
bmt/metadata/enableDefaultSTVPosterCompatibility=false

# When importing TV Shows, try to add them into the Sage Recordings.
bmt/metadata/importTVAsRecordedShows=false

# -- Group: Media Configuration
# Stacking Model regex (taken from xbmc group)
bmt/media/stackingModelRegex=[ _\\.-]+(cd|dvd|part|disc)[ _\\.-]*([0-9a-d]+)

# Regular expression for the file extensions that are recognized
bmt/media/videoExtensionsRegex=avi|mpg|divx|mkv|wmv|mov|xvid|ts|m2ts|m4v|mp4|iso

# Regular expression for the directory names to ignore
bmt/media/excludeVideoDirsRegex=null

# -- Group: URL Settings
# Cache Directory where cached URLs are stored
bmt/urlconfiguration/cacheDir=cache/url

# How long, in seconds, URLs remain in the cache
bmt/urlconfiguration/cacheExpiryInSeconds=86400

# URL Factory class name for creating new Url objects
bmt/urlconfiguration/urlFactoryClass=org.jdna.url.CachedUrlFactory

# HTTP User Agent that is sent with each hew http request
bmt/urlconfiguration/httpUserAgent=Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008072820 Firefox/3.0.1

# -- Group: DVD Profiler
# Directory where the local dvd profiler data will be indexed
bmt/dvdprofilerLocal/indexDir=cache/indexDVDProfLocal/

# Local DVD Profiler image directory
bmt/dvdprofilerLocal/imageDir=\\ORION\Shared\Users\Kevin\My Documents\DVD Profiler\Databases\Default\Images

# Local DVD Profiler xml file
bmt/dvdprofilerLocal/xmlFile=\\ORION\Shared\Videos\Collection.xml

# Data/Time the xml file was modified as a long value.  Should not be set directly.
bmt/dvdprofilerLocal/xmlFileLastModified=0

# -- Group: IMDB Configuration
# Preferred height or width of an imdb thumbnail
bmt/imdb/forcedIMDBImageSize=512

# -- Group: MyMovies
# Directory where the MyMovies metadata will be indexed
bmt/mymovies/indexDir=cache/indexMyMovies/

# MyMovies xml file
bmt/mymovies/xmlFile=null

# Data/Time the xml file was modified as a long value.  Should not be set directly.
bmt/mymovies/xmlFileLastModified=0

# -- Group: Sage Persistence
# How each Actor will be written.  This mask will be applied to each actor, and then appeneded into a single line for the properties file. {0} - Actor Name, {1} - Actor Role
bmt/sageMetadata/actorMask={0} -- {1};


# Description Mask (note ${PROP_FIELD_NAME} field names are looked up in the property file)
bmt/sageMetadata/descriptionMask=${Description}
User Rating: ${UserRating}


# Title to use for multi volume vidoes (_disc is disc # 1,2,3,etc)
bmt/sageMetadata/multiCDTitleMask=${Title} Disc ${DiscNumber}

# Title to use for single volume vidoes
bmt/sageMetadata/titleMask=${Title}

# Number genre levels to write.  -1 means all levels.
bmt/sageMetadata/genreLevels=1

# Title mask to use for TV Files
bmt/sageMetadata/tvTitleMask=${Title} - S${SeasonNumber}E${EpisodeNumber} - ${EpisodeTitle}

# Title mask to use for TV on Dvd
bmt/sageMetadata/tvDvdTitleMask=${Title} - S${SeasonNumber}D${DiscNumber}

# Rewrite titles so that 'A Big Adventure' becomes 'Big Adventure, A'
bmt/sageMetadata/rewriteTitle=false

# A Search/Replace Regex containing 2 groups that will rewrite the title
bmt/sageMetadata/rewriteTitleRegex=^(in\s+the|in\s+a|i\s+am|in|the|a|an|i|am),?\s+(.*)

# -- Group: Automatic Plugin Configuration
# Enable/Disable the Automatic Plugin (does not remove it)
bmt/plugin/enabled=true

# Overwrite Existing Metadata Configuration
bmt/plugin/overwriteMetadata=true

# Overwrite Existing Fanart
bmt/plugin/overwriteFanart=true

# -- Group: Fanart
# Fanart Implementation (Do Not Change this unless you KNOW what you are doing)
phoenix/mediametadata/fanartSupportClass=sagex.phoenix.fanart.PhoenixFanartSupport

# Phoenix Metadata Implementation.  Default is Metadata Tools (bmt), don't change unless you KNOW what you are doing.
phoenix/mediametadata/mediaMetadataSupportClass=bmt.BMTMetadataSupport

# This should never need to be enabled.  But if it is, then it parse the Display Title for the TV Series Name
phoenix/mediametadata/parseMediaFileForTVSeries=false

# Regex to use when parsing the Series Name from the Display Title
phoenix/mediametadata/tvSeriesRegex=.*[\\/](.*)s([0-9]{1,2})\s*[exd]([[0-9]]{1,2})

# Regex to use when parsing the Season Number from the Display Title
phoenix/mediametadata/tvSeasonRegex=.*[\\/].*s([0-9]{1,2})\s*[exd]([[0-9]]{1,2})

# -- Group: Logging
# -- Group: Metadata Settings
# Class Files for SageTV Metadata Parser Plugins
mediafile_metadata_parser_plugins=

# Extra Metadata Fields that SageTV will store in its Wiz.bin database
custom_metadata_properties=MediaProviderDataID;MediaTitle;MediaType;OriginalAirDate;EpisodeTitle;EpisodeNumber;SeasonNumber;DiscNumber;UserRating

# When SetMediaFileMetadata() apis are used, and this is set to true, then SageTV will store the metadata value directly to the .properties file for the media item as well
update_external_properties_files_on_metadata_changes=true

# -- Group: System Settings
# Classes that SageTV will start automatically when it starts
load_at_startup_runnable_classes=net.sf.sageplugins.webserver.StartServer;com.shadeblue.sage.InfoPopup.StartListener

# -- Group: Debug Settings
# ???
debug_logging=true

# ???
debug_capture_progress=false

# ???
debug_exif_parser=false

# ???
debug_msgs=false

# -- Group: New Search

[lovingHDTV]

Originally Posted by lovingHDTV

For me the little swirling icon never goes away. Does that mean it is still scanning or that it completed?

OK if I hit status again I can see that it did scan things. I can also browse so it is finding everything. I was just confused when the scanning swirling icon never went away.

this is really cool!

thanks,
dave

[stuckless]

Originally Posted by lovingHDTV

For me the little swirling icon never goes away. Does that mean it is still scanning or that it completed?

Swirling icon should indicate that it is doing a scan... there should also be some #s there as well... Like "Scanning 1 of 183 (1%)"

[stuckless]

Originally Posted by KJake

okey dokey - any ideas why the command-line isn't saving the search? here's my .properties file.

I noticed that you are using 3.0.3... I would upgrade to 3.1, since 3.0.3, had some severe issues. But back to your question... I don't know. I just scanned your show using the command line, which returned 0 results. The metadata-titles will only be updated if it prompts you with a list of results and you select 1 of them. It is not updated for auto-selected searches, or for searches that return 0 results.

[KJake]

Originally Posted by stuckless

I noticed that you are using 3.0.3... I would upgrade to 3.1, since 3.0.3, had some severe issues. But back to your question... I don't know. I just scanned your show using the command line, which returned 0 results. The metadata-titles will only be updated if it prompts you with a list of results and you select 1 of them. It is not updated for auto-selected searches, or for searches that return 0 results.

Sorry, that's my fault for not reporting my error properly

I am actually running 3.1 - the properties file just has an old version in it.

C:\Program Files\SageTV\SageTV>java -Xmx1024m -jar MetadataTool.jar
Batch MetaData Tools (3.1)
Batch MetaData Tools (3.1)
Import/Update Movie MetaData from a MetaData Provider.

and see attachment (the web page title reports 3.0.4, but page shows 3.1)

See other attachment for commandline process output - as I understand, I shouldn't be prompted that second time?

[stuckless]

Originally Posted by KJake

Sorry, that's my fault for not reporting my error properly

I am actually running 3.1 - the properties file just has an old version in it.

C:\Program Files\SageTV\SageTV>java -Xmx1024m -jar MetadataTool.jar
Batch MetaData Tools (3.1)
Batch MetaData Tools (3.1)
Import/Update Movie MetaData from a MetaData Provider.

and see attachment (the web page title reports 3.0.4, but page shows 3.1)

See other attachment for commandline process output - as I understand, I shouldn't be prompted that second time?

This is probably a misunderstanding of how the metadata-titles work vs how they probably should work

When you search, metadata titles is only ever consulted during the automatic phase. ONce the automatic phase is complete, and you are in the manual phase, metadata titles is never consulted. I'd be ok with calling it a bug... and hopefully i'll eventually put in a system that works in both automatic and manual mode.

But, for now, if you are willing to rename your media files to be "stargate sg" instead of "stargate sg-1", then it will work. (ie, Stargate SG - S9E15.avi)