Utility: Batch Metadata Importer 3.x

[kevine]

Originally Posted by stuckless

You could look in the mailcheck.log file and see if there are any errors in that file. Or, if you don't feel comfortable looking through log files, then just email the log file to metadatatools at gmail dot com and i'll look through it when I get a chance. if you send the log file, could you also send me the complete filename of a few media files that are not being scanned.

Also, how are you doing the scan? command line? web ui? automatic scan via plugin?

sometimes, I find using the commandline to be most effective for troubleshooting, because you can fully specify the file on the commandline and then review it's generated property file / log.

commandline would be like...

C:\SageTV\> java -jar MetadataTools.jar --metadataOnly "Full Path To File.avi"

Thanks for your help. I know when I get this working, it is going to be a great tool for me.

The mailcheck.log file does not exist. I have all the debug logging setup (I think). And I read in a post it is supposed to be in the SageTV program directory but it's not. Do I need to restart the SageTV service?

Also, the command like worked perfectly. It created a properties file for the MPG I sent it.

[stuckless]

Originally Posted by kevine

The mailcheck.log file does not exist. I have all the debug logging setup (I think). And I read in a post it is supposed to be in the SageTV program directory but it's not. Do I need to restart the SageTV service?

That is odd... is there a log4j.properties in the sagetv home directroy. if so, then check that file to see what the logfile is called.

Also, the command like worked perfectly. It created a properties file for the MPG I sent it.

Keep in mind... there are lots of commandline args... just run it without any paramters to get a full list. the command that I sent will NOT download fanart, because I gave it --metadataOnly, which I use for testing.

[kevine]

Originally Posted by stuckless

That is odd... is there a log4j.properties in the sagetv home directroy. if so, then check that file to see what the logfile is called.

The file name is sage-plugins-log4j.log and Sage Job Queue is writing to it as well. I do not have debug logging on SJQ on. However, I just ran a full reindex and it went from 250 MB to 500+MB. Maybe I should not have this much logging on. I would be happy to provide you with a file but not that big. Can you tell me what logging I should have on? Thanks.

[stuckless]

Originally Posted by kevine

The file name is sage-plugins-log4j.log and Sage Job Queue is writing to it as well. I do not have debug logging on SJQ on. However, I just ran a full reindex and it went from 250 MB to 500+MB. Maybe I should not have this much logging on. I would be happy to provide you with a file but not that big. Can you tell me what logging I should have on? Thanks.

I would need debug logging turned on, to see what is really going on. You can look at the logging setting that I link to from the first post in this thread, for an overview on the log4j loggin and bmt.

or, if you open the file, and search for your filename, then you'll get the logging for the start of that filename, until it starts to process the next file.

[kevine]

Originally Posted by stuckless

I would need debug logging turned on, to see what is really going on. You can look at the logging setting that I link to from the first post in this thread, for an overview on the log4j loggin and bmt.

or, if you open the file, and search for your filename, then you'll get the logging for the start of that filename, until it starts to process the next file.

After updating the logging per the post, the file size limit does not work. The file is 300 MB after a full refresh and I have it set for 100MB.
Here is the contents of the logging properties:

#log4j configuration created using BMT Web UI
#Mon Nov 30 09:47:31 EST 2009
log4j.appender.log=org.apache.log4j.DailyRollingFileAppender
log4j.appender.log.DatePattern='.'yyyy-MM-dd
log4j.appender.log.File=mailcheck.log
# Set the maximum log file size (use KB, MB or GB)
log4j.appender.log.MaxFileSize=100MB
# Set the number of log files (0 means no backup files at all)
log4j.appender.log.MaxBackupIndex=5
# Append to the end of the file or overwrites the file at start.
log4j.appender.log.Append=false
log4j.appender.log.layout=org.apache.log4j.PatternLayout
log4j.appender.log.layout.ConversionPattern=%5r %-5p [%t] %c - %m%n
log4j.logger.bmt=debug
log4j.logger.org.jdna=debug
log4j.logger.phoenix=debug
log4j.logger.sagex.phoenix=debug
log4j.rootCategory=debug, log

I did change the name of the log file to be mailcheck.log also. I restarted the SageTV service after the changes.

Lastly and more important, I checked a few files in my recording directory (Only have 1 recording directory) and none are in the log file. Even the one I did manually using the command line does not appear in the full refresh log.

[toricred]

I just re-installed BMI on my system because I was having problems getting the web page to come up (dumb problem on my fault, but fixed now). I could have sworn that previously I was getting Metadata for CSI: NY, but now it won't look it up. I remember there being a problem with the : character. What was the fix for this again?

[bluenote]

I took the plunge and re-scanned my whole video library into sage recordings, and it worked really well. Unfortunately I guess I should have taken it bite by bite, because it turns out I needed to change the "TV title mask" because I can't get sagemc to show me episode order (S##E## or episode ID, etc).

EDIT: I managed to re-scan these properly (or so I thought), but as it turns out, it's still futile, because Sage MC recorded TV screen groups on title and not show title.

To everyone who is using the 'import to TV' feature, how are you managing not to end up with a ridiculously messy display in recorded tv view?

I would settle for properly sorting on Show title, but what would be even better is something that is S##E## aware.

If I have to go back, as it looks like I will have to, how the heck do I get all these entries out of sage recordings and back into Video Library folder view?

thanks

Cory

[kevine]

I have determined that BMT is not finding my recordings directory at all. The log shows it is not even trying to scan the files. Therefore, I have some configuration problem. Here are a few questions to help solve the problem:

1. Does the recordings directory need to be in SageTV Video Sources in the BMT configuration? Because if it does, I found it strange that when I installed BMT it was not there. I have added it and tried making it the only directory in there. None of which worked.

2. Does the fact that \\MYSERVER\SageTV is the name of my network share have anything to do with why it might be confused?

I was going to attach my log but it is scanning my photos share (\\MYSERVER\Photos). That is what is making it such a huge file. SageAlert and Sage Job Queue are sharing the same log file.

Thanks for all your help.

[razrsharpe]

Originally Posted by cnovak

To everyone who is using the 'import to TV' feature, how are you managing not to end up with a ridiculously messy display in recorded tv view?

I do not use the "import to TV" however you need to set the sage recording TV title mask to ${MediaTitle}. Then SageTV or SageMC will group TV appropiately. From the BMT webui homescreen go -> Configuration -> Metadata Tools -> Sage Persistence ->
Sage Recording TV Title Mask -> set to "${MediaTitle}" (without quotes).


To have access to Season/Episode Info you can look at TVE (link in my signature) or write a plugin to support that information with the default SageMC interface.

[bluenote]

Originally Posted by razrsharpe

I do not use the "import to TV" however you need to set the sage recording TV title mask to ${MediaTitle}. Then SageTV or SageMC will group TV appropiately. From the BMT webui homescreen go -> Configuration -> Metadata Tools -> Sage Persistence ->
Sage Recording TV Title Mask -> set to "${MediaTitle}" (without quotes).


To have access to Season/Episode Info you can look at TVE (link in my signature) or write a plugin to support that information with the default SageMC interface.

Thank you for this suggestion, your plugin really does for sage (and sagemc) what its needed for a very long time. thank you

Cory

[bluenote]

What is the best way to force BMT to re-generate .properties files, and for sage to re-import them? For video library files, that is? A lot of my properties files have full path show titles, I guess left over from an earlier time or even perhaps a different tool, but I want to re-download art, meta, and re-import to sage.
(And a lot of them are marked as movies, too).

What I've tried so far isn't giving me the results I expect.

thanks for any pointers

Cory

[MrFusion]

Hi,

I have a wierd problem -- I am trying to change the BMT URL cache directory from the default cache/url to another drive (D:\SageCache\url).

I changed the location in the BMT Web GUI, & confirmed that sage.properties had the corresponding update ("bmt/urlconfiguration/cacheDir=D\:\\SageCache\\url").

Restarted the SageTV service and... no dice . BMT still caches in the original location.

Any ideas as to what I've done wrong?

Many thanks!

Oh... I have BMT v 3.1, phoenix 1.44, sagetv 6.6.2.218, and java 1.6.0_15.

Cheers.

[robogeek]

Originally Posted by MrFusion

Hi,

I have a wierd problem -- I am trying to change the BMT URL cache directory from the default cache/url to another drive (D:\SageCache\url).

I changed the location in the BMT Web GUI, & confirmed that sage.properties had the corresponding update ("bmt/urlconfiguration/cacheDir=D\:\\SageCache\\url").

Restarted the SageTV service and... no dice . BMT still caches in the original location.

Any ideas as to what I've done wrong?

Many thanks!

Oh... I have BMT v 3.1, phoenix 1.44, sagetv 6.6.2.218, and java 1.6.0_15.

Cheers.

Open the sage.properties file and do a search for cacheDir. Update the pathnames to your new cache location for any cacheDir lines that have phoenix or bmt in it. Be sure to do this with the SageTV Service shut down.

[MrFusion]

Originally Posted by robogeek

Open the sage.properties file and do a search for cacheDir. Update the pathnames to your new cache location for any cacheDir lines that have phoenix or bmt in it. Be sure to do this with the SageTV Service shut down.

Hi,

Thanks - both Phoenix & BMT cachedir's point to D:\SageCache.

Phoenix honours this (ie all the SMW cache is there), but BMT ignores it.

As per above, I checked sage.properties and it does indeed point to the right place ("bmt/urlconfiguration/cacheDir=D\:\\SageCache\\url").

And yes, I did restart the server

[gdrider77]

Can i setup a custom mask for tv shows grouped in folders? So basically this is what i have as an example.

American dad>Season01>American DadS1E05 - The Episode.avi

The problem is when i display it using file view in sage movie wall, i navigate the folders correctly, it shows a single image/folder for the Series(American dad), then when i click on that, it shows the proper season folders, but then when i am in a season, it shows the full name of the file, and in the list view you can really only see the beginning of the filename, which is identical in all of them so in list view if i have 10 episodes, it shows as:

American DadS4E01 - E..
American DadS4E02 - E..
American DadS4E03 - E..
American DadS4E04 - E..
American DadS4E05 - E..
etc..

I would like to simply show the episode number and the episode name.

E05 - The Episode.avi

I have tried in the web ui to change the sage persistence "TV DVD Title Mask" and "TV DVD Title Mask" to "${EpisodeNumber} - ${EpisodeTitle}", but no go. Do i need to reset certain things or something? or is it possible this is not considered a dvd as i ripped these as episodes, and not as a dvd with menus, and if so, where should i change this setting?

Also, how can i reset the bmt settings to a "default"? I have played around with the mask stuff so much i cant remember the default and need to put it back, as i can no longer go into my tv show folders. here is what i have currently.

bmt/imdb/IMDbDomain=www.imdb.com
bmt/imdb/forcedIMDBImageSize=512
bmt/media/stackingModelRegex=[ _\\\\.-]+(cd|dvd|part|disc)[ _\\\\.-]*([0-9a-d]+)
bmt/media/videoExtensionsRegex=avi|mpg|divx|mkv|wmv|mov|xvid|ts|m2ts|m4v|mp4|iso
bmt/metadata/backgroundImageWidth=-1
bmt/metadata/bannerImageWidth=-1
bmt/metadata/defaultProviderId=tvdb,imdb-2,themoviedb.org,imdb.xml
bmt/metadata/enableDefaultSTVPosterCompatibility=false
bmt/metadata/goodScoreThreshold=0.9
bmt/metadata/importTVAsRecordedShows=false
bmt/metadata/maxDownloadableImages=10
bmt/metadata/posterImageWidth=200
bmt/metadata/scoreAlternateTitles=true
bmt/metadata/videoMetadataProviders=org.jdna.media.metadata.impl.tvdb.TVDBMetadataProvider,org.jdna.media.metadata.impl.imdb.IMDBMetaDataProvider,org.jdna.media.metadata.impl.nielm.NielmIMDBMetaDataProvider,,org.jdna.media.metadata.impl.dvdproflocal.LocalDVDProfMetaDataProvider,org.jdna.media.metadata.impl.themoviedb.TheMovieDBMetadataProvider,org.jdna.media.metadata.impl.mymovies.MyMoviesMetadataProvider
bmt/metadata/wordsToClean=1080p,720p,480p,1080i,720i,480i,dvd,dvdrip,cam,ts,tc,scr,screener,dvdscr,xvid,divx,avi,vrs,repack,mallat,proper,dmt,dmd,stv,HDTV,x264
bmt/plugin/enabled=true
bmt/plugin/overwriteFanart=false
bmt/plugin/overwriteMetadata=true
bmt/sageMetadata/actorMask={0} -- {1};\n
bmt/sageMetadata/genreLevels=1
bmt/sageMetadata/multiCDTitleMask=${MediaTitle} Disc ${DiscNumber}
bmt/sageMetadata/rewriteTitle=false
bmt/sageMetadata/sageTVTitleMask=${MediaTitle}
bmt/sageMetadata/titleMask=${MediaTitle} - S${SeasonNumber}E${EpisodeNumber} - ${EpisodeTitle}
bmt/sageMetadata/tvDvdTitleMask=${MediaTitle} - S${SeasonNumber}E${EpisodeNumber} - ${EpisodeTitle}
bmt/sageMetadata/tvTitleMask=${MediaTitle} - S${SeasonNumber}E${EpisodeNumber} - ${EpisodeTitle}
bmt/urlconfiguration/cacheDir=cache/url
bmt/urlconfiguration/cacheExpiryInSeconds=86400
bmt/urlconfiguration/httpUserAgent=Mozilla/5.0 (X11; U; Linux i686; en-US; rv\:1.9.0.1) Gecko/2008072820 Firefox/3.0.1
bmt/urlconfiguration/urlFactoryClass=org.jdna.url.CachedUrlFactory
bmt/xbmc/imdb.xml/fanart=true
bmt/xbmc/imdb.xml/fullcredits=true
bmt/xbmc/imdb.xml/imdbscale=512
bmt/xbmc/imdb.xml/impawards=true
bmt/xbmc/imdb.xml/movieposterdb=true
bmt/xbmc/imdb.xml/trailer=true
bmt/xbmc/imdb.xml/url=akas.imdb.com

Thanks.

[razrsharpe]

Originally Posted by cnovak

What is the best way to force BMT to re-generate .properties files, and for sage to re-import them? For video library files, that is? A lot of my properties files have full path show titles, I guess left over from an earlier time or even perhaps a different tool, but I want to re-download art, meta, and re-import to sage.
(And a lot of them are marked as movies, too).

What I've tried so far isn't giving me the results I expect.

thanks for any pointers

Cory

use the commandline (or make a .bat file that executes something this):
open up command prompt
cd to directory where bmt exists
and run this commandline:

java -jar metadatatool.jar --overwriteMetadata --overwriteFanart %1

where %1 is the full path to the mediafile or directory

that will redownload the metadata, redownload the fanart, touch the file (update its timestamp) and then you can trigger a library import scan and the new data will exist in sage

For bat file examples that i posted a while back see this post:
http://forums.sagetv.com/forums/show...bat#post372103

None of them do exactly what you want but you can get the idea

[ChaOConnor]

Originally Posted by MrFusion

Hi,

Thanks - both Phoenix & BMT cachedir's point to D:\SageCache.

Phoenix honours this (ie all the SMW cache is there), but BMT ignores it.

As per above, I checked sage.properties and it does indeed point to the right place ("bmt/urlconfiguration/cacheDir=D\:\\SageCache\\url").

And yes, I did restart the server

I too am having this problem... Just to clarify, it has to be changed in both the Sage.Properties file as well as through the BMT web interface?

[stuckless]

Originally Posted by toricred

I just re-installed BMI on my system because I was having problems getting the web page to come up (dumb problem on my fault, but fixed now). I could have sworn that previously I was getting Metadata for CSI: NY, but now it won't look it up. I remember there being a problem with the : character. What was the fix for this again?

I think that the fix is to add an entry to the metadata-titles files. (you have to search back for posts that contain metadata-titles, since I don't recall off hand the specifics )

Originally Posted by kevine

I have determined that BMT is not finding my recordings directory at all.

If bmt is invoked using the automatic plugin, then it is passed the filename to scan. If bmt is invoked by the webui, then it scans the files using the sage api... and you have to check off the options for scanning. If a file is NOT yet recognized by sage, then it will not be scanned. If you use the commandline, then you specify the directory/file on the commandline.

Originally Posted by cnovak

What is the best way to force BMT to re-generate .properties files, and for sage to re-import them?

Either find and remove all .properties files and then rescan... or use the Overwrite option from the web ui, or set the overwrite option for the automatic plugin (if you use that), or use --overwrite on the commandline.

Originally Posted by MrFusion

Restarted the SageTV service and... no dice . BMT still caches in the original location.

Originally Posted by ChaOConnor

I too am having this problem... Just to clarify, it has to be changed in both the Sage.Properties file as well as through the BMT web interface?

It may be that bmt is NOT using the cache property. bmt was written first, then along cam phoneix, and that cache value may actually be the cache property for phoenix realted cache items. BMT 4.0 will leverage much more of the phoenix apis, and so this will probably be addressed in a 4.x release.

[kevine]

Originally Posted by stuckless

If bmt is invoked by the webui, then it scans the files using the sage api... and you have to check off the options for scanning.

That is what I am saying, it is not working. Why would it be scanning my photo directory? I have check off the proper options. I generally am using the web ui although I have the plugin installed.

[tchapin]

I think I had an old version of BMT installed, but today decided to install the latest version including Jetty.

So, I downloaded and installed Jetty. After shutting down the sage service, I unzipped the Jetty install into the proper location. I changed the username and password from the default. I changed the port # to 8086. I updated the sage.properties file.

How do I verify that Jetty is working correctly? Is there a separate support thread for it? I couldn't find one.

Then I downloaded and installed BMT. I unzipped it into the proper location.

Then I restarted sage.

When I go to the BMT URL (http://192.168.0.7:8086/bmt/), I see a toolbar across the top of the page, then below that, "Phoenix", "Metadata Tools", and "SageTV".

I have not knowingly installed any other Jetty plugins.

In any case, when I visit the page, I am not asked for the user name or password. I then get three pop up messages "Failed to get staus for:" "Phoenix", "Metadata Tools", and "SageTV".

So no joy there.

Using Placeshifter, I imported the BMT STVi for SageMC. In the extras menu, I now have a "BMT Options" option. When I go in there, it takes a while to process (w/ the spinning circle). I see two options "Fanart Enabled" and "Automatic Metadata/Fanart", both with blank options boxes next to them. When I click on either of the options box, it takes a while to process, then nothing happens.

Thoughts?

Thanks.

Todd