Utility: Batch Metadata Importer 3.x

[stuckless]

tmiranda,

currently, a few developers, including myself, are working to solve your issue. Well, maybe we are not working specifically to solve "your" issue but we are working together to provide a common api that can be used by sagemc and the default stv plugins to ensure that everyone is able to read the fanart correctly. There is a defined filename/format for how the fanart has to be named, and the scraper tools will all use the same format to ensure consistency with the api. I believe that Tiki will also be using the phoenix fanart apis in his plugin as well.

[tmiranda]

Thanks. I've been following that thread.

[Tiki]

Originally Posted by stuckless

I believe that Tiki will also be using the phoenix fanart apis in his plugin as well.

I like the idea of a consistent way of storing and retrieving the fan art so that people can easily switch between the various STV's and take advantage of the automatic scrapers. So, yes, I am keeping an eye on these threads and asking questions / making suggestions. I plan on implementing the phoenix apis, but I will probably wait a few weeks before I jump into it.

Right now I don't have a lot of time to experiment with the code, plus I'm hoping that within a couple of weeks some of these new tools will start to stabilize as the initial kinks get ironed out. Also, I'm waiting for a few things to happen before I start coding - (1) a tool to convert existing fanart on my hard drive to the new format/naming convention, (2) I'm waiting to see how the automatic fan art grabbers work out

The new storage format should make it possible to do a lot of neat stuff, but without the automated tools, it will be a lot more tedious to manually create and add fanart. I'm also a little concerned about how the automatic metadata importers will handle grabbing data for guide data. I've noticed that at certain times, the websites that these scrapers search for fanart can be really slow. I would hate for Sage to become unresponsive waiting for fan art to download. On the other hand, I don't want to have to manually tell Sage to download an image for each show individually.

So, it will be interesting to see how they handle that and also how it all integrates together (will it be an STVi that we can import, or will it be a command line tool or a set of APIs in a JAR that the STV / STVi creators will have to call from Sage to make it work)? So far the discussion has mostly focused on how the fanart should be named and stored and on the API's for retrieving the fanart and setting the preferred images. There hasn't been much discussion that I've seen on how the scrapers will work.

Anyway, sorry if I'm rambling a little off topic...

[tmiranda]

Tiki,

I like your train of thought. I really do NOT want to manually convert all of the fanart I've collected to this new format and I do not want my system to come to a crawl while scraping fanart.

Tom

[stuckless]

My apologies Tiki... I didn't mean to speak on your behalf.

I have a beta of the metadata tools available for any developer that wants to try out the new folder layout/metadata properties. I'm pretty sure EP is making available a beta of his scraper tools as well.

As for some of your thoughts/ideas... BMT does run as a sagetv metadata plugin that will fetch metadata/fanart on the fly as vidoes are added to the library. This isn't new, it's been there for a long time. What is new with the 2.0 release is that it will act as a metadata/fanart implementation to the phoenix metadata/fanart apis, which means that stv developers can fetch metadata interactively for any media item. .

For 2.0 release of bmt, i'll be including a simple stvi for the default stv and sagemc that show how the phoenix metadata/fanart apis can be used interactively to search for and download metadata/fanart.

[jaminben]

Originally Posted by stuckless

For 2.0 release of bmt, i'll be including a simple stvi for the default stv and sagemc that show how the phoenix metadata/fanart apis can be used interactively to search for and download metadata/fanart.

This sounds very nice indeed and should tie in with my new import quite well.

[tmiranda]

Originally Posted by stuckless

For 2.0 release of bmt, i'll be including a simple stvi for the default stv and sagemc that show how the phoenix metadata/fanart apis can be used interactively to search for and download metadata/fanart.

Will this download banners as well as backgrounds?

[stuckless]

support for downloading posters, backgrounds and banners has been added.. but not all providers will have banners... I'm sill have some testing to do... but yeah.. it should pull down banners as well.

[stanger89]

Sean,

Have you looked into supporting Blu-ray folders?

Oh, and really stupid question, but where does the metadata.properties go? I'm trying to run this from my desktop so I've just got the zip file extracted and I'm running it from there. I created a metadata.properties file in the same directory I'm running the jar from but it doesn't look like it's using it.

[onlydarksets]

That's the right spot. Try listing the properties via command-line - that will tell you if it is picking it up.

java -jar MetadataTool.jar --showProperties

For BR, you could just create an empty "VIDEO_TS" folder in the BR folder. It's a workaround, but it does work.

[stuckless]

Originally Posted by stanger89

Have you looked into supporting Blu-ray folders?

I haven't... but i guess it will have to happen... I'll need to dig up some information of the folder/file structures. If you have any information, feel free to post it here.

Oh, and really stupid question, but where does the metadata.properties go? I'm trying to run this from my desktop so I've just got the zip file extracted and I'm running it from there. I created a metadata.properties file in the same directory I'm running the jar from but it doesn't look like it's using it.

The metadata.properties must be in the same directory from which the application is started. Also, the log file, metadataupater.log, will show the location of the property file and whether or not it exists. And as another user pointed out, if you run with --showProperties, you can see whether or not your properties are getting picked up. Also the --showProperties will dump all properties along with a brief description of what it does and it's datatype.

[stanger89]

Originally Posted by onlydarksets

That's the right spot. Try listing the properties via command-line - that will tell you if it is picking it up.

Originally Posted by stuckless

The metadata.properties must be in the same directory from which the application is started. Also, the log file, metadataupater.log, will show the location of the property file and whether or not it exists. And as another user pointed out, if you run with --showProperties, you can see whether or not your properties are getting picked up. Also the --showProperties will dump all properties along with a brief description of what it does and it's datatype.

Nope it's not. Hm, my file is metadata.properites. Wonder if that's the problem

Woohoo! it works.

I haven't... but i guess it will have to happen... I'll need to dig up some information of the folder/file structures. If you have any information, feel free to post it here.

Well "BDMV" is the BD equivalent of VIDEO_TS.

R:\Blu-ray\Quantum of Solace\BDMV
R:\Blu-ray\Quantum of Solace\CERTIFICATE
R:\Blu-ray\Quantum of Solace\BDMV\AUXDATA
R:\Blu-ray\Quantum of Solace\BDMV\BDJO
R:\Blu-ray\Quantum of Solace\BDMV\JAR
R:\Blu-ray\Quantum of Solace\BDMV\META
R:\Blu-ray\Quantum of Solace\BDMV\CLIPINF
R:\Blu-ray\Quantum of Solace\BDMV\PLAYLIST
R:\Blu-ray\Quantum of Solace\BDMV\STREAM
R:\Blu-ray\Quantum of Solace\BDMV\BACKUP
R:\Blu-ray\Quantum of Solace\BDMV\index.bdmv
R:\Blu-ray\Quantum of Solace\BDMV\MovieObject.bdmv
R:\Blu-ray\Quantum of Solace\BDMV\CLIPINF\00000.clpi
R:\Blu-ray\Quantum of Solace\BDMV\PLAYLIST\00000.mpls
R:\Blu-ray\Quantum of Solace\BDMV\STREAM\00000.m2ts
R:\Blu-ray\Quantum of Solace\BDMV\BACKUP\CLIPINF
R:\Blu-ray\Quantum of Solace\BDMV\BACKUP\PLAYLIST
R:\Blu-ray\Quantum of Solace\BDMV\BACKUP\BDJO
R:\Blu-ray\Quantum of Solace\BDMV\BACKUP\index.bdmv
R:\Blu-ray\Quantum of Solace\BDMV\BACKUP\MovieObject.bdmv
R:\Blu-ray\Quantum of Solace\BDMV\BACKUP\CLIPINF\00000.clpi
R:\Blu-ray\Quantum of Solace\BDMV\BACKUP\PLAYLIST\00000.mpls
R:\Blu-ray\Quantum of Solace\CERTIFICATE\BACKUP

Everything of interest is in the BDMV folder, the STREAM subfolder contains the actual movie, and PLAYLIST contains the playlist file. So if you're detecting off VIDEO_TS, you could just detect of VIDEO_TS or BDMV and that should work. Though I'm not 100% sure what exactly Sage looks for.

[stuckless]

ok... well i've added a check for BDMV as well as VIDEO_TS... it'll go out in the 2.0 build, so we'll see if that does the trick

[dblaine2]

Originally Posted by stuckless

Java can be a little forgiving here as well... you can for neatness, use forward slashes

//server/fanart

or

c:/videos/fanart

My central fanart directory is on C: but I found through looking at the logs that it was munging up the path by sticking the directory where my script is before this path.

ie c:\utilsc:\central\backgrounds

So I just specified the property as relative (use as many ../ as you need):

../central/backgrounds

Dave

[stuckless]

Thanks Dave... that's obviously a bug so I entered bug report for that. This may actually already be fixed when the 2.0 release goes out... but I'll do a quick test for it.

[dblaine2]

Hi Stuckless,

It also seems that while using the central backdrops folder that backdrops are downloaded everytime (although depending on the overwrite setting it won't actually update the files). But it seems alot of wasted bandwidth if it could just check for the existance first.

Dave

[dblaine2]

BTW is there a good provider builtin that will work with Music?

Dave

[stuckless]

Originally Posted by dblaine2

Hi Stuckless,

It also seems that while using the central backdrops folder that backdrops are downloaded everytime (although depending on the overwrite setting it won't actually update the files). But it seems alot of wasted bandwidth if it could just check for the existance first.

Dave

I'll certainly look at the issue... but I don't think that it is downloading the image unless it intends to write it... ie, the code never stores the actual image file, just a reference forom where the image can be downloaded.... which means that unless the intention to overwrite is there, it will not even attempt to download the image. But i'll verify this.

As for the music... i haven't implemented anything specific to do with music yet... but I did pull in the xbmc scrapers in hopes of actually using those to provider music metadata/fanart.

[dblaine2]

Hi Stuckless,

Got another question this time about multi disk DVD support. Say I have 24 Season 1 Disk 1 (its in VIDEO_TS format) so I have the directory named "24 - S01D01" and then using the command:

java -jar MetadataTool.jar --recurse --provider=imdb "\\pvr\imports\DVD\24 - S01D01"

I get no matches.

Am I doing something wrong?

EDIT: I tried other TV providers with same results.

Dave

[stuckless]

did you try --provider=tvdb.xml

not all providers support tv. Actually only the tvdb.xml provider can be used for TV.

I just tried it with tvdb.xml and it worked.