Utility: Batch Metadata Importer 3.x

[razrsharpe]

stuckless, great import... overall I like it very much. It may be in this thread somewhere already but I can't seem to find it. How do I force sage to update its metadata (from existing .properties files created previously by metadatatool CLI). I also can't get sage to see some .properties files that exist... others have outdated metadata I want updated. I am using dvd profiler to provide the metadata and it appears that the .properties files are all being created properly. I just can't seem to get sage to see them. I tried adding the --refreshsage option but got an error that i needed to specify the sage api (i am sure im missing something it was kind of late when i tried that...). I also tried rescanning the import directories; also removing the import directory, and then immediately readdding it and rescanning. I did not try remove, rescan, add, rescan (maybe that's what I'm missing).

[stuckless]

razrsharpe, I assume since you are using DVD Profiler, then your movies are actually dvd folders. I don't have a lot of dvd folders personally, but the way it generally works, is that when the tool runs, it updates the timestamp on the DVD forcing sage to refresh that media item on it's next refresh cycle. Again that's how it "should" work, but with the 2.x release, there have been so many changes, that it's likely something is broken. It's my intention to spend some time this weekend working testing a bunch of issues that have been reported, and try to get out a new build.

Something that you can try.... if you have the "automatic" plugin enabled, try disabling the plugin (requires a server restart). Then rescan the collection from the command line, and then tell sage to "refresh" it's media. That's the basics of what needs to happen. If it doesn't work, maybe you could send me your log files as well (sagetv_0.txt and maillog.log)

[sflamm]

I'm having the same problem.

After hours of trying all combination/options (just like razrsharpe) - the .properties files were produced but nothing was being picked up in the video library regardless.

FYI - my folders consist of standard DVD folders (with video_ts etc), Blu-ray folders (which contain only a .m2ts), Blu-ray folders (which contain only a .iso) and Blu-ray folders (containing the BDMV folder structure). None of them are working.

I'd like to try the fanart features as well, but seems risky if I cant even get basic DVD info/metadata to work (works fine without using the plug-in).

Can you please investigate fix? I'm sure I'm missing out on some great stuff here.

[stuckless]

ok.... i'll dedicate some time this weekend to testing the dvd/bluray issues and the plugin.

[razrsharpe]

Yes my movies are all folder structure (Standard DVD: MOVIE NAME\VIDEO_TS, blu-ray: MOVIE NAME\BDMV, and blu-ray: MOVIE NAME\movie name.m2ts)

When you say the CLI updates the timestamp is that the timestamp of the folder, the VIDEO_TS/BDMV folder, the files inside those folders, something in the sage database, or something else? It does look like the folder is getting its modified date timestamp updated. I will confirm this evening when I am home and can run some more extensive tests.

I do have the automatic plugin installed i will try disabling that and refreshing sage (i assume you mean the right click in the video screen and "rescan import folders" or is the process something more involved?

What does the --refreshsage option do? and how do i link it to the sage api?

FYI the fanart features are all working great. I have only tried this using the sage plugin. I want to try to get it working from the CLI too. Hopefully I can get to that tonight as well.

Thanks again for all the fanart/metadata work its a great addition to the sage experience

[stuckless]

Originally Posted by razrsharpe

Yes my movies are all folder structure (Standard DVD: MOVIE NAME\VIDEO_TS, blu-ray: MOVIE NAME\BDMV, and blu-ray: MOVIE NAME\movie name.m2ts)

The bluray stuff is going to take some time work out, especially since I don't actually have any bluray media at the moment

When you say the CLI updates the timestamp is that the timestamp of the folder, the VIDEO_TS/BDMV folder, the files inside those folders, something in the sage database, or something else? It does look like the folder is getting its modified date timestamp updated. I will confirm this evening when I am home and can run some more extensive tests.

I think it is the folder's timestamp that get's updated, but maybe I need to update all the children in the folder for sage to recognize it as being updated.

I do have the automatic plugin installed i will try disabling that and refreshing sage (i assume you mean the right click in the video screen and "rescan import folders" or is the process something more involved?

That's pretty much it.

What does the --refreshsage option do? and how do i link it to the sage api?

That is supposed to be the same as going into sage and hitting the refresh/rescan media button. Requires that the sage remote APIs be installed on the server.

[razrsharpe]

Originally Posted by stuckless

The bluray stuff is going to take some time work out, especially since I don't actually have any bluray media at the moment

I was only running the CLI against my bluray folder. I will try against the DVD folder and see what happens.

I did create some .properties files all from within sage using the plugin that did update the fanart/metadata for some bluray titles. Then I decided i wanted to use dvd profiler for the description, etc because i like the descriptions better than the moviedb ones and decided i would try to make the dvd profiler import work, that's when i ran into issues.

I think it is the folder's timestamp that get's updated, but maybe I need to update all the children in the folder for sage to recognize it as being updated.

I confirmed that the parent folder (folder containing the VIDEO_TS or BDMV folder) is the timestamp that is updated. Again I was only running against my bluray folder. I will test again tonight against the DVD folder.

That is supposed to be the same as going into sage and hitting the refresh/rescan media button. Requires that the sage remote APIs be installed on the server.

thanks i will give that a try too.

[sflamm]

According to this post, it is the .properties file's timestamp that needs to be updated, NOT the folder timestamp.

http://forums.sagetv.com/forums/showthread.php?t=26374

If the metadata in the properties file changes, you'll need to update the time stamp on its media file so SageTV sees that the media file is newer & reimports it during the next library import directory scan.

Then the library needs to be re-scanned/imported.

And if it is a DVD:

Note: As of v6.4, imported DVDs can use .properties files. To add a metadata properties file for a DVD, place the <DVDName>.properties file in the same folder where the DVD folder is located, so that the <DVDName>.properties file and the <DVDName> folder are in the same parent folder.

I think the rescan will happen automatically when the SageTV auto-scans (assuming you have that enabled)


Questions:

1. How do you install the SageTV remote API's to enable the --refreshoption (please provide instructions)

2. For SageMC users shouldnt you be generating a ".my" file instead of the ".properties" file?

[stuckless]

Originally Posted by sflamm

According to this post, it is the .properties file's timestamp that needs to be updated, NOT the folder timestamp.

http://forums.sagetv.com/forums/showthread.php?t=26374

That post says (as you quoted), that the Sage
Media File needs it's timestamp updated if the properties change. I think that the issue is.... if DVDs are not be recognized as being changed, then it's most like because BMT considers the movie folder to be the "Media File", while sage considers the "VIDEO_TS" or "BDMV" folder to the media file. So, in the next release, i'm updating the timestamp on the child folders as well.

Questions:
1. How do you install the SageTV remote API's to enable the --refreshoption (please provide instructions)

In my signature, click on "Remote APIS", then the "Wiki" link.

2. For SageMC users shouldnt you be generating a ".my" file instead of the ".properties" file?

Since sage doesn't read the .my files, I'm not sure what value it would add? BMT is an open project.... anyone is free is implement a .my persisentence layer, but I don't think that I'll ever get to it.

[razrsharpe]

ok so i think i made some progress getting sage to read the .properties files. I have only been working on DVD folders for this (no bluray).

1. Remove the dvd folder from the media import locations in detailed setup
2. Rescan import folders to remove reference to folders you just removed
3. Run the metadatatools CLI on the dvd folder using dvdprofiler as the source and create the .properties. Use the --fanartFolder and --overwriteFanart option to copy the posters from dvd profiler db to each movies fanart posters folder
4. using windows explorer search for all *.jpg in the dvd folder (recursive) and delete them all. Do this to use the poster from the fanart folder. also MUCH faster displaying posters/coverart (at least with my initial testing... have not tested on the hd200 yet.
5. Add the dvd folder back in to the media import locations
6. rescan import folders
7. Wala everything shows up

Now some things I noticed.

When creating the fanart folders it seems to only remove the first illegal character in a title (not the 2nd, 3rd, etc) for the folder name. Creating the .properties files do not have this problem. For example "Waiting..." should be in a folder "Waiting" but the cli throws an exception. Contrast that with "W." which ends up in a folder "W". The same thing is true with titles with more than 1 ":", they throw and exception.

On a couple movies (less than 10 out of 350) i got some weird image corruption on the poster that was copied from dvd profiler (see the attached jpgs). I would be surprised if it was caused by the CLI but want to see if anyone else has seen this.

[razrsharpe]

Originally Posted by stuckless

That post says (as you quoted), that the Sage
Media File needs it's timestamp updated if the properties change. I think that the issue is.... if DVDs are not be recognized as being changed, then it's most like because BMT considers the movie folder to be the "Media File", while sage considers the "VIDEO_TS" or "BDMV" folder to the media file. So, in the next release, i'm updating the timestamp on the child folders as well.

for now you can get around this maybe limitation by removing the folder, rescanning, adding the folder, rescanning (as detailed in my previous post). Extra steps but it does work.

Since sage doesn't read the .my files, I'm not sure what value it would add? BMT is an open project.... anyone is free is implement a .my persisentence layer, but I don't think that I'll ever get to it.

I concur that this doesn't add much benefit and isn't worth the effort.

[razrsharpe]

I think I'm finally getting the hang of it and I must say I REALLY like this tool. Thanks a bunch

I have a small feature request for the CLI and dvd profiler. If you have both the dvd and bluray of the same movie (same title) you can not distinguish between them in the search options. It would be awesome if the search choices included the media type (dvd, bluray, hddvd)... i know its a field in the collection.xml database so theoretically it shouldn't be that hard (of course i could be way off base too). So the search choices would be something like Movie [year, media_type] or some such

One more feature request/enhancement. When matching against movies with remakes (both the original and remake are in the collection) it seems to always choose the first one. For example my folders (media) are "MOVIE_NAME RELEASE_YEAR" for movies with remakes. For flight of the pheonix it matched both the original and the remake with the metadata from the original. It's not such a huge deal because i just made a short batch file to that always prompts your for a selection and i can drag and drop the folder onto it (via right click send to) but it would be awesome to either auto figure out which ones it's supposed to be or not auto match it and prompt the user.

Again thanks for the great tool, I really like it

[sflamm]

razrsharpe: are you running the tool manually or are you using the plugin? (last I checked the plugin was still problematic)

I assume once the plugin has been debugged that would be the easiest method ... just dump a blu-ray folder in the directory and it should take care of the rest, yes?

[razrsharpe]

Originally Posted by sflamm

razrsharpe: are you running the tool manually or are you using the plugin? (last I checked the plugin was still problematic)

I assume once the plugin has been debugged that would be the easiest method ... just dump a blu-ray folder in the directory and it should take care of the rest, yes?

i have been running the tool manually from the command line and not using the automatic feature of the plugin. But yes i would assume once all the bugs (or operator errors) are worked out it should be pretty straight forward. Although I'm not sure how to setup the providers to get the metadata from dvdprofiler and the fanart from moviedb...

[sflamm]

That's an interesting point. From my experience with the tool it seemed that you can only specify an ordered list of the providers.

Perhaps it hasnt been considered that the providers themselves should be demarcated to provide a certain type of information only -- that way the tool could follow the hierachy to collect all the right information.

i.e. providerlist = dvdprofiler (metadata provider), moviedb (metadata & fanart provider)

then if the tool was run, it would search for the title in order, find that dvdprofiler had the meta data, but was not a fanart provider. then continue down the list and find that moviedb is a fanart provider and grab the fanart from it instead. it would continue down the list until all the required information was found or unavailable.

seems like this would fit into the current implementation model they have and provide the right kind of functionality we need.

thoughts?

[stuckless]

Originally Posted by razrsharpe

I think I'm finally getting the hang of it and I must say I REALLY like this tool. Thanks a bunch

I have a small feature request for the CLI and dvd profiler. If you have both the dvd and bluray of the same movie (same title) you can not distinguish between them in the search options. It would be awesome if the search choices included the media type (dvd, bluray, hddvd)... i know its a field in the collection.xml database so theoretically it shouldn't be that hard (of course i could be way off base too). So the search choices would be something like Movie [year, media_type] or some such

I'll see what i can do in this case... I'll simply add an enhancement request for now.

One more feature request/enhancement. When matching against movies with remakes (both the original and remake are in the collection) it seems to always choose the first one. For example my folders (media) are "MOVIE_NAME RELEASE_YEAR" for movies with remakes. For flight of the pheonix it matched both the original and the remake with the metadata from the original. It's not such a huge deal because i just made a short batch file to that always prompts your for a selection and i can drag and drop the folder onto it (via right click send to) but it would be awesome to either auto figure out which ones it's supposed to be or not auto match it and prompt the user.

That's an excellent point, the problem is that the source filename would have to have a year in it's title. I may add an option to compare years, and if enabled, it would parse the year from the filename and find a match based on the title and year. That's going to require some reworking. I've added it as an enhancement req.

[razrsharpe]

Originally Posted by sflamm

Perhaps it hasnt been considered that the providers themselves should be demarcated to provide a certain type of information only -- that way the tool could follow the hierachy to collect all the right information.

i.e. providerlist = dvdprofiler (metadata provider), moviedb (metadata & fanart provider)

then if the tool was run, it would search for the title in order, find that dvdprofiler had the meta data, but was not a fanart provider. then continue down the list and find that moviedb is a fanart provider and grab the fanart from it instead. it would continue down the list until all the required information was found or unavailable.

I like it and that makes sense... maybe it's already like that i haven't found it or experimented with it enough yet...

[razrsharpe]

Originally Posted by stuckless

I'll see what i can do in this case... I'll simply add an enhancement request for now.

That's an excellent point, the problem is that the source filename would have to have a year in it's title. I may add an option to compare years, and if enabled, it would parse the year from the filename and find a match based on the title and year. That's going to require some reworking. I've added it as an enhancement req.

Awesome, thanks a bunch.

Originally Posted by razrsharpe

When creating the fanart folders it seems to only remove the first illegal character in a title (not the 2nd, 3rd, etc) for the folder name. Creating the .properties files do not have this problem. For example "Waiting..." should be in a folder "Waiting" but the cli throws an exception. Contrast that with "W." which ends up in a folder "W". The same thing is true with titles with more than 1 ":", they throw and exception.

Have you ever noticed this? I posted it yesterday morning. I don't want to be a nag but wanted to make sure you saw in case it got missed in the flurry of posts

[stuckless]

I was just rescanning the thread as you pointed this out... I'm adding a bug for it now, so it doesn't get lost. Thanks for pointing it out... I usually try to acknowledge a request or bug, and if I don't there's a good chance that I didn't notice it. No harm is pointing it out again

[sflamm]

stuckless:

Did you see my idea/suggestion on allowing providers to have their capabilities demarcated and to use them in order to return all the appropriate information?