Utility: Batch Metadata Importer 3.x

[mkanet]

Thanks Sean. Yeah, it definitely looks like its required for me to enable the Overwrite Existing Fanart option in Scan (even when Overwrite Fanart is already selected under the Automatic feature settings). Maybe this is easy enough to reproduce on any platform.

Edit: Whats the quickest/easiest way to remove metadata/fanart so my recordings will be at virgin state? The whole reason I deleted the fanart is so I can see the automatic feature at work.

Is it possible for the automatic feature to not add fanart, but the scan feature does if its a show that's never been recorded before? I could have sworn I never recorded a show before and now both the scan with overwrite feature and full rescan feature will not add fanart/metadata to something that the on demand feature detects. Its hard for me to know which show I've not recorded which is why I wanted to return things to virgin state to it to start working like the first time again. I sure hope that my automatic feature didnt stop working again. I know for sure it works in version 3.03 (at least it was for sure in 3.03).

Originally Posted by stuckless

It's sad... but I only just noticed that about an hour before you posted it A classic example of my brain seeing what I want it to see

As for the overwrite flags for the plugin, they should work, but I haven't tested them explicitly for a while. I'll take a quick look and see if they are still be honoured.

[razrsharpe]

Originally Posted by mkanet

Edit: Whats the quickest/easiest way to remove metadata/fanart so my recordings will be at virgin state? The whole reason I deleted the fanart is so I can see the automatic feature at work.

Don't know if it's the easiest/quickest... but this is what i do

1. remove the import directory from sage
2. rescan import folders
3. delete any .properties files and/or fanart
4. readd the import directory
5. rescan import folders

[mkanet]

Thanks! I will try that when I get home. Will these instructions also work on the TV recordings directory? I've never tried to tell SageTV to NOT have a TV recordings directory. Thanks again.

Originally Posted by razrsharpe

Don't know if it's the easiest/quickest... but this is what i do

1. remove the import directory from sage
2. rescan import folders
3. delete any .properties files and/or fanart
4. readd the import directory
5. rescan import folders

[razrsharpe]

Originally Posted by mkanet

Thanks! I will try that when I get home. Will these instructions also work on the TV recordings directory? I've never tried to tell SageTV to NOT have a TV recordings directory. Thanks again.

not sure about that... ive never done that (removed / readded the recording directory). Personally i would be hesitent to remove the tv recording directory for fear of losing the recording history, etc. I would look into moving those files (the ones with BMT metadata) out of that directory into an import directory... There's an FAQ how to do this correctly somewhere... i think its a sticky...

[mkanet]

Actually, now that I think about this, this solution isn't going to work; as I already had lost all my TV recordings. Doing a full rescan after doesnt remove metadata.

Sean? What's the easiest way to remove metadata/fanart information from my TV recordings? Sorry if I missed it somewhere in this thread. I didnt see it.

Originally Posted by razrsharpe

not sure about that... ive never done that (removed / readded the recording directory). Personally i would be hesitent to remove the tv recording directory for fear of losing the recording history, etc. I would look into moving those files (the ones with BMT metadata) out of that directory into an import directory... There's an FAQ how to do this correctly somewhere... i think its a sticky...

[stuckless]

Originally Posted by mkanet

Thanks Sean. Yeah, it definitely looks like its required for me to enable the Overwrite Existing Fanart option in Scan (even when Overwrite Fanart is already selected under the Automatic feature settings). Maybe this is easy enough to reproduce on any platform.

Keep in mind, the web ui scanner doesn't honour the system wide flags for overwrite fanart/metadata, and the web ui scanner adds it's own checkboxes for those options. So, if you have overwrite metadata enabled in the configuration, but you don't enable it on the web ui scan, then the metadata will not be overwritten.

Edit: Whats the quickest/easiest way to remove metadata/fanart so my recordings will be at virgin state? The whole reason I deleted the fanart is so I can see the automatic feature at work.

Restore your backup wiz.bin

Is it possible for the automatic feature to not add fanart, but the scan feature does if its a show that's never been recorded before? I could have sworn I never recorded a show before and now both the scan with overwrite feature and full rescan feature will not add fanart/metadata to something that the on demand feature detects.

When the automatic scanner scans a file, the first thing it does, is check to see if the file has existing metadata (ie, reads the .properties). If there is a .properties for a given file, then it will not scan it again. This is default behaviour for the automatic feature. On-demand has a different behaviour... it assumes that if you are doing on-demand searches, then you want to overwrite metadata/fanart.

Its hard for me to know which show I've not recorded which is why I wanted to return things to virgin state to it to start working like the first time again.

NOt sure a "virgin" state is ever possible unless you have a backup of your wiz.bin.

[mkanet]

Sean, thanks! Restoring an older wiz.bin allowed BMT/sageTV to automatically add TV shows and their respective fanart pictures and metadata.

However, there's an issue with movies. For some reason, none of the TV movies I'm recording have fanart (over 15 movies I've tried). The metadata is there when viewing it using the "brose" feature , but fanart is not downloaded. This is for movies that have never been recorded before. In my other video imports directories that I have manual simple names such as "The Pursuit of Happyness.ts" do get fanart.

A big possible clue is that the On Demand feature, as always, can download the fanart for any of these movies (that will not work with automatic). This is true for even movies that only have one simple entry to select from in the On Demand list.

Is there anything I can do so the movies get fanart too? Unless I'm mistaken, I dont have too much control over movie names. Here are some examples that dont work with automatic, but work with On Demand. Bare in mind, that TV show work pretty good.

Here's an example of TV Movies that do not work:
AnalyzeThat-5074192-0.mpg
NurseryUniversity-5046198-0.mpg

Here's an example of a TV show that works just fine:
KeepingUpWiththeKardashians-LeavingtheNest-5077520-0.mpg

PS: Here's another clue, although I can see metadata information for these movies (like in the screenshot below), I dont get any properties files for these movies, just TV shows).



EDIT: I included a screenshot of the web configuration interface browsing the files. It will metadata for a movie, but no fanart pictures... like the way they are for TV series such as the Kardashians. I also included a link to full debugging on while recording a some movies.

screenshot

debug

Thanks,
Michael

Originally Posted by stuckless

Keep in mind, the web ui scanner doesn't honour the system wide flags for overwrite fanart/metadata, and the web ui scanner adds it's own checkboxes for those options. So, if you have overwrite metadata enabled in the configuration, but you don't enable it on the web ui scan, then the metadata will not be overwritten.

Restore your backup wiz.bin

When the automatic scanner scans a file, the first thing it does, is check to see if the file has existing metadata (ie, reads the .properties). If there is a .properties for a given file, then it will not scan it again. This is default behaviour for the automatic feature. On-demand has a different behaviour... it assumes that if you are doing on-demand searches, then you want to overwrite metadata/fanart.

NOt sure a "virgin" state is ever possible unless you have a backup of your wiz.bin.

[KJake]

Originally Posted by stuckless

This is probably a misunderstanding of how the metadata-titles work vs how they probably should work

When you search, metadata titles is only ever consulted during the automatic phase. ONce the automatic phase is complete, and you are in the manual phase, metadata titles is never consulted. I'd be ok with calling it a bug... and hopefully i'll eventually put in a system that works in both automatic and manual mode.

But, for now, if you are willing to rename your media files to be "stargate sg" instead of "stargate sg-1", then it will work. (ie, Stargate SG - S9E15.avi)

Thanks! I renamed to Stargate SG1, no dash and all is well. They still display as "Stargate SG-1" in Sage, so not a big deal at all! Thanks to perl, renaming was quick

[Fuzzy]

Originally Posted by stuckless

In the sage home dir, delete the cache/url/ directory and then try this again. I'm thinking that bmt may be using a older cached copy of the xml metadata from tvdb. And, make sure that tvdb is in your list of providers (it is by default, unless you've manually removed it)

Thanks Sean, That seemed to do it. Any idea wht the expiration time on the cache is? Seems a day would suffice, otherwise, you risk not getting updated show info for 1st run recordings and such for a while afterwards.

[Nick_W]

Hi Sean,

Since upgrading to 3.1 all my TV shows (not Sage recordings, just downloaded ones) are being marked as movies. It does this with every one of them, Man V Food, The Wire, Hell's Kitchen etc.

It still gets the correct metadata, but it adds fanart under movies, and it never uses the tvdb any more because it's marked it as a movie!

Any ideas?

TIA,

Nick.

EDIT: This was fixed by removing the cache folder and rescanning.

[Nick_W]

Hi Sean,

Every time I do a scan from the web interface the existing metadata properties files are updated, and some of the metadata is added to the bottom of the file again, so the file just keep growing. Here is an example of a file that's been scanned twice:

#Generator: Batch Metadata Tools (3.1); MediaFile: file:////NAS1/Video/TV/Hell's%20Kitchen/Season%2006/Hell's%20Kitchen%20S06E09.avi
#Tue Sep 15 07:45:26 BST 2009
Actor=Gordon Ramsay -- ;
Description=The remaining eight chefs are asked to make crepes, where the winners receiving a night out dining at a French restaurant and the losers creating a French-themed dinner service. During the prep time one contestant is injured, and the other three can't make it through the service.
EpisodeNumber=9
EpisodeTitle=8 Chefs compete
Genre=Reality
MediaProviderDataID=tvdb\:74897
MediaTitle=Hell's Kitchen
MediaType=TV
OriginalAirDate=2009-09-08
Rated=TV-14
RunningTime=3600000
SeasonNumber=6
Title=Hell's Kitchen - S06E09 - 8 Chefs compete
UserRating=0
Year=2009
x-MetadataProviderDataUrl=74897;;;;SEASON\=06&EPISODE\=09
x-MetadataProviderId=tvdb
x-ser-Cast=0|Gordon Ramsay||null
x-ser-Genres=Reality

MediaProviderDataID=tvdb:74897

MediaTitle=Hell's Kitchen

MediaType=TV

OriginalAirDate=2009-09-08

EpisodeTitle=8 Chefs compete

EpisodeNumber=9

SeasonNumber=6

UserRating=0

MediaProviderDataID=tvdb:74897

MediaTitle=Hell's Kitchen

MediaType=TV

OriginalAirDate=2009-09-08

EpisodeTitle=8 Chefs compete

EpisodeNumber=9

SeasonNumber=6

UserRating=0

If I scan again this code will be added to the bottom of the file for each scan:

MediaProviderDataID=tvdb:74897

MediaTitle=Hell's Kitchen

MediaType=TV

OriginalAirDate=2009-09-08

EpisodeTitle=8 Chefs compete

EpisodeNumber=9

SeasonNumber=6

UserRating=0

Any ideas?

TIA,

Nick.

[stuckless]

Originally Posted by Fuzzy

Thanks Sean, That seemed to do it. Any idea wht the expiration time on the cache is? Seems a day would suffice, otherwise, you risk not getting updated show info for 1st run recordings and such for a while afterwards.

Cache expiry is supposed to happen every 24hrs (configurable in the web ui).... I'll make a note to check the expiry code and see if there's an issue.

[stuckless]

Originally Posted by Nick_W

Hi Sean,

Every time I do a scan from the web interface the existing metadata properties files are updated, and some of the metadata is added to the bottom of the file again, so the file just keep growing.
Any ideas?

This is the SageTV implementation for SetMediaFileMetadata(). When that api is called, then it updates it wiz.bin and the .properties (this is core SageTV, not bmt). But you can turn this off, from the webui by unchecking
Configuration -> SageTV -> Metadata Settings -> Update .properties when changed.

By unchecking this, you risk that the .properties my become out of sync with the wiz.bin.

[Nick_W]

Originally Posted by stuckless

This is the SageTV implementation for SetMediaFileMetadata(). When that api is called, then it updates it wiz.bin and the .properties (this is core SageTV, not bmt). But you can turn this off, from the webui by unchecking
Configuration -> SageTV -> Metadata Settings -> Update .properties when changed.

By unchecking this, you risk that the .properties my become out of sync with the wiz.bin.

Understood; thanks Sean

Regards,

Nick.

[Taddeusz]

I'm going to have to go back to the previous version. 3.1 just isn't working out for me. I had to trash all my metadata and start from scratch as everything all of a sudden showed up in the root of the file system view in SageMC.

I have a feeling it has something to do with the Display Title. The way I determined this is I have the movie Amélie. The default Display Title is Movies/0-F/Amélie. When it is shown in SageMC it is displayed properly as "Amélie". But when I change the Display Title it moves it into the root of the File System view. Changing the Display Title back to Movies/0-F/Amélie puts the file back into the correct place in the File System view but then the listed title is the whole path, Movies/0-F/Amélie, and not just the title by itself. There's some value that's being modified when the Display Title is changed that shouldn't be.

Also, the web interface is still not giving me the full list of of imported videos when I do a general scan. If I scan specifically for a title it will find it but doing just a general scan only brings up a limited list.

[stuckless]

Originally Posted by Taddeusz

I'm going to have to go back to the previous version. 3.1 just isn't working out for me. I had to trash all my metadata and start from scratch as everything all of a sudden showed up in the root of the file system view in SageMC.

I have a feeling it has something to do with the Display Title. The way I determined this is I have the movie Amélie. The default Display Title is Movies/0-F/Amélie. When it is shown in SageMC it is displayed properly as "Amélie". But when I change the Display Title it moves it into the root of the File System view. Changing the Display Title back to Movies/0-F/Amélie puts the file back into the correct place in the File System view but then the listed title is the whole path, Movies/0-F/Amélie, and not just the title by itself. There's some value that's being modified when the Display Title is changed that shouldn't be.

This has been discussed, but I'll give a more technical description here. Basically as of 3.1, it will directly write to the wiz.bin via the AddShow() api. When I add a show, I set the "Title" of the show to be the show's title. When SageTV imports a show, it must put the relative path of the Show in the Show Title field, because in SageMC, when it builds the filesystem view, it calls the GetShowTitle() api to return the relative path of the Show. I don't know why GetShowTitle() would return the relative path of the title when SageTV imports the vidoe, but BMT only has 1 place to set the Title, and my option was to set it as just the Title. All that being said, I've been working with MeInMaui to provide an update to SageMC that will present the folder view in the correct way, since there is nothing that I can do in BMT to fix this, short of removing the ability to update the metadata in the wiz.bin via the AddShow api.

Also, the web interface is still not giving me the full list of of imported videos when I do a general scan. If I scan specifically for a title it will find it but doing just a general scan only brings up a limited list.

This has been answered, but here it is again.... When you "Scan", ONLY videos that are not updated, are returned. Videos that are scanned automatically are not returned. Why? Because if you have hundreds of videos, but only a couple that are not scanned correctly, do you really want to get all videos back and then manually click on each one to find out which ones are scanned and which ones are not. My feeling is that you only want to work on videos that a missing metadata/fanart, so only those are returned. Browse, on the other hand, will let your return all videos.

[Taddeusz]

Sorry, I guess my skimming of the thread missed those details.

[stuckless]

Originally Posted by Taddeusz

Sorry, I guess my skimming of the thread missed those details.

That's not a problem... with 1776 posts... it can very easy to miss details... which is why I repeated it.... I could have tried to find the posts where I had already discussed it... but it was faster to just rewrite the details

[Taddeusz]

Originally Posted by stuckless

That's not a problem... with 1776 posts... it can very easy to miss details... which is why I repeated it.... I could have tried to find the posts where I had already discussed it... but it was faster to just rewrite the details

This may be beside the point. I downgraded my Metadata Tools until a fix for SageMC becomes available. But I noticed that the Display Title in the web interface of 3.1 shows the relative path but in 3.02 does not. In reference to what you stated that SageTV saves the relative path in the title and from what I gather essentially hides it when it is displayed in the STV. What appears to be happening to me in reference to 3.1 is that the relative path is being removed from from the Title in the wiz.bin. Would this not also affect the default STV?

[mkanet]

Hi Sean, I'm not sure if you saw my reply earlier; my apologies if you did and havent had a chance to fully analyze all the stuff in my debug files before replying back. Anyway, I just thought Id put post this again just in case somehow this got overlooked. It did take a while for me to setup everything up for debugging full information on what is needed (tracing only TV recordings directory), hopefully you found out what to do to get fanart images for my movies as well.

Thanks,
-Michael

Originally Posted by mkanet

Sean, thanks! Restoring an older wiz.bin allowed BMT/sageTV to automatically add TV shows and their respective fanart pictures and metadata.

However, there's an issue with movies. For some reason, none of the TV movies I'm recording have fanart (over 15 movies I've tried). The metadata is there when viewing it using the "brose" feature , but fanart is not downloaded. This is for movies that have never been recorded before. In my other video imports directories that I have manual simple names such as "The Pursuit of Happyness.ts" do get fanart.

A big possible clue is that the On Demand feature, as always, can download the fanart for any of these movies (that will not work with automatic). This is true for even movies that only have one simple entry to select from in the On Demand list.

Is there anything I can do so the movies get fanart too? Unless I'm mistaken, I dont have too much control over movie names. Here are some examples that dont work with automatic, but work with On Demand. Bare in mind, that TV show work pretty good.

Here's an example of TV Movies that do not work:
AnalyzeThat-5074192-0.mpg
NurseryUniversity-5046198-0.mpg

Here's an example of a TV show that works just fine:
KeepingUpWiththeKardashians-LeavingtheNest-5077520-0.mpg

PS: Here's another clue, although I can see metadata information for these movies (like in the screenshot below), I dont get any properties files for these movies, just TV shows).



EDIT: I included a screenshot of the web configuration interface browsing the files. It will metadata for a movie, but no fanart pictures... like the way they are for TV series such as the Kardashians. I also included a link to full debugging on while recording a some movies.

screenshot

debug

Thanks,
Michael