Utility: Batch Metadata Importer 3.x

[jaminben]

Ok, So I got a bit impatient and decided to restart the server. I've now noticed that by doing this it also initiates a scan of my imported mediafiles. I can keep stopping/starting the service and it will always restart a scan of these mediafiles from the beginning. It still only affects the files I've imported into the Sage DB and not regular movies.

I've attached an updated log file including a SageTV log as well.

Cheers

Ben

[jdg]

Tried to find this by surching but could not find a solution. after a scan all my videos say disc at the end of the title

Cars Disc
Taken Disc
A Bugs Life Disc

Is there a way to stop this from happening. I know there is something I'm overlooking.

[stuckless]

Originally Posted by jdg

Is there a way to stop this from happening. I know there is something I'm overlooking.

Are you running 3.1? This was one issue that I discovered the "broken" 3.0.3 release.

[gplasky]

I upgraded to 3.1 and I am seeing the same thing to with disc being appended to DVDs and TV movies that the auto-plugin can't find. I also had a SageTV recording rename itself with season-episode number. I think I know how to fix that. (But I fixed it previously and now it's back.)

Gerry

[stuckless]

Originally Posted by gplasky

I upgraded to 3.1 and I am seeing the same thing to with disc being appended to DVDs and TV movies that the auto-plugin can't find. I also had a SageTV recording rename itself with season-episode number. I think I know how to fix that. (But I fixed it previously and now it's back.)

Gerry

Gerry, these were all specific issues that I spent a week testing a fixing in 3.1. Part of the problem in the 3.0 release was that the title code was fragmented, and each subsystem had some small parts of code that would affect the title. All these pieces now rely on a single function to set the title. In my testing, I had lots of titles that had "Disc" at the end, or sage recordings being renamed with the season episode, but only in some cases, ie, using the automatic plugin, etc. With the changes that are in place, you should be able to reproduce this issue by using the web ui to scan the file for metadata (ie, edit metadata and then use the find metadata button). But, could you verify that you are in fact running 3.1, since all these issues were fixed in the 3.1 build (the status screen in the web ui should show versions of all jars). But if they are not, then this is not going to be easy to find

Could you send me the filenames of the mediafiles that are giving you a problem. Also, could you tell me if they are sage recordings, and if they are sage recordings, are they movies or shows, etc. The more information the better for my testing.

[Fuzzy]

It may be that since the data was changed with the 3.03 bugged build, and not that the Disc is in the name, the only way to get them to scan properly again, is to go in and manually remove the Disc from the names. I know I went through the painful process of removing S1D, S2D, etc from all my recordings after I unfortunately did a full scan with 3.03.

[stuckless]

Originally Posted by Fuzzy

It may be that since the data was changed with the 3.03 bugged build, and not that the Disc is in the name, the only way to get them to scan properly again, is to go in and manually remove the Disc from the names. I know I went through the painful process of removing S1D, S2D, etc from all my recordings after I unfortunately did a full scan with 3.03.

I know... I recanned my collection using 3.0.3 and then I spent a week fixing the code to ensure that never happens again (my apologies to you for having to fix this manually)

Using 3.1, I rescanned my collection with the Overwrite Metadata flag turned on from the web ui. If you scanned files using 3.0.3, then you'll need to enable the overwrite metadata flag or else bmt just re-reads what was previously there. The automatic plugin, by default, will never overwrite metadata, but you and enable the overwrite flag from the plugin configuration in the web ui.

The good news, for me, was that by using overwrite metadata option, I was able to completely fix ALL of my media, including DVDs, Sage Recordings, and Movies.

Also, in 3.1, the scan results that you see from the commandline, automatic plugin and web ui, are identical. Prior to this release, you could have gotten subtly different titles depending on which system scanned the files (ie, commandline, web ui, stv on-demand, or automatic plugin).

And one final note about the web ui. When you click "Find Metadata" and you select a result, that information is NOT saved until you hit the save button. YOu can reload the old metadata by simply clicking on the video again item from left media list. When you search for metadata and it loads it into the UI, it will format the title accordingly. But if you change the title to something different in the web ui, and then hit save, then it wil save whatever you put in there. This was very different than the 3.0x releases. In the 3.0x releases, if you changed the title, then bmt would then rewrite the title again upon saving.

[r00st3r]

I seem to be having issues with this new release.

Metadata tools: 3.1
SageTV: 6.6.2.218
Java: 1.6.0_13
SageMC: 6.3.9a pre-release 41
sagemc.jar: 6.3.8
Phoenix.jar: 1.43

Here's what is happening.

In Sage I am using file system with the following structure:
Movies
...12 Rounds
...Ace Ventura
...Across the Universe
...etc etc..

TV
...24
.......Season 1
...................Episode 01
...................Episode 02
...Boston Legal
.......Season 1
...................Episode 01
...................Episode 02
...Chuck
.......Season 1
...................Episode 01
...................Episode 02
...Etc...

which has worked great so far. BMT was working as well. I decided to upgrade both Sage TV and BMT as you can see in the above versions.

I scanned the videos using the web gui and it came up with a bunch of files which it couldn't resolve the metatdata. That's fine. But for every other file, movie or tv, it changed something to where the file listing isn't like it was before in Sage. Now it's all movies, TV shows and Folders with no sub menus like this:

12 Rounds
24
3 Babes, 100 Guns, and a Fat Chick (Las Vegas Episode, not porn)
Ace Ventura
Across the Universe
Boston Legal
Chuck
Chuck Verses the Best Friend (Chuck Episode)
Chuck Verses the First Kill(Chuck Episode)
Coraline
Movies
Sympathy For The Devil (Supernatural Episode)
Transformers
Tremors
Two Guys A Girl and a Pizza Place
*************************************
But it didn't do it to all the files.

The only thing at this top level should be:
...Bluray Movies
...Movies
...Tv
...Workout


All the other movies and tv shows should be listed in the respective "File Listing" just like the directory structure. If I look at the directory structure of the import directory it's still

Movies
...12 Rounds
...Ace Ventura
...Across the Universe
...etc etc..

TV
...24
.......Season 1
...................Episode 01
...................Episode 02
...Boston Legal
.......Season 1
...................Episode 01
...................Episode 02
...Chuck
.......Season 1
...................Episode 01
...................Episode 02
...Etc...

So is it the properties file that drives how the file structure is shown in Sage? I thought File System actually used the file system, not something else.

It's long winded, but I didn't know how else to explain it. Any suggestions?

[stuckless]

Originally Posted by r00st3r

I seem to be having issues with this new release.

Thanks for the verbose detail. Unfortunately, you are not alone. Other people seem to be experiencing the same issue. I don't understand why the File System view in SageMC would not correctly show the File System layout. Have you tried the filesystem view in the default stv to see if it also is affected?

BMT does change metadata, but it does not move files around.

[tmiranda]

Sean,

I just updated to 3.1 and the upgrades are superb. I have not enabled the automatic plugin yet, but the web UI is fantastic.

One small issue. When I download Fanart for archived recordings it clears the archived flag. Not a big issue, but thought you would like to know.

I'll try the automatic plugin over the weekend.

Thanks,

Tom

[stuckless]

Originally Posted by tmiranda

One small issue. When I download Fanart for archived recordings it clears the archived flag. Not a big issue, but thought you would like to know.

I think i know the reason. When new metadata is attached, I have to replace the old show object with a new one (with metadata), and I don't preserve the archive flag. I'll add a bug for that.

[Tofu]

I am using the Metadata Tools web interface, and I am having problems saving the duration/running time. A Metadata find will return the movie length in milliseconds, but once you save it, it defaults back to 1.

Anyone else run into this issue?

Thanks

[stuckless]

Originally Posted by Tofu

I am using the Metadata Tools web interface, and I am having problems saving the duration/running time. A Metadata find will return the movie length in milliseconds, but once you save it, it defaults back to 1.
Anyone else run into this issue?

Is this a DVD? For some reason, I cannot get sagetv to accept a duration value for a DVD.

[Fuzzy]

I've got some general questions about thetvdb and BMT, as well as using the metadata-titles file. My problems stems from this. I've got quite a few (pretty much whole season) of "Special Agent Oso" recorded (disney channel show). It was not finding anything for it, so I checked thetvdb, and sure enough, it wasn't there yet. So, being the swell guy I am, I added it to thetvdb. It is now populated with episode info, and even a banner I made. However, this has not been able to be found with BMT. It just returns no results. I have added Special Agent Oso to my metadata-titles file in hopes that it would point it to the proper database entries on thetvdb, but it has made no differnce. I've also tried to use the metadata-titles to help with some other issuea I have been having with shows with ampersands... ("Max & Ruby" in guide vs. "Max And Ruby" on thetvdb). Is there a way to have BMT search with & swapped interchangably with AND? the problem with having it retitle it to the Max and Ruby from the guides Max & Ruby info, is that it loses association with sage's favorite settings.

my metadata-titles.properties file:

#Title to MediaProviderDataID mappings
#Wed Sep 09 08:57:23 PDT 2009
Max\ &\ Ruby=tvdb\:71741
Merlin\ (2008)=tvdb\:83123
Special\ Agent\ Oso=tvdb\:112931
The\ Bill\ Engvall\ Show=tvdb\:80434
The\ Daily\ Show\ With\ Jon\ Stewart=tvdb\:71256
Word\ World=tvdb\:81312
WordWorld=tvdb\:81312

[r00st3r]

BMT does change metadata, but it does not move files around.

I didn't think it did as my directory structure is the same as it was.
It seems that when there is a properties file generated by BMT, it doesn't know how to organize it in the File System view.

I will test the File System view in default Sage stv and report back.

Thanks!

[stuckless]

Originally Posted by Fuzzy

I've got some general questions about thetvdb and BMT, as well as using the metadata-titles file. My problems stems from this. I've got quite a few (pretty much whole season) of "Special Agent Oso" recorded (disney channel show). It was not finding anything for it, so I checked thetvdb, and sure enough, it wasn't there yet. So, being the swell guy I am, I added it to thetvdb. It is now populated with episode info, and even a banner I made. However, this has not been able to be found with BMT. It just returns no results. I have added Special Agent Oso to my metadata-titles file in hopes that it would point it to the proper database entries on thetvdb, but it has made no differnce. I've also tried to use the metadata-titles to help with some other issuea I have been having with shows with ampersands... ("Max & Ruby" in guide vs. "Max And Ruby" on thetvdb). Is there a way to have BMT search with & swapped interchangably with AND? the problem with having it retitle it to the Max and Ruby from the guides Max & Ruby info, is that it loses association with sage's favorite settings.

Could you post a filename special angent oso and I'll give it try and see what's up. As for the &/And issue... i'm not sure what to do about this. I don't want to blindly replace & and And or vice versa, because that's just going to cause me headaches in other areas It almost needs it's own special configuration file that will lookup a title that will override the current title for a given show... either way, not exactly trivial. I'll add a enhancement request... and I'll see what I can come up with.

[gplasky]

Originally Posted by gplasky

I upgraded to 3.1 and I am seeing the same thing to with disc being appended to DVDs and TV movies that the auto-plugin can't find. I also had a SageTV recording rename itself with season-episode number. I think I know how to fix that. (But I fixed it previously and now it's back.)

Gerry

Originally Posted by stuckless

Gerry, these were all specific issues that I spent a week testing a fixing in 3.1. Part of the problem in the 3.0 release was that the title code was fragmented, and each subsystem had some small parts of code that would affect the title. All these pieces now rely on a single function to set the title. In my testing, I had lots of titles that had "Disc" at the end, or sage recordings being renamed with the season episode, but only in some cases, ie, using the automatic plugin, etc. With the changes that are in place, you should be able to reproduce this issue by using the web ui to scan the file for metadata (ie, edit metadata and then use the find metadata button). But, could you verify that you are in fact running 3.1, since all these issues were fixed in the 3.1 build (the status screen in the web ui should show versions of all jars). But if they are not, then this is not going to be easy to find

Could you send me the filenames of the mediafiles that are giving you a problem. Also, could you tell me if they are sage recordings, and if they are sage recordings, are they movies or shows, etc. The more information the better for my testing.

OK. It looks like I was wrong. I just THOUGHT I updated to 3.1 Turns out it was 3.03. Sorry about that. ANd that explains most of it.

My current issue is on a PC Client.

Phoenix
Version 1.43
sagex.api Version 6.6.1
Fanart Enabled true
Fanart Folder \\MAIN\FANART

Metadata Tools
Version 3.1
Automatic Plugin Enabled (Server) true
Custom Fields Configured true
Last Scan Date Sat Sep 12 12:06:35 EDT 2009
Total Scanned 3
Total Success 2
Last Scanned MediaFile 10th & Wolf
Total Failed 1
(1) Failed MediaFile AFewGoodMen-1664242-0.ts

SageTV
Version 6.6.2.218
Required Version 6.6.1
Java Version 1.6.0
Server true
Recordings Used Diskspace 305.42 G
Library Import Scan in Progress false
JVM Max Memory 780,416 M
JVM Allocated Memory 337,160 M
JVM Total Free Memory 541,089 M

I can't enable the metadata configuration after importing either the STVI in the default or the xml in SageMC. This was working last week. I've looked at it a couple of times now, backed up and did a clean install with ONLY the metadata importer being the only plugin. It says "Disabled" and when I click it or push "enter" it doesn't change. I've made sure none of the files are read-only now. The HD100 and HD200 and the WHS server plugins are fine and can be changed with no issues. My next step is to roll back java unless I'm missing something obvious.

Gerry

[stuckless]

Gerry, try sending me the log file (mailcheck + sagetv_0.txt) and I'll see if I can see an error in there.

[gplasky]

Originally Posted by stuckless

Gerry, try sending me the log file (mailcheck + sagetv_0.txt) and I'll see if I can see an error in there.

Thanks. Give me a minute or so. I put Java 1.6.14 back on there.


File attached below.


Gerry

[stuckless]

Gerry, I've looked at the logs, and I can see this...

Sat 9/12 15:14:56.794 [AWT-EventQueue-0@193c0cf] EXCEPTION in getValueAsObject:sage.a.e: Error in method reflection of bmt_api_IsBMTPluginInstalled of java.lang.reflect.InvocationTargetException for:If(bmt_api_IsBMTPluginInstalled(), "Enabled", "Disabled")

Which is the root of your issue. Unfortunately I don't know why that bmt api call would be throwing an exception... so i'll run down some possible scenarios.

1. You are running the stv plugin on a client, but you haven't installed the bmt plugin on the client machine. (bmt has to be extraced on both the server and client machines)
2. You installed the bmt plugin over an existing plugin, but you didn't "overwrite" the previous jars.
3. You are running vista and somehow UAC is messing with the files
4. something else