Metadata Tools 4.0 Beta Thread (aka BMT 4.0)

[stuckless]

Just wanted to give a quick update...

- I had to take a few days away from bmt (was getting burned out between this, and some other stuff). But I have noticed the same issue, that others have reported, in that since b15, the automatic plugin is not scanning media. I especially noticed this this weekend, when I added 4 new movies to my collection, and none of them scanned. I'll try to resolve this issue later this week and post up a new build. (it looks like my files were not accepted by the filter, which is what @peterreader noted above)

- MeInMaui suggested that I add a Wiz.bin backup, so I've added that. You can now easily backup the wiz.bin from th webui. (most people probably do this in some form already, using some other means)

Many thanks to everyone that has been helping with this beta so far. And thanks to those that have used the new "Support" tool to create support zips... that's been handy. BTW, when you create the zip, you don't need to include anything other than the logs, unless I specifically ask for the imported files details. Adding the imported filenames makes for a large zip file

[newpath]

BMT Version: 4.0 Beta 15
SageTV Version: 6.6.2.218
SAGEMC Version: 6.3.9a
Java Version: 1.6.0_17
Phoenix Version: 1.62
FanArt Folder Enabled - Local
SageTV Automatic Scans: 1 hour
Import New Videos: Yes
Commandline Tool: no success
On-Demand Plugin: Imported and Enabled
Automatic Tool properties set in sage.properties

I am trying to get BMT to work properly in supplying metadata for TVExplorer and, for the most part, it works OK. However, I have run into a problem that is driving me mad and it is related to TV series that are repeats of past series of the same name. Battlestar Galactica is one example that is giving me problems. It was televised in the 1970s and updated in the 2000s. I have the 2003+ series and the folder/file structure as follows: E:\Television\Battlestar Galactica\Season 01\Battlestar Galactica - s01e01 - 33.mkv (as an example).

Whenever I do a scan in the webui, it always retrieves metadata from the 1970s series, even though it lists the correct file names as it scans. I have tried the following in the MediaTitles.xml file:

<titles>
<match>
<regex>[\\/]Battlestar Galactica[\\/]</regex>
<title>Battlestar Galactica</title>
<year>2003</year>
<metadata type="tv" name="tvdb">73545</metadata>
</match>
</titles>

This did not effect the problem at all. There is precious little documentation for BMT and I can find nothing the forums related to this issue. Is there anything I can do to get the correct metadata for this, and other, repeated TV series? It is obviously something I do not know or am overlooking, but I would sure like some help with the proper procedure.

Thanks in advance.

[stuckless]

@newpath - my apologies for the lack of documentation... being a 1 man show, it tough to provide adequate support/documentation in addition developing the actual application

When you changed the mediatitles, did you restart sagetv server? Currently that file in only read on startup.

I'll make a note to test the media titles again tonight... I know they have worked in the past, but something may be broken.

[newpath]

Sean,

I reread my previous post and apologize for the negativity regarding the comment about documentation. I did not mean it to be critical, but rather to say that I could not find a solution to my problem. I very much appreciate your work with BMT and it's great software. Again, I am sorry.

To update, the problem I described occurs if I do a scan on the Battlestar Galactica folder or on individual season folders. I can get the correct metadata if I do a scan of each individual episode because it pauses to let you select which series (1970s or 2000s) you would like to download. This is pretty time consuming, however.

Let me know how your tests come out and thanks for your efforts. By the way, I did restart SageTV after modifying the MediaTitles.xml file.

[petereader]

This may be a dumb question, but I'm curious enough to ask anyways. Would it be possible to do something like the movie file naming convention, but for tv series, i.e. Battlestar Galactica (2003) - S##E##.avi ?

[stuckless]

Originally Posted by petereader

This may be a dumb question, but I'm curious enough to ask anyways. Would it be possible to do something like the movie file naming convention, but for tv series, i.e. Battlestar Galactica (2003) - S##E##.avi ?

Hey Peter, it's not a dumb question, but I'm not sure I understand it Are you asking if the year, for a tv filename, could be parsed out of the filename? If, so, then yeah, I think that's possible... I don't currently do that, but it's a good idea. I added that ability for the movies, but not tv.. i guess that's more of an oversight on my part.

[petereader]

Yeah, sorry. Spent so much time thinking about it (after all, I'm at work, what else have I got to do ) that I think half the question never made it to post. Dusting off the cobwebs in my head sometimes just reveals the spaghetti!

I was thinking about newpath's issue with Battlestar Galactica (I have files from this series as well); rather than using a MediaTitles.xml file to distinguish between the 70's series and the one from 2003, I figured it might be easier to use the file's title to make the distinction (or at least another option if the mediatitles.xml isn't working for a particular show for whatever reason).

I was really more curious if the functionality was already there. I believe mediatitles.xml should handle the issue just as well (plus provides the ability to include other parameters than just the date if that becomes necessary - more flexibilty is always better, right?).

[Tiki]

The mediatitles.xml file should allow you to force a particular match for a series. So, if you only care about the "new" Battlestar Galactica, you can force it to always use the new fanart.

However, the tricky part is what to do if you want to be able to present correct fan art for both the old and new Battlestar Galactica series. I don't think that anyone has come up with a way to fully address this issue. And of course, there seems to be a popular trend of remaking old TV series, so there are several other shows that have the same issue as B.S.G.

There are three pieces to this puzzle.
1. First, when obtaining fanart from the internet or some other provider, the "Scraper" must determine which fan art to grab.
2. The Scraper must have a way to store both the old (1970's episodes) and the new (2000's episodes) fan art even though both series have the same name.
3. There must be a mechanism for the STV to retrieve the correct fan art so it can be displayed.

I think it should be possible to do, but it would probably take a fair amount of effort. I think it would have to rely on decent metadata from the EPG. Assuming the Original Air Date was correct, it should in theory be possible to set up a rule in the metadata.xml file (or a similar file) that compares that date to some pre-defined date range (eg: if original air date < Jan 1, 1985 use "BSG-old"; if original air date > Jan 1, 1985 use "BSG-new).

This would take care of Sage Recordings. Imports would have to rely on the files or folders being labeled with a year.

I don't know if this is a task that Stuckless is willing to take on, but it would be nice.

[stuckless]

Here's beta 16

Fixes
1. Fixed Bug where automatic plugin was skipping files ALWAYS
2. Fixed Bug where skipped items were being consider "success" items
3. Fixed Year bug in imdb... will have better hits for years like (2009/I)
4. Fixed Bug in the Processor where when there was a year in the query, but a year match wasn't found in the list, then it return nothing, but it should have still returned a "good match"
5. Fixed Bug in how multi-volume titles were being processed (should increase hits for Movie cd1, Movie cd2, etc)
6. Added backup to the tools menu

Basically this build adds better fuzzy logic support for title matching. I also found a number of bugs in the process.

I did do some unit testing on the MediaTitles.xml and they do appear to be working in my environment, so no changes there.

I added wiz.bin backup tool to the tools menu (thanks to MeInMaui for the suggestion). I'll do more with it later, but for now, you can use to to make a quick backup of your wiz.bin before doing a scan. I'm considering adding an autobackup feature, but I haven't gotten around to that yet.

Good luck and thanks for testing.

[stuckless]

Originally Posted by Tiki

The mediatitles.xml file should allow you to force a particular match for a series. So, if you only care about the "new" Battlestar Galactica, you can force it to always use the new fanart.

However, the tricky part is what to do if you want to be able to present correct fan art for both the old and new Battlestar Galactica series. I don't think that anyone has come up with a way to fully address this issue. And of course, there seems to be a popular trend of remaking old TV series, so there are several other shows that have the same issue as B.S.G.

There are three pieces to this puzzle.
1. First, when obtaining fanart from the internet or some other provider, the "Scraper" must determine which fan art to grab.
2. The Scraper must have a way to store both the old (1970's episodes) and the new (2000's episodes) fan art even though both series have the same name.
3. There must be a mechanism for the STV to retrieve the correct fan art so it can be displayed.

I think it should be possible to do, but it would probably take a fair amount of effort. I think it would have to rely on decent metadata from the EPG. Assuming the Original Air Date was correct, it should in theory be possible to set up a rule in the metadata.xml file (or a similar file) that compares that date to some pre-defined date range (eg: if original air date < Jan 1, 1985 use "BSG-old"; if original air date > Jan 1, 1985 use "BSG-new).

This would take care of Sage Recordings. Imports would have to rely on the files or folders being labeled with a year.

I don't know if this is a task that Stuckless is willing to take on, but it would be nice.

Tiki, I have thought about this, and what I was intending to do, was adding the year to the Fanart folder. I don't know if it's as easy as that or not. Also, I'll be adding YEAR support to the tv scrapers so that if you have the year in the show filename.

[newpath]

Sorry I took so long to get back, but I have been away.

Sean,

I never got the MediaTitles.xml method to work.

@petereader - I agree, not a dumb question and I had several file renaming options I was going to try and this was one of them. On your suggestion, I moved it to the top and did the following:

Old Name: Battlestar Galactica-e01s01-33.mkv
New Name: Battlestar Galactica (2003)-e01s01-33.mkv

I did this for all the BS mkvs and even used 2003 for all seasons just to see what would happen. Well, jump and shout and knock yourself out! It worked perfectly for all seasons. I did the same with The Prisoner, which was a remake of an earlier seires, by using (2009) in the file name and it worked as well.

These are the only two TV series that I have that presents this particular problem and in the future I can account for it by including the year in the title name. However, it would be easier to use the MediaTitles.xml. I did not try it, but I wonder if using the title "Battlestar Galactica (2003)" in the xml file would have worked. Now that I have perfect metadata I am reticent to try that approach.

Thanks for everyones' help. If I can be of assistance, let me know.

[LehighBri]

With beta16, I still get the following issues that I had in beta10:

System Messages
Failed to find metadata for: D:\Media\Movies\DVDs\9 (2009)\VIDEO_TS using provider imdb-2,themoviedb.org,imdb.xml
MY RESPONSE: THIS SHOWS UP AS 9 (2009/I) ON IMDB

Failed to find metadata for: D:\Media\Movies\DVDs\Entourage Season 2 Disc 3\VIDEO_TS using provider imdb-2,themoviedb.org,imdb.xml
MY RESPONSE: TV SERIES SPANNING MULTIPLE CDS STILL NOT WORKING FOR ME

Failed to find metadata for: D:\Media\Movies\DVDs\Entourage Season 2 Disc 2\VIDEO_TS using provider imdb-2,themoviedb.org,imdb.xml

Failed to find metadata for: D:\Media\Movies\DVDs\Entourage Season 2 Disc 1\VIDEO_TS using provider imdb-2,themoviedb.org,imdb.xml

Failed to find metadata for: D:\Media\Movies\DVDs\Entourage Season 1 Disc 2\VIDEO_TS using provider imdb-2,themoviedb.org,imdb.xml

Failed to find metadata for: D:\Media\Movies\DVDs\Entourage Season 1 Disc 1\VIDEO_TS using provider imdb-2,themoviedb.org,imdb.xml

Failed to find metadata for: D:\Media\Movies\DVDs\Tudors S01 D04\VIDEO_TS using provider tvdb

Failed to find metadata for: D:\Media\Movies\DVDs\Tudors S01 D03\VIDEO_TS using provider tvdb

Failed to find metadata for: D:\Media\Movies\DVDs\Tudors S01 D02\VIDEO_TS using provider tvdb

Failed to find metadata for: D:\Media\Movies\DVDs\Tudors S01 D01\VIDEO_TS using provider tvdb

[stuckless]

[QUOTE=LehighBri;412624] Failed to find metadata for: D:\Media\Movies\DVDs\9 (2009)\VIDEO_TS using provider imdb-2,themoviedb.org,imdb.xml
MY RESPONSE: THIS SHOWS UP AS 9 (2009/I) ON IMDB
[quote]

This'll be fixed in the next build. There was a bug in the similarity scorer where titles less than 2 chars would not be scored.

Failed to find metadata for: D:\Media\Movies\DVDs\Entourage Season 2 Disc 3\VIDEO_TS using provider imdb-2,themoviedb.org,imdb.xml
MY RESPONSE: TV SERIES SPANNING MULTIPLE CDS STILL NOT WORKING FOR ME

What's interesting about this, is that it doesn't detect this as being a TV style show, and it should. (I'll check this tonight)

Failed to find metadata for: D:\Media\Movies\DVDs\Entourage Season 2 Disc 2\VIDEO_TS using provider imdb-2,themoviedb.org,imdb.xml

Failed to find metadata for: D:\Media\Movies\DVDs\Entourage Season 2 Disc 1\VIDEO_TS using provider imdb-2,themoviedb.org,imdb.xml

Failed to find metadata for: D:\Media\Movies\DVDs\Entourage Season 1 Disc 2\VIDEO_TS using provider imdb-2,themoviedb.org,imdb.xml

Failed to find metadata for: D:\Media\Movies\DVDs\Entourage Season 1 Disc 1\VIDEO_TS using provider imdb-2,themoviedb.org,imdb.xml

Failed to find metadata for: D:\Media\Movies\DVDs\Tudors S01 D04\VIDEO_TS using provider tvdb

Failed to find metadata for: D:\Media\Movies\DVDs\Tudors S01 D03\VIDEO_TS using provider tvdb

Failed to find metadata for: D:\Media\Movies\DVDs\Tudors S01 D02\VIDEO_TS using provider tvdb

Failed to find metadata for: D:\Media\Movies\DVDs\Tudors S01 D01\VIDEO_TS using provider tvdb

In 4.0, I basically throw an error if i can't get actual season and episode information. I am looking at adding just series metadata, for these cases, but I changed it to the current way, because sometimes if a season and episode wasn't found then it could overwrite more specific information will less speciif (series) information. I guess I need to revist the series stuff.

[petereader]

Originally Posted by newpath

I did this for all the BS mkvs and even used 2003 for all seasons just to see what would happen. Well, jump and shout and knock yourself out! It worked perfectly for all seasons. I did the same with The Prisoner, which was a remake of an earlier seires, by using (2009) in the file name and it worked as well.

newpath-Glad you got the problem worked out. Now that Sean has the automatic plugin working again, I'll try this on my Battlestar Galactica shows and see if I get the same result.

Originally Posted by stuckless

Also, I'll be adding YEAR support to the tv scrapers so that if you have the year in the show filename.

Sean,
Did you actually get this in for Beta16 or is this just something that was previously incorporated?

[stuckless]

Originally Posted by petereader

Sean,
Did you actually get this in for Beta16 or is this just something that was previously incorporated?

Not yet... I don't see this as being a huge effort (since I already do it for the movie scrapers). I'll see if I can get it into b17.

[petereader]

I was just wondering how it seemed to work for newpath. I'm going to try it myself when I get home tonight.

[stuckless]

This works well for newpath, because the tvdb entry is actually called, "Battlestar Galactica (2003)", which results in an exact match.

[petereader]

Gotcha. So this may not work well for other shows until you update the scraper.

[Tiki]

Originally Posted by stuckless

Tiki, I have thought about this, and what I was intending to do, was adding the year to the Fanart folder. I don't know if it's as easy as that or not. Also, I'll be adding YEAR support to the tv scrapers so that if you have the year in the show filename.

I think that's at least part of what you need to do. What makes it tricky is that with TV shows, they can run for many years, so what year do you append to the show title? Most likely you would want to use the year of the 1st episode (provided you can get that info from the TVdb.com or other provider). Or, would you make a separate folder for each year that the series ran? This would lead to a lot of duplication (think about how many shows run for 5 or 10 years)...

Assuming you choose to stick with a single fanart folder for the entire series, once you append a year to the folder name it will no longer match the show title exactly. The Phoenix APIs will need a way to match the show to the fanart. Either you need to do some fuzzy logic to compare the original air date to the year label that you appended to the folder name, or you will need another piece of metadata stored somewhere that points to the fanart folder location.

The problem with relying on a metadata pointer is that if you choose to store it in Wiz.bin as a custom metadata item, it will only be present on episodes that you specifically scan. Right now, I just have to scan one episode of a show and after that the fanart will be there for all future airings and episodes.

The problem with the fuzzy logic approach is that the EPG data isn't always complete or 100% accurate, so original air date is not always set.

Do you have another idea that I haven't thought of that would get around these problems?

[ranger]

Originally Posted by Tiki

I think that's at least part of what you need to do. What makes it tricky is that with TV shows, they can run for many years, so what year do you append to the show title? Most likely you would want to use the year of the 1st episode (provided you can get that info from the TVdb.com or other provider). Or, would you make a separate folder for each year that the series ran? This would lead to a lot of duplication (think about how many shows run for 5 or 10 years)...

I vote for sticking with the convention set by TVDB.com as it effectively differentiates between different series with the same name as opposed to different seasons of the same series.