New XMLTV Import Tool

[silkshadow]

Holy cow! Thank you so much StephaneM! Can't wait to give it a try tonight!

[pilt down man]

A few questions if I may about your very fine importer:
Installation-
The importer seems to assume that SageTV is installed on the C: drive; mine is installed on the D: drive.
The classic interface does not allow one to select a drive other than C:!
During installation I had pointed to SageTV on the D: drive. Your importer created the 3rdPartyGuideData folder within a new sage directory structure on the C: drive but the xmltv.properties, which was in the SageTV folder on D:, pointed to the 3rdPartyGuideData folder on the D: drive which had not been created.

Suffice it to say the program data was not picked up by SageTV
I moved the the 3rdPartyGuideData from the C: to the D: drive and hey presto all is now sunshine and roses, it works.

Next problem occured when sage did an epg update- the program guide was gone! All the imported info had not been refreshed!
Tried removing capture source and reinstating it after first setting it to no guide data and then to XMLTV again- still no program guide!

This is where I am upto right now- I liked what I saw when it was working so am happy to go with this but there seems to be a problem when the epg gets updated. Your importer again created the new tv guide xml files in the 3rdPartyGuideData on C: drive, eventhough xmltv.properties on the D: drive pointed to D:...3rdPartyGuideData.

So what would your suggestion be?
Should I reinstall the importer again and change xmltv.properties to point to the 3rdPartyGuideData on C:?

Sorry for the long post I hope you understood the problem area.

Regards

[StephaneM]

Hello,

Originally Posted by pilt down man

The importer seems to assume that SageTV is installed on the C: drive; mine is installed on the D: drive.
The classic interface does not allow one to select a drive other than C:!

Yes, in fact the Importer use the "Program Files" environment path, so it default to it. In the classical interface, the "Browse for folder" dialog is also rooted to this path, but you can type the path in the text box without problems.

During installation I had pointed to SageTV on the D: drive. Your importer created the 3rdPartyGuideData folder within a new sage directory structure on the C: drive but the xmltv.properties, which was in the SageTV folder on D:, pointed to the 3rdPartyGuideData folder on the D: drive which had not been created.

That's because the "SageTV path" property used in the Installer is not carried over to the Importer after the install. So if you only change the path manually in the Importer to your SageTV directory it will be fine.

I'll change the Installer so as it correctly set the SageTV path for the Importer.

Regards,
Stéphane.

[popechild]

Stephanie, I've got a question about the way the EPGID is working.

I'm on 1.8 (full) running basically at default settings. I've been having a problem lately where certain shows are getting non-unique showIDs (the "EP" variety) even though they're new, first-run episodes.

Two examples I've noticed in the last few days:
1) Rock Star Supernova - there's a "performance show" and a "results show" each week. The results show seems to (usually) have a new EP showID every week, the competition show is re-using the same showID for every episode. Unfortunately, this caused me to miss a recording (the wife was NOT happy!) because I didn't remember to manually record the episode - thinking it would be automatically picked up by the series recording - but it didn't record since Sage saw it as "Watched" already.

2) Access Hollywood - fortunately I have no desire to do a series recording on this, because it seems it would be impossible, as every single episode (at least in the next two weeks of my EPG) uses the exact same showID (EP).

I've been talking to Andy (Opus4) in another thread about this and he's told me that there's no way within Sage to have it record "all episodes, regardless of showID" in a series recording. He's also seeing unique showIDs for every episode of Rock Star. So apparently the way my importer is set up, I'm somehow getting non-unique episode IDs for unique episodes, and I need to figure out how to stop this from happening.

What's new :
Changed (1.7) : Zap2it EPGID support is disabled by default. This is because import times are way longer when this is enabled and apart keeping history of watched shows you'll have no real benefits when using Zap2it EPGID (Also the Importer is sometime forced to not use the exact same EPGID)

Does this reference what I'm dealing with? Is the "apart from keeping history of watched shows" what I'm missing that is causing me problems?

Can you recommend a way for me to setup the importer so that I won't have this problem anymore?

Thanks!

[StephaneM]

Hello,

Originally Posted by popechild

I've been talking to Andy (Opus4) in another thread about this and he's told me that there's no way within Sage to have it record "all episodes, regardless of showID" in a series recording. He's also seeing unique showIDs for every episode of Rock Star. So apparently the way my importer is set up, I'm somehow getting non-unique episode IDs for unique episodes, and I need to figure out how to stop this from happening.

Basically, the Importer try to generate ShowID with the information he can found about shows : title / episode title or number / description / airing date.

The Importer offers three detection methods for ShowID generation :

* based on episode title / numbers only : the Importer will create a new ShowID for episodes with different ep. title / numbers

* based on ep. title or description : the Importer will create a new ShowID for episodes with different ep. title or ep. description (it means that if an ep.title is found the ep. description is not used)

* based on ep.title or airing date : the Importer will create a new ShowID each day (if an ep.title is not found).

There is also the possible use of the Zap2it ShowIDs. Zap2it does include ShowID in the guide data. However as the Importer is supposed to work with other data sources and be "conservative" about ShowIDs, it always try to work with its own logic, that is creating ShowIDs with the help of ep.title / description etc...

So when it see a Zap2it ShowIDs the Importer will need to validate it against its own ShowID database (just in case you are using another datasource without Zap2it ShowID and still retreive the right ShowID).

Unfortunately for the same show Zap2it can issue more than one ShowID, especially when the show is aired on multiple channels and is of type "News". While it does make sense, this is not what the Importer is expecting and as such the Importer is constantly rewriting his ShowID database.

For instance, the Importer will import the show "Late News" and find the a ShowID AAXX, it accepts the ShowID and store it for later use, then it import a new "Late News" and this time the ShowID is BBYY (whereas it should have been AAYY), the Importer must then lookup all AA-- ShowIDs in its database and change them to BB-- (and this take a very very long time).

That's why the Zap2it ShowID are not used by default in the latest version of the Importer, and I recommend to not use it.

So what you can try is to use the latest method of the Importer regarding unique episode detection : "use ep.title or airing date". This will guarantee you that at least one episode by day will be recorded (when show informations are not sufficient to differentiate unique episodes).

Regards,
Stéphane.

[popechild]

Okay Stephanie, I'll give that a try...

[popechild]

That did the trick! Of course, I guess now the downside is that if the same episode of a show actually does air multiple times on different days, my series recording will record it every time, right?

Is there a "middle road" somehow that would allow it to work the way it's supposed to (not record episodes that have already been watched) for shows that have the proper guide data, but allow it to record all episodes (for safety) of shows that I realize don't get good guide data?

Does the "first run/rerun" option when setting up a series recording make any differnce for me now or is just going to record at least one episode a day no matter what? Sorry for the confusion, I'm just over my head a little in understanding exactly how all the showID stuff works.

Thanks.

[StephaneM]

Originally Posted by popechild

That did the trick! Of course, I guess now the downside is that if the same episode of a show actually does air multiple times on different days, my series recording will record it every time, right?

Yes it will record it every time. But this is only for shows that doesn't have an episode title or episode number (and movies are not concerned).

So this is already the "middle road" you suggest.

Does the "first run/rerun" option when setting up a series recording make any differnce for me now or is just going to record at least one episode a day no matter what?

First run / rerun, will work the same way. For shows with enough data, the Importer either use the XMLTV information or its history, for shows whose ShowID is generated using the airdate they will be all first run (as they should).

To make it short : movies and series will be ok if they are correctly handled in the XMLTV file (that is Movies does have a "movie" category and Series does have an episode title or number), all other shows will have a unique ShowID per day (so : news, sports etc...)

Regards,
Stéphane.

Regards,
Stéphane.

[popechild]

Originally Posted by StephaneM

Yes it will record it every time. But this is only for shows that doesn't have an episode title or episode number (and movies are not concerned).

So this is already the "middle road" you suggest.



First run / rerun, will work the same way. For shows with enough data, the Importer either use the XMLTV information or its history, for shows whose ShowID is generated using the airdate they will be all first run (as they should).

To make it short : movies and series will be ok if they are correctly handled in the XMLTV file (that is Movies does have a "movie" category and Series does have an episode title or number), all other shows will have a unique ShowID per day (so : news, sports etc...)

Regards,
Stéphane.

Regards,
Stéphane.

Fantastic! Thanks for the great product!

[CPA]

Was wondering if there is a way to display the year of release for movies in the Guide and detail info. I see on zap2it that the year is included in the description of each movie. Any ideas?

[StephaneM]

Hello,

Originally Posted by CPA

Was wondering if there is a way to display the year of release for movies in the Guide and detail info. I see on zap2it that the year is included in the description of each movie. Any ideas?

The year of the movie is already handled by the Importer, so you should see it in the guide and when viewing program information (Though SageTV display the date only for movies, so the Zap2it XMLTV must describe the show as a movie and I think this is not the case)

Regards,
Stéphane.

[CPA]

That would explain another mystery. I am useing SageMC which has a movie filter in the MY TV section, but it has yet to populate. I would imagine this is becasue it is looking for the movie tag and there isn't one to be found from zap2it. Is there any fix for this or am I out of luck?

[StephaneM]

Originally Posted by CPA

Is there any fix for this or am I out of luck?

Currently there is nothing you can do about this, I'll issue an updated version of the Importer to take care of this.

Regards,
Stéphane.

[CPA]

That's great! Thanks again for all your help. I'll keep my eyes open for the update.

[popechild]

Originally Posted by StephaneM

Yes it will record it every time. But this is only for shows that doesn't have an episode title or episode number (and movies are not concerned).

So this is already the "middle road" you suggest.

Well, I thought this would solve my problem but it only took about two days for it to bite me in the a** again.

Last night there was a two hour finale of the show Treasure Hunters, which I had a series recording set up for. The finale was broken up in the EPG data into two one-hour blocks though, and they were both listed as the same title/description. But apparently because of the "one unique episode per day" thing it only recorded the first hour and not the second - which was the finale!

So that brings me right back to where I was before - I'd really like to be able to have an option to just "record all" of a particular series, regardless of whether it's new/old/has been recorded/watched/etc. Looking at zap2it.com they show different info for the two hours last night, so I don't know if I would've gotten it recorded properly using the "built in" EPG data that Sage gets or not, but somewhere along the way in using the xml importer I'm having problems that continue to confuse Sage as to what it should record.

Is there just a way for me to "record all episodes?"

Or should I just switch over to the zap2it showIDs? What does it mean that this will take a "long, long time?" I let it run overnight automatically, so it could take a while and still be okay as long as that's still the only time it did processing...

Thanks for any help...

[StephaneM]

Originally Posted by popechild

Well, I thought this would solve my problem but it only took about two days for it to bite me in the a** again.

Well, using XMLTV (and even builtin SageTV EPG), you should always review what is scheduled to record (other issues exist even with the built in EPG, for instance when a show is misspelled). If you want you can use the Nielm Web Server Plugin: this is very handy to check all your recordings from a Web Browser.

Is there just a way for me to "record all episodes?"

No

should I just switch over to the zap2it showIDs? What does it mean that this will take a "long, long time?"

Yes you can try, just edit the advanced_settings.xml file (in the Install folder of the Importer) and change <use_zap2it_epgid value="False" /> to <use_zap2it_epgid value="True" /> and do the Import again.

The time it will take depends on the number of channels. I guess it will take twice the time it used to take (or more).

Regards,
Stéphane.

[popechild]

Originally Posted by StephaneM

Well, using XMLTV (and even builtin SageTV EPG), you should always review what is scheduled to record (other issues exist even with the built in EPG, for instance when a show is misspelled). If you want you can use the Nielm Web Server Plugin: this is very handy to check all your recordings from a Web Browser.

Yeah, I know I should, but that's kind of like saying I should floss every day, and that certainly doesn't happen either. Not to mention, it sort of takes something away from the "set it and forget it" nature of series recordings.

The lack of a "record all" option is one of the most frustrating things about Sage to me (at least compared to my old Tivo, which had that as a standard option). Especially now that I've missed two recordings in as many weeks, after just starting down the "series" road about a month ago. It doesn't give me a lot of hope for the fall tv season...

I know that's an issue inherant with Sage, not just your importer, but it would be nice to have an option, say, to just give every show a unique showID no matter what. Then Sage would record them all for people that wanted it that way (like me ). Don't know how hard that would be to add...

Yes you can try, just edit the advanced_settings.xml file (in the Install folder of the Importer) and change <use_zap2it_epgid value="False" /> to <use_zap2it_epgid value="True" /> and do the Import again.

The time it will take depends on the number of channels. I guess it will take twice the time it used to take (or more).

I'll give it a shot. If that's all it takes it shouldn't be a problem at all, since I'm only using OTA HD and only have about 20 channels.

[popechild]

I don't appear to have an "advanced_settings.xml" only a "settings.xml" which doesn't have the setting you described. This is in the main install folder where the executable is...

(I also have "channels.xml, clone_channels.xml, import-log.xml, lineup.xml, lookup.xml)

[StephaneM]

Originally Posted by popechild

I don't appear to have an "advanced_settings.xml" only a "settings.xml" which doesn't have the setting you described. This is in the main install folder where the executable is...

Sorry, the "advanced_settings.xml" file is available for download on the FAQ page.

Regards,
Stéphane.

[popechild]

Originally Posted by StephaneM

Sorry, the "advanced_settings.xml" file is available for download on the FAQ page.

Regards,
Stéphane.

Thanks. I used it to import successfully and it only took 5 minutes or so. Plenty short enough since I run it overnight. We'll see if this helps!

Thanks...