New XMLTV Import Tool

[StephaneM]

What's new

• Improved : Movies detection for Zap2it EPG data.
• Added : support for sound format and aspect ratio XMLTV nodes (unfortunately, those properties are attached to the show and not on particular airings)

[rsagetv99]

Have you given anymore thought to allowing a text file to store or import the XMLTV channel numbers? I have a script that reads the XMLTV and MyTheatre XMLTV files and derives the channel numbers and names automatically. I can create one formatted to your specs, just let me know if it is do-able.

[StephaneM]

Originally Posted by rsagetv99

Have you given anymore thought to allowing a text file to store or import the XMLTV channel numbers?

No, as the channel editor now is perfectly responding to keyboard input (that is there is no need to use a mouse).

I have a script that reads the XMLTV and MyTheatre XMLTV files and derives the channel numbers and names automatically. I can create one formatted to your specs, just let me know if it is do-able.

If you have automatic requirements, then you can easily edit (manually or with a script) the channels.xml file (and clone_channels.xml if needed).

The lineups.xml / channels.xml / clone_channels.xml are never changed by the Importer unless you are using the channel editor. So any modifications you can make to those files will be left untouched and used as is by the Importer.

lineups.xml : this one defines the lineups to be created. Each lineup is described by a "lineup" node, with attributes (mandatory) "id" and "name". "id" is the lineup number (the one you want) and "name" is for the lineup name as it will appear in SageTV when configuring tuners.

channels.xml : this one is the main channel list, it should contain all the channels found in all your XMLTV files (the Importer will include all channels found, but you are not required to include them all). Each channel is described by a "channel" node with attributes : "id", "name", "call-sign", "channel" and "lineup". "id" is for the XMLTV id of that channel, "name" and "call-sign" are self explanatory, "channel" is the channel number used for tuning, "lineup" is the lineup id where the channel should be (the lineup id is one of the "id" found in the lineups.xml)

clone_channels.xml : this one may contains duplicate channels. The Importer put duplicate channels in this file so as only the main channel list is to be in synch with the one found in the XMLTV files. The structure of the XML document is the same as channels.xml except that each channel node can have a "same-as" attribute. This attribute may be emtpy or contains the XMLTV id of the channel to be duplicated.

You can even add additionnals attributes to nodes if you want, these attributes (or even nodes) will not creates any problems for the Importer (and they will be retained even if you use the Importer channel editor).


Regards,
Stéphane.

[ekiwi]

I have upgraded to the new beta and now 2 days in a row when using this importer I loose all scheduled recordings

on looking at the logs

immediately before the import i have things scheduled to record immediately after i don't

I posted the following in the beta forum

http://forums.freytechnologies.com/f...ad.php?t=20461

Nothing is scheduled to record!!!!!
this has happened twice since i installed the beta

i open view recording schedule and there is nothing in the list. (I checked for filter but I haven't set any)

i look at a program in the guide that should be set to record, it is not highlighted with a white outline and if i look at its record options I can Add it as a new favorite rather than options to edit or delete

If i look at manage favorites all the favorites are there and if i select any under additional airings I can see all upcoming programs but none of them are set to record

restarting the service/client and reimporting the guide makes no difference.

The first time this happened i had modified the stv using reported ok stvis but the second time i had nothing other than the original stvi that comes with the beta

in both cases it worked for more than 24 hours before they disappeared and during that time i had forced manual imports of the xmltv data just to check if that was working ok

to resolve the problem i just replaced the wiz.bin with the old one that i backed up before upgrade to beta.

My system

java 1.5.0.02

the following load at startup
net.sf.sageplugins.webserver.StartServer;
net.sf.sageplugins.getstatus.GetStatus;
net.sf.sageplugins.sagexmlinfo.RecordingXmlGenerator

and the StephaneM's xmltv importer

just remembered about the above sageplugins so i will remove them as well in case they are the issue

any other ideas? If no one can think of anything stupid i might be doing I will submit it as a bug

-ekiwi

[StephaneM]

Yes, that's right, I noticed this too this morning.

Actually the guide data is imported and all seems to be fine (the guide is populated) but you cannot make any search through the data, so the favorites are not working.

I've submitted a bug report to SageTV, until then you should revert to a non beta version of SageTV.

Regards,
Stéphane.

[ukmgranger]

Any news on this problem yet?

Originally Posted by StephaneM

Yes, that's right, I noticed this too this morning.

Actually the guide data is imported and all seems to be fine (the guide is populated) but you cannot make any search through the data, so the favorites are not working.

I've submitted a bug report to SageTV, until then you should revert to a non beta version of SageTV.

Regards,
Stéphane.

[StephaneM]

Originally Posted by ukmgranger

Any news on this problem yet?

There is some news but it's only a temporary fix until a new SageTV beta comes out.

Also if you made a donation you should have received an e-mail about this.

Regards,
Stéphane.

[owilsky]

Does the new beta (6.0.12) already contain the fix you need?

[StephaneM]

Originally Posted by owilsky

Does the new beta (6.0.12) already contain the fix you need?

No, it should be in SageTV beta 6.0.13 (when all be ready a new version of the XMLTV Importer will be available).

Regards,
Stéphane.

[AtariJeff]

I've just started playing with this and it's pretty slick. One thing I noticed though, was that it doesn't seem to bring in the extended ratings such as language, etc. Should that be working? They are in the XML file from MyTheatre. With young kids in the house I am using the parental controls.

[StephaneM]

Hello,

Originally Posted by AtariJeff

One thing I noticed though, was that it doesn't seem to bring in the extended ratings such as language, etc. Should that be working?

No, only MPAA ratings are retrieved for now. Parental ratings will be included in the next version.

Regards,
Stéphane.

[Mark SS]

Can I wave a potential solution to the series/episode identification problem?

In http://forums.sagetv.com/forums/showthread.php?t=20838 this TV database site is discussed - http://tvdb.zsori.com/.

So, where we get EPG data that includes an episode number and name, it would be possible to query the zsori database to retrieve a unique series and episode id?

[silkshadow]

Can I wave a potential solution to the series/episode identification problem?

In http://forums.sagetv.com/forums/showthread.php?t=20838 this TV database site is discussed - http://tvdb.zsori.com/.

So, where we get EPG data that includes an episode number and name, it would be possible to query the zsori database to retrieve a unique series and episode id?

I would second that! Maybe you could find a way to make it work with the media center (imported video)?

[StephaneM]

Originally Posted by Mark SS

Can I wave a potential solution to the series/episode identification problem? .../... So, where we get EPG data that includes an episode number and name, it would be possible to query the zsori database to retrieve a unique series and episode id?

Well, for my Importer this is not required : it already retreive a unique Series and Episode ID if you have the show title and episode title or number.

Querying this database would raise significantly the import duration and is not useful for non English speaking countries.

Again the only problems any XMLTV tools will have regarding unique IDs is for shows that doesn't carry episode title or meaningfull description, for all other shows the XMLTV Importer does a great job.

Other XMLTV plugins may benefit from this database (at least for English speaking countries) because they doesn't keep an ID database on their own (whereas my Importer does it).

Regards,
Stéphane.

[xred]

So with the hour change this weekend here in Canada, I woke up to find that all of the EPG data imported from XMLTV sources were now off by an hour (ie shows that start at 4pm now show as 5pm). This happened only to data brought in by the importer, the data the Sage EPG is grabbing is still correct after the hour change.

So to resolve this issue, I fired up the Importer and unchecked "Use Daylight Savings time" on both my XMLTV sources and re ran the importer.... to find that everything is still off by an hour. Anyone else having this problem with the importer? I would rather NOT change the offset to "4" (Atlantic time) from "5" (EST) though I'm sure it would solve the problem.

TIA

[StephaneM]

Originally Posted by xred

So with the hour change this weekend here in Canada, I woke up to find that all of the EPG data imported from XMLTV sources were now off by an hour (ie shows that start at 4pm now show as 5pm). This happened only to data brought in by the importer, the data the Sage EPG is grabbing is still correct after the hour change.

Daylight saving time can be an issue because everything must be in sync :

* SageTV : it does it jobs right so no problem here
* The Importer : it will, if you ask it to, add/remove an hour when it needs to
* XMLTV data : may or may not add/remove an hour for DST.

The Importer was designed with the assumption that the time in your XMLTV file is ALWAYS local time and that it have to convert this time into universal time by adding/removing hours (adjusting for DST if needed).

So let's say your actual time zone is GMT-5. Your XMLTV file will display time as 2000 GMT-5, for a show that is really airing at 20h.

Then you have to put "5" in the Importer, so as it can add five hours to 20h00 to have universal time.

When DST is in use your time zone will change to GMT-4 (don't matter if it is correct it's just an example). Then your 20h00 show should be displayed 2000 GMT-4, and the Importer will indeed add only 4 hours if you checked the "dst" check box.

However, your XMLTV file may very well use 2100 GMT-5 instead, and in this case, you do not need the check the "dst" check box.

So to resolve this issue, I fired up the Importer and unchecked "Use Daylight Savings time" on both my XMLTV sources and re ran the importer.... to find that everything is still off by an hour. Anyone else having this problem with the importer? I would rather NOT change the offset to "4" (Atlantic time) from "5" (EST) though I'm sure it would solve the problem.

Are you sure that SageTV reloaded the guide data? Running the Importer is not enough : it just convert the data, so as SageTV can load them quickly on its own.

You can force the EPG update in SageTV by disabling / enabling a channel or use the Web Server plugin.

Regards,
Stéphane.

[StephaneM]

What's new

• Fixed : The setup wizard now set up the SageTV path for the Importer.
• Added : Parental Ratings support (with ratings editor : you must edit ratings to correctly translate any ratings used in your XMLTV file to something SageTV will understand)
• Added : Support for XMLTV writer / producer... nodes
• Added : Support for SageTV beta 6.0.13

[xred]

Originally Posted by StephaneM

Daylight saving time can be an issue because everything must be in sync :
Are you sure that SageTV reloaded the guide data? Running the Importer is not enough : it just convert the data, so as SageTV can load them quickly on its own.

You can force the EPG update in SageTV by disabling / enabling a channel or use the Web Server plugin.

OK I tried this. Actually removed both tuners using the importer from Sage then added them back in an re-imported the lineups. Didn't work.

I then ran the source XMLTV config and realized it was set to -0400 (Eastern Daylight). Reset it to -0500, downloaded new data, removed the tuners from Sage again an imported the whole lineup again.... and its still off by an hour!

In any case I do believe its the importer at some level since both the Zap2it sourced XMLTV and MyTheatre XMLTV are both showing an hour off, even though I changed the zap2it one from 0400 to 0500. I'll keep trying and update.

[StephaneM]

Originally Posted by xred

OK I tried this. Actually removed both tuners using the importer from Sage then added them back in an re-imported the lineups. Didn't work.

To be really on the safe side:

* Delete all content from the C:\Program Files\SageTV\SageTV\3rdPartyGuideData directory
* Do Import your XMLTV data one more time, and check you are really using the time offset you should use. That is if you have something like this in your XMLTV file : <programme start="20061123205000 -0500"> then you should put "5" in the time offset (I suppose that when you are in DST this should change to <programme start="20061123205000 -0400"> and the Importer will do it right if the DST check box is checked)
* When the Importer has finished its job, force SageTV to update the guide data :
- either by using Nielm's Web Server plugin
- either by disabling / enabling a channel in your lineups

(Note : I don't think that revoving / adding a source with the same lineup force the guide data update. You will see if SageTV really triggered a guide update by looking at the C:\Program Files\SageTV\SageTV\xmltv.log file).

You can also use the Importer Wizard to see if the choosen time offset is right. The Wizard select a show that should airs around 8 PM. When you adjust the time offset it will update the show hours as it will be in SageTV after the Import. So make the adjustment until you see the right schedule for the show.

Ultimately, check that your Windows time zone is ok.


Regards,
Stéphane.

[xred]

Originally Posted by StephaneM

(Note : I don't think that revoving / adding a source with the same lineup force the guide data update. You will see if SageTV really triggered a guide update by looking at the C:\Program Files\SageTV\SageTV\xmltv.log file).

You can also use the Importer Wizard to see if the choosen time offset is right. The Wizard select a show that should airs around 8 PM. When you adjust the time offset it will update the show hours as it will be in SageTV after the Import. So make the adjustment until you see the right schedule for the show.

OK I finally figured out at least part of this, of course user error was a major part of it

I was running Sage as a service in the background. The importer can import data while Sage is running (like it does nightly for me), but it can't shift the hour while running like the native Sage EPG can.

Once I removed the sources using the importer lineup, shut down Sage fully and then ran the import again it finally shifted the data from the Zap2it source to the correct time. It didn't do this for the MyTheatre sourced data for some reason, but I am pretty sure at this point that its the data coming out of Mytheatre thats the root cause. I am going to try flushing out all the Mytheatre data and starting with a fresh set to see if it resolves this.