New XMLTV Import Tool

[StephaneM]

Hello All,

I've managed to port my Beyond TV XMLTV Importer to SageTV.

So here it is a brand new XMLTV Importer for SageTV.

This tools among ohter features will :

• Give you an easy Wizard to set it up: including a lineup/channel editor
• Automatically launch your XMLTV Grabber (if you automate the Importer : you can do this from the Wizard)
• Generate real EPGID : the Importer has a lookup table in which it keeps tracks of all programs and thus can generate the same ID for a show (even after a year)
• Should handle well First Runs and Reruns even if your XMLTV file doesn't have the required information (again thank to the lookup table). Note: if your XMLTV file doesn't have any information about first runs and re-runs, then the Importer will first consider that all is new and progressively will show re-runs (the Importer will think a show it doesn't have in its lookup table is new otherwise a rerun).
• It supports color coding in the SageTV EPG
• It offloads the XMLTV transformation : that is SageTV will only load the result of the transformation the XMLTV Importer is doing, so when SageTV update it's guide data, it only load a cached copy (so it is fast).
• It supports multiple lineups, duplicate channels and dummy channels.
• It doesn't need the xerces.jar file (in fact you'll have to get rid of it), but you must use Java Runtime 1.5 (06)

Current version :

What's new :

• Added (2.0) : Support for SageTV beta 6.0.13
• Added (2.0) : Support for Parental Ratings (with Ratings Editor)
• Improved (1.9) : Movies detection for Zap2it EPG data.
• Fixed (1.8) : import-log.xml file possible corruption prevent the Importer from running.
• Improved (1.7) : BelowNormal Process priority is used for the Importer and grabbers (when running unattended)
• Changed (1.7) : Zap2it EPGID support is disabled by default. This is because import times are way longer when this is enabled and apart keeping history of watched shows you'll have no real benefits when using Zap2it EPGID (Also the Importer is sometime forced to not use the exact same EPGID)
• Fixed (1.7) : Show descriptions were not imported in some situations.
• Added (1.7) : Hidden options (See the FAQ).
• Fixed (1.6) : Some content were truncated in SageTV program guide in version 1.5

Download and how to set it up : SageTV XMLTV Importer

License : Donationware

Note: during the Setup of the Importer, you must shutdown SageTV completely, as during Setup, the Setup Wizard will copy the Java classes and edit your Sage.properties file (if needed). As always : you should backup SageTV before Setup. Uninstalling the XMLTV Importer, should restore all your old settings (Previous XMLTV Plugin if any and xerces.jar if found during setup)

Best Regards,
Stéphane.

[ukmgranger]

This looks like it is going to be amazing. Will it work for UK XMLTV?

[StephaneM]

Originally Posted by ukmgranger

Will it work for UK XMLTV?

I don't see why it wouldn't

The Importer is the same one as Beyond TV and it exists from nearly two years. I don't have seen yet a XMLTV file that is not handled well by the Importer.

There are only three new features in this Importer :

* The ability to launch the grabber automatically by the Importer (the grabber will be hidden as well the Importer : so no problems if you run SageTV in FSE : it will not steal the focus or drop it out from FSE)

* The handling of first runs and re-runs : in this version the Importer keeps its own history as well as looking at what the XMLTV file is anouncing.

* The setup process and the Java classes (that effectively "glue" it to SageTV)

So more or less: upon setup of the XMLTV Importer the XMLTV selection in SageTV should be up and running (no more copying / editing properties files / copying xerces.jar etc..). But as setup is always a tricky process, I'm expecting to have to give some support on this part of the process, but the whole conversion routine should be fine and robust.

Regards,
Stéphane.

[johnnytyler]

I look forward to using this importer, tell me can you incorporate zap2it info with other xmltv lineups and if so how do you go about setting it up?

[xred]

Originally Posted by johnnytyler

I look forward to using this importer, tell me can you incorporate zap2it info with other xmltv lineups and if so how do you go about setting it up?

Your about to get this question 1000 times, so let me clarify what I think a lot of users are after!

Can we use this importer to take data from multiple XML files and combine them in to a single EPG file for Sage? e.g:

Channels 0-100 on Tuner #1 from zap2it.xml
Channels 100-400 on Tuner #2 from zap2it.xml
Channels 401-699 on Tuner #2 from epg.xml

I cannot give you my money for the full version fast enough if it can do the above

[StephaneM]

Originally Posted by johnnytyler

I look forward to using this importer, tell me can you incorporate zap2it info with other xmltv lineups and if so how do you go about setting it up?

You will have first to do whatever it takes to configure the XMLTV grabber tv_grab_na_dd to download the proper XMLTV file.

Then run the Importer Wizard and check the "don't show this wizard again", click on the "Next >" button and configure the Importer with the Zap2It XMLTV file you just downloaded, run the Wizard until it begins to do the transformation and click on the "Cancel" button.

Then open the Importer again and you'll have access to the "Classical Interface" where you can add another XMLTV file.

To add another XMLTV file to be handled by the Importer click on the "+" button and another dialog will open : type all information needed and click on "OK". Repeat operations for other XMLTV files.

When you have added all your XMLTV files, click on the "Edit Channels" button, and define your lineups and edit your channels : basically the Importer will list all the channels found in all your XMLTV file, you just have to give them a name / call sign / number and choose their lineup (You can also create dummy channels as well as duplicate some of them if you need a channel on multiple lineup).

Regards,
Stéphane.

[StephaneM]

Originally Posted by xred

Can we use this importer to take data from multiple XML files and combine them in to a single EPG file for Sage?

Yes you can do it : the Importer can also trigger silently each grabber needed to download each XMLTV file.

Channels 0-100 on Tuner #1 from zap2it.xml
Channels 100-400 on Tuner #2 from zap2it.xml
Channels 401-699 on Tuner #2 from epg.xml

Yes you can do whatever you want to. It's just a matter of creating the lineups in the Importer and dispatch channels between them.

However it can be tricky to setup because of the presentation of the channels editor :

* The editor displays all the channels found in all your XMLTV files
* You are then supposed to choose for each channel : a name / call sign / number and lineup
* But if you want a channel to be in two lineups : you just have to use the "additionnal channels" tab and duplicate your channel here, be careful when doing this to create the duplicate channel with the same ID as the original and to choose the original channel in the "Same as" drop down list (this will duplicate the shows for the new channel), then as with the first channel list : give it a name / call sign / channel number and choose the lineup.

Give it a try, you don't need to make a donation to run the Importer, the only feature that doesn't work without donation is the "automation" of the Importer : during the Wizard you can choose if you want to run the Importer automatically (for instance each sunday at 6:00am), this schedule a task in Windows task scheduler.

With the "free" version, the task will execute but the Importer will just open and do nothing, with the "donator" version, the Importer will load, then launch each XMLTV grabbers and wait until every download is completer and finally will begin the transformation process (during the transformation you can see an animated notification icon in your taskbar).

SageTV will continue to load the data every 24 hours as usual, it uses the Java classes installed by the Importer, those classes only load XML files produced by the Importer (a straight and quick load)

Regards,
Stéphane.

[heffe2001]

Hopefully someone will make a simple 'how-to' for using this with XML files exported from xmltv2mtepg123 (from Mytheatre) with this.

[StephaneM]

Originally Posted by heffe2001

Hopefully someone will make a simple 'how-to' for using this with XML files exported from xmltv2mtepg123.

Do you find the "How to" for using the Importer useful? I know it doesn't cover neither the XMLTV grabber configuration, nor the SageTV setup process.

But anyway the most difficult part of the whole thing is to create the right lineups.

Otherwise :

* XMLTV grabbers are not that complicated to setup generally (I don't know the MyTheater plugin tough). You have a couple of command line to type and usually you create a batch file to launch the grabber. Now instead of using a batch file you can type directly the command line in the XMLTV Importer. Depending on the grabber it may or may not work (it works with the regular XMLTV grabber, for MyTheater, if there is a command line, it should work also). Also you are not required to put the grabber command line in the XMLTV Importer, you can use whatever you use today to launch your grabber.

* SageTV setup is straightforward : just pick up the XMLTV option and choose the lineup. The most important point is to pick the right lineup and that the lineup contains the channels that are related to the current tuner and of course that their numbers are the good one. For MyTheater the channel number is the channel favorite number (and this is this number you'll have to type in the XMLTV Importer).

Regards,
Stéphane.

[heffe2001]

My main problem is understanding how to set up the channel lineup, I'll have another look at it when I have more time this weekend.

I've got the mytheatre epg export working (at least I think it is), I just need to get the channel lineup created (I think) and it should work. What I'm wanting to do is just use the MT epg for the channels that zap2it doesn't serve (PPV, and some of the higher channels), so it would require 2 seperate xml's being loaded.

[grooves12]

The generate Show ID... how is it handling deciding if a show is the same episode or not. Currently I am using Demedius' importer, and importing all channels from Zap2it that have guide data for them, and the Mytheatre Exporter for the remainder. This causes conflicts for channels that carry the same programming between the two lineups, as the ShowID's don't match.

Will this get around that problem, even if the guide listing is formatted slightly different??

[grooves12]

Also, is there a way to determine first run by the Original Air Date?? I would rather not have ALL episodes marked as new if they haven't been seen by this system before.

[Mark SS]

Originally Posted by StephaneM

Generate real EPGID : the Importer has a lookup table in which it keeps tracks of all programs and thus can generate the same ID for a show (even after a year)

You'll be getting a donation for this feature alone as soon as I get this up and running. I was getting so frustrated by the EGPID issues that I was starting to look at writing a plugin myself!

Thank you in advance!

[rsagetv99]

Can the importer derive the channel number and channel names from the XMLTV file? The data is in the file, but for some reason I've noticed a couple of importers do not pull in this data.

If it cannot do this, is there an alternative way to add the channel numbers, like importing from a text file? The reason I ask is that I have a script that parses the XMLTV file and generates the channel name/number for each channel ID and puts it in a text file; I can modify that script to meet whatever format you might need for your importer...

I also was wondering about the Episode ID. I believe the internal EPG passes the Episode ID untouched (except for removing the "."). If your importer will do the same, then people who switch from the internal EPG to your importer can retain their "watched" status. This will also eliminate a problem many of us are having with Demedius' importer creating unique Episode IDs for identical episodes from different sources.

Not sure if you are willing to continue the development of the software and tweaking/customizing it for some of the Sage users, but if you are up to the task I'm sure you will get quite a few donations.

I have a few more ideas, but I'll try not to bombard you yet.

[StephaneM]

Originally Posted by grooves12

The generate Show ID... how is it handling deciding if a show is the same episode or not.

To make it short : the Importer compute (as other XMLTV Plugin it seems) a hash for a show, but instead of computing one hash, it computes three hashes. All are not case / space / non alpha char dependant. Instead of using the hash directly to generate a EPGID, it match them to a Lookup table to find the proper series and episode as well as the first air date. (So the lookup table here is the key : you'd better not delete it and as well backup it : it is located in the install folder of the Importer, it will get bigger and bigger with time and cleaning is done automatically every year)

So from one import to another the Importer know how to keep track of all your shows on all your XMLTV file.

It will even work if one XMLTV file show "That '70s show" and the other "That 70's show" (notice the quote).

The Importer works well if any of this is verified :

* shows have an episode title
* shows have an episode number
* shows have an episode description

So depending on your XMLTV files : if they all share the same episode title or episode number : it will be ok. But if you only have an episode description : it is more problematic (at least all shows from a series will belong to the same series).

Currently I am using Demedius' importer, and importing all channels from Zap2it that have guide data for them, and the Mytheatre Exporter for the remainder. This causes conflicts for channels that carry the same programming between the two lineups, as the ShowID's don't match.

The EPGID should match, at least the will be from the same series, I cannot guarantee that the same episode in your two XMLTV files will be the same, it really depends on the information in the XMLTV files.

Regards,
Stéphane.

[StephaneM]

Originally Posted by grooves12

Also, is there a way to determine first run by the Original Air Date??

Yes the Importer does handle the <premiere> node and <previously-shown> node. So if these data is here, it will import it. When these nodes are not present the Importer does try to figure it out all alone (it will work better though as time goes by)

For instance : I don't have any <premiere> neither <previously-shown> nodes in my XMLTV files, so I rather record "All" episodes, but for some of my shows now, the Importer does know that the new season is "First run" rather that "re-runs".

So if you have nothing in your XMLTV regarding original air date : all show will be new, but as the same shows will be aired over and over : you'll see the real new ones as "First run" (typically you'll have to wait one season).

Regards,
Stéphane.

[StephaneM]

Originally Posted by rsagetv99

Can the importer derive the channel number and channel names from the XMLTV file? The data is in the file, but for some reason I've noticed a couple of importers do not pull in this data.

Yes it will read the channel name, but not the channel number as there is no node dedicated to this usage. That's why there is a channel editor in the Importer : this a grid showing you all your channels, it should takes you five minutes to edit proprely your channels.

You need to edit your channels the first time and when you have a new channel (the Importer will not import any new channel unless you first edited it in the channel editor).

If it cannot do this, is there an alternative way to add the channel numbers, like importing from a text file? The reason I ask is that I have a script that parses the XMLTV file and generates the channel name/number for each channel ID and puts it in a text file; I can modify that script to meet whatever format you might need for your importer...

All the Importer configuration is written to xml files, the one you will want to edit is : channels.xml and also clone_channels.xml (for duplicated / dummy channels) they are in the install folder of the Importer.

For instance :

<channels>
  <channel id="tf1.fr" name="TF1" call-sign="TF1" channel="1" lineup="1" majorchannel="" minorchannel="" />
  <channel id="france2.fr" name="France 2" call-sign="France2" channel="2" lineup="1" majorchannel="" minorchannel="" />
  <channel id="canalplussport.fr" name="Canal+ Sport" call-sign="Canal+Spor" channel="3" lineup="1" majorchannel="" minorchannel="" />
  <channel id="canalplusconfort.fr" name="Canal+ Décalé" call-sign="Canal+Déca" channel="4" lineup="1" majorchannel="" minorchannel="" />
  <channel id="bbcprime.fr" name="BBC Prime" call-sign="BBCPrime" channel="5" lineup="1" majorchannel="" minorchannel="" />
</channels>

You'll need to edit : the name / call-sign and channel attributes (the majorchannel / minorchannel are not used for SageTV). The "lineup" attribute is the number of the lineup you created in the Importer channel editor.

But I really don't think a script is needed for this : you don't have to change channel name and number every day

I also was wondering about the Episode ID. I believe the internal EPG passes the Episode ID untouched (except for removing the ".").

To which Episode ID are you referring to? The SageTV one?

If your importer will do the same, then people who switch from the internal EPG to your importer can retain their "watched" status. This will also eliminate a problem many of us are having with Demedius' importer creating unique Episode IDs for identical episodes from different sources.

No the Importer doesn't rely on Episode ID that may be in the XMLTV file (there is no such property in the XMLTV specs), so the Importer does generate it's own EPGID (I use a very strong algorithm to generate those : to make them consistent and for the transformation process to be real fast).

Not sure if you are willing to continue the development of the software and tweaking/customizing it for some of the Sage users

Of course I am. I currently use SageTV and so decided to use my own Importer rather than the available ones : that's why I ported my XMLTV Importer to sage (and that was not pleasant : I don't know Java at all )

I have a few more ideas, but I'll try not to bombard you yet.

You are welcome.

Regards,
Stéphane.

[rsagetv99]

Originally Posted by StephaneM

To which Episode ID are you referring to? The SageTV one?

I am referring to the <episode-num> element. Maybe it is not in the spec, but it is definitely present in the Zap2it files. I am pretty sure that this number, minus the "." is passed to Sage as the ShowID and that is what Sage uses to determine if a show has been watched, not liked, and how it keeps from recording duplicate episodes.

To have the best of both worlds it would be great if your importer could only generate an EPGID for shows that do not already have one.

<episode-num system="dd_progid">SH808754.0000</episode-num>

As for the daily channel editing, it is sometimes needed if you use the MyTheatre exporter since the Channel ID sometimes changes in MyTheatre. I'll take a crack at editing the XML file via my script.

BTW, in case you ever decide to implement this, I know the number does not have its own node in the xml file, but in the Zap2it files it is consistently in the same place. The way I am doing it with the script is that I look for the "display-name" nodes and pull the number from the 3rd "display-name" node for each channel ID. I do a similar thing for the long and short channel names, except I crop the extra data that is not needed. Pretty rudimentary but it gets the job done.

Thanks again.

[Madcoder]

So if someone has donated for the BeyondTV version, but these days uses Sage, can they still obtain the new version for Sage, or is there a new donation required?

[StephaneM]

Originally Posted by rsagetv99

I am referring to the <episode-num> element. Maybe it is not in the spec, but it is definitely present in the Zap2it files. I am pretty sure that this number, minus the "." is passed to Sage as the ShowID and that is what Sage uses to determine if a show has been watched, not liked, and how it keeps from recording duplicate episodes.

The <episode-num> is recognized by the Importer but only the one with XMLTV numbering system, so the "dd_progid" is ignored.

To have the best of both worlds it would be great if your importer could only generate an EPGID for shows that do not already have one.

I don't know, I'm thinking the exact opposite and this may be the source of duplicate EPGID for the same show you see with other XMLTV Plugin. To be efficient, the EPGID generation must give the same ID to the same show whatever the XMLTV file the show is in. So by not depending on the sole dd_progid found in Zap2it XMLTV file, you had a better chance of having the same EPGID if you use another XMLTV file.

For instance, it is frequent that a grabber for a country use another web site as a data source, or you as a user can change to use another grabber at any time. I am in this situation and the Importer always do it's job : it gives me consistent EPGID and in two years I never recorded a duplicate (even if there is a slight name change in the show name or episode title).

As for the daily channel editing, it is sometimes needed if you use the MyTheatre exporter since the Channel ID sometimes changes in MyTheatre. I'll take a crack at editing the XML file via my script.

Be careful then : the Importer doesn't expect the channel ID to change. As such the channel will be a new channel to SageTV (this however is not important for the EPGID).

Regards,
Stéphane.