New XMLTV Import Tool

[StephaneM]

What's new :

• Improved (1.5) : The Java Classes now use the SAX parser to improve load speed and fix memory usage problems on some PC.
• Added (1.5) : Warning messages in Channel editor.

[xred]

OK after several attempts at getting this XMLTV importer working, I am deep deep in the mouth of madness....

Its taken me 3 attempts to try and get this working, and each time I have to MANUALLY punch in all 650+ of my channels...EVERY TIME The most frustrating part is that the correct channel numbers are right there and space delimited in the Name column just to the left!

Would it not be possible to add shift - select / Insert channel number functionality in to the channel editor? It would save a TON of time, especially since I am planning to add 2 more satellites soon... and that will put me well over 1000 channels

EDIT: The error message in 1.5 just made me realize whats wrong (and that I am dense...). I have 2 XML files each with the channels witch about 600 channels and I am picking and choosing single channels from each (e.g. 1-300 from File A and 300-500 from File B). The editor wants me to number all 1300+ channels or it won't go.... argh

[StephaneM]

Hello,

Originally Posted by xred

Would it not be possible to add shift - select / Insert channel number functionality in to the channel editor?

Acutally you can completely manage the channel grid with the keyboard, here are some of the shortcuts :

F2 : Unselect the content of the cell and move the insertion point at the end of the cell content
HOME : move to the first cell in the row
END : move to the last cell in the row
TAB : move from cell to cell
SHIFT+TAB : move from cell to cell (backward)
CTRL+HOME : first cell in the grid
CTRL+END : last cell in the grid
CTRL+0 : enter a null value in the cell

All other shortcuts are working (SHIFT+LEFT / CTRL+C / CTRL+V)

The only thing that is not easily selectable with keyboard only is the lineup dropdown list, I will try to see if I can improve it.

Regards,
Stéphane.

[xred]

Originally Posted by StephaneM

Hello,

Acutally you can completely manage the channel grid with the keyboard, here are some of the shortcuts :

F2 : Unselect the content of the cell and move the insertion point at the end of the cell content
HOME : move to the first cell in the row
END : move to the last cell in the row
TAB : move from cell to cell
SHIFT+TAB : move from cell to cell (backward)
CTRL+HOME : first cell in the grid
CTRL+END : last cell in the grid
CTRL+0 : enter a null value in the cell

All other shortcuts are working (SHIFT+LEFT / CTRL+C / CTRL+V)

The only thing that is not easily selectable with keyboard only is the lineup dropdown list, I will try to see if I can improve it.
.

I finally got it running after realizing I had to manually put "0" in every channel I am not using. The importer picked up the call sign as "###ABC" instead of just the letters, which made me lose my channel icons. I'm dreading having to manually rename them to get them back!

Still the importer did its job so you have a donation from me waiting for you in Paypal =). If you're taking suggestions, my wishlist of features for the Channel Editor would be:

- The ability to select multiple rows and perform the following actions on them (maybe via right click?);

1. Extract channel number from the Name column. In my case (and I suspect most North American users) the channel number is the first set of characters in the name column. A space delimited search for numeric characters in the Name field should have a 99% plus accuracy rate.

2. The ability to REMOVE the channel number from the call sign field. This one is a bit more tricky as there is generally no space between the number and letters. If the previous (#1) feature was implemented then it might be possible to simply do a string search and add a "remove channel number from call sign" feature.

3. The ability to either delete channels or automatically "zero out" the channel number on unused channels. Channels with a 0 in the channel number should NOT be imported in to Sage (ie 0 = no import)

I'm sure theres more I am not thinking of, but the above would slice about 30 minutes off how long it takes me to setup my channel list!

[StephaneM]

The ability to select multiple rows and perform the following actions on them (maybe via right click?)

Actually I don't know if the DataGrid used in the channel editor is able to do this..

1. Extract channel number from the Name column. In my case (and I suspect most North American users) the channel number is the first set of characters in the name column. A space delimited search for numeric characters in the Name field should have a 99% plus accuracy rate.

2. The ability to REMOVE the channel number from the call sign field. This one is a bit more tricky as their is generally no space between the number and letters. If the previous (#1) feature was implemented then it might be possible to simply do a string search and add a "remove channel number from call sign" feature.

Actually I want to adhere to the XMLTV specification and channel numbers and call-signs are not in the specification. So at some point I will probably change the channel editor to something else more community oriented. What I want to achieve is something like that : select your reception mode (Analog / Digitial), your reception mean (Antenna / Cable / Satellite), your country, your area, and provider and it will select appropriate lineups from the already existing user submitted lineups...

Of course you would still have access to the channel editor.

3. The ability to either delete channels or automatically "zero out" the channel number on unused channels. Channels with a 0 in the channel number should NOT be imported in to Sage (ie 0 = no import)

I would prefer that you don't grab the unused channel (this is why you can't delete the channels you don't want in the editor and also because each time you open the channel editor, it will try to stay in sync with the XMLTV file : adding channels that are not yet in the editor and removing channels that are no more in the XMLTV file).

Regards,
Stéphane.

[xred]

Originally Posted by StephaneM

Actually I don't know if the DataGrid used in the channel editor is able to do this..

Have you considered breaking the channel editor out as a separate app? Its where all the magic happens as far as setup is concerned. There are tons of XMLTV plugins, but virtually no channel editors that I can find! The ability to take 2 (or more) XMLTV sources, edit the channel data as needed and then output to a single xml file is the real key to getting XMLTV listings working 99% of the time. With all the HTPC apps out today I am surprised no one has written one of these already, since one app could be used to generate custom XML files that Sage / BeyondTV / MCE with their respective plugins could all use.

Originally Posted by StephaneM

I would prefer that you don't grab the unused channel (this is why you can't delete the channels you don't want in the editor and also because each time you open the channel editor, it will try to stay in sync with the XMLTV file : adding channels that are not yet in the editor and removing channels that are no more in the XMLTV file).

I've done this as much as possible. I have hand picked my channels in the Zap2it lineup to just the bare minimum, but the big problem is in MyTheatre where I have to have ALL the channels I want to watch in the "SageTV" favorite and therefore in the EPG data it exports. If Mytheatre allowed you to have the same channel in different favorite groups I might be able to get around this, but no joy as far as I can tell.

[Alfiegerner]

Hi Stephane,

There may already be away to do this but if not this may be a feature request.

All of my movies are currently being imported so they display stars according to how good they are (e.g. 1 star for a bad film, 5 stars for a good one). Is there some way I could disable that? The problem is that this breaks my imdb lookups using SageMC.

[StephaneM]

Originally Posted by xred

Have you considered breaking the channel editor out as a separate app? Its where all the magic happens as far as setup is concerned. There are tons of XMLTV plugins, but virtually no channel editors that I can find! The ability to take 2 (or more) XMLTV sources, edit the channel data as needed and then output to a single xml file is the real key to getting XMLTV listings working 99% of the time. With all the HTPC apps out today I am surprised no one has written one of these already, since one app could be used to generate custom XML files that Sage / BeyondTV / MCE with their respective plugins could all use.

I don't think that a channel editor carry such a value :

* channel setup is dependant on the PVR Software used. From my experience, BTV and SageTV needs an editor, but not GBPVR and MediaPortal. I don't know what's needed for MCE but as it only support one lineup (or two with analog + digital) this limit the need of an editor

* merging XMLTV file is not the purpose of the editor (for instance my Importer doesn't merge XMLTV file, it doesn't really need to). Also it seems that many grabbers are already capable of merging XMLTV and that other PVR software can load more than one XMLTV file. Also a really simple script can do it.

For me the real value is what the Importer is doing globally : it loads XMLTV data, build lineups and regroup program logically. In a sense it already do what you suggest : the same Importer code base is used for Beyond TV and SageTV, when the Importer do its work, it produces other XML files and those files are used to populate the program guide of BTV and SageTV.

The only differences between the two version of the Importer are in the setup program and in the way data are loaded.

But basically if someone want to use the XML data produced by the Importer to load them into MCE, it is possible.

If Mytheatre allowed you to have the same channel in different favorite groups I might be able to get around this, but no joy as far as I can tell.

If you want I can send you a quick and dirty script that will delete any channels you don't want (as well as programs). You can the run this before submitting the data to the Importer.

Regards,
Stéphane.

[StephaneM]

Originally Posted by Alfiegerner

There may already be away to do this but if not this may be a feature request.

No, there is no way to disable the star rating.

I will add some advanced options at some point (they will be hidden though), because I don't want to have the Importer overloaded with options, my point is to sort of offer a really simple to use Importer.

Regards,
Stéphane.

[Alfiegerner]

Originally Posted by StephaneM

No, there is no way to disable the star rating.

I will add some advanced options at some point (they will be hidden though), because I don't want to have the Importer overloaded with options, my point is to sort of offer a really simple to use Importer.

Regards,
Stéphane.

Ah okay, I think you are right to want to kep the main interface simple. This would be a bonus for me though.

[henk99]

I have been fiddling around with it for a few days, but i don't have it working yet. I thought i did, as there was data in the tvguide, but this proved to be old data. Now, a week later, the old data is gone and i see it is not importing.
An epgdata.xml file is grabbed every day by 'tv-to-xml' program for dutch tvguide. This files looks good to me and still gets grabbed every night at 4am. I have run the importtool wizzard with sagetv off (service as well). I have slected the proper epgdata.xml file and did the channel line up and all appears to be fine, according to the tool.
When the wizzard is done i start up the sagetv service and run sagetv to the the videosource setup. It all looks ok, the channel names show in the proper order i set-up in the tool.
However, there is nothing in the program guide..

Any idea what i'm missing here?

Henk

[StephaneM]

Originally Posted by henk99

I thought i did, as there was data in the tvguide, but this proved to be old data.

I suppose that these data were Importer with my tool?

Now, a week later, the old data is gone and i see it is not importing.

Are you sure you have the right version of the Importer installed, the setup program that end with a "s" is the free version that doesn't run automatically, for having the Importer running unattended you must have downloaded and installed the version linked in the e-mail you receive after donation.

Also, be sure to have installed the latest 1.5 version (in previous ones, the Java classes were using the DOM XML parser and with a lot of guide data the DOM parser may not have enough memory to load the data file).

An epgdata.xml file is grabbed every day by 'tv-to-xml' program for dutch tvguide. This files looks good to me and still gets grabbed every night at 4am. I have run the importtool wizzard with sagetv off (service as well). I have slected the proper epgdata.xml file and did the channel line up and all appears to be fine, according to the tool.

Usually once you have setup the Importer and installed the full version you do not need to run the Wizard again (and by the way you do not need to shut down SageTV to run the Importer, shut down on SageTV is only required when installing or uninstalling the Importer).

You can verify a few things:

* check the xmltv.log file in the C:\Program Files\SageTV\SageTV directory, this one will tell you if SageTV has correctly loaded the data with the help of the Importer Java Classes.

* check your Event Viewer log (SageTV Utilities) to see if the Importer does have encountered some issues during the conversion process of your XMLTV file.

* check that you do have XML files in the C:\Program Files\SageTV\SageTV\3rdPartyGuideData directory

Usually this is how the whole process is supposed to be working:

1) XMLTV file is downloaded by your grabber. Either you scheduled the grabber to run by itself or you do use the Importer to launch it.

2) The Importer will run unattended to transform your XMLTV data (and optionally launch your grabber). It will run unattended only if you have the donator version; otherwise you have to launch it manually

3) Every 24 hours, SageTV will load guide data. It uses the Java Classes installed by the Importer. Those Java Classes will load the XML files produced by the Importer (C:\Program Files\SageTV\SageTV\3rdPartyGuideData directory)

Regards,
Stéphane.

[henk99]

Originally Posted by StephaneM

I suppose that these data were Importer with my tool
Are you sure you have the right version of the Importer installed, the setup program that end with a "s" is the free version that doesn't run automatically, for having the Importer running unattended you must have downloaded and installed the version linked in the e-mail you receive after donation.

Also, be sure to have installed the latest 1.5 version (in previous ones, the Java classes were using the DOM XML parser and with a lot of guide data the DOM parser may not have enough memory to load the data file).


Usually once you have setup the Importer and installed the full version you do not need to run the Wizard again (and by the way you do not need to shut down SageTV to run the Importer, shut down on SageTV is only required when installing or uninstalling the Importer).

You can verify a few things:

* check the xmltv.log file in the C:\Program Files\SageTV\SageTV directory, this one will tell you if SageTV has correctly loaded the data with the help of the Importer Java Classes.

* check your Event Viewer log (SageTV Utilities) to see if the Importer does have encountered some issues during the conversion process of your XMLTV file.

* check that you do have XML files in the C:\Program Files\SageTV\SageTV\3rdPartyGuideData directory

Usually this is how the whole process is supposed to be working:

1) XMLTV file is downloaded by your grabber. Either you scheduled the grabber to run by itself or you do use the Importer to launch it.

2) The Importer will run unattended to transform your XMLTV data (and optionally launch your grabber). It will run unattended only if you have the donator version; otherwise you have to launch it manually

3) Every 24 hours, SageTV will load guide data. It uses the Java Classes installed by the Importer. Those Java Classes will load the XML files produced by the Importer (C:\Program Files\SageTV\SageTV\3rdPartyGuideData directory)

Regards,
Stéphane.

No, the epg data is imported with a tool called tvgids_to_xml which is working fine.

Yes, i have donated and i am using version 1.5, so that isn't the problem.

The xml log is showing this:
17-05-2006 22:40:19,541 Start EPG update for lineup n°867507149
17-05-2006 22:40:19,541 Reading configuration
17-05-2006 22:40:19,541 Reading Channels File
17-05-2006 22:40:19,557 Creating channels array (22)
17-05-2006 22:40:19,557 Setting SageTV lineup
17-05-2006 22:40:19,557 Reading Lineups File
17-05-2006 22:40:19,557 End Update

Also there are 4 files in the 3rd party directory. All appears to be fine for sagetv to import it.

I have tried again and the importer runs through it's steps fine, however, nothing is showing in sagetv.
When i go into source setup and go though all the steps, it doesn't even show the channels anymore. So, at this point sagetv is unusable as i have no channels to select from. Also, as there is no epg info, the scheduled programs are not recorded. This is becoming quite a problem.

Please let me know if anything else can be done.

[StephaneM]

Originally Posted by henk99

Please let me know if anything else can be done.

From your xmltv.log file I think that the problem is that you don't have selected the right lineup.

The log begin with Start EPG update for lineup n°867507149

I doubt that you created a lineup with such a number (usually it's 1 or 2 or 3). So I think that somehow the XMLTV lineup to be used is stayed the same as you was using before my tool (867507149 is typically a lineup number from another XMLTV plugin).

So do your video source setup, and this time choose "Do not use any guide data", this will clear any guide data / lineup associated with the video source and finish the source setup.

Then redo the source setup and this time choose "Use XMLTV data" and select the right lineup, it should work.

Verify in your xmltv.log file that it starts with "Start EPG update for lineup n°1" and that n°1 is indeed the lineup number as it is in the Importer channel editor.

Regards,
Stéphane.

[henk99]

Originally Posted by StephaneM

From your xmltv.log file I think that the problem is that you don't have selected the right lineup.

The log begin with Start EPG update for lineup n°867507149

I doubt that you created a lineup with such a number (usually it's 1 or 2 or 3). So I think that somehow the XMLTV lineup to be used is stayed the same as you was using before my tool (867507149 is typically a lineup number from another XMLTV plugin).

So do your video source setup, and this time choose "Do not use any guide data", this will clear any guide data / lineup associated with the video source and finish the source setup.

Then redo the source setup and this time choose "Use XMLTV data" and select the right lineup, it should work.

Verify in your xmltv.log file that it starts with "Start EPG update for lineup n°1" and that n°1 is indeed the lineup number as it is in the Importer channel editor.

Regards,
Stéphane.

Hi Stephane

That did the trick. I followed your guide by selecting 'no epg' first. After that it worked properly. Thank you for your quick help on this!

I have another question though. When making the line-up, there is a column called 'channel name' and 'call sign' this call sign appears to be the one i see in the program guide. The thing is that this call sign column does not accept spaces in the names. For instance one channel has channel name RTL 4. I the call sign column i have to type it as RTL4. I would like it to show up in the sagetv program guide as RTL 4 (with a space). Is that possible? If so, how can i do that.

Cheers,
Henk

[StephaneM]

Originally Posted by henk99

The thing is that this call sign column does not accept spaces in the names. For instance one channel has channel name RTL 4. I the call sign column i have to type it as RTL4. I would like it to show up in the sagetv program guide as RTL 4 (with a space). Is that possible? If so, how can i do that.

Then only constraint for the call-sign is the length (I will remove it in the upcoming version of the Importer). So normally you can type a space in it (I will also add some validity constraints for characters not allowed in a filename as this is the call-sign that is used for channel logos).

Also there is a bug in version 1.5 with the new SAX Parser : some strings imported are truncated (it's random), So if you see program title, ep.title, description with a part missing : there is nothing to worry about it.

Regards,
Stéphane.

[henk99]

Originally Posted by StephaneM

Then only constraint for the call-sign is the length (I will remove it in the upcoming version of the Importer). So normally you can type a space in it (I will also add some validity constraints for characters not allowed in a filename as this is the call-sign that is used for channel logos).

Also there is a bug in version 1.5 with the new SAX Parser : some strings imported are truncated (it's random), So if you see program title, ep.title, description with a part missing : there is nothing to worry about it.

Regards,
Stéphane.

You are right...it is a length issue. It only happens with long names. I'll use it in the new version then

[StephaneM]

What's new :

• Fixed (1.6) : Some content were truncated in SageTV program guide in version 1.5
• Improved (1.6) : Comboboxes in channel editor are now keyboard friendly, call-signs are no more limited to 10 chars

[Gabriel]

Hi Stephane,

Is it possible with your tool to set a time offset in a zap2it lineup on a per-channel basis? Most of my channels are on time but a few are in a different time zone or DST, so a global offset will not work.

[StephaneM]

Hello,

Originally Posted by Gabriel

Is it possible with your tool to set a time offset in a zap2it lineup on a per-channel basis?

No this is not possible. You'll have to split your XMLTV file in two distinct files for this to work. Then in the Importer you can import these files as two sources and as such you can define a different time offset.

It seems to me that the XMLTV grabber for Zap2it allows mutliple lineups (and can produce one XMLTV file for each lineup). So it should be easy to set up.

To process multiple lineups, specify a different "--config-file". To
    prevent re-downloading the data on subsequent passes, the --reprocess
    option is recommended.

    Here's an example: (the = sign is optional, but helps readability)

     tv_grab_na_dd --config-file=lineup1.dat --output=lineup1.xml --dd-data=dd.xml
     tv_grab_na_dd --config-file=lineup2.dat --output=lineup2.xml --dd-data=dd.xml --reprocess
     tv_grab_na_dd --config-file=lineup3.dat --output=lineup3.xml --dd-data=dd.xml --reprocess

    Each config file specifies the desired lineup and channel list. The
    login information is is only used in the first command, but it doesn't
    hurt in the other entries.

Regards,
Stéphane.