Updated XMLTV Plugin with compile steps

[hvymetal]

In an effort for some IPTV channels I am working for importing into SageTV I updated the plugin for more use cases. The changes below should work with most XMLTV grabbers. Also for future development there are easy steps for compiling the plugin on the Unraid Docker image. The Unraid Docker Image already has the required java_jdk installed for compiling. In the future I might look into the the ICON working as the data is also stored inside the XMLTV file.

(Warning me venting)
The original was not touched since the beginning of SageTv and impossible to use without remapping or using scripting for remapping channels. It is almost like the developers of SageTV wanted everyone to use their guide data or SD. As for SD the plus is that it gives 2 weeks of data with more info than XMLTV grabbers thus being a good value. Also to be fair to SageTV they also did give a good run(~15 years) with their guide data. Any case with SageTV on its last leg this needs to change as other DVR development have XMLTV native. Viva La SageTV

1. Stop SageTV Server
2. Rename on add/modify/ xmltv_EXAMPLE.properties examples
3. Copy all files and folder contents(not folder) of folder SAGETV_SERVER_ROOT to SageTV folder. Only Jar and *.proerties are required if not compiling required
4. Add the following line in Sage.properties epg/epg_import_plugin=xmltv.XMLTVImportPlugin
5. Start SageTV Server
6. Monitor in server folder xmltv.log, sagetv_0.txt,Sage.properties. If epg/epg_import_plugin=xmltv.XMLTVImportPlugin is removed from Sage.properties something is installed not correctly
7. In SageTV guide setup use XMLTV with zipcode 00000. If ask for license use TRIAL


#######Commnds for UnRaid Docker to Moditfy Sage.properties for Plugin
# Stop Sage Server
sudo -E "PATH=$PATH" -u sagetv /usr/local/bin/stopsage &
# Property Setting for EPG Plugin
sudo sed -i 'epg/epg_import_plugin' /opt/sagetv/server/Sage.properties
sudo echo 'epg/epg_import_plugin=xmltv.XMLTVImportPlugin' >> /opt/sagetv/server/Sage.properties
# Start Sage Server
sudo -E "PATH=$PATH" -u sagetv /usr/local/bin/startsage &

#########Commnds for UnRaid Docker Compile of Plugin
cd /opt/sagetv/server/xmltv
sh ondocker_build.sh

 
 /* Change Log
##V2.0  01/13/2022
1.  Add code if display-name or channel id number(example 5, 5.1 5-1) then add as the channel number for import.  Examples below import as channel 2.1 and 1162 respectively below
##2.02  01/14/2022
1.  Fixed first channel * bug
2.  Fixed double year in title when title already had a date
3.  add tms filed for episode-num instead of creating unique id
4.  Do not generate SxEx if default case is found ("onscreen", "common")
5.  Fix Display for SxEx in descriptions
6.  Fix short name being set as channel number
7.  Start code for reading Logo Icon for channel (WIP) 
##2.03  01/14/2022
1.  Fixed new Integer((int) to remove compile deprecated warning
2.  Switched org.xml.sax.helpers.XMLReaderFactory to javax.xml.parsers.SAXParserFactory; to remove compile deprecated warning
##2.04 01/16/2022
1.  Added .properties entry for xmltv.channel.display-name.ShortNameIndex 
		 default=0 with no entry .properties file
		 init.ChannelShortNameIndex0:=shortest found without spaces
		 init.ChannelShortNameIndex>0: XMLTV file index found in order starting with 1 <display-name>CBS</display-name>
2.  Added .properties entry for "xmltv.channel.xmltv.channel.ShortNameRegEx"		
		default=.* with no entry .properties file will find anything
3.  Added .properties entry for xmltv.channel.display-name.LongNameIndex 
		default=0 with no entry .properties file
		init.ChannelLongNameIndex0:=longest found with spaces
		init.ChannelLongNameIndex>0: XMLTV file index found in order starting with 1 <display-name>CBS</display-name>
4.  Added .properties entry for xmltv.channel.NumberTag 
		 default="" No entry .properties file.  
		 init.ChannelNumberTag="" None will try to detect channel number from <channel id=XXX> or <display-name>XXX<display-name>
		 init.ChannelNumberTag=channel then <channel id=XXX> 
		 init.ChannelNumberTag=TAG_NAME then <TAG_NAME>xxx<TAG_NAME>
5.  Added .properties entry for init.ChannelNumberTagIndex		
		default=0 with no entry .properties file
		init.ChannelNumberTagIndex=0 None will try to detect channel number from <channel id=xxx> or <display-name>
		init.ChannelNumberTagIndex>0 XMLTV file index found in order starting with 1
6.  Added .properties entry for "xmltv.channel.xmltv.channel.NumberTagRegEx"		
		default=.* with no entry .properties file will find anything
7.  Added .properties entry for "xmltv.channel.IconDownload"
		 default=false with no entry .properties
		 false=Do Note download image file
		 true=Download image file from channel <icon src= >
##2.06 01/18/2022
1.  Added the following .properties entry  for enabling or disabling logging parts.  Defualt with no entry is true
		log.show=false
		log.channel=false
		log.configuration=false
		log.defaults=false
2.	Changed the way ChannelId is calulated. The previous way caused collections.  The new way uses the ProviderID as part of the encoding. The new limit limits the Provider ID to 998. 
		 2147483647	Max Number
		 1000000000
		 1030002100
		      99999	SD MAX it Sends
		  998           Upper Part is Listing ProviderID 1-998 999 used if none provided (* 1000000)
		     123456     Max XMLTV Number without Decimal 999999   
		 1   123456     Max XMLTV Number with Decimal 9999.99 When Decimal (* 100 +1000000000)  
3.  Remove non file characters so Channel icons could be downloaded the following examples where failing: "Cats 24/7" or "Dogs 24/7"	
##2.08 01/28/2022
1.  Filter if desc tag is from channel section.  This would crash the plugin if not done.
2.  Add show if it is up to 8 hours previous. Fixes on first import no data problem
3.  Add setting this.show.showId on <episode-num system="pluto"> tag
4.  Fixed freeFormEpisodeNumber on <episode-num system="onscreen"> and <episode-num system="common"> tags
5.  Add new classs to hold all the init. variables from .properties file
6.  Added xmltv.channel.NumberOffset to offset channel numbers on import.
7.  Added if *.xmltv.properties is found in server directory use it as properties without requirement of configurations= in xmltv.properties file
 ##2.09 011/13/2022
1. Added if no <date> tag is found use <start> time.  This means it is new or live
2. Switch function from addAiringPublic to addAiringPublic2.  This allows the following mask to be set:
	LIVE_MASK and NEW_MASK
	audio: STEREO_MASK, SAP_MASK, DD51_MASK, DOLBY_MASK, DUBBED_MASK, SURROUND_MASK, SUBTITLE_MASK, CC_MASK, TAPE_MASK
	quality: HDTV_MASK, THREED_MASK, LETTERBOX_MASK, WIDESCREEN_MASK
	premiere: PREMIERE_MASK, CHANNEL_PREMIERE_MASK, SEASON_FINALE_MASK, SERIES_FINALE_MASK, SEASON_PREMIERE_MASK, SERIES_PREMIERE_MASK
	show part 
	total parts
##2.10 11/13/2022	
1.  Fixed <date> logic to use date in <previously-shown>
##2.11 11/13/2022	
1.  For Strings use isEmpty not isBlank
*/

[boukmandutty]

The auto mapping works great.

The problem that remains for me is how sagetv reads the epg provided by Channels dvr. As @nyplayer pointed out over in the hardware thread, sagetv does some strange things with the names of movies and the titles of series when the epg comes form channels dvr. See the attached pictures. The same thing happens when I send the epg from channels dvr through xteve.

[hvymetal]

Originally Posted by boukmandutty

The auto mapping works great.

The problem that remains for me is how sagetv reads the epg provided by Channels dvr. As @nyplayer pointed out over in the hardware thread, sagetv does some strange things with the names of movies and the titles of series when the epg comes form channels dvr. See the attached pictures. The same thing happens when I send the epg from channels dvr through xteve.

Have you inspect the XMLTV file if is store like that? Also do you sample XMLTV file?

[boukmandutty]

I have attached a sample. I dont know where to access the Schedules Direct version for comparison though. I can confirm from experimentation that the current format from the channels dvr server works flawlessly with nextpvr and even with mythtv.

[nyplayer]

Originally Posted by boukmandutty

The auto mapping works great.

The problem that remains for me is how sagetv reads the epg provided by Channels dvr. As @nyplayer pointed out over in the hardware thread, sagetv does some strange things with the names of movies and the titles of series when the epg comes form channels dvr. See the attached pictures. The same thing happens when I send the epg from channels dvr through xteve.

You got it working on linux ?

[boukmandutty]

Originally Posted by nyplayer

You got it working on linux ?

Yeah the auto-mapping works with the files provided in the attachment above. The only problem continues to be the funky thing that happens with the names.

[nyplayer]

Originally Posted by boukmandutty

Yeah the auto-mapping works with the files provided in the attachment above. The only problem continues to be the funky thing that happens with the names.

I guess I will wait ...
Yeah the names and epgid etc ...also season and episode even though it is there is not recognized by SageTV. So I went with a sched direct west coast and east coast lineup.... and manually mapped channels I wanted.

[hvymetal]

Originally Posted by boukmandutty

I have attached a sample. I dont know where to access the Schedules Direct version for comparison though. I can confirm from experimentation that the current format from the channels dvr server works flawlessly with nextpvr and even with mythtv.

Since I do not know which entries are not correct can you provide and example file and test case is wrong? Also it would help if in the photos you highlight the problem.

Thanks

[hvymetal]

Originally Posted by nyplayer

You got it working on linux ?

The plug in not Linux dependent it should also work with windows.

[nyplayer]

Originally Posted by hvymetal

The plug in not Linux dependent it should also work with windows.

Great I will test it on my Windows test machine ...


I know for sure there is no original airdate parsed season episode parsed as misc.... and epgid data is wrong ...

https://forums.sagetv.com/forums/showthread.php?t=66885

[boukmandutty]

The pictures are below.

The xmltv info for The Angry Birds is:

<programme start="20220114020000 +0000" stop="20220114040000 +0000" channel="6081">
<title>The Angry Birds Movie 2 (2019)</title>
<desc>The birds unite with the green pigs when fowl from an ice-covered island plan to destroy them both.</desc>
<category>Movie</category>
<category>Feature Film</category>
<category>Animated</category>
<category>Children</category>
<category>Comedy</category>
<category>Adventure</category>
<icon src="https://tmsimg.fancybits.co/assets/p14568986_v_v12_ac.jpg?w=480&amp;h=720"></icon>
<date>20190802</date>
<previously-shown></previously-shown>
<episode-num system="tms">MV010390220000</episode-num>
<credits>
<director>Thurop Van Orman</director>
<actor>Jason Sudeikis</actor>
<actor>Josh Gad</actor>
<actor>Leslie Jones</actor>
</credits>
</programme>

The Family Guy episode:

<programme start="20220114023000 +0000" stop="20220114030000 +0000" channel="6082">
<title>Family Guy</title>
<sub-title>Peter's Progress</sub-title>
<desc>A psychic reveals that Peter had a past life in the 17th century as Griffin Peterson.</desc>
<category>Episode</category>
<category>Series</category>
<category>Animated</category>
<category>Sitcom</category>
<icon src="http://tmsimg.fancybits.co/assets/p184483_b_h6_bb.jpg"></icon>
<series-id system="tms">184483</series-id>
<date>20090517</date>
<previously-shown></previously-shown>
<episode-num system="tms">EP002960010132</episode-num>
<episode-num system="onscreen">S7E16</episode-num>
<episode-num system="xmltv_ns">6.15.</episode-num>
<credits>
<director>Brian Iles</director>
<director>Peter Shin</director>
<actor>Seth MacFarlane</actor>
<actor>Alex Borstein</actor>
<actor>Seth Green</actor>
</credits>
</programme>

I have attached for each a marked-up picture of what it looks like with the xmltv plugin. There is also a picture of what it looks like when Schedules Direct is used through the tedious mapping.

@nyplayer I didn't see your recent posts on your attempt to fix this until just now when you linked to it. If you have managed to fix any part of what I have here let me know.

[hvymetal]

WTF

         String title = this.show.title;
                // FIXME handling rerun shows correctly with SageTV 9
                // String rerunTitle = null;
                String rerunTitle = this.show.title;
                String episodeName = this.show.episodeName;
                if (this.initcapAllChannelIds
                        || this.initcapChannelIds
                                .contains(this.channel.xmltvId)) {
                    if (this.initcapTitle) {
                        title = initcap(title);
                    }
                    if (this.initcapEpisodeName) {
                        episodeName = initcap(episodeName);
                    }
                }
                if (title != null) {
                    if (this.show.date != null
                            && isActivated(categories,
                                    this.dateTitleDecorationCategories)) {
                        title = this.dateTitleDecoration.format(
                                new Object[] {title, this.show.date},
                                new StringBuffer(), null).toString();
                    } else if (this.titleAddYear
                            && this.show.year != null
                            && isActivated(categories,
                                    this.titleAddYearCategories)) {
                        title += " (" + this.show.year + ")";
                    }
                    if (this.show.quality != null
                            && this.show.quality.startsWith("HD")
                            && isActivated(this.channel.id,
                                    this.hdTitleDecorationChannels)) {
                        title = this.hdTitleDecoration.format(
                                new Object[] {title}, new StringBuffer(), null)
                                .toString();
                    }
                }
                if (this.show.rerun) {
                    rerunTitle = title;
                    // FIXME handling rerun shows correctly with SageTV 9
                    //title = null;

[hvymetal]

Originally Posted by boukmandutty

@nyplayer I didn't see your recent posts on your attempt to fix this until just now when you linked to it. If you have managed to fix any part of what I have here let me know.

Update should fix the problems listed. Update is in top post.

[hvymetal]

Originally Posted by nyplayer

https://forums.sagetv.com/forums/showthread.php?t=66885

Not sure if I fixed the problem because I could not find it but some of the other changes might have solved this one also.

[nyplayer]

I am getting this error ... on Windows. All I want to test is bringing in epgdata.xml which is in the SageTV root directory ... no XTEVE yet.

14-01-2022 04:47:29,860 java.lang.NoClassDefFoundError: xmltv/Channel
at xmltv.XMLTVImportPlugin.startElement(XMLTVImportPlugin.java:1431)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator.startElement(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)
at xmltv.XMLTVImportPlugin.updateGuide(XMLTVImportPlugin.java:1299)
at xmltv.XMLTVImportPlugin.updateGuide(XMLTVImportPlugin.java:1204)
at sage.EPG.pluginExtractGuide(EPG.java:1625)
at sage.WarlockRipper.extractGuide(WarlockRipper.java:973)
at sage.EPGDataSource.expand(EPGDataSource.java:709)
at sage.EPG.run(EPG.java:773)
at java.lang.Thread.run(Unknown Source)
14-01-2022 04:47:29,860 java.lang.NoClassDefFoundError: xmltv/Channel
at xmltv.XMLTVImportPlugin.updateGuide(XMLTVImportPlugin.java:1311)
at xmltv.XMLTVImportPlugin.updateGuide(XMLTVImportPlugin.java:1204)
at sage.EPG.pluginExtractGuide(EPG.java:1625)
at sage.WarlockRipper.extractGuide(WarlockRipper.java:973)
at sage.EPGDataSource.expand(EPGDataSource.java:709)
at sage.EPG.run(EPG.java:773)
at java.lang.Thread.run(Unknown Source)

[hvymetal]

Originally Posted by nyplayer

I am getting this error ... on Windows. All I want to test is bringing in epgdata.xml which is in the SageTV root directory ... no XTEVE yet.

I think this was caused because the docker I'm using is Java16 and I did not compile for lower version of java. Can you test the attached version it was compiled for 8 or higher of java.

[nyplayer]

These are the error logs and my xmltv.properties... I might be doing something wrong in the setup.

[hvymetal]

What Java are using? Also how did you install? You should only need the .Jar file copied to you servers JARs folder. Make sure you do not have older versions of the plugin in the same folder(.jar or folders). Also any xmltv folder in the root of the Server or in the JARs referencing the older version need to be renamed

[nyplayer]

Originally Posted by hvymetal

What Java are using? Also how did you install? You should only need the .Jar file copied to you servers JARs folder. Make sure you do not have older versions of the plugin in the same folder(.jar or folders). Also any xmltv folder in the root of the Server or in the JARs referencing the older version need to be renamed

jre1.8.0_191


I am going to try a full fresh install of SageTV there might be some stuff from the old XMLTV around.

[hvymetal]

Originally Posted by nyplayer

jre1.8.0_191


I am going to try a full fresh install of SageTV there might be some stuff from the old XMLTV around.

It worked with the following setup in a Window 7 64bit VM
1. Installed with latest SageTv 64bit installer
2. Ran SageTV First time
3. Stop SageTV
4. copy the XMLTVImportPlugin.jar in SageTv Jars folder
5. Add the following line in Sage.properties epg/epg_import_plugin=xmltv.XMLTVImportPlugin
6. Copied a XMLTV FILE name epgdata.xml
7. Start SageTV
8. Source Setup->EPG Lineup
9. Zip Code 0000
10. Will list XMLTV Lineup
11. Use extended

The plugin will work with out any .properties file if epgdata.xml in the root of the SageTV server directory. It will give error in xmltv.error.log for not finding the .properties but still work(This is a bug and need correcting). The .properties are used if you want to do customizations and add multiple lineups. The examples I provided 2 examples that give the guide a different name and location of the XMLTV file.