ananova has stopped there xml listings service

[nielm]

XMLTV 0.5.28 has just been released with source and Win32 EXE versions.

This should fix tv_grab_uk_rt (at least the release notes say so, I have not tested it), and include my patch for getting lots more channels than before.

Note that tv_grab_uk_rt now has a fast mode by default (where only program times and titles are downloaded) and an optional slow mode (--slow command line arg) -- giving same behaviour as before (with detailed descriptions, actors etc, but very slow download times).

[jimbobuk]

Kind of working but boy is it slow in detailed mode... yet far too sparse in default mode... you only get show name, not epsiode name.. hasn't this got huge potential to screw even more with Sage's recording decisions..

Anyways this post seemed the more appropriate place to ask, but nielm you said this on another thread

ecause uk_rt is so slow, I run it daily with --offset 7 --days 1 saving the output into dated files so that I only get 1 days' data at a time... I then combine these dated files with tv_cat and tv_sort into a single epgdata.xml file...

I was wondering if you could elaborate on this a little.. if you have a batch file that you use could you post it up online.. or if not could you just explain the syntax of tv_cat and tv_sort.. I can't find any help files on it.. there is on on tv_check.. i'll read over that tomorrow.. presumably it goes something like this?!

First Run:

xmltv tv_grab_uk_rt --days 1 --slow > epgdata00.xml
xmltv tv_grab_uk_rt --days 1 --slow --offset 1 > epgdata01.xml
xmltv tv_grab_uk_rt --days 1 --slow --offset 2 > epgdata02.xml
xmltv tv_grab_uk_rt --days 1 --slow --offset 3 > epgdata03.xml
xmltv tv_grab_uk_rt --days 1 --slow --offset 4 > epgdata04.xml
xmltv tv_grab_uk_rt --days 1 --slow --offset 5 > epgdata05.xml
xmltv tv_grab_uk_rt --days 1 --slow --offset 6 > epgdata06.xml
xmltv tv_grab_uk_rt --days 1 --slow --offset 7 > epgdata07.xml

xmltv tv_cat epgdata00.xml epgdata01.xml epgdata02.xml epgdata03.xml epgdata04.xml epgdata05.xml epgdata06.xml epgdata07.xml epgdata.xml

xmltv tv_sort epgdata.xml

Then daily runs of:

xmltv tv_grab_uk_rt --days 1 --slow --offset 7 > epgdata07.xml
xmltv tv_cat epgdata07.xml epgdata.xml epgdata.xml
xmltv tv_sort epgdata.xml

assuming the cat has syntax of <source1> <source2> <source3> <destination> ...

Is tv_sort required!?? I thought i'd read somewhere where it said that xmltv files dont really have an order to them!?

Also with something like this isn't your main xml file getting larger and larger each time?!? I make out my channel selection for a day to be about 800k it could get quite unwierdly fairly quickly.. is this a problem for Sage too causing more CPU spikes as it tries to parse a monster 50meg epg file?

All in all though it looks quite good if you could detail how to correctly use tv_cat and why you have to use tv_sort then i plan to do this

i) do a slow rip of my favourite channels that have 90% of my recordings on (bbc1, 2, itv, channel4, five, sky one, sky one mix, living, e4 etc.) to one epg file

ii) do a normal rip of remaining channels to another epg file

iii) tv_cat them together

iv) sort them (when i know why we have to

and that way reduce the time and stress on their site...

One final thing.. are you guys all doing the really bad manual channel configuration stuff to setup your channels.. it was slow and horrible with tv_grab_uk but having to type with the rt one is really annoying... from what i can see the file that is created is fairly trivial text.. surely we could just work on a file that has EVERY channel added and then we manually delete it ourselves.. this would be far more methodical and quicker and easier... is this how all you guys work with your config? why doesn't .xmltv ship with example files listing all possible channels in the same format ready for direct editing..

Phew a biggie sorry about that.. should pretty much list all my queries and concerns about this at the moment... I hope we get some official detailed EPG data sometime soon, if only to give poor RT and friends a rest from our pillaging

[jimbobuk]

for anyone about to start configuring as a test.. not sure if this will work but as mentioned above here is my tv_grab_uk_rt.conf file for most of the important channels on sky and midlands terrestrial... I assume (unless someone else says otherwise) that you can just at least, delete entries on here and have it setup for you.. save you 5 mins of pain at least..

edit: [appended .txt onto it so it would attach]

[nielm]

Re episode name -- if you dont have episode names, you may find that Sage always records episodes of everything, even if you have already recorded it -- this is because the XMLTV plugin uses the program start time in place of the episode name to generate a programs unique ID

--

For channel ids for your config file, there are all listed in the file XMLTV_dir/share/xmltv/tv_grab_uk/channel_ids, you can copy/paste from this file into a config file...

--

For getting daily listings I use a unix-style cygwin shell script that is a bit cleverer that runs multiple grabbers, saves the files in dated filename (eg epgdata_20040203_uk_rt.xml), checks that grabs completed sucessfully and gets missed dates.

But something similar can be done in a BAT file..

all tv_cat/tv_sort/tv_grep commands need to be run in the form:
tv_command source(s) > destination

tv_sort is not strictly required (tv_cat is quicker), but performs a lot of cleaning up, checking for overlaps, and adding stop times to programs that do no have them (Sage does not care that shows are not in any particular order, but it a show does not have a stop time, the XMLTV plugin will not recognise it).

I would also recommend keeping the 'daily' files in a separate directory from the sage directory, and only copy the completed file over (cleaner, easier for debugging!)

The commands that you listed would nearly work, but with minor modifications (note none of this has been tested!)

First Run:

xmltv tv_grab_uk_rt --days 1 --slow > epgdata00.xml
xmltv tv_grab_uk_rt --days 1 --slow --offset 1 > epgdata01.xml
xmltv tv_grab_uk_rt --days 1 --slow --offset 2 > epgdata02.xml
xmltv tv_grab_uk_rt --days 1 --slow --offset 3 > epgdata03.xml
xmltv tv_grab_uk_rt --days 1 --slow --offset 4 > epgdata04.xml
xmltv tv_grab_uk_rt --days 1 --slow --offset 5 > epgdata05.xml
xmltv tv_grab_uk_rt --days 1 --slow --offset 6 > epgdata06.xml
xmltv tv_grab_uk_rt --days 1 --slow --offset 7 > epgdata07.xml

xmltv tv_sort epgdata00.xml epgdata01.xml epgdata02.xml epgdata03.xml epgdata04.xml epgdata05.xml epgdata06.xml epgdata07.xml > epgdata.xml 
copy epgdata.xml SAGE_DIR

Daily runs:

rem Keep 2 days of 'old' programs in _m1 -- minus1 and _m2 -- minus2 files

del epgdata_m2.xml
ren epgdata_m1.xml epgdata_m2.xml
ren epgdata00.xml epgdata_m1.xml
ren epgdata01.xml epgdata00.xml
ren epgdata02.xml epgdata01.xml
ren epgdata03.xml epgdata02.xml
ren epgdata04.xml epgdata03.xml
ren epgdata05.xml epgdata04.xml
ren epgdata06.xml epgdata05.xml
ren epgdata07.xml epgdata06.xml

xmltv tv_grab_uk_rt --days 1 --slow --offset 7 > epgdata07.xml
xmltv tv_sort epgdata00.xml epgdata01.xml epgdata02.xml epgdata03.xml epgdata04.xml epgdata05.xml epgdata06.xml epgdata07.xml > epgdata.xml 
copy epgdata.xml SAGE_DIR

--

It is not perfect, but it is difficult with the limited features of DOS batch programming do do anything more advanced...

For instance if it was run multiple times on one day, it would remove a day of programs each time (which is why I added the keeping of history above).
If anybody knows a better Windoes scripting language they could probably improve on it. There is also talk on the XMLTV mailing lists of writing something generic in perl that could be included into the next release.

Enhancements could be:

running the grabber with different config files for different channels, one with --slow, one not:

xmltv tv_grab_uk_rt --days 1 --slow --offset 7 > epgdata07_a.xml
xmltv tv_grab_uk_rt --days 1 --offset 7 --config-file fast_channels.conf > epgdata07_b.xml
tv_cat epgdata07_a.xml epgdata07_b.xml > epgdata07.xml
del epgdata07_a.xml epgdata07_b.xml

You could also add the command to remove 'clumps' (see this post):

(instead of tv_grab and tv_sort above

xmltv tv_grab_uk_rt --days 1 --slow --offset 7 > epgdata07_c.xml
tv_grep --not --clumpidx "" -or --clumpidx "0/" epgdata07_c.xml > epgdata_07.xml
del epgdata07_c.xml

The daily script could be improved to handled missed days using IF EXISTS commands -- in this way, the daily script can also perform the task of the 'first run' script:

instead of

   ren epgdata01.xml epgdata00.xml
   ren epgdata02.xml epgdata01.xml
   ren epgdata03.xml epgdata02.xml

etc, put

    IF EXIST epgdata01.xml (
        ren epgdata01.xml epgdata00.xml
    ) ELSE (
        xmltv tv_grab_uk_rt --days 1 --slow --offset 0 > epgdata00.xml
    )
    IF EXIST epgdata02.xml (
        ren epgdata02.xml epgdata01.xml
    ) ELSE (
        xmltv tv_grab_uk_rt --days 1 --slow --offset 1 > epgdata01.xml
    )
    IF EXIST epgdata03.xml (
        ren epgdata03.xml epgdata02.xml
    ) ELSE (
        xmltv tv_grab_uk_rt --days 1 --slow --offset 2 > epgdata02.xml
    )

etc, addding the tv_grep for clumps if necessary.

Finally, if you really feel like a challenge, you could use find to verify that the daily grab command succeded or failed, and then delete the file so that it would be re-retrieved at next run time:

xmltv tv_grab_uk_rt --days 1 --slow --offset 7 > epgdata07.xml
rem See if epgdata file contains any programmes...
find  epgdata07.xml "<programme" >nul:
if errorlevel 1 (
   echo "epgdata07 does not contain programmes -- deleting"
   del epgdata07.xml
)

[nielm]

Note: the tv_sort command above is missing the 'm1' and 'm2' files. Should be:

xmltv tv_sort epgdata_m1.xml epgdata_m2.xml epgdata00.xml epgdata01.xml epgdata02.xml epgdata03.xml epgdata04.xml epgdata05.xml epgdata06.xml epgdata07.xml > epgdata.xml

And the help files for all XMLTV commands in the Win32 EXE are in the docs/man subdirectory of the zipfile.

[jimbobuk]

I've just tried to get this working.. ran a minimal pass of channels first off, followed by a slow one today for the first time.. It fails on the sort command.. as a 1st pass i assume i can do tv_cat... I will look into the docs.. here is the error ...

C:\Program Files\xmltv-0.5.28-win32>xmltv tv_sort epgdata00.xml epgdata01.xml epgdata02.xml epgdata03.xml epgdata04.xml

Timezone is +0000

epgdata00.xml:
junk after document element at line 4550, column 0, byte 248052 at /PerlApp/XML/
Parser.pm line 187

I've been looking at the file.. not exactly by line yet but i had ran the xmltv grab on another machine in another directory and had

adding '--share=C:/Program Files/xmltv-0.5.28-win32/share/xmltv'

or similar on the end with the incorrect path.. correcting this though and its not helping.. I'll get a decent text editor installed and have a proper look at where its complaining..

[jimbobuk]

yeah its this share line.. not sure exactly what the problem is.. the path is right now... anyone?!

[jimbobuk]

removing this share line from each file seems to make it work but these are files generated by the program.. it'll stop my batch file working.. i could trim it but i just dont understand.. if anyone can help please do, i'm getting near the end of my current set of data

[jimbobuk]

as i said removing the shareline made both tv_cat and tv_sort work.. I tried using the new tv_sort-ed version with Sage and on loading it used CPU for a long time but then going into live tv and it had a few programs listed.. other channels were blank.. wait a little more and they all turned into nodata... now also my recordings have ALL lost their names... not exactly brilliant..

Whats going off?!?

[jimbobuk]

one final update.. i'm giving up for tonight.. my recordings are still screwed.. but live tv has SOME data in it but only at and around Sunday.. this will perhaps be the time that was done with --slow today.. the rest would of been these dodgy ones i made up on another machine with the quick grab to try and populate the epg to begin with ...

I really hope i can get my recording information back

A quick glance and my wiz.bak is 2meg, my wiz.bin is 600k.. I've copied the wiz.bak just in case this may hold my info... it looks like Sage could of gone a bit crazy... can i just copy the bak over the bin and hopefully get back to where we are.. I may try to get some cyclical backups going if the wiz file is THAT important

[nielm]

(dont run the EXE myself, so don't have the --share problem)

the 'share' line is some debug output from when you run the grabber specifying where XMLTV files it's helper files (eg channel_id's). It should not get added to the XMLTV file (I think this is a bug in the EXE). It should normally point to the share/xmltv subdir of the xmltv directory where you unpacked the win32 zipfile.

You could get rid of it either by adding that argument it to the command line of the xmltv tv_grab_uk_rt (meaning that XMLTV.exe does not have do to it and report it itself), or by usign the --output option instead of '>' to indicate the output file:
xmltv tv_grab_uk_rt --days 1 --slow --output epgdata00.xml


Cyclic backups (I do them daily) of sage.properties and wiz.bin are VERY important, especially when testing with the Beta version! wiz.bin holds all EPG info, all favourites settings and all info for recorded files, plus all watched programs for intelligent recording etc, any loss or corruption of this file (for whatever reason, eg bugs in the Beta) would lose all this data.

WRT loss of EPG data, do you get any Java exception messages in your sagetv_0.txt logfile at about the time you update the EPG?

It is very surprising that all recordings have lost their names, as normally this info is not affected by updates to the EPG...

[MrManson]

I had the --share problem also - added it to the command and it seems to work ok now (although positioning is important - I think I had to put it ahead of --days to get it to work).

I'll find out when I get home if it's run ok overnight...

[jimbobuk]

well i've tried restoring the bak file but its not really done anything.. I get a bit of the EPG back but my recordings have lost all their details.. I'm just going to have to watch them within windows not sage.. brilliant

I wish Sage performed the .bak cycling itself.. I'd happily allocate 10-15 .baks or more to isolate this myself... time to get my batch file doing it on startup..

Still not sure whats up with my EPG

[jimbobuk]

How do you use the -share option i've tried

xmltv -share
xmltv --share

and it says its not a valid command for both..

what is the proper syntax of this in use, the same as what is on the end of my EPG files

so

xmltv tv_grab_uk_rt --share=C:/Program Files/xmltv-0.5.28-win32/share/xmltv --days 1 --slow --offset 4 > epgdata04.xml

is what i need to do?!?

[MrManson]

Here is the batch file that I use to run xmltv...

xmltv tv_grab_uk_rt --days 7 --share="C:\Program Files\xmltv\xmltv-0.5.27-win32\share\xmltv" > listings.xml
copy listings.xml "C:\Program Files\Frey Technologies\SageTV\epgdata.xml"
exit

Hope it helps.

**obviously I should have said I put it AFTER --days - that'll teach me for surfing when I should be working

[jimbobuk]

Ok well my EPG looks ok from where i've been grabbing slow rips 4 days ahead.. the mix of fast and slow must be messing with it... or has anyone tried with the fast details in Sage.. maybe it can't handle it..

losing my recording info will perhaps help me to flush it out and start again fresh... I will watch stuff first of course.. the other thing thats bugging me is as reported somewhere else i've still lost 20gig on my recording drive.. it says i have 150gig but i can only find ~135-140gig on the disc itself.. i've turned off system restore and even the index searching to no avail.. may try a reformat but i think i will probably need too many of my programs to be saved than i can fit elsewhere on another drive..

anyways back on topic.. i remember this being talked about somewhere else to do the cyclic backups but i decided to write my own.. its not exactly rocket science but it appears to work.. will have to see how it pans out.. I'll attach it here for anyone who wants to just use it.. edit for directory paths accordingly.. ohh and make a backup directory in your sage dir.. its not at all smart

[nielm]

It Depends who is saying a disk is 150Gb... Disks are sold using 'marketing gigs', ie a 150 gig disk has 150,000,000,000 bytes of unformatted space on the disk, which is equivalent to about 139.7 real gigabytes... ie 139.7*1024*1024*1024.

So your 135-140GB of real space is normal (some additional space is lost in formatting) for a 150Gig drive.

[nielm]

Fast mode seems to have problems with the start date of the first program in the day:

<programme start="20040205232000" stop="20040206002000 +0000" channel="london.bbc2.bbc.co.uk" clumpidx="0/1">

Note that the start time does not have a timezone -- this is not handled by the XMLTV plugin, which expects that all times have a time zone, so it cannot parse the time, so it fails...

This is probably a bug in the XMLTV plugin rather than a bug in the uk_rt grabber, because the date is still in a valid date format...

Workaround: Believe it or not, passing the generated EPG data through the tv_grep command in my next post -- which adds timezone to start dates which are missing it should help...

After the end of march, you may need to change +0000 to +0100, if the bug has not already been fixed)

I will raise it on the XMLTV mailing list anyway, when I have figured out where the bug is... I also have a performance boost patch for tv_grab_uk_rt that I want to add...

[nielm]

tv_grep command to fixum programs with no start timezone:

xmltv tv_grep --eval "if ( ${$_}{start} =~ /^[0-9]{12,14}$/ ) { ${$_}{start} .=' +0000'; }  1" no_tz.xml > fix_tz.xml

(replace no_tz.xml and fix_tz.xml with your input and output files.

[jimbobuk]

Thanks Nielm...

I did wonder about the timezones.. i've never fiddled with it before (tv_grab_uk) and its always seemed to be right.. why are timezones important.. surely the data will be correct and so will your machine.. and both will be in the same zone....!?

As for the hdd.. i appreciate this difference in who reports sizes BUT generally this is reflected in the sizes windows reports.. windows reports a given size and my total files on it is signifantly less (15gig or so) yet i only have 1-2gig free... this hdd did die in another machine, i wonder if its declared a lot of space as dead..!?? i dunno.. i've just had an identical drive (model, size) die in that same computer just last week, so there is something going on.. always happens when i'm ripping a dvd with dvd decrypter, something that i wont be doing again in this machine at all.. I thought i'd cracked what had caused it, the whole lack of support for 48bit LBA hdds in windows 2000 (if you dont enable a registry tweak and have post sp4 or 3 then your OS wont handle them properly) .. but i've since repartitioned this new drive that has just failed into partitions less than the limit for 48bit.. in the microsoft fix this was claimed to fix any problems... its just annoying

My xmltv grab didn't work as i didn't put "" round the path for share/source ... dammit

Also i seem to get occasional files created that are epgdata04.xml~ I just can't see what creates it... I'm doing the ren commands before as described above to make space.. i wonder if the OS isn't able to move them in time (i really hope not as it sounds ridiculous) and xmltv is creating the ~ file instead of going over the one that still exists there..

Thank god its friday is all time to get a beer