RFC: XML show information format

[nielm]

A lot of STVs are now outputting XML show info, and I have defined a standard XML show information format that would contain all the information in Sage's DB.

This format is documented on the Sage Wiki, and the XML data can be generated by either the v2.7 (or higher) version of the sage webserver, or by the Sage Recordings XML file info generator.

The Java library used to generate the XML show information has public APIs that can be used by other STVs to generate XML files for use for other purposes.

The intention is also to create a Sage XML reader class so that the webserver can be used to both export + import favorite + show information and make DB backups...

[KJake]

Most people say that by setting the charset to UTF-8 will make it universal, however I've heard that may not be true and the charset may need to be configurable. Lets hope someone who knows more on that topic jumps in...

Anyways, I think this is a great idea.

[nielm]

A couple of people have asked for the Episode Number to be in the XML... I cannot add this as a separate item because this information is not explicitly available from Sage's DB...

However, in Zap2It's data, the Epsiode ID seems to be the last 4 digits of the EPGID (provided that it starts with EP and has 10 or more digits)...

[owilsky]

Originally Posted by nielm

as well as an external backup of the Sage DB in case the wiz.bin gets smoked...

yepp, yepp, huraaaah... yes! Definately a good idea! I lost a lot of shows because of a corruption. I have daily backups, but I did not notice this coruption at first. And several days/ recordings later a backup is useless because the new recordings would be gone...

I know nothing about how XML works with non-latin-1 characters, so if there are any Java/XML experts there, can you give me any tips on writing eg, Hebrew/Greek/Arabic/etc XML? (Java is unicode internally, so it 'should' just be a matter of setting the charset in the XML writer...)

Normally I would suggest to encode the xml file in UFT-8... but if we do UTF-16 be are on the secure side, right? And it is still editable by notepad (on Win2k/XP).

A couple of people have asked for the Episode Number to be in the XML... I cannot add this as a separate item because this information is not explicitly available from Sage's DB...

I suggest that more fields in XML should be possible than in Wiz.bin. Just ignore those fields when you import the xml to wiz.bin.

More problems will be for your external "STV-independant JAVA plugin", it has to read the xml and wiz.bin and update only the changed fields without touching those fields which aren't available in wiz.bin.

[owilsky]

One note to the <description> tag although this might be important for other tags as well.

Nielm, do you remember the line break problem? How to cope with these?

[eobiont]

Could chapter information be optionally encoded here?

I think you would want to call them chapters or something because they would not necessarily be comercials, but could just mark the start of a movie or where it ends - you know to lop off the first 5 and last 8 minutes of a HBO movie. I could also see a STV developed that would let you set chapter markers. So you could jump to a next marker. Programs that are designed to burn DVDs or clip out show secions could use this information.

<chapterList>
<chapter
chapterDescription="Monologue"
startFrame=0
endFrame=625
/>
<chapter
chapterDescription="Commercials"
startFrame=626
endFrame=3996
/>
<chapter
chapterDescription="First guest"
startFrame=3997
endFrame=10333
/>
</end chaperList>

[bluenote]

I think this is a *great* idea niel. Im afraid I dont have specific comments about the XML because I'm only an amateur programmer but I commend you RFC'ing -- this kind of feature increases its functionality greatly the more standard it becomes. I look forward to the day that I can fully record and play in mpeg-4 ... no matter how much disk I get, its not long after that I am hunting for more disk space

anyways kudos on the idea and on the openness of asking for suggestions

Cory

[nielm]

thanks, if only I had the time to implement it

eobiont: cool idea... Not sure how I would make it work for Sage Recordings though: it would mean 2 programs needing to write to the same file (Sage for updating watched flags, etc; SA or whatever to update the chapter section...).
Maybe 2 separate files is still best for the moment...

[Jere_Jones]

Originally Posted by nielm

thanks, if only I had the time to implement it

As soon as I get some time, I'll help.

Originally Posted by nielm

eobiont: cool idea... Not sure how I would make it work for Sage Recordings though: it would mean 2 programs needing to write to the same file (Sage for updating watched flags, etc; SA or whatever to update the chapter section...).
Maybe 2 separate files is still best for the moment...

I've been working on giving SA the ability to add info to an xml for exactly this purpose. This isn't hard. The real problem (as I see it) is that opening and parsing an XML is more demanding than opening and parsing the current txt file. This would come into play when reloading the commercials if doing live detection.

Another problem you might have is if you edited the information in sage (via the webserver or some other future means) then the xml will need to be rewritten. Any process doing this will need to basically load the xml and only change what it knows, leaving the rest. Unfortunately, it seems that most XML libraries are centered more around writting or reading, not updating.

Just my $0.02.

Jere

[ajuhawk]

I too definately like the idea of a universal xml format. One thing I've noticed with the latest beta of v3 is that the hidden features allow you to "export xml" but you have to do it manually (i.e. not automatically generated with every new recording). My point is that if a truly universal format is going to be created, Sage almost needs to be involved as well, or better yet, they'll do it for us

-AJ

[MTuckman]

nielm,

I noticed this thread, and thought I would post what I had worked up for your web server. I am using TVedia as my front end, as I am using an Xcard and the response time of the plugin was too slow. TVedia does a great job with the interface, but doesn't hold a candle to Sage's EPG and PVR functionality.

I have created two java classes that I added to your webserver. One returns all of the recordings that currently exist, the other returns general status of Sage, including disk space and currently recording.

They are a little rough, but I am working on "porting" as much of the Sage environment through XML to TVedia. Eventually I would like to be able to control favorites, etc. from TVedia, using the XML as the data driver, and the webserver handling the requests.

Here are the two classes I wrote, for what it is worth.

Thanks,

Mike

[stanger89]

FWIW, this was just posted on AVS:
http://mcgrathinfosolution.com/mis_en.htm

[KJake]

Any chance this could be ported/used in a stv import so that XML files are written for all shows in your recording directories? I think I had an STV that did this for me back with Sage2 or 3, but not any more from what I can tell.

The reason I'm asking is, that if we can agree to keep using this DTD, then I will use this format in the coming versions of my CreateDVD program. That way, anyone can load the import into whatever STV they're running and I'd have a way to work with any STV.

[alon24]

Malore has an option in the Unofficial Secrets - for creating an XML for a show.

That option as it is written now, does not support any other lang other than English.

I have a fix for that, for any 1 who is willing to implement it (2 lines).

please contact me, as I do think this should change in the main Sagetv, since it comes basicly from the company in the original STV.

Again, using the fix (2 lines) will fix the creation of XML files for show from the Unofficial Secrets used by the default STV.

[nielm]

My idea was to have an STV-independant Java plugin that would create XML files for all sage recordings on start up, and every so often check to see if any new recordings had been created and write those too...

But, I have been too busy to do anything

[aperry]

If you're not already going there, I'd suggest that for at least the show information (non-Sage specific info), that you use the XMLTV format for consistency. That would make porting it to anything else much easier.

[KJake]

Originally Posted by alon24

Malore has an option in the Unofficial Secrets - for creating an XML for a show.

That is good, but it will be specific to that STV. I'm really looking for something we can standardise and make it importable to any STV.

[KJake]

Originally Posted by nielm

My idea was to have an STV-independant Java plugin that would create XML files for all sage recordings on start up, and every so often check to see if any new recordings had been created and write those too...

I like your approach, it is exactly what I'm looking for. What do you need help with, or is it just time?

[alon24]

Originally Posted by KJake

That is good, but it will be specific to that STV. I'm really looking for something we can standardise and make it importable to any STV.

When you turn on the unofficial secrets then its there in any stv, so its there baisicly anytime

[KJake]

Originally Posted by alon24

When you turn on the unofficial secrets then its there in any stv, so its there baisicly anytime

I just tried on SageMC and it doesn't do anything. I'm certain that the unofficial secrets are only built into the official STV (and its derivatives).