Docker sage upgrade script and new user questions

[Madz]

I've been a long time Sage on windows server user, but time to retire my ancient WHS2011 so I've bought a HDHomeRun quadro and want to move to running sage inside a linux docker container

I'm looking through the docker containers that stuckless has created - what an awesome resource thank you so much! Unfortunately this is also my first experience with docker, so I have a few n00b questions if stuckless or anyone could please help me:

I'm confused about a few things:

1) Looking at the sage updgrade script (https://github.com/stuckless/sagetv-...upgrade-sagetv) I can see it references https://dl.bintray.com/opensagetv/sagetv/sagetv/ but I'm confused about a couple of things:
Firstly, it looks like https://github.com/OpenSageTV/sagetv was forked off https://github.com/google/sagetv but there seems to be activity on the original google one ? I don't know what the history was here when this became open source, but which one of these is the update script link above pointing to? And secondly, when I try to manually access that link it just says forbidden - looks like maybe bintray is moving ?? or maybe I'm doing something wrong - stuckless can you confirm if that updgrade script is still correct??

2) I can see there's a number of containers with different java versions, is everything rosy on the latest java11 or are there any reasons I might stick with an older java version?

3) I have a comskip license so will have to figure out how to use registered version of comskip - there used to be a difference (on windows) not sure if that still applies

4) I use a custom STV based on the old SageMCE - I guess the sage tv files all end up exposed on the host and I can just copy that into place on the host?

5) For the config options - do I need to worry about PUID, PGID, VIDEO_GUID if I'm only running the network tuner??

For v9 do I need to worry about LICENCE_DATA ??

Any guidance on an appropriate java memory size for linux??

Thanks in advance

[Striker:WG]

well, i'm not as familiar with installing docker straight on Linux. I know it can be done (I briefly messed around with it before switching to Unraid) but it is far easier on Unraid.

Bintray has shutdown their old services and it has broken a few things around here. The people responsible for maintaining it (and that would have received notices about the pending closures) have moved on. We've tried reaching out to Bintray but they don't seem to be too interested in helping us rectify the situation.

I'm currently running the Java8 docker. I don't remember exactly why (maybe it was plugin related?) but there were reasons that Java8 was more compatible.

Comskip is based on having a donators version downloaded to enable processing of HD files. That being said, I honestly don't remember if I needed to replace the file that came with the docker container, I think it just worked.

With a docker container, it will have its own internal self contained file system. You can then setup mount points that expose portions of the host file system. You will want to have configuration files, your wiz.bin and any media/recording files in the host file system so they don't get overwritten if the docker container ever gets rebuilt. On Unraid, its pretty simple as everything is pre-setup with parameterized options that make it pretty easy to adjust. Not as sure how it works with a standard docker install.

V9 does not require license data unless you are trying to use an old mpeg demux from version 7 to keep support for subtitles (I think there's a few threads around here on the topic).

I run 3GB of memory allocated to my Sage docker to ensure there's plenty of memory for Java.

[Madz]

Thanks for the quick reply Striker! I'm looking to run on a Synology NAS eventually, but just trying to get it up and running on a basic linux install for now. I'm using Portainer so have a GUI to help with the config.

When you say this, you don't mean Stuckless, right - you mean at bintray end?

Originally Posted by Striker:WG

Bintray has shutdown their old services and it has broken a few things around here. The people responsible for maintaining it (and that would have received notices about the pending closures) have moved on

I'm worried at the moment the container create won't work at all as it won't be able to pull down the sage files, but will test it out shortly and see.

I'm still confused with the PUID/PGID side of things - if that needs to point to a sage tv user on the host, does that mean I should be running the container in the context of that user?? (eg not sure about unraid, but in docker there's a --user option I think).

The other question I have - I saw stuckless had written that to upgrade your existing installation you can copy the files into the server folder (host-side) before starting the container - if converting from windows to linux can I copy over my existing wiz.bin and sage properties or will they not be compatible??

Thanks

[wnjj]

Before Bintray shut down I copied the latest Linux install over to GitHub. If the upgrade script is modified with the new path it should be able to run. The only problem is the automatic build for Linux is also broken so for now Linux is stuck at version 9.2.2.

The server install package the script needs is located here:
https://github.com/OpenSageTV/sagetv...2_amd64.tar.gz

[Striker:WG]

Originally Posted by Madz

Thanks for the quick reply Striker! I'm looking to run on a Synology NAS eventually, but just trying to get it up and running on a basic linux install for now. I'm using Portainer so have a GUI to help with the config.

When you say this, you don't mean Stuckless, right - you mean at bintray end?


I'm worried at the moment the container create won't work at all as it won't be able to pull down the sage files, but will test it out shortly and see.

I'm still confused with the PUID/PGID side of things - if that needs to point to a sage tv user on the host, does that mean I should be running the container in the context of that user?? (eg not sure about unraid, but in docker there's a --user option I think).

The other question I have - I saw stuckless had written that to upgrade your existing installation you can copy the files into the server folder (host-side) before starting the container - if converting from windows to linux can I copy over my existing wiz.bin and sage properties or will they not be compatible??

Thanks

Yup, Stuckless is still around, my comment was in reference to the folks over at bintray and some plugin developers that have moved on. We've been enjoying the benefits of their hard work from years ago and now the house of cards has toppled. Still trying to figure out how (if it all) it can be rebuilt. There's another thread on the topic here: https://forums.sagetv.com/forums/sho...t=66721&page=2

Hopefully Stuckless can respond, as I'm not sure if the docker container has any reliance on bintray or not.

In regards to the PGID/PUID that is something that is abstracted from the Unraid experience, so not sure what to tell you on that one.

Yes, you can copy your windows wiz.bin into your linux environment and they will work. I've done that several times. I did have to mess around with CHOWN and CHMOD to ensure the wiz.bin I was copying in had the same settings as the original one.

[stuckless]

For #1 - the master source is from google's repo. Th opensagetv upgrade script was basically downloading from bintray, which has others have pointed out, is now dead. So likely even if you start this up, it will not get/install sagetv.

For #2 - I think any of the containers work, I think I was using the Java 11 one.

For #3 - There is no difference between licensed and unlicensed comskip on Linux (it is the full version on Linux)

For #4 - Files are exposed via the volume mounts

For #5 - those should map the userid, grouped on your host system (I don't think I set them on unraid)

You don't care about the license key, it's not used.

But, since bintray is shut down, my guess is that these docker containers will ultimately stop working as well (at least they won't be able to download/upgrade sagetv)

[wnjj]

Originally Posted by stuckless

But, since bintray is shut down, my guess is that these docker containers will ultimately stop working as well (at least they won't be able to download/upgrade sagetv)

Version 9.2.2 is hosted on Github/OpenSageTV because I uploaded it there but Travis doesn't seem to be running on pull requests anymore. Even if it did, there's no upload method for Github so updates for Linux are currently stalled. I don't have a Linux setup so cannot build them manually for upload.

[Madz]

Thanks everyone for your responses, much appreciated

wnjj - I actually came across those binaries last night before I saw your reply - very glad you put them there, thank you!

It seems I've picked a pretty difficult time to try and do this as in addition to the base binaries not being available for the docker container scripts to work, most plugins can't be installed either as they reference bintray too.

Anyway I was able to get partially up and running using the binaries wnjj set aside and copying the plugins off my windows installation, although I've still got a couple of issues with my old STV3 MCE theme not working (metal mania) it's showing a black and white screen with no selection box, so not sure how successful that's been. More work to do there.

The most concerning I issue I have atm though is that all channels for one particular DVB-T station on the HD Home Run always switches to playing in slow motion after a few seconds - it affects all streams/channels on that station/base frequency (sorry not sure the correct terminology). The channels on the other stations seem to play fine. Playing the same station via VLC direct off the HD Home Run works fine, so it's some sort of SageTV/linux/docker issue? Anyone seen that before??

[Madz]

So I have my SageMC theme issues resolved - was just a bad path with backslashes rather than forward slashes that slipped through the cracks. Everything is now working well except for this issue with the HDHR Quatro tuner which is going to end up being a deal breaker unfortunately.

It's only one channel that's affected strangely - although interestingly the frequency scan did use side bands ? for that particular channel (eg I *think* it was 1846250000 rather than 184500000 - but I am trying it on 184500000 and it's working just the same *shrug*) in any case, the windows sage server can use the HDHR tuner on the same channel, same frq file etc without any issues so this is something linux or linux-docker specific.

On windows I had to install HDHR software and there was a Sage TV BDA compatibility option - on linux I didn't need to install anything? Does that mean the drivers are built into Sage or built into linux??
Could they need updating or would there be a similar compatibilty setting I need to adjust - I don't know what that option does really on windows??

Have contacted Silicon Dust support but expect will be told Sage TV isn't supported so am really hoping for some help on here

[wnjj]

It looks like Silicondust drivers/librariesa are built in to Linux SageTV. https://github.com/google/sagetv/tre...ty/SiliconDust

It’s possible there’s some kind of bug that was fixed in the Windows drivers that only affects something on that one channel. The Linux library is a few years old.

[Madz]

Originally Posted by wnjj

It looks like Silicondust drivers/librariesa are built in to Linux SageTV. https://github.com/google/sagetv/tre...ty/SiliconDust

It’s possible there’s some kind of bug that was fixed in the Windows drivers that only affects something on that one channel. The Linux library is a few years old.

Hm interesting!! Yeah as expected SiliconDust got back to me saying SageTV has it's own code to interact with the device and it's nothing to do with them...

wnjj you wouldn't happen to have a working build environment would you? If so any chance you can do me a huge favour and substitute in the libhdhomerun source from here: https://www.silicondust.com/support/linux/ for the above for a trial build. I've compared and there are some minor changes, looks like it should still build though - if you're able to send me the generated libHDHomeRunCapture.so file I can try it on my system to see if it fixes the issue...I'm just really hoping the bug is in there and not the SageTV wrapper code as I'm in a world of trouble then!

[wnjj]

Originally Posted by Madz

Hm interesting!! Yeah as expected SiliconDust got back to me saying SageTV has it's own code to interact with the device and it's nothing to do with them...

wnjj you wouldn't happen to have a working build environment would you? If so any chance you can do me a huge favour and substitute in the libhdhomerun source from here: https://www.silicondust.com/support/linux/ for the above for a trial build. I've compared and there are some minor changes, looks like it should still build though - if you're able to send me the generated libHDHomeRunCapture.so file I can try it on my system to see if it fixes the issue...I'm just really hoping the bug is in there and not the SageTV wrapper code as I'm in a world of trouble then!

PM sent.

[Madz]

Ah yeah - I managed to build the SiliconDust library but compared to the .so supplied with sage there's a bunch of sage specific stuff in it, so must be some wapper code packaged up with it. Appreciate you taking the time anyway... unless Stuckless happens to see this, might be time for me to learn how to build Sage TV

[stuckless]

Originally Posted by Madz

Ah yeah - I managed to build the SiliconDust library but compared to the .so supplied with sage there's a bunch of sage specific stuff in it, so must be some wapper code packaged up with it. Appreciate you taking the time anyway... unless Stuckless happens to see this, might be time for me to learn how to build Sage TV

Building sagetv (Linux) is fairly easy... I used the docker container to do the builds on my Linux workstation, but, this could run on mac or windows as well.

What needs to happen, in the long term, is that someone just needs to convert to the .travis.yml file that builds and deploys to bintray to github actions and build and deploy to github releases. This is not huge task, since the yml files for actions would be fairly straight forward and someone would just need to update the deploy.sh script to figure out how to create a github release (again there are scripts for that sort of stuff existing, so, should not be too hard to figure out)

[Madz]

Love your work stuckless, the docker container is great!! I was able to build sage no worries, but unfortunately the .so binary didn't change with the new libhdhomerun code in place so I'm guessing the changes are not in functions relevant to sage, though will spend a bit more time tomorrow to confirm this

Would also like to test the libhdhomerun library with another client to ensure it can correctly record the channel just as a double-check the bug is definitely on the sage side.

Assuming it is as appears though, that's pretty bad news for me - not liking my chances of tracking down a quirky issue in the recording code.... will have a bit of a look at it as time permits but I can see myself ending up having to run a windows VM just because of this which sucks...

Appreciate your thoughts on fixing the wider build process too - will try to look at it as time permits if no-one else gets there first but this is all new ground for me and I've got to get to somewhere definitive with this recording issue asap as the new NAS is on the way and I was wanting to decommission the existing sage server this week...

[KeithAbbott]

Would OpenDCT be an option to resolve this?

[Madz]

Originally Posted by KeithAbbott

Would OpenDCT be an option to resolve this?

Hmmm - interesting. I'd seen that mentioned in places but assumed it was something cable TV specific. I need to tune a DVB-T channel. But just having a quick search this morning I came across this discussion on advantages to using OpenDCT vs the built-in support for HDHR in SageTV

https://forums.sagetv.com/forums/sho...99&postcount=6

Originally Posted by nyplayer

You can use Tuner pooling ... and share the tuners with other systems. You can have SageTV to force an unlock and take over a tuner that is in use by another system if all tuners are in use... Also I have found that the muxing of the TS files in SageTV to be unreliable on certain channels. You are using the latest FFMPEG to mux the streams instead of the outdated SageTV way of muxing.

That sounds very much like what's happening to me

I'll definitely look into this further - thanks heaps for the tip Keith

[wnjj]

Originally Posted by stuckless

Building sagetv (Linux) is fairly easy... I used the docker container to do the builds on my Linux workstation, but, this could run on mac or windows as well.

What needs to happen, in the long term, is that someone just needs to convert to the .travis.yml file that builds and deploys to bintray to github actions and build and deploy to github releases. This is not huge task, since the yml files for actions would be fairly straight forward and someone would just need to update the deploy.sh script to figure out how to create a github release (again there are scripts for that sort of stuff existing, so, should not be too hard to figure out)

I looked into this a bit already. I don't think Github actions will work since they seem to be built for auto releasing within the same repo as the source (unless Narflex opens the main repo for releases). If we want to continue on publishing to OpenSageTV, the Github REST API looks simple enough to script with curl. I'm starting to play around with it a bit right now in a debug repo but haven't gotten anything working yet.
I think the edits can be limited to the deploy.sh script.

Or course right now there's still an issue with the Travis compile that Narflex is working on.

[Madz]

Oh right, I didnt realise we wouldn't be able to put the binaries in the google/sagetv repo

[wnjj]

Originally Posted by wnjj

I looked into this a bit already. I don't think Github actions will work since they seem to be built for auto releasing within the same repo as the source (unless Narflex opens the main repo for releases). If we want to continue on publishing to OpenSageTV, the Github REST API looks simple enough to script with curl. I'm starting to play around with it a bit right now in a debug repo but haven't gotten anything working yet.
I think the edits can be limited to the deploy.sh script.

Or course right now there's still an issue with the Travis compile that Narflex is working on.

I found a Gradle plugin for publishing GitHub releases. I’ve tested it out on a temp repo and have it working to build the release and add binaries. It should be able to just replace the Bintray one in build.gradle with no other changes needed. The last piece needed is to set up a user & token for Travis to use to access OpenSageTV.