SageTV Server Linux with Comskip for Docker (and unRAID) Support Thread

[BobPhoenix]

Originally Posted by Narflex

I was very careful about reviewing the filesystem changes that affected reading/writing the Wiz.bin...but of course I could have missed something. If somebody wants to get me a Wiz.bin file that's loading as corrupt I'm more than happy to look into exactly why that's occurring...that's one of the worst things that can go wrong with SageTV and I always want to understand why it happened when it does so it can be fixed.

I could give you mine but even compressed it is 65MB and I don't think I can post one that large. I was planing on starting over again now that I have (or had until this) a stable Linux platform. The size could be part of the reason why it corrupts for me too (477MB uncompressed).

[Fuzzy]

Originally Posted by Narflex

I was very careful about reviewing the filesystem changes that affected reading/writing the Wiz.bin...but of course I could have missed something. If somebody wants to get me a Wiz.bin file that's loading as corrupt I'm more than happy to look into exactly why that's occurring...that's one of the worst things that can go wrong with SageTV and I always want to understand why it happened when it does so it can be fixed.

Yeah, I was already going to ping you on that, in hopes you can decipher the problem.. like I said, Wizard is still a black box to me. I'll shoot you a link to my corrupt file.

[EnterNoEscape]

Originally Posted by stuckless

There were 3 separate issues that I've found with this latest release...
1. corrupt wiz.bin
2. massive plugin failures (which I think is related to a a corrupt Sage.properties..ie plugins like sagex that can't be easily uninstalled because of dependencies was basically showing as not install)
3. I noticed this the other day... but a client that I use a lot in testing, when connecting to the server, was forced to go through the client setup, again... which leads me to thing that perhaps the client properties somehow got corrupted??

Issues #2 and #3 are particularly odd since at least from what I can tell, it's completely unaffected by the file reading and writing changes.

[stuckless]

Originally Posted by EnterNoEscape

Issues #2 and #3 are particularly odd since at least from what I can tell, it's completely unaffected by the file reading and writing changes.

Yeah, that's why I was wondering if it was something with the container shutdown/restart... But #2 has been experienced in windows as well. In all fairness I've experienced #2 several times since long before sagetv went open source.... I just never really notice it in development, and I only ever update my production system when there is something that I need, and in this case there was I also touched the plugin system in this latest build, so I could be something that I've introduced, as well.

#3 was odd as well... since I routinely blow away my clients directory in my dev environment, but I never remove them from my production server.... I have about 50+ file in my production clients directory... almost all of which are not used, but since I never clean it out, it keep growing.

[Narflex]

Thanks Fuzzy, I'll try to take a look at that today. And if anybody has logs from either of the other 2 problems, please post them so I can take a look.

[stuckless]

Originally Posted by Narflex

Thanks Fuzzy, I'll try to take a look at that today. And if anybody has logs from either of the other 2 problems, please post them so I can take a look.

I probably do still have my logs on my production system, although I suspect they are quite large since I had had to do a lot of plugin updates to get things working again.

My goal, if I can get time, is to create a similar situation, ie, phoenix/bmt install (older version) and then do the upgrade and figure out what is happening during the upgarde. As I said, I've seen this a happen a few times, even in v7, where a plugin would upgrade/install but not actually install/upgrade all files. I've seen it especially when a plugin is upgrading a jar file with the same name. It's odd that linux would even fail here, since, in linux there are no file locks, so you can overwrite/replace a file and not get a "write" error because you can't update it.

Anyways, if I can do some testing and reproduce it... I'll capture those clean logs and maybe I'll even figure out what is going on.

[Narflex]

Thanks Fuzzy for providing the file. I found the bug and just submitted a fix to GitHub. The problem was in the buffered file reader that was added, there was a case where if the read request was larger than the read buffer size that it would first take from the buffer what it could and then do a direct read from the file source after that. When it was doing that direct read it was not properly updating the file pointer.

https://github.com/google/sagetv/com...3f92587c170d24

[EnterNoEscape]

That was subtle, but very a important detail. Thanks!

What it's doing is pulling what it can from the buffer (if anything), then if the remaining data needed would not actually fit in the buffer if we were to fill it again, it reads it from disk directly. This is an effort to not need to manually tell the reader when it should optimize for bulk reads and when it should fully buffer.

[Fuzzy]

Originally Posted by Narflex

Thanks Fuzzy for providing the file. I found the bug and just submitted a fix to GitHub. The problem was in the buffered file reader that was added, there was a case where if the read request was larger than the read buffer size that it would first take from the buffer what it could and then do a direct read from the file source after that. When it was doing that direct read it was not properly updating the file pointer.

https://github.com/google/sagetv/com...3f92587c170d24

So I'm guessing it only affected Wiz.bin's with an exceptionally large record or something then? Just wondering why it would only affect some users, and not others.

[EnterNoEscape]

Originally Posted by Fuzzy

So I'm guessing it only affected Wiz.bin's with an exceptionally large record or something then? Just wondering why it would only affect some users, and not others.

Correct.

[stuckless]

In light of Jeff's fix, I've updated the SageTV build to 9.0.7.369 to include this fix.

The Docker container now supports are VERSION variable. By default it is set to 'latest' and it will check and grab the latest on every restart. If you set to a version, such as, 9.0.7.369, then it will ALWAYS use that version until you change it. Keep in mind, if you give it an invalid version, like, 9.0.7.360, then it will just bomb out since it won't find it.

[cburbs]

I just installed the docker.

It loads to the Guacamole page but that is it.

If I look at the log I see

Changing to SageTV directory /opt/sagetv/server
cat: activkey: No such file or directory
Executing pre-scripts
Starting server mode
./startsagecore: line 47: /var/run/sagetv.pid: Permission denied
Java HotSpot(TM) 64-Bit Server VM warning: -XX:ThreadPriorityPolicy requires root privilege on Linux

[stuckless]

Originally Posted by cburbs

I just installed the docker.

It loads to the Guacamole page but that is it.

If I look at the log I see

Changing to SageTV directory /opt/sagetv/server
cat: activkey: No such file or directory
Executing pre-scripts
Starting server mode
./startsagecore: line 47: /var/run/sagetv.pid: Permission denied
Java HotSpot(TM) 64-Bit Server VM warning: -XX:ThreadPriorityPolicy requires root privilege on Linux

So, if you've never used SageTV before... then after installing this, you'll need to use a client in order to access to sagetv and configure it. The Web UI link in the container will only be valid AFTER you setup sagetv and install the Web UI plugin. Configuring SageTV requires a client (Windows or the Android MiniClient, or a hardware client like the HD300).

[mibsy]

Originally Posted by mibsy

One other note that is likely to get boo's from some in the crowd. I consolidated all my storage into my Synology NAS and it worked great under Windows and my Intel NUC (which is running unRAID now). I would like to do the same thing with this unRAID server (all external media Music/Video/Photos as well as the live/recording directory would be only on the NAS). I know performance conscious folks just screamed out loud, but it works very well for me. I would like to make this set-up work the same way. Any thoughts on how to do it in an automated way? I see a few barriers:

So I found a solution to my SMB/NFS mounting problem for Sagetv. I located a plugin that does exactly what I want to connect to two shares for my Sagetv running as a docker under unRAID. The plugin is called "Unassigned Devices" and makes it easy to mount SMB/NFS shares or volumes. They mount prior to the docker starting up and the drives are visible under the "Plugins" and "Main" tab of the unRAID WebUI. There are a few minimal configuration settings under the "Settings' tab as well. I was able to map my recording directory and my "other" media on my NAS to the default directories configured under the docker. For example:

/mnt/disks/myNAS_Fileshare/directory/ to /var/media
/mnt/disks/myNAS_Fileshare2/directory/ to /var/mediaext

The Unassigned Plugin takes care of mounting on boot and unmounting prior to shutdown or restart.

Now I have to get my IR blasting working for changing channels on my DISH receivers. I have both USB UIRT and HD PVR's with IR tuners and the proper kernels on unRAID see them, but I have more work to do with lirc.

[hvymetal]

I am really getting frustrated with SageTV's wiz.bin always getting corrupted. Originally I had a windows based machine and it gotten to the point that the wiz.bin would get corrupted once a month. Fast forward 4 months ago I upgraded using unraid and the docker. Also before upgrading I used the webpage plugin to export the favorites so I could start with fresh wiz.bin.

Today the wiz.bin somehow got corrupted again on the unraid server losing my favorites . On my old windows server I would backup the wiz.bin daily with a batch file. Unfortunately I never script the unraid server to backup it up automatically. This means the only backup I had was about a week old from when I upgraded to SageTV V9.0.4.287. This lost any data changes (watched, new fav, etc) that has happened before it.

Currently I'm using version SageTV V9.0.7.369 and my wiz.bin file is about 18mb. Can someone can tell me why the wiz.bin keeps getting corrupted? Also can someone help with a cron script that can be added to the docker to allow backup wiz.bin and sage.properties daily?


I wish that the guide data and the favorites where not in the same database. I feel the constant updates of the guide data is what causes the file to get corrupted.

[stuckless]

Originally Posted by hvymetal

I am really getting frustrated with SageTV's wiz.bin always getting corrupted. Originally I had a windows based machine and it gotten to the point that the wiz.bin would get corrupted once a month. Fast forward 4 months ago I upgraded using unraid and the docker. Also before upgrading I used the webpage plugin to export the favorites so I could start with fresh wiz.bin.

Today the wiz.bin somehow got corrupted again on the unraid server losing my favorites . On my old windows server I would backup the wiz.bin daily with a batch file. Unfortunately I never script the unraid server to backup it up automatically. This means the only backup I had was about a week old from when I upgraded to SageTV V9.0.4.287. This lost any data changes (watched, new fav, etc) that has happened before it.

Currently I'm using version SageTV V9.0.7.369 and my wiz.bin file is about 18mb. Can someone can tell me why the wiz.bin keeps getting corrupted? Also can someone help with a cron script that can be added to the docker to allow backup wiz.bin and sage.properties daily?


I wish that the guide data and the favorites where not in the same database. I feel the constant updates of the guide data is what causes the file to get corrupted.

If you are running the latest version of this container, it will do a backup before every update. Check if you have a backups/ directory in the server/ directory.

[hvymetal]

Originally Posted by stuckless

If you are running the latest version of this container, it will do a backup before every update. Check if you have a backups/ directory in the server/ directory.

I found the the backup directory. Is there anyway give an option to backup daily? Also any ideas what can be corrupting the file?

Just in case if anyone else wants a daily backup Script for Unraid open shell and do the following

1. nano /etc/cron.daily/SageTV.sh
2. **PASTE BELOW
tar -cf /mnt/user/apps/sagetv/server/backups/SageTVBackup-`date +%Y%m%d-%s`.tar /mnt/user/apps/sagetv/server/Wiz.bin /mnt/user/apps/sagetv/server/Sage.properties
3. **SAVE CTRL-X
4. chmod 777 /etc/cron.daily/SageTV.sh

[alfi33]

Originally Posted by hvymetal

I found the the backup directory. Is there anyway give an option to backup daily? Also any ideas what can be corrupting the file?

Just in case if anyone else wants a daily backup Script for Unraid open shell and do the following

1. nano /etc/cron.daily/SageTV.sh
2. **PASTE BELOW
tar -cf /mnt/user/apps/sagetv/server/backups/SageTVBackup-`date +%Y%m%d-%s`.tar /mnt/user/apps/sagetv/server/Wiz.bin /mnt/user/apps/sagetv/server/Sage.properties
3. **SAVE CTRL-X
4. chmod 777 /etc/cron.daily/SageTV.sh

Another easy way to backup up SageTV server data in unRAID is to use the "Backup/Restore" feature that's part of the Community Applications plugin.

[stuckless]

Originally Posted by alfi33

Another easy way to backup up SageTV server data in unRAID is to use the "Backup/Restore" feature that's part of the Community Applications plugin.

Originally Posted by hvymetal

I found the the backup directory. Is there anyway give an option to backup daily? Also any ideas what can be corrupting the file?

I would use the backup/restore feature in the as alfi33 suggested.. it stops the container before doing a backup, which ensures that sagetv should have committed all data to Wiz.bin before doing a backup.

As for the what is causing it... I'm not sure. The 9.0.7 release there was a bug that caused Wiz.bin corruption on some Wiz.bin files. If your's is corrupting everytime, then, I'm not sure what could be causing that. BTW the bug is fixed in the 9.0.8 release.

[Narflex]

Originally Posted by hvymetal

I found the the backup directory. Is there anyway give an option to backup daily? Also any ideas what can be corrupting the file?

There was a bug introduced recently that could corrupt it; but it is fixed in the latest versions. Aside from that, there is no known reason the Wiz.bin file would get corrupted....I use one that's about 14 years old now.

If you're on the latest version, next time it happens, feel free to send it to me and I'll take a look at why.