SageTV Server Linux with Comskip for Docker (and unRAID) Support Thread

[stuckless]

Originally Posted by Taddeusz

Thought I had all the dependencies installed since it compiles and runs from bash. However, don't mind at all since for now this is strictly for testing.

Could you do some testing with this version.
https://github.com/stuckless/sagetv-...v-base/comskip

I'll updating the docker containers later this week (maybe tomorrow), and this version of comskip is a version an unRAID user (saarg) informed me about, which is used in his tv container, and I think kodi as well, since the build instructions were from the kodi site.

[Taddeusz]

Originally Posted by stuckless

Could you do some testing with this version.
https://github.com/stuckless/sagetv-...v-base/comskip

I'll updating the docker containers later this week (maybe tomorrow), and this version of comskip is a version an unRAID user (saarg) informed me about, which is used in his tv container, and I think kodi as well, since the build instructions were from the kodi site.

That one is even worse. They error out in mpeg2video with "ac-tex damaged at xx xx" where xx are different numbers. It appears this is based on a really old version.

I think the issue with comskip must have something to do with unRAID's disk subsystem. XFS works fine with Ubuntu. I also added a 100GB vdisk to my unRAID Ubuntu vm and formatted it ext4. Recording to it and doing live comskip worked fine.

[Taddeusz]

Originally Posted by stuckless

Could you do some testing with this version.
https://github.com/stuckless/sagetv-...v-base/comskip

I'll updating the docker containers later this week (maybe tomorrow), and this version of comskip is a version an unRAID user (saarg) informed me about, which is used in his tv container, and I think kodi as well, since the build instructions were from the kodi site.

Not running it live still seemed to work.

[stuckless]

There is a new set of unRAID docker images that have been uploaded. The changes are...

- Both Java 7 and Java 8 images use Oracle Java. This should fix the TimeZone issue in Java 7.
- Comskip has been updated to latest build + ffmpeg 3.1

I've also included an alternate comskip binary, comskip.xilka, which appears to be an older comskip with ffmpeg2... this appears to be what some kodi users are using. I think in the sagetv properties you can set the location to

cd/comskip_location=/opt/sagetv/comskip/comskip.xilka

IF you do use this version, you will need to update the container to get the latest files and start sagetv, then shut it down, edit the properties. AND, you will need to modify your unRAID docker configuration and set OPT_COMSKIP=N. That last part is important, since if it is Y then it will overwrite your configuration.

[tmiranda]

Originally Posted by Taddeusz

Thought I had all the dependencies installed since it compiles and runs from bash. However, don't mind at all since for now this is strictly for testing.

You'll be good to go. I'll send you the manifest in a few days. Sorry for the delay.

[Taddeusz]

Saw another thread about modifying the sagesettings file to increase the heap size. This doesn't seem to work with the docker image. If I stop SageTV, modify the file, then start it again then Docker will overwrite my changes and it gets set back to 768MB.

[Taddeusz]

In my testing I've also noticed that I get quite a few halts from my HDHR's vs my current main Windows system using the same tuners. Not sure why that would be?

[Taddeusz]

Another observation is that comskip seems to have finished the scan (e.g. all the breaks are present) but the tasks seem to be taking an inordinate amount of time to complete. Looking at TOP I see the comskip tasks as moderately active but the stats plugin I got shows no notable disk throughput. I've not seen this happen under Windows/WINE or native Linux running manually.

[stuckless]

Originally Posted by Taddeusz

Saw another thread about modifying the sagesettings file to increase the heap size. This doesn't seem to work with the docker image. If I stop SageTV, modify the file, then start it again then Docker will overwrite my changes and it gets set back to 768MB.

That doesn't seem right... I've been running 1G memory since I set this up, and I've restarted my docker container many times for testing, and my sagesettings has never been overwritten. That's the purpose of sagesettings is that it's not a file in the tgz or deb packages so it's a safe area to put configuration settings.

The only way that I could see this happening is if the /opt/sagetv/ mapping was not pointing to a valid share, and in which case the files would then get mapped internal to the docker container... but that would mean your wiz.bin would be deleted as well during a docker update, etc.

[Taddeusz]

Originally Posted by stuckless

That doesn't seem right... I've been running 1G memory since I set this up, and I've restarted my docker container many times for testing, and my sagesettings has never been overwritten. That's the purpose of sagesettings is that it's not a file in the tgz or deb packages so it's a safe area to put configuration settings.

The only way that I could see this happening is if the /opt/sagetv/ mapping was not pointing to a valid share, and in which case the files would then get mapped internal to the docker container... but that would mean your wiz.bin would be deleted as well during a docker update, etc.

Definitely pointed to a valid share.

[alfi33]

Originally Posted by Taddeusz

Saw another thread about modifying the sagesettings file to increase the heap size. This doesn't seem to work with the docker image. If I stop SageTV, modify the file, then start it again then Docker will overwrite my changes and it gets set back to 768MB.

Can't you just edit the Environmental Variable for JAVA_MEM_MB in the Docker Container settings? (or is that what you're doing and I'm just confused)

[Taddeusz]

Originally Posted by alfi33

Can't you just edit the Environmental Variable for JAVA_MEM_MB in the Docker Container settings? (or is that what you're doing and I'm just confused)

Oh, well, I swear I looked at those options. My brain didn't see it.

[alfi33]

Originally Posted by stuckless

There is a new set of unRAID docker images that have been uploaded. The changes are...

- Both Java 7 and Java 8 images use Oracle Java. This should fix the TimeZone issue in Java 7.
- Comskip has been updated to latest build + ffmpeg 3.1

I've also included an alternate comskip binary, comskip.xilka, which appears to be an older comskip with ffmpeg2... this appears to be what some kodi users are using. I think in the sagetv properties you can set the location to

cd/comskip_location=/opt/sagetv/comskip/comskip.xilka

IF you do use this version, you will need to update the container to get the latest files and start sagetv, then shut it down, edit the properties. AND, you will need to modify your unRAID docker configuration and set OPT_COMSKIP=N. That last part is important, since if it is Y then it will overwrite your configuration.

A few observations...

This latest version of Comskip in the Docker seems to be a bit better for me for live_tv. In my limited (and unscientific testing), comskip did not crash for 4 out of 5 recordings that were processed live.

In the previous version, it seemed to crash consistently in the first 5-10 minutes.

I was not able to get comskip.xilka to work at all. I tried renaming/replacing the comskip file with the comskip.xilka file...but the comskip process would not start up after doing this.

Then I tried using the method you described (editing the sage.properties file with the location of the comskip.xilka file with the Docker stopped)...but the comskip process still would not start when I started a recording. (I put the path to the comskip file back to what it was, and then the comskip process fired up when I started a recording as expected.)

I didn't spend very long on this so it could have just been me doing something wrong...


Also, I am not able to edit the sage.properties file (from my Windows workstation) file without going into MidnightCommander and adjusting the permissions.

[stuckless]

Originally Posted by Taddeusz

Saw another thread about modifying the sagesettings file to increase the heap size. This doesn't seem to work with the docker image. If I stop SageTV, modify the file, then start it again then Docker will overwrite my changes and it gets set back to 768MB.

Originally Posted by Taddeusz

Oh, well, I swear I looked at those options. My brain didn't see it.

That's ok... I wrote the template and scripts for the docker container and I totally forgot that I actually do OVERWRITE the sagesettings with the value that is set there So your original statement is correct... it does get overwritten

[Taddeusz]

I have a "bare metal" Ubuntu 16.04.1 64-bit system with the latest native comskip that I've been testing on with help from tmiranda. All my comskip live tv tests have been successful. Not sure how we can go about troubleshooting this on unRAID. I guess my next step would be to install Docker along with the SageTV docker image and see if that makes a difference. That way it rules out anything Docker specific.

Unfortunately, that won't be happening today. Apparently one of my kids powered that system off at some point so I don't currently have access to it.

[Taddeusz]

Ok, I just keep on plugging away at this. I have some positive news. My suspicions about uNRAID's disk subsystem being the problem seem to have been confirmed. I started digging into the disk settings. Particularly the tuning settings. I found some information indicating that the default md_num_stripes, md_write_limit, & md_sync_window settings are set to be optimized for systems with only 512MB of RAM.

My system has 24GB so I don't have such limited constraints. I decided to double the values to 2560, 1536, & 768 respectively and so far have my most successful test with live comskip detection. I just started the test so it might be a little early to tell but so far it's been running for almost 30 minutes without any of the comskip processes dropping out.

I may continue to try and tweak those values as out of the 4 recordings that I started one of them finished already and the UI still shows it's processing. However, the fact that I'm only using 2 drives to do these tests may be limiting read performance while it's trying do the writes.

I'll post again later but so far these results are looking very positive.

BTW, the md_write_limit setting cannot be modified through the web interface. I had to ssh in and modify the /boot/config/disk.cfg file to get at that setting. Information about them can be had here: http://lime-technology.com/forum/ind...2767#msg142767

[Taddeusz]

I've recorded another full hour with live detection and it all completed successfully. None of the comskip processes failed.

I also determined the reason for comskip taking so long to complete. In my troubleshooting I had set live_tv_retries=600. So when it got to the end of the file it was retrying 600 times before deciding to fail. Put it back to 8 and it works normally now.

I'm not sure what the optimal settings would be. In that forum post there's a mention of the risk of saturating the PCI or PCIe bus if md_sync_window is set too high.

[BobPhoenix]

Originally Posted by Taddeusz

I've recorded another full hour with live detection and it all completed successfully. None of the comskip processes failed.

I also determined the reason for comskip taking so long to complete. In my troubleshooting I had set live_tv_retries=600. So when it got to the end of the file it was retrying 600 times before deciding to fail. Put it back to 8 and it works normally now.

I'm not sure what the optimal settings would be. In that forum post there's a mention of the risk of saturating the PCI or PCIe bus if md_sync_window is set too high.

Don't know if this will make a difference to you or not. But I went with separate drives mounted outside of the array and NOT the cache drive to record to. Then I move the recordings to the array periodically to my normal imported folders that are defined in the unRAID array.

[Taddeusz]

Originally Posted by BobPhoenix

Don't know if this will make a difference to you or not. But I went with separate drives mounted outside of the array and NOT the cache drive to record to. Then I move the recordings to the array periodically to my normal imported folders that are defined in the unRAID array.

IMHO, simpler is better. I'll just stick to recording to the pool. At least I now know and can pass on the knowledge about the pool settings. Realistically, the 2x increase I made only increased the needed RAM from 5MB to 10MB. Just to give some headroom I then increased 8x from the default settings to 40MB. That should give enough headroom for other things to happen.

I understand why under limited resources you would set lower limits. However, my system is a dual quad core Xeon server with 24GB RAM. Increasing the amount of RAM needed for the pool pipeline shouldn't impact things a whole lot. It can only really increase performance.

[Fuzzy]

My newly commissioned unraid system has it recording to the pool (user/Recordings), no cache, but honestly I haven't watched it enough to actually monitor the performance. That said, I think I may set it up to individual disks instead (disk1/Recordings, disk2/Recordings, etc.) and let sage spread out the load with it's bandwidth management.