SageTV Server Linux with Comskip for Docker (and unRAID) Support Thread

[stuckless]

Originally Posted by stanger89

Back to debugging comskip, which seems to not run at all for me, I get this in the log when I try to scan/queue unmarked recordings:

Sat 8/20 18:40:51.217 [Thread-454@6590ea08] CD: ComskipJob.executeComskipLinux: comskip failed with return code = 126
Sat 8/20 18:40:51.217 [EventRouter-305a3a536ac0@5d91fe73] CD: SystemStatus: No jobs in queue.
Sat 8/20 18:40:51.217 [Thread-454@6590ea08] java.lang.Exception
Sat 8/20 18:40:51.217 [Thread-454@6590ea08] 	at tmiranda.cd.Log.write(Log.java:64)
Sat 8/20 18:40:51.218 [Thread-454@6590ea08] 	at tmiranda.cd.ComskipJob.executeComskipLinux(ComskipJob.java:386)
Sat 8/20 18:40:51.218 [Thread-454@6590ea08] 	at tmiranda.cd.ComskipJob.executeComskip(ComskipJob.java:173)
Sat 8/20 18:40:51.218 [Thread-454@6590ea08] 	at tmiranda.cd.ComskipJob.executeUsingDefault(ComskipJob.java:187)
Sat 8/20 18:40:51.218 [Thread-454@6590ea08] 	at tmiranda.cd.ComskipJob.run(ComskipJob.java:130)
Sat 8/20 18:40:51.218 [Thread-454@6590ea08] 	at java.lang.Thread.run(Thread.java:745)
Sat 8/20 18:40:51.218 [EventRouter-305a3a536ac0@5d91fe73] processOptionsMenu optionsMenu=default:OPUS4A-186818|OptionsMenu:Unable to set plugin config value
Sat 8/20 18:40:51.218 [Thread-454@6590ea08] CD: ComskipJob.adjustJobTime: Null Times.
Sat 8/20 18:40:51.219 [Thread-454@6590ea08] CD: ComskipJob.updateRatio: Old, Current and new Ratios 0.36450315:2.7790686E-6:0.3280531
Sat 8/20 18:40:51.219 [Thread-458@59d1ee0c] CD: ComskipJob.executeComskipLinux: comskip failed with return code = 126
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] java.lang.Exception
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] 	at tmiranda.cd.Log.write(Log.java:64)
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] 	at tmiranda.cd.ComskipJob.executeComskipLinux(ComskipJob.java:386)
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] 	at tmiranda.cd.ComskipJob.executeComskip(ComskipJob.java:173)
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] 	at tmiranda.cd.ComskipJob.executeUsingDefault(ComskipJob.java:187)
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] 	at tmiranda.cd.ComskipJob.run(ComskipJob.java:130)
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] 	at java.lang.Thread.run(Thread.java:745)
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] CD: ComskipJob.adjustJobTime: Null Times.
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] CD: ComskipJob.updateRatio: Old, Current and new Ratios 0.36450276:2.2553572E-6:0.3280527

I wonder if I have a permissions issue? The default sagetv user in Commercial Detector was sagetv. I tried changing it to "nobody", but it didn't seem to make a difference.

In terms of USERs the unRAID image has a varialbe or PUID and GUID where you set the user/group ids... normally this is 99 and 100. On startup, the docker container will reset all permissions to be that user/group, and sagetv process will run as that user, etc. Internally to the docker container, 'sagetv' user name maps to that PUID (which maps to the unRAID nobody user). This is all done so that files created by SageTV container can be read/written by other unRAID apps/connections.

Comskip binaries are installed in /mnt/user/appdata/sagetv/comskip and on Startup, the container will configure comskip (if the comskip variable is set to Y). This includes setting paths, etc. In all my testing this just works, but I'm wondering if there is something going on with your media files, or if you've reconfigured comskip after the container is started.

[stanger89]

Originally Posted by tmiranda

comskip is running but returning an error (126). I really don't know what that error is. I did a quick Google and didn't find much. Your best bet is to search the comskip forums http://www.kaashoek.com/comskip/index.php

Well I also get a null pointer exception popup in SageTV when I select "Scan All Recordings Without comskip info"

"There was an error setting the plugin configuration value of"
java.lang.NullPointerException"

Originally Posted by stuckless

In terms of USERs the unRAID image has a varialbe or PUID and GUID where you set the user/group ids... normally this is 99 and 100. On startup, the docker container will reset all permissions to be that user/group, and sagetv process will run as that user, etc. Internally to the docker container, 'sagetv' user name maps to that PUID (which maps to the unRAID nobody user). This is all done so that files created by SageTV container can be read/written by other unRAID apps/connections.

Comskip binaries are installed in /mnt/user/appdata/sagetv/comskip and on Startup, the container will configure comskip (if the comskip variable is set to Y). This includes setting paths, etc. In all my testing this just works, but I'm wondering if there is something going on with your media files, or if you've reconfigured comskip after the container is started.

Other than changing the User in the CD settings yesterday, I haven't reconfigured anything.

Though I wonder if it might be related to me still running unRAID 6.1.

[Ryanagon]

New to unraid but I haven't had much trouble installing other dockers. I can't get the SageTV one to install though. If I try to add the repository it just never gives me the option to select SageTV from the template drop down and if I use the Community Applications plugin on unRaid, when hitting "Create" after it fills in the container information, nothing happens.

Am I missing something?

[stuckless]

Originally Posted by Ryanagon

New to unraid but I haven't had much trouble installing other dockers. I can't get the SageTV one to install though. If I try to add the repository it just never gives me the option to select SageTV from the template drop down and if I use the Community Applications plugin on unRaid, when hitting "Create" after it fills in the container information, nothing happens.

Am I missing something?

I'm not sure 100% sure, but, you are not the first to experience this. I built this docker image for unRAID 6.2-beta21, and I think that if you are 6.1, then "sometimes" you get the probems you describe. I'm not sure what about my unRAID template is causing the problems. You can always upgrade to the 6.2 betas.

[Ryanagon]

Originally Posted by stuckless

I'm not sure 100% sure, but, you are not the first to experience this. I built this docker image for unRAID 6.2-beta21, and I think that if you are 6.1, then "sometimes" you get the probems you describe. I'm not sure what about my unRAID template is causing the problems. You can always upgrade to the 6.2 betas.

Yep that fixed it for me. Updated to 6.2 RC4 and it's adding the container now. Thanks so much for the help.

[alfi33]

Originally Posted by Ryanagon

New to unraid but I haven't had much trouble installing other dockers. I can't get the SageTV one to install though. If I try to add the repository it just never gives me the option to select SageTV from the template drop down and if I use the Community Applications plugin on unRaid, when hitting "Create" after it fills in the container information, nothing happens.

Am I missing something?

I'm on unRAID 6.1.9 and I think I remember having a similar issue when I was trying to get this Docker installed. I think what I ended up doing is adding this to the Template Repositories
on the Dockers tab in unRAID:
https://github.com/stuckless/sagetv-...uckless-sagetv

[Taddeusz]

I believe I have determined the cause of my problems with video conversion. ffmpeg requires the libfaad2 library. On my regular Ubuntu test vm I was able to install the package and ffmpeg was then able to do a conversion. This library should be included in the Docker image. I'm unsure if there are other libraries that also need to be included. Libfaad2 is the only one that was preventing my conversions from running.

[stuckless]

Originally Posted by Taddeusz

I believe I have determined the cause of my problems with video conversion. ffmpeg requires the libfaad2 library. On my regular Ubuntu test vm I was able to install the package and ffmpeg was then able to do a conversion. This library should be included in the Docker image. I'm unsure if there are other libraries that also need to be included. Libfaad2 is the only one that was preventing my conversions from running.

I'll update the Docker image later this week or on the weekend to include libfaad2.

[stuckless]

Originally Posted by Taddeusz

I believe I have determined the cause of my problems with video conversion. ffmpeg requires the libfaad2 library. On my regular Ubuntu test vm I was able to install the package and ffmpeg was then able to do a conversion. This library should be included in the Docker image. I'm unsure if there are other libraries that also need to be included. Libfaad2 is the only one that was preventing my conversions from running.

It's interesting... I just checked the base and it has libfaad2 installed.

https://github.com/stuckless/sagetv-...Dockerfile#L21

RUN set -x \
  && apt-get update \
  && apt-get install -y wget curl libfaad2 net-tools file \
    less vim software-properties-common unzip cifs-utils

[Taddeusz]

Originally Posted by stuckless

It's interesting... I just checked the base and it has libfaad2 installed.

https://github.com/stuckless/sagetv-...Dockerfile#L21

RUN set -x \
  && apt-get update \
  && apt-get install -y wget curl libfaad2 net-tools file \
    less vim software-properties-common unzip cifs-utils

Then maybe it's failing in the Docker for a different reason.

[Taddeusz]

Originally Posted by stuckless

It's interesting... I just checked the base and it has libfaad2 installed.

https://github.com/stuckless/sagetv-...Dockerfile#L21

RUN set -x \
  && apt-get update \
  && apt-get install -y wget curl libfaad2 net-tools file \
    less vim software-properties-common unzip cifs-utils

Yeah, it's a different error. I looked in both logs. My Ubuntu install says that it couldn't find libfaad2. The Docker log says: "FFMPEG has crashed with signal 11 !!!". Not sure what that means.

[Taddeusz]

Here are the transcode portions of my log.

Wed 8/24 10:42:48.896 [XcodeStderrConsumer@46228101] FFmpeg version git-e035fbb, Copyright (c) 2000-2010 the FFmpeg developers
Wed 8/24 10:42:48.898 [XcodeStderrConsumer@46228101] built on Jun 23 2016 16:11:18 with gcc 4.8.4
Wed 8/24 10:42:48.898 [XcodeStderrConsumer@46228101] configuration: --disable-ffserver --disable-ffplay --enable-gpl --enable-pthreads --enable-nonfree --enable-libfaac --enable-libx264 --enable-libxvid --disable-devices --disable-bzlib --disable-demuxer=msnwc_tcp --enable-libfaad --extra-cflags='-I. -I/build/sagetv/third_party/codecs/faad2/include -I/build/sagetv/third_party/codecs/faac/include -I/build/sagetv/third_party/codecs/x264 -I/build/sagetv/third_party/codecs/xvidcore/src' --extra-ldflags='-L/build/sagetv/third_party/codecs/faac/libfaac/.libs -L/build/sagetv/third_party/codecs/faad2/libfaad/.libs -L/build/sagetv/third_party/codecs/x264 -L/build/sagetv/third_party/codecs/xvidcore/build/generic/=build'
Wed 8/24 10:42:48.899 [XcodeStderrConsumer@46228101] libavutil     50.22. 0 / 50.22. 0
Wed 8/24 10:42:48.899 [XcodeStderrConsumer@46228101] libavcodec    52.83. 0 / 52.83. 0
Wed 8/24 10:42:48.899 [XcodeStderrConsumer@46228101] libavformat   52.73. 0 / 52.73. 0
Wed 8/24 10:42:48.899 [XcodeStderrConsumer@46228101] libavdevice   52. 2. 0 / 52. 2. 0
Wed 8/24 10:42:48.901 [XcodeStderrConsumer@46228101] libavfilter    1.22. 0 /  1.22. 0
Wed 8/24 10:42:48.901 [XcodeStderrConsumer@46228101] libswscale     0.11. 0 /  0.11. 0
Wed 8/24 10:42:48.907 [XcodeStderrConsumer@46228101] [NULL @ 0x13114c0] Probed with size=2048 and score=52
Wed 8/24 10:42:49.050 [XcodeStderrConsumer@46228101] [mpeg @ 0x13114c0] Probe buffer size limit 5000000 reached
Wed 8/24 10:42:49.055 [XcodeStderrConsumer@46228101] Input #0, mpeg, from '/mnt/disks/192.168.22.100_video2/2020-22882623-0.mpg':
Wed 8/24 10:42:49.055 [XcodeStderrConsumer@46228101] Duration: 00:58:56.89, start: 1.352122, bitrate: 15372 kb/s
Wed 8/24 10:42:49.055 [XcodeStderrConsumer@46228101] Stream #0.0[0x80], 81, 1/90000: Audio: ac3, 48000 Hz, 5.1, s16, 384 kb/s
Wed 8/24 10:42:49.055 [XcodeStderrConsumer@46228101] Stream #0.1[0x81], 80, 1/90000: Audio: ac3, 48000 Hz, stereo, s16, 192 kb/s
Wed 8/24 10:42:49.056 [XcodeStderrConsumer@46228101] Stream #0.2[0x1e0], 77, 1/90000: Video: mpeg2video, yuv420p, 1920x1080, 1001/60000, AR: 16:9, interlaced, 24000 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
Wed 8/24 10:42:49.086 [XcodeStderrConsumer@46228101] [libx264 @ 0x1313fe0] using mv_range_thread = 72
Wed 8/24 10:42:49.087 [XcodeStderrConsumer@46228101] [libx264 @ 0x1313fe0] using SAR=1/1
Wed 8/24 10:42:49.087 [XcodeStderrConsumer@46228101] FFMPEG has crashed with signal 11 !!!

[stuckless]

Originally Posted by Taddeusz

Here are the transcode portions of my log.

I wonder if it's a network filesystem issue when transcoding. I notice your file appears to be on a network share.

[Taddeusz]

Originally Posted by stuckless

I wonder if it's a network filesystem issue when transcoding. I notice your file appears to be on a network share.

Yes, it is. I hadn't thought about that. I'll try copying something to the local filesystem and attempt to convert it again.

[Taddeusz]

Originally Posted by stuckless

I wonder if it's a network filesystem issue when transcoding. I notice your file appears to be on a network share.

I moved the file to the local file system and got the same error.

[panteragstk]

I looked around and it doesn't seem that this has been answered, but how do I get HTTP tuning for my directv boxes to work with the docker?

I found this thread but am unsure if it will work using the docker.

I don't see a "common" folder in the sage directory on my server to place my existing files in so I'm assuming that is a windows only folder.

Any ideas?

[Fuzzy]

Originally Posted by panteragstk

I looked around and it doesn't seem that this has been answered, but how do I get HTTP tuning for my directv boxes to work with the docker?

I found this thread but am unsure if it will work using the docker.

I don't see a "common" folder in the sage directory on my server to place my existing files in so I'm assuming that is a windows only folder.

Any ideas?

I have never used http tuning with sage - but I believe this would be via calls to whatever tuning utility linux has from the gentuner plugin, which is included with the docker: https://github.com/jwittkoski/GenericTunerPlugin

[panteragstk]

Originally Posted by Fuzzy

I have never used http tuning with sage - but I believe this would be via calls to whatever tuning utility linux has from the gentuner plugin, which is included with the docker: https://github.com/jwittkoski/GenericTunerPlugin

Thanks. I guess I missed that.

[alfi33]

Originally Posted by Fuzzy

My newly commissioned unraid system has it recording to the pool (user/Recordings), no cache, but honestly I haven't watched it enough to actually monitor the performance. That said, I think I may set it up to individual disks instead (disk1/Recordings, disk2/Recordings, etc.) and let sage spread out the load with it's bandwidth management.

Originally Posted by EnterNoEscape

The only downside to what you're planning is that you're still limited roughly to the performance of one disk since even when you write to two different disks, the parity is still being written to the designated 1-2 other parity disks simultaneously and it would only get worse if the files are on distant parts of the data disks translating into a lot of seeking on the parity drive(s). I can however see that possibly helping with read performance. All of that being said, the recordings are usually a very light task, so maybe it doesn't matter.

I wonder if it does make good sense to do what Fuzzy suggested. Although it probably wouldn't help writes at all, it could help with reads (like you said). When you start doing things like Comskip, which can be a very read intensive and a potentially disk thrashing activity, maybe it would be beneficial to let Sage try to spread out recordings over a few disks.

But I don't think I have yet seen any issues on my small (6 disk) unRAID box, so maybe you're right and it really doesn't matter. I'm just trying to figure out how I want to lay it all out. I guess I'll just need to do some experimenting...

[Taddeusz]

Woohoo! After a week of waiting the place from which I ordered my parity drive finally decided to ship it. Should be waiting for me when I get home. I'll install it tonight and start progressively moving my recordings over drive by drive. Should be finished in 3 days. Or one if I decide to take a day off to do it. But I think I might just stick with the progressive approach.