SageTV Server Linux with Comskip for Docker (and unRAID) Support Thread

[Taddeusz]

Originally Posted by Fuzzy

My newly commissioned unraid system has it recording to the pool (user/Recordings), no cache, but honestly I haven't watched it enough to actually monitor the performance. That said, I think I may set it up to individual disks instead (disk1/Recordings, disk2/Recordings, etc.) and let sage spread out the load with it's bandwidth management.

Last I checked that setting is only available in Windows. I could not find it in my tests of the Linux version.

[EnterNoEscape]

Originally Posted by Fuzzy

My newly commissioned unraid system has it recording to the pool (user/Recordings), no cache, but honestly I haven't watched it enough to actually monitor the performance. That said, I think I may set it up to individual disks instead (disk1/Recordings, disk2/Recordings, etc.) and let sage spread out the load with it's bandwidth management.

The only downside to what you're planning is that you're still limited roughly to the performance of one disk since even when you write to two different disks, the parity is still being written to the designated 1-2 other parity disks simultaneously and it would only get worse if the files are on distant parts of the data disks translating into a lot of seeking on the parity drive(s). I can however see that possibly helping with read performance. All of that being said, the recordings are usually a very light task, so maybe it doesn't matter.

[stuckless]

Originally Posted by Fuzzy

My newly commissioned unraid system has it recording to the pool (user/Recordings), no cache, but honestly I haven't watched it enough to actually monitor the performance. That said, I think I may set it up to individual disks instead (disk1/Recordings, disk2/Recordings, etc.) and let sage spread out the load with it's bandwidth management.

I wonder if a SSD cache drive would be simpler (and better). I realize that SSD has a finite lifespan, and I fully suspect that in about 3-4 years I'll be replacing the SSD drive, but performance wise I would think it would work well for the recordings. The drive is transparent to the pool and the unRAID "mover" physically moves the files to the parity enabled drives, later. A 120G SSD drive is pretty cheap these days. (I also keep my docker.img and virtual machines on the SSD -- cache only shares). There is a chance that I could lose data between the time it is recorded on the SDD and the time that it is copied to the physical pool, but I'm ok with that.

[stanger89]

Originally Posted by stuckless

I wonder if a SSD cache drive would be simpler (and better). I realize that SSD has a finite lifespan, and I fully suspect that in about 3-4 years I'll be replacing the SSD drive, but performance wise I would think it would work well for the recordings.

I wouldn't worry about it, I monitored my recording activities for a while and determined I run about 90GB/day recording.

With good drives running into the petabytes, I don't really see it as an issue.

With the stripe setting information, maybe it's really time to finally "commission" my unRAID Sage docker....

[stuckless]

Originally Posted by stanger89

I wouldn't worry about it, I monitored my recording activities for a while and determined I run about 90GB/day recording.

With good drives running into the petabytes, I don't really see it as an issue.

With the stripe setting information, maybe it's really time to finally "commission" my unRAID Sage docker....

You are probably right... I had a couple of SSD lying around, so I decided to pop them into the server... I ran the server for a couple weeks without the SSD and I can't say that I noticed any diference, performance wise.

[Fuzzy]

Originally Posted by EnterNoEscape

The only downside to what you're planning is that you're still limited roughly to the performance of one disk since even when you write to two different disks, the parity is still being written to the designated 1-2 other parity disks simultaneously and it would only get worse if the files are on distant parts of the data disks translating into a lot of seeking on the parity drive(s). I can however see that possibly helping with read performance. All of that being said, the recordings are usually a very light task, so maybe it doesn't matter.

I do have my fastest drive as my parity drive - but I was under the impression, however, that UnRaid handled the parity in a lazy fashion, so it can still write at full speed to the disk(s), and the parity would be updated as it was able to - not immediate protection, but likely quick enough.

I do have an SSD as my cache currently, but I have not enabled caching for the recording share - time will tell whether I think it will be required.

[hugenbdd]

Originally Posted by Taddeusz

Ok, I just keep on plugging away at this. I have some positive news. My suspicions about uNRAID's disk subsystem being the problem seem to have been confirmed. I started digging into the disk settings. Particularly the tuning settings. I found some information indicating that the default md_num_stripes, md_write_limit, & md_sync_window settings are set to be optimized for systems with only 512MB of RAM.

My system has 24GB so I don't have such limited constraints. I decided to double the values to 2560, 1536, & 768 respectively and so far have my most successful test with live comskip detection. I just started the test so it might be a little early to tell but so far it's been running for almost 30 minutes without any of the comskip processes dropping out.

I may continue to try and tweak those values as out of the 4 recordings that I started one of them finished already and the UI still shows it's processing. However, the fact that I'm only using 2 drives to do these tests may be limiting read performance while it's trying do the writes.

I'll post again later but so far these results are looking very positive.

BTW, the md_write_limit setting cannot be modified through the web interface. I had to ssh in and modify the /boot/config/disk.cfg file to get at that setting. Information about them can be had here: http://lime-technology.com/forum/ind...2767#msg142767

I have made the changes you mentioned above but I still can't get Live TV detection working. Any chance you can post your comskip.ini file?

Thanks
Dave

[Taddeusz]

Originally Posted by hugenbdd

I have made the changes you mentioned above but I still can't get Live TV detection working. Any chance you can post your comskip.ini file?

Thanks
Dave

Ok, here is my comskip.ini file. The differences you might look for are live retries, standoff, and threads.

Also make sure when you modify those pool settings that you are stopping and starting the array. They don't have any effect otherwise.

[hugenbdd]

Originally Posted by Taddeusz

Ok, here is my comskip.ini file. The differences you might look for are live retries, standoff, and threads.

Also make sure when you modify those pool settings that you are stopping and starting the array. They don't have any effect otherwise.

Thanks!!!

That seems to have solved it. I did stop the array, update the disk.cfg file to the same values you used, and restarted the array. I'm to late to try it live on the Olympics (Long Live Recording) but the short testing I have done seems to have it working. Should know more tomorrow evening. And will update with my results.

[BobPhoenix]

Originally Posted by Fuzzy

I do have my fastest drive as my parity drive - but I was under the impression, however, that UnRaid handled the parity in a lazy fashion, so it can still write at full speed to the disk(s), and the parity would be updated as it was able to - not immediate protection, but likely quick enough.

I do have an SSD as my cache currently, but I have not enabled caching for the recording share - time will tell whether I think it will be required.

Not really. unRAID is real time parity that is why the writes are so much slower to the array than the reads. It take 4 I/Os to write to the array. Read parity read array drive compute changes write data and write parity (note I may have the order slightly scrambled but you get the idea). This allows only 2 drives to be spinning during a write to the array. That is where the cache drive comes in. It was the first way to speed up writes to the array. It is not part of the array but is included in the shares (if setup that way). Then unRAID will write at drive speed to the cache drive and at 3:40am (default configuration) will move the files from the cache drive to the array at the slower speed. That is where you could say it lazily does the parity except all the data on the array is fully protected. It is only the new data on the cache drive that is unprotected. With a "cache pool" you can setup raid 1 style parity protection for the cache drive. Another more recent change will skip two of the I/Os and speed up the writes - not as fast as writing to the cache drive but much faster than the default speeds to the array. It basically causes all drives to spin when a write to the array occurs. All the data drives except the one where the new data is to be written are read. Then parity is calculated with the new data and all the other drives and the new data and the parity drive are written. I'm sure the unRAID Wiki will explain this a little better.

[EnterNoEscape]

Originally Posted by Fuzzy

I do have my fastest drive as my parity drive - but I was under the impression, however, that UnRaid handled the parity in a lazy fashion, so it can still write at full speed to the disk(s), and the parity would be updated as it was able to - not immediate protection, but likely quick enough.

I do have an SSD as my cache currently, but I have not enabled caching for the recording share - time will tell whether I think it will be required.

If by lazy, you mean it will try to write a full stripe (or at least read cache the stripe so it doesn't need to do a read, write) and Linux will do it's own write caching, you're probably correct. My understanding is that the type of raid in use in Unraid is a modified version of mdadm. If that's completely true, I can say that the parity is being written at the exact same time as the data.

[stanger89]

So, I've been actually using my docker a bit. Mostly so far so good, need to figure out comskip....

But what I ran into last night is more of a showstopper. None of my MKVs appear to have their audio/subtitles detected. Sage just plays the first audio track and no subs, and provides no listing of available tracks on the info page, and no menu to change them. This is all via my HD300. My Windows server detects the tracks just fine.

[Taddeusz]

I'm taking baby steps towards switching completely to unRAID. Now running SageTV server from unRAID but currently using the Windows shares for the drives through the Unassigned Devices plug-in.

Had a small hiccup in ordering a parity drive. Thought I had ordered a 3TB drive when I really ordered 2TB. Waiting for the pending authorization to expire on the cancelled order before ordering again. Hopefully I'll be able to order another soon.

Once I get the parity drive I'll move the recordings over. I have enough free space on the current drives to do that. I think I'll have to order another drive to start moving all the other media over. Before I do that I'll probably have to upgrade the license to Plus since I have more than 6 drives. That will also let me define a cache drive. I think I'll use the 120GB SSD currently being used as the Windows OS drive for that.

I'm really excited about this transition. I've had this server hardware essentially doing nothing that I had originally intended to use to virtualize SageTV. But I discovered that it wasn't quite practical to do with the vmware ESXi I had used. unRAID and the open sourcing of SageTV have not only made it practical but a reality.

[stuckless]

Originally Posted by stanger89

So, I've been actually using my docker a bit. Mostly so far so good, need to figure out comskip....

But what I ran into last night is more of a showstopper. None of my MKVs appear to have their audio/subtitles detected. Sage just plays the first audio track and no subs, and provides no listing of available tracks on the info page, and no menu to change them. This is all via my HD300. My Windows server detects the tracks just fine.

Do you suspect the issue is related to the unRAID + SageTV?

[stanger89]

Originally Posted by stuckless

Do you suspect the issue is related to the unRAID + SageTV?

I'm thinking it's a Linux vs Windows SageTV issue. The files play fine, but it seems when Sage scanned them it didn't detect any of the tracks.

[EnterNoEscape]

Originally Posted by stanger89

I'm thinking it's a Linux vs Windows SageTV issue. The files play fine, but it seems when Sage scanned them it didn't detect any of the tracks.

I worked a lot around the external subtitle parts of the SageTV core. External subtitles are usually detected and loaded when you start playback. If your subtitles are external to the .mkv files, it's simply a matter of for whatever reason SageTV literally doesn't see the appropriately named subtitle files. You can enable further debugging specifically for subtitles in Sage.properties. Add the property debug_subtitles=true

[stanger89]

Originally Posted by EnterNoEscape

I worked a lot around the external subtitle parts of the SageTV core. External subtitles are usually detected and loaded when you start playback. If your subtitles are external to the .mkv files, it's simply a matter of for whatever reason SageTV literally doesn't see the appropriately named subtitle files. You can enable further debugging specifically for subtitles in Sage.properties. Add the property debug_subtitles=true

These aren't external subtitles, they're in the MKV file. The bigger problem is that Sage doesn't detect the multiple audio tracks, and so if the first audio track in the file isn't the right one, there's no way to switch to it, because Sage doesn't know there's more than one.

[Taddeusz]

I'm having trouble converting recordings. I added the transcoder debugging properties. In the log I see:

Thu 8/18 11:00:32.320 [XcodeStderrConsumer@499d1c4b] FFMPEG has crashed with signal 11 !!!

[stanger89]

Back to debugging comskip, which seems to not run at all for me, I get this in the log when I try to scan/queue unmarked recordings:

Sat 8/20 18:40:51.217 [Thread-454@6590ea08] CD: ComskipJob.executeComskipLinux: comskip failed with return code = 126
Sat 8/20 18:40:51.217 [EventRouter-305a3a536ac0@5d91fe73] CD: SystemStatus: No jobs in queue.
Sat 8/20 18:40:51.217 [Thread-454@6590ea08] java.lang.Exception
Sat 8/20 18:40:51.217 [Thread-454@6590ea08] 	at tmiranda.cd.Log.write(Log.java:64)
Sat 8/20 18:40:51.218 [Thread-454@6590ea08] 	at tmiranda.cd.ComskipJob.executeComskipLinux(ComskipJob.java:386)
Sat 8/20 18:40:51.218 [Thread-454@6590ea08] 	at tmiranda.cd.ComskipJob.executeComskip(ComskipJob.java:173)
Sat 8/20 18:40:51.218 [Thread-454@6590ea08] 	at tmiranda.cd.ComskipJob.executeUsingDefault(ComskipJob.java:187)
Sat 8/20 18:40:51.218 [Thread-454@6590ea08] 	at tmiranda.cd.ComskipJob.run(ComskipJob.java:130)
Sat 8/20 18:40:51.218 [Thread-454@6590ea08] 	at java.lang.Thread.run(Thread.java:745)
Sat 8/20 18:40:51.218 [EventRouter-305a3a536ac0@5d91fe73] processOptionsMenu optionsMenu=default:OPUS4A-186818|OptionsMenu:Unable to set plugin config value
Sat 8/20 18:40:51.218 [Thread-454@6590ea08] CD: ComskipJob.adjustJobTime: Null Times.
Sat 8/20 18:40:51.219 [Thread-454@6590ea08] CD: ComskipJob.updateRatio: Old, Current and new Ratios 0.36450315:2.7790686E-6:0.3280531
Sat 8/20 18:40:51.219 [Thread-458@59d1ee0c] CD: ComskipJob.executeComskipLinux: comskip failed with return code = 126
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] java.lang.Exception
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] 	at tmiranda.cd.Log.write(Log.java:64)
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] 	at tmiranda.cd.ComskipJob.executeComskipLinux(ComskipJob.java:386)
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] 	at tmiranda.cd.ComskipJob.executeComskip(ComskipJob.java:173)
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] 	at tmiranda.cd.ComskipJob.executeUsingDefault(ComskipJob.java:187)
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] 	at tmiranda.cd.ComskipJob.run(ComskipJob.java:130)
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] 	at java.lang.Thread.run(Thread.java:745)
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] CD: ComskipJob.adjustJobTime: Null Times.
Sat 8/20 18:40:51.220 [Thread-458@59d1ee0c] CD: ComskipJob.updateRatio: Old, Current and new Ratios 0.36450276:2.2553572E-6:0.3280527

I wonder if I have a permissions issue? The default sagetv user in Commercial Detector was sagetv. I tried changing it to "nobody", but it didn't seem to make a difference.

[tmiranda]

comskip is running but returning an error (126). I really don't know what that error is. I did a quick Google and didn't find much. Your best bet is to search the comskip forums http://www.kaashoek.com/comskip/index.php