Pegged CPU for unRaid docker

[MacDaddy]

I was watching the Texans football tonight (delayed watching an active recording) when the playback stuttered, stalled and then went black. It might have been a mercy killing given the way they were playing at the moment. I brought up the unRaid dashboard and saw that out of the 8 cores, core 0 was pegged 100%. Looking at cAdvisor and scanning in to each docker, I did not see anything that was even near 100%, although the sage docker had higher utilization that I am used to seeing. When I returned to the dashboard I saw that CPU 0 was normal, but CPU 3 was pegged. I restarted the sage docker and was able to resume using although all my in-progress recordings were of course gone.

I've attached the Sage and OpenDCT logs. It looks like something started to happen around the 8/9 21:01:22 mark. I see complaints about Remuxer out of sync and later a hang detected. I would sure appreciate any insights from the experts on what might be causing this and where I can make adjustments to prevent it in the future.

I know that I had two sports recordings going on the same time with SRE involved. One program was Astros on ATTSWHD starting at 7:00. The second program was Texans on KTRKDT starting at 6:30. There may have been some other shows fire up while I was delayed watching the Texans.

Let me know if I can provide any other information. Thanks in advance for any insights you all may be able to offer.

[EnterNoEscape]

This is interesting in that it looks like SageTV is holding up OpenDCT a lot for some reason. On the surface it looks like a throughput problem. Those messages about the remuxer being out of sync means that the packets coming in from OpenDCT are not properly spaced TS packets. When you then look at the OpenDCT logs, you can see why. OpenDCT is getting backed up because SageTV can't receive the data fast enough. There are also some dropped packets that never made it to OpenDCT from the HDHomeRun which doesn't help either.

My advice would be to stop using MediaServerConsumerImpl and change over to FFmpegTransSageTVConsumerImpl. FFmpegTransSageTVConsumerImpl is the default in a normal installation of OpenDCT because it's often more consistent than using the internal remuxer.

[MacDaddy]

Thanks very much for the insights. I will change over to FFmpeg as you suggested. I'll do some forum searches on where that might be, but I'm gonna guess in Sage somewhere.

The drops are interesting. I have a dual NIC with one side attached to my backbone network (router, clients, etc.) and the other side connected directly to my capture devices. The capture side does not serve DHCP and the devices use defaulted 169.254.x.x addresses. There are five capture devices - 2xHDHR discovery (old white box, config for OTA), 2xHDHR connect (config for OTA) and 1 HDHR Prime (Xfinity cablecard, 3 streams).

I deliberately put the capture devices on their own isolated segment to control the traffic. When I look at my interfaces, I see eth0 (backbone) has

 RX packets 667278003  bytes 321603524745 (299.5 GiB)
 RX errors 0  dropped 24428  overruns 179  frame 0

The eth1 (capture devices) has

 RX packets 787784975  bytes 1075247471656 (1001.4 GiB)
 RX errors 0  dropped 817213  overruns 0  frame 0

I'm not sure what an acceptable drop rate would be, but a comparison of the two sides shows that for the amount of data received eth1 has 10x greater drop rate. Much depends on the transmission design of the capture devices to determine if this is an issue, but I think it worth exploring some more.

When I see a CPU at 100%, I usually think that either memory is overcommitted and the swap rate is blocking it, or some lock mechanism is in a race, or a massive IO operation has encountered an error and is looping.
It was interesting to me that in the unRaid dashboard I could see the 100% bounce to different cores. Also the Cadvisor did not show any of the individual docker containers getting anywhere near 100%. I wonder if there was a condition with the docker engine itself. I'm not really sure where to look for diagnostic information on that. I'll have to see if any of the unRaid folks have ideas.

Thanks again for your insights and advice. This is a rare occurrence. I've had recordings on every single capture device as a stress test when I first began using the sage docker. It passed the stress test, but I didn't run it for hours. If you can think of other logs that I should preserve should this happen again, please let me know.

[MacDaddy]

Originally Posted by MacDaddy

but I'm gonna guess in Sage somewhere

Nope. It appears to be in opendct.properties. I stopped OpenDCT, edited the opendct.properties for this line:

consumer.dynamic.channels.media_server=consumer.dynamic.channels.raw=consumer.dynamic.default=opendct.consumer.FFmpegTransSageTVConsumerImpl

Then stopped Sage, started OpenDCT and started Sage. I'll let you know how it works.

[glenner]

Originally Posted by MacDaddy

It looks like something started to happen around the 8/9 21:01:22 mark.

Does your mover run at 9pm? I've seen heavy CPU use, and out of memory issues when the mover runs. There are tweaks that seem to help. But the first step is to determine if the mover is involved. And you can check that in your scheduler settings for the mover, or in the unraid syslog.