SageTV Docker Freezes When Moving Large Files

[Taddeusz]

I'm not sure how to categorize this issue. I originally thought this was a some kind of problem with my hardware so I've been chasing it by continuing to spend money on upgrades. Then I thought it was an underlying problem with Unraid, it may still be, but now I'm not so sure.

Over the weekend I installed a larger NVMe SSD cache drive to replace my SATA SSD cache. This morning I was moving one of my VM's to the new drive and noticed that SageTV became near unresponsive. Specifically the web interface would not respond in a timely manner. The Unraid interface itself and a couple of my other Docker containers were quickly responsive like normal.

So to me this falls back on the SageTV container itself. I'm not entirely sure what makes this container different from all the others?

Makes me wonder if the problems I've been experiencing with SageTV freezing while my Unraid array drives are spinning up is a result of something unique with the SageTV container rather than something with Unraid itself?

I may be way off but I can't otherwise explain my experience today.

[stuckless]

typically freezing (temporarily) falls into 2 categories, from my experience.
1. cpu starvation
2. io

Moving a file won't likely hit #1, but it could hit #2. SageTV could be susceptible to IO, especially since it would be reading files from the filesystem and it couldn't get enough IO to locate and read the file then it would appear to pause. I actually see this happen from time to time on my linux system when the "tracker" daemon kicks in and starts to index my HD, and it kills my system for about 5 minutes Tracker is not running on unraid or the sagetv container.

[Taddeusz]

Originally Posted by stuckless

typically freezing (temporarily) falls into 2 categories, from my experience.
1. cpu starvation
2. io

Moving a file won't likely hit #1, but it could hit #2. SageTV could be susceptible to IO, especially since it would be reading files from the filesystem and it couldn't get enough IO to locate and read the file then it would appear to pause. I actually see this happen from time to time on my linux system when the "tracker" daemon kicks in and starts to index my HD, and it kills my system for about 5 minutes Tracker is not running on unraid or the sagetv container.

Thanks for the reply. I was actually messing around with this further earlier. Apparently Placeshifter still works fine as I can connect and navigate through my VM just fine. When I try to browse to the web UI it takes a long time to respond and sometimes times out.

At one point I tried to restart the container while the file was being moved and it froze at "*** Running /etc/my_init.d/30-setperms..." until Unraid was finished moving the file.

Anyway, neither of those should be the case. I'm now Running an i5 8400 6 core CPU. My LSI 9211 card is running on the CPU's x16 slot so save for my parity drive and now the NVMe drive most of the data doesn't cross the DMI link. With my 9 array drives it pushes up to 1.3GB/s on rebuilds and parity checks without breaking a sweat. The system is still perfectly usable. So copying a file that appears to top out at about maybe 250-300MB/s burst writing to the NVMe drive and a continuous 45-50MB/s writing back to the array, this shouldn't be the issue. In either direction this still causes problems. It is definitely not starved of CPU. I think CPU usage is about 10-20%, if that, when the mover is copying this file back and forth.

[stuckless]

I can see setperms being blocked if the IO is maxed out. It's reading every file in your media drive and sagetv install drive and and then checking and changing the permission on those files to be readable/writeable by sagetv container. Sometimes this process can take a few minutes. So, sagetv and the webserver will not be online until AFTER the setperms script runs. I think, though, there is an option in the unraid template for sagetv to tell it NOT to run setperms. It will then only run on the sagetv install dir after a new installation is downloaded.

[Taddeusz]

Originally Posted by stuckless

I can see setperms being blocked if the IO is maxed out. It's reading every file in your media drive and sagetv install drive and and then checking and changing the permission on those files to be readable/writeable by sagetv container. Sometimes this process can take a few minutes. So, sagetv and the webserver will not be online until AFTER the setperms script runs. I think, though, there is an option in the unraid template for sagetv to tell it NOT to run setperms. It will then only run on the sagetv install dir after a new installation is downloaded.

Yeah, I was looking at the Docker code earlier. I saw the check for OPT_SETPERMS.

I just don't get how such a relatively low mount of I/O would cause this considering I've seen my system push a lot more without issue.

Out of curiosity have you modified your disk tunables from the defaults?

I guess the major part of my confusion at this point is why SageTV's web server seems to be the only thing affected when a large file is being moved? None of my other web apps, including my Let's Encrypt docker which I'm using to proxy, seem to be affected.

[stuckless]

Originally Posted by Taddeusz

Yeah, I was looking at the Docker code earlier. I saw the check for OPT_SETPERMS.

I just don't get how such a relatively low mount of I/O would cause this considering I've seen my system push a lot more without issue.

Out of curiosity have you modified your disk tunables from the defaults?

I guess the major part of my confusion at this point is why SageTV's web server seems to be the only thing affected when a large file is being moved? None of my other web apps, including my Let's Encrypt docker which I'm using to proxy, seem to be affected.

i haven't done any other tuning to my drives, and I could be hitting the same issues as you, but not really noticing it. All my recordings get recorded to my cache drive, and then the mover moves then to the main drives later (not sure when that happens).

The reason the sagetv webserver might be more affected by this, vs, the let's encrypt container could simply be because of how how file intensive the sagetv webserver is. The sagetv webserver will be accessing .jsp files, images, and java jars, classes, and groovy scripts, and even requiring some on the fly compiles in some cases, which is going to require more IO. In most cases, this happens incredibly fast, but if the IO is maxed out, then it can appear to hang.

If you run windows and sometimes you notice your HD light is solid red because some process is hammering the HD, and then the rest of windows becomes unusable until that process stops. I think this can be a very similar case here.

You might want to post something in the unRAID forums, or even see if there are tuning options for the mover to see if it can be tuned to NOT consume all the IO. Or, maybe it's something unrelated, that the unRAID folks have encountered.

[Taddeusz]

Originally Posted by stuckless

i haven't done any other tuning to my drives, and I could be hitting the same issues as you, but not really noticing it. All my recordings get recorded to my cache drive, and then the mover moves then to the main drives later (not sure when that happens).

Yeah, I do the same thing. My vdisk images are permanently stored on my cache.

I've been messing around with my tunables and I've discovered something. I did have mine modified to recommended for larger arrays:

md_num_stripes: 4096
md_sync_window: 2048
md_sync_thresh: 2000

When moving the vdisk from the array to my NVMe cache it was bursting reading and writing. I've seen that before when I'm moving around large files. I have 16GB of RAM so I decided to double md_num_stripes to 8192 which uses 256MB of RAM for stripes (4096*8 drives*8192). Now when that same file is being moved to the cache drive it's reading continuously from the array. I didn't modify the sync window or threshold. When I was writing the file back to the array I was getting a maximum of 150MB/s to the target drive so I don't think it's affecting things. It's my understanding that the sync tunables mostly affect sync and parity check operations. I wish there were a better description of the disk tunables.