unRAID / SageTV / OpenDCT Problem

[KeithAbbott]

I migrated to unRAID / SageTV / OpenDCT over the weekend. So far, I have only migrated the recordings, which seemed to go fine. I was able to use SageTV all Sunday with no real issues. So to get ready to migrate the ripped DVDs, I added one 6TB hard drive and began the preclear on it. The pre-read verification step was 23% complete when I went to bed last night. This morning, the unRAID server was totally unreachable, using either the computer name or the ip address. I was forced to hard-reboot the machine. I started SageTV back up, and all seemed well initially. However, within 4 hours the same thing has happened. I do not have enough experience yet with unRAID to know how to diagnose and repair the root cause of the problem, particularly when I am unable to bring up the web console.

I do have a monitor connected to the machine, when I turned the monitor on, there are thousands of messages being displayed so fast that they are unreadable. Can anyone suggest next steps, or point me to some sort of debugging guide?

[BobPhoenix]

Have you limited the memory and cores allowed to the SageTV docker? That may not have anything to do with your problem but it is something you can try. On mine I have the following in Advanced settings for the SageTV docker:

--cpuset-cpus="4,5,6,20,21,22" --memory="6G"

Note my server has 32 cores (Dual Xeons with 8 cores and 8 hyper threads).

[MacDaddy]

I'm also using unRaid/Sage docker/OpenDCT. When I was kicking the tires in my first experience with the docker approach I used my decommissioned unRaid box (old Celeron E1400 and 4GB memory). I was trying many different dockers and plugins. At one point I experienced a similar effect to what you describe (massive rolling error messages on the local monitor). I was able to reboot and during one of the stable periods, I expanded the space available for my docker. While I'm not able to be completely sure that was the root cause of the problem, it did allow me to continue testing with stability.

If you can reboot and get a responsive console, you might also want to check the resource monitor on the Apps tab (it's the small blue circle on the right hand side that looks like a speedometer). It will show you cpu/mem for the running dockers and also calculate the space taken up in appdata.

Good luck.

[KeithAbbott]

Originally Posted by MacDaddy

I'm also using unRaid/Sage docker/OpenDCT. When I was kicking the tires in my first experience with the docker approach I used my decommissioned unRaid box (old Celeron E1400 and 4GB memory). I was trying many different dockers and plugins. At one point I experienced a similar effect to what you describe (massive rolling error messages on the local monitor). I was able to reboot and during one of the stable periods, I expanded the space available for my docker. While I'm not able to be completely sure that was the root cause of the problem, it did allow me to continue testing with stability.

If you can reboot and get a responsive console, you might also want to check the resource monitor on the Apps tab (it's the small blue circle on the right hand side that looks like a speedometer). It will show you cpu/mem for the running dockers and also calculate the space taken up in appdata.

Good luck.

Thanks for the information. Checking the resource monitor, it's showing 10%-15% cpu for OpenDCT, 3%-5% for SageTV. 3% memory for OpenDCT, 50% memory for SageTV. Seems like there is enough space, but if I wanted to increase it, how would I go about doing that?

[MacDaddy]

From memory: Go to the Settings tab. Select Docker. Stop the Docker. You should be able to increase the size of the image (keep in mind that you can only grow it - you can't shrink). Start the Docker and you should have a larger image.

Check the logs for your dockers (go to Dockers tab and click the links under the logs heading - may need to be in Advanced view for this). Also go back to the resource monitor and have it calculate space on appdata share for you. You should be able to tell with your new image size how close you are.

[MacDaddy]

There are some great tutorials on unRaid from SpaceInvaderOne on youtube.

Check out https://www.youtube.com/watch?v=ISJczs06pD8. It is not Sage specific but will give you some great context.

[KeithAbbott]

Originally Posted by MacDaddy

From memory: Go to the Settings tab. Select Docker. Stop the Docker. You should be able to increase the size of the image (keep in mind that you can only grow it - you can't shrink). Start the Docker and you should have a larger image.

Check the logs for your dockers (go to Dockers tab and click the links under the logs heading - may need to be in Advanced view for this). Also go back to the resource monitor and have it calculate space on appdata share for you. You should be able to tell with your new image size how close you are.

I bumped it up from 20GB to 30GB this morning, I'll wait and see if the problem goes away.

[KeithAbbott]

It's been almost 16 hours, and everything has been behaving. I even restarted the preclear, it finished the pre-read and is zeroing the disk now.

[KeithAbbott]

Looks like I spoke too soon. The server was unresponsive when I checked this morning. I am going to restart the server, and see if I can make it at least 24 hours before initiating another pre-clear.

In the meantime, any more tips on how to determine the root cause would be appreciated.

[EnterNoEscape]

If when you say the server is unresponsive, you mean the unRAID server itself (not just SageTV), you might fair a little better in the unRAID forums as far as troubleshooting goes.

[graywolf]

Yes. Go to the unRaid forum and find the preclear plugin thread. Think others have encountered this and think there are suggestions
Even running a preclear script from the command line

[KeithAbbott]

Originally Posted by EnterNoEscape

If when you say the server is unresponsive, you mean the unRAID server itself (not just SageTV), you might fair a little better in the unRAID forums as far as troubleshooting goes.

Yes, the same as in my original post, the unRAID server is unresponsive, at least via the web gui. In all fairness, I have not yet tried PuTTY.

I will be taking graywolf's advice and looking up preclear problems in the unRAID forum. And in the meantime, I will hold off initiating any more preclears, in the hopes that I can isolate where the problem is.

[graywolf]

I haven't done a preclear via the gui. I use my original unRaid (tower cas with 3x5 hot swap) to do my preclears since it isn't currently used for anything else.

[MacDaddy]

FWIW I've precleared maybe a half dozen drives (both original script and GUI). I've had only one instance where the script "didn't work". It turned out the disk was actually bad from the box. I don't want to introduce any more work, but is there a way to preclear another disk (even an older lower capacity throw down drive) just to gain confidence in the preclear?

Would also echo advice for help from the unRaid forums directly.

[KeithAbbott]

Originally Posted by graywolf

I haven't done a preclear via the gui. I use my original unRaid (tower cas with 3x5 hot swap) to do my preclears since it isn't currently used for anything else.

I assumed that the preclear had to be done on the unRAID server that the disk would ultimately be installed in. Evidently the preclear signature does not have anything machine-specific/thumb drive-specific in it? Could I do the preclear from a second machine using a trial version of unRAID?

Originally Posted by MacDaddy

FWIW I've precleared maybe a half dozen drives (both original script and GUI). I've had only one instance where the script "didn't work". It turned out the disk was actually bad from the box. I don't want to introduce any more work, but is there a way to preclear another disk (even an older lower capacity throw down drive) just to gain confidence in the preclear?

Would also echo advice for help from the unRaid forums directly.

This is actually the third disk that I've attempted to preclear. The first one (8TB parity drive) precleared the first time with no issue. The second one (8TB array drive) initially failed, but precleared successfully on the second attempt. The failed attempt did not cause the unRAID server to become unresponsive. The third drive (6TB array drive) has failed three times, and either caused or was a victim of unRAID becoming unresponsive.

I do have an additional 6TB drive that I can try next time, instead of re-trying the first one.

[KeithAbbott]

From googling and reading some of the unRAID forums, some are saying that preclearing is no longer a necessity like it used to be. With older versions of unRAID, adding a new/used drive into the array that had not been precleared took the array down while the new drive got zeroes written to it. Newer versions of unRAID can handle a new drive without taking the array down.

The disks that I am adding to unRAID have been used in my Windows Sage server for several years, so the likelihood of problems with them seem small. Should I just skip the preclear and add the drives (one-by-one) into the array directly?

[EnterNoEscape]

Originally Posted by KeithAbbott

From googling and reading some of the unRAID forums, some are saying that preclearing is no longer a necessity like it used to be. With older versions of unRAID, adding a new/used drive into the array that had not been precleared took the array down while the new drive got zeroes written to it. Newer versions of unRAID can handle a new drive without taking the array down.

The disks that I am adding to unRAID have been used in my Windows Sage server for several years, so the likelihood of problems with them seem small. Should I just skip the preclear and add the drives (one-by-one) into the array directly?

I still like to preclear drives because it's like giving the drive a once over and letting you know if any SMART values changed between before it started and after it ended. Sometimes you can draw pretty good conclusions about how the drive will perform or even if the drive should probably be returned before you have it loaded up with real data.

[EnterNoEscape]

Also, in your case, there's likely an underlying cause you'll want to find out now instead of later such as a controller that needs newer firmware or just doesn't play nicely with your configuration/unRAID. Maybe the hard drive is to blame. It could even just be a bad SATA cable (I've had that happen a few times to me involving larger arrays).

[Fuzzy]

Preclearing is not required, but I believe adding a noncleared drive will invalidate the parity, meaning a loss of protection from the time out added to the array and when it is done rebuilding the parity (could be a while). Preclearing sets the new drive to all zeros, so adding it to the array doesn't change the parity bits, so the parity remains valid, and keeps you protected the whole time.

[BobPhoenix]

You are not using a Marvel controller with the latest unRAID version are you? Just about any v6 unRAID server can (but not always/necessarily will) have problems with Marvel controllers. I can't use the 9230 Marvel controller on my MB because it drops drives. I don't remember your problems being related to a Marvel controller but I could believe it after my problems even though I wasn't using it with unRAID. I was passing it through to a Windows VM running on unRAID.