Appropriate hardware

[EnterNoEscape]

Originally Posted by Fuzzy

I don't believe the unRAID array needs to be offline to rebuild, though I haven't dug that deep into it. Although, use while rebuilding will certainly slow down the rebuilding process.

It does not need to be offline to rebuild. Array access does slow down a bit while it's rebuilding, but that's normal for RAID in general.

[stanger89]

Originally Posted by mlcarson

You seem to be agreeing but then saying that you haven't paid for the duplication cost of 40TB of data and have only purchased a 4TB parity drive. If you've done both then bravo - most people don't do that.

I have all my important and irreplaceable data backed up off site. That's not everything. All my media is already backed up by the fact that I have the originals.

I was responding to where you said "If I had 40TB of data and most of it was non-changing, I'd probably still just do backups of the static data to 10TB drives and then just have these drives stored some place offsite."

I was pointing out that that's $1600 to buy drives to duplicate that much storage. That's a lot more than the cost of a single drive, especially for stuff like ripped content that is recoverable, however having some sort of redundant storage greatly reduces the chance that you'll have to spend a lot of time recovering it.

The point of the rebuilding rant is that it can potentially bring down your raid to an unusable status for longer than the time than it would have taken to just restore a backup.

It does not, the array is up and usable for the whole rebuild time.

That was true for RAID5 -- not sure how Unraid behaves in that scenario. I suspect it has a lot to do with the CPU power available to the Unraid server. If RAID6 double parity is used, the rebuild should take even longer than with RAID5. So if you have decent backups, the question is what does the additional RAID system give you? If performance degradation during a rebuild is negligible then I get it. Back when I was doing RAID5, it was anything but.

The performance degradation is negligible.

My point is, when you start getting to the point where you have tens of TB of content, that is recoverable, thus doesn't need a "backup", parity protection is cheap protection against a drive failure and potentially weeks of effort recovering (reripping/organizing) that much content.

[Fuzzy]

Originally Posted by mlcarson

If RAID6 double parity is used, the rebuild should take even longer than with RAID5. So if you have decent backups, the question is what does the additional RAID system give you? If performance degradation during a rebuild is negligible then I get it. Back when I was doing RAID5, it was anything but.

unRAID with dual-parity (equating to your RAID6 comment) shouldn't really be any slower than single parity. The CPU that would likely be used in any modern unRAID system can calculate the parity FAR faster than the drives can access the data - and newer controllers can process data far faster than a spinning drive can provide it, so unRAID can often read from all drives simultaneously, and process parity very near the speed of the slowest drive involved.

Regarding the performance hit, I'm pretty sure unRAID puts the priority to the shares access, and does parity checks and rebuilds as drivetime allows. Shouldn't really affect operation all that much.

[panteragstk]

I've been through a catastrophic failure with unraid. Only one parity drive and had two drives die. No biggie. Found out what data was missing that I couldn't restore from the failed drives and put it back into unraid via backup.

All working data drives were just put into a "new" array and new parity data was created. Minimal data loss, minimal downtime.

Can't say the same for traditional raid.

I used to use unraid just as a NAS. Once they added all the new VM and Docker features I was able to stop using my Windows server all together (with a few exceptions) and haven't looked back.

Having said that, I used Windows successfully for many years without issues (once I went to a server OS) so if you go that route, great. We all have experience doing a lot with sage so if you run into issues we're all here to help out.

[Skirge01]

Originally Posted by Fuzzy

Also bear in mind, that the parity drive is not required in unRAID. If you don't want to spend the money on an n+1 drive, just don't install one. It will still work with all the pool, docker and VM capabilities without having a parity drive - and one can always be added later too if you just want to spread out the expense.

Good point, Fuzzy. This is also true for tRAID.

[mlcarson]

It definitely sounds like Unraid/Flexraid has advantages over traditional RAID. Love the story of multiple drive failures - that would have normally have been catastrophic. Luckily my storage needs are low enough where I can do backups duplicating the data. Any type of RAID in my case would mainly be for data that hadn't made it to backup yet in the event of a disk failure.

For now, I'm just sticking with Windows for SageTV and see if that continues to work. I thought that would be easy this time but I actually ran into more issues than when I originally did the setup.

Typing in the Schedules Direct password with numbers and a mix of upper/lower case just wasn't working. I eventually had to put it in the sage.properties file and delete the sdauth file so that the new integrated Schedules Direct routine would find it and import it. If I had to guess as to the problem, it would be that capitalization wasn't working or that typing in letters was working how numbers were but not showing it as such. Maybe it would have worked fine using a remote but something was definitely broken with keyboard input.

The larger problem was that my HDHomerun Prime wasn't being detected properly and I didn't catch it. It identified the HDHomerun Prime as being connected to my old SageTV server and used that IP address rather than the local server and then complaining that no tuners/encoders were available.

Things are working now but I'm tempted to blow it all away and redoing it from scratch again to make sure that everything is done right.

[EnterNoEscape]

Originally Posted by mlcarson

Typing in the Schedules Direct password with numbers and a mix of upper/lower case just wasn't working. I eventually had to put it in the sage.properties file and delete the sdauth file so that the new integrated Schedules Direct routine would find it and import it. If I had to guess as to the problem, it would be that capitalization wasn't working or that typing in letters was working how numbers were but not showing it as such. Maybe it would have worked fine using a remote but something was definitely broken with keyboard input.

I'm curious what you actually did there since I did not design core SD to import your username and password from Sage.properties. If one really does want to do this outside of the UI, the formatting of the sdauth file is very simple. It's basically your username, then a space, then your password converted to a SHA1 hash in lowercase hexadecimal.

[mlcarson]

I just blew away the installations of both SageTV and OpenDCT and started over knowing where the gotchas were. Everything went smoothly including the Schedules Direct password entry so I don't know what was going on with it last night.


What I did to get it import last night was to add the following entries to the Sage.properties file and blew away the sdauth file.

sdepg/sdUser= <userid>
sdepg/sdPassword= <password>

The <userid> and <password> would of course be the actual userid and password I use for Schedules direct. These entries are not in the current Sage.properties file so must be remnants from the previous plugin. The installation routine detected them and asked if I wanted to use them and worked last night.

This is what I did this go around:
Installed the Microsoft Visual C++ 2015 Redistributable (x64)
Installed the LAVFilters-0.68.1
Installed Java8 Update 111
Installed HDHomerun_win_20161019beta.exe
Installed SageTVSetup_9.0.9.441_R1.005.exe
Ran the HDHomeRun Setup program and chose SageTV for BDA Compatibility Mode. Also disabled any old units no longer installed.
Installed the OpenDCT_0.5.13_x86.msi
Ran OpenDCT as console - killed after it connected to Prime DCT for a minute.
Started OpenDCT as a service
Ran SageTVv9 program installed previously.

Defaulted through most of the prompts but removed all import directories and defined my own. Removed Video Recording Directory and defined my own.
Added video source/schedules direct data and defined channel lineup on one source and then used that lineup for the other 2 tuners of the HDHomerun prime.
Changed Video Renderer to EVR.
Modify registry to increase Java heap to 1GB RAM
HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Frey Technologies\SageTV\JVMMaxHeapSizeMB = 400h (1024).

This gets me to a point with a basic working configuration. I should now be able to point my Nvidia Shield AndroidTV miniclients at the new installation.

A couple of things of note. Uninstalling SageTV and OpenDCT doesn't blow away the installation directories or all of the contents within them. I manually deleted them to ensure a completely new installation.

[Fuzzy]

Originally Posted by mlcarson

I just blew away the installations of both SageTV and OpenDCT and started over knowing where the gotchas were. Everything went smoothly including the Schedules Direct password entry so I don't know what was going on with it last night.


What I did to get it import last night was to add the following entries to the Sage.properties file and blew away the sdauth file.

sdepg/sdUser= <userid>
sdepg/sdPassword= <password>

The <userid> and <password> would of course be the actual userid and password I use for Schedules direct. These entries are not in the current Sage.properties file so must be remnants from the previous plugin. The installation routine detected them and asked if I wanted to use them and worked last night.

This is what I did this go around:
Installed the Microsoft Visual C++ 2015 Redistributable (x64)
Installed the LAVFilters-0.68.1
Installed Java8 Update 111
Installed HDHomerun_win_20161019beta.exe
Installed SageTVSetup_9.0.9.441_R1.005.exe
Ran the HDHomeRun Setup program and chose SageTV for BDA Compatibility Mode. Also disabled any old units no longer installed.
Installed the OpenDCT_0.5.13_x86.msi
Ran OpenDCT as console - killed after it connected to Prime DCT for a minute.
Started OpenDCT as a service
Ran SageTVv9 program installed previously.

Defaulted through most of the prompts but removed all import directories and defined my own. Removed Video Recording Directory and defined my own.
Added video source/schedules direct data and defined channel lineup on one source and then used that lineup for the other 2 tuners of the HDHomerun prime.
Changed Video Renderer to EVR.
Modify registry to increase Java heap to 1GB RAM
HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Frey Technologies\SageTV\JVMMaxHeapSizeMB = 400h (1024).

This gets me to a point with a basic working configuration. I should now be able to point my Nvidia Shield AndroidTV miniclients at the new installation.

A couple of things of note. Uninstalling SageTV and OpenDCT doesn't blow away the installation directories or all of the contents within them. I manually deleted them to ensure a completely new installation.

Few points.

1. The core SDEPG support does not use those sdepg/sdUser and password properties, so setting those has no impact in any way on sage's built-in support (they were used for the plugin made by slugger).

2 You do not need to install the HDHomeRun drivers/software if you are using openDCT to talk to your HDHomeRun devices. It talks to them directly instead of via windows BDA - simplifying things considerably in my view.

3. Uninstall only removes what the installer installed, that way you can reinstall later and configurations would remain.

[mlcarson]

Well, I'm not making up what those sdepg entries did. I copied just those two entries while trying to get the connection to Schedules Direct to work. This was a new installation on a PC that I hadn't installed SageTV on so the old Schedules Direct plugin was never there. This was confirmed by the complete absence of any other sdepg entries. The routine for adding the video sources asked if I wanted to copy these entries over. I replied Yes and it magically connected without having to type the credentials. Up to this point, it had never successfully connected. It did require deletion of the sdauth file. Deleting the sdauth file up to that point just allowed me to reenter the Schedules Direct credentials.

The installation of the HDHomerun drivers/software allowed deletion/disabling of old cards that were still being seen by the Windows driver and also updated the firmware of any HDHomerun device in the system so shouldn't be overlooked. You're right in that specifying SageTV for the BDA type makes no difference but the OpenDCT software was looking at the Windows registry for discovery of the HDHomerun hardware. As soon as the old stuff was deleted, OpenDCT no longer detected it and cleaned up the SageTV installation as a result. Before this, I was seeing multiple HDHomerun Connect and HDHomerun Prime devices for video sources which didn't really exist.

[EnterNoEscape]

Originally Posted by mlcarson

Well, I'm not making up what those sdepg entries did. I copied just those two entries while trying to get the connection to Schedules Direct to work. This was a new installation on a PC that I hadn't installed SageTV on so the old Schedules Direct plugin was never there. This was confirmed by the complete absence of any other sdepg entries. The routine for adding the video sources asked if I wanted to copy these entries over. I replied Yes and it magically connected without having to type the credentials. Up to this point, it had never successfully connected. It did require deletion of the sdauth file. Deleting the sdauth file up to that point just allowed me to reenter the Schedules Direct credentials.

The installation of the HDHomerun drivers/software allowed deletion/disabling of old cards that were still being seen by the Windows driver and also updated the firmware of any HDHomerun device in the system so shouldn't be overlooked. You're right in that specifying SageTV for the BDA type makes no difference but the OpenDCT software was looking at the Windows registry for discovery of the HDHomerun hardware. As soon as the old stuff was deleted, OpenDCT no longer detected it and cleaned up the SageTV installation as a result. Before this, I was seeing multiple HDHomerun Connect and HDHomerun Prime devices for video sources which didn't really exist.

I double-checked the code involved there in case another developer improved on my last commit and I didn't catch any changes, so while I'm glad it's working for you, I couldn't say how/why.

If you saw devices available in SageTV that disappeared after cleaning up the HDHomeRun drivers, they did not come from OpenDCT. SageTV doesn't remove network encoder capture devices automatically. What it will remove is anything else it detected via other means such as BDA. OpenDCT sends a broadcast out to your network and all HDHomeRun devices reply to it. Any time you restart OpenDCT this is what happens. It stores your preferences for each device, but it only loads and presents each capture device if it was detected on your network.