Help setting up new unRAID SageTV9 server

[glenner]

Originally Posted by graywolf

Stop sagetv docker
Edit sagetv docker
(Going from memory) should have near bottom Options for additional paths
Should be able to create a new path to /mnt/disks
Start sagetv docker
Should now be able to get to it

Thanks yes... this basically works. My docker volume mappings are like this now.

/unraid --> /mnt/user
/opt/sagetv --> /mnt/user/appdata/sagetv/
/var/media --> /mnt/user/sagemedia/
/var/mediaext --> /mnt/disks/

These are defaults... except the last one. I changed /var/mediaext to point to /mnt/disks. Here it can find any externally mounted windows SMB volumes I may have mounted.

[glenner]

Originally Posted by Fuzzy

FYI, the /unraid volume mapping to /mnt/user is always called unraid. It was put into the docker template for just this reason, the ability for the SageTV docker to find any of your local shares via direct access instead of needing to go through samba to get there.

Thanks... good to know.

Originally Posted by Fuzzy

Regarding the earlier question about why sagemedia was created with cache=no, realize that SageTV is not creating a share. The docker template maps it's internal /var/media to the host's /mnt/user/sagemedia which in doing so, it creates the /mnt/user/sagemedia folder. Creating a folder in /mnt/user is what creates the sharepoint, and it does so using unRAIDs default settings (configured in Global Share Settings).

I see... another useful concise explanation offered up using the minimal number of words. Thanks. :-) In any event, it sounds like enabling cache=yes on /mnt/user/sagemedia, or any newly created share created by any means, needs to be a manual step for those who are running unraid with a cache pool. The default seems to cache=no in all cases.

I don't want my recordings initially going through the parity disk as that makes little sense and does sound like a i/o performance bottleneck.

On my existing windows setup, I recall having i/o problems until I reformatted my drives with an allocation unit of 64KB. That solves most i/o issues on windows. Writing live recordings to the cache pool sounds smart to me. I'm not clear how xfs works exactly and whether is has the allocation unit issues that NTFS has. Need to look into that...

[glenner]

So an update on my progress.... I've created these shares so far in unRAID. I'm guessing this is most of what I will be needing.

appdata	- cache=prefer, application data	
backups - cache=no, backups (folders, system images, vms, norton ghost, macrium, etc.)	
domains - cache=prefer, saved VM instances	
downloads - cache=no, download folder	
isos - cache=yes, ISO images
movies - cache=no, personal movie collection	
music - cache=no, personal music collection	
pictures - cache=no, home pictures, iphone photos, camera photos, etc.	
sagemedia - cache=yes, sagetv recordings, and default media import directories	
system - cache=prefer, system data
temp - cache=no, temporary storage	
tvshows - cache=no, personal tv show archive (downloaded)
videos - cache=no, home videos, iphone movies, etc.

domains, isos, appdata, and system were created out of the box for me. sagemedia is created via the docker install (and manually changed cache=yes). The rest I created myself.

I've since been taking my NTFS drives out of my windows setup one by one and attached them internally to the unRAID box. The unRAID box seems to support hot swap, so I can just plug the power and SATA cables into the drive while hot. Then unRAID will see the drive in the "Unassigned Drive" plugin. I can then mount the NTFS volume under Unassigned Drives. I then started learning how to use Midnight Commander ("mc" from ssh window) to copy the contents of each drive to my various unRAID shares. I populated movies, pictures, tvshows, videos, music, sagemedia/tv folders in this way copying everything over. Copying the media files over this way via SATA is definitely better than trying to transfer everything over the network: a sure path to early insanity. It definitely helps to have a couple of large blank new data disks in the unRAID box when you are trying to migrate a large amount of media from another system. It also helps to set cache=no on all these shares initially, and to not have a parity drive setup initially. I just want my media to be migrated over quick and easy... and so I don't need it going through the cache (it's actually slower this way if you have a raid1 cache pool, and then need to call the mover), or taking the parity hit.

I noticed when I copied files to a share and the cache was enabled, I was getting about 60MB/s write throughput at the cache. But if I write to the data disc directly I was getting 130MB+/s. I'm not sure why the cache is slower than the data disk since that seems to defeat the purpose. But I guess the cache pool in raid1 setup is actually slower than just writing to a data disk. I'll see what data disk write speeds I get when I enable the parity disk later.

I may yet change my cache pool setup to be raid0. This would allow me to double the size of my cache from 250GB to 500GB. And presumably would double my write speeds to the cache as both SSDs are used to stripe the write. The spaceinvader dude describes how to set that up here: https://www.youtube.com/watch?v=ij8AOEF1pTU (see around the 17 minute mark). Raid0 also kind of make sense to me here. I'm not clear what the failure rate is on SSDs, but I thought it was extremely low, as there are no moving parts. And presumably maybe there is a reliable way to backup the appdata and system folders on the cache so you can recover from an SSD failure if you needed to. I wonder if anyone has ever lost a cache pool SSD in an unRAID setup?

[glenner]

One thing I need to resolve... is when I root ssh to unRAID, and copy/move files around, or use midnight commander, all my files are getting root.root ownership. I only have the root user on this system right now, and realize I probably need to fix that and not use root for everything.

It seems like the default ownership is nobody.users. ie. If I copy a file from windows explorer over to an unRAID share, the file is owned by nobody.users.

So far, I've been using chown to correct ownerships to nobody.users when doing stuff as root via ssh.

What is the recommended user setup on unRAID? Can I create a nobody.users user and use that for ssh? What user should I use/create to ssh into the box, and manipulate files?

[graywolf]

For the permission issue. On unRaids tool tab you should use the docker safe permission tool

Will change ownership and permission properly
Be sure to use the docker safe one

[jusjoken]

Originally Posted by graywolf

For the permission issue. On unRaids tool tab you should use the docker safe permission tool

Will change ownership and permission properly
Be sure to use the docker safe one

I do not see this on the tools tab. Was it installed by a plugin?

k

[stuckless]

When sagetv starts up, it takes ownership of all media files and all sagetv intallation files that are configured via the "apps", "media" and "mediaext" path mappings. So any files that are placed there are already reset on every sagetv restart. If you have other files in other locations, then sagetv won't touch those, even if they are configured inside of sagetv.

[graywolf]

Originally Posted by jusjoken

I do not see this on the tools tab. Was it installed by a plugin?

k

Not sure if it was a plugin.
Tools tab => UnRaid OS section => Docker Safe New Perms

On my screen it is just to the right of Diagnostics and left of Hardware Profile

[HuMan321]

Originally Posted by Fuzzy

If you are trying to get sage to map to a samba share that is NOT on you unraid server, you should have sage talk to that share directly via samba. Do so by browsing to it in sage. Go up in folders until you find Network, and then browse down to your server and share from there.

Originally Posted by Fuzzy

The SageTVService has to have access to the network shares you are trying to access. This means it has to be logged in as a User with those permissions (as opposed to the default user of 'LocalSystem'. LocalSystem doens't have access to network shares. Also, browsing for shares is somewhat inconsistant from a service, so you'll be better off using the Specify... button and typing ni the UNC path of the share (\\Server\Share).

Hi, I think the LocalSystem is my issue on my unRaid SageTV setup. I cannot figure out how to log in as a user on unRaid - probably need an "idiots" guide...

Cannot browse to network folders - says invalid directory
Thanks,

[Fuzzy]

Originally Posted by glenner

I've since been taking my NTFS drives out of my windows setup one by one and attached them internally to the unRAID box. The unRAID box seems to support hot swap, so I can just plug the power and SATA cables into the drive while hot. Then unRAID will see the drive in the "Unassigned Drive" plugin. I can then mount the NTFS volume under Unassigned Drives. I then started learning how to use Midnight Commander ("mc" from ssh window) to copy the contents of each drive to my various unRAID shares. I populated movies, pictures, tvshows, videos, music, sagemedia/tv folders in this way copying everything over. Copying the media files over this way via SATA is definitely better than trying to transfer everything over the network: a sure path to early insanity. It definitely helps to have a couple of large blank new data disks in the unRAID box when you are trying to migrate a large amount of media from another system. It also helps to set cache=no on all these shares initially, and to not have a parity drive setup initially. I just want my media to be migrated over quick and easy... and so I don't need it going through the cache (it's actually slower this way if you have a raid1 cache pool, and then need to call the mover), or taking the parity hit.

The 'parity hit' can be minimized by turning on 'Turbo Write' in unRAID. This is done on the Disk Settings page, by setting Tunable (md_write_method) to "reconstruct write", instead of the default "auto" which is synonymous with "read/modify/write" in current versions. Reconstruct write will require all pool drives to spin up to write to the array, but instead of doing the slower read data and parity, calculate new parity, then write data and parity, it writes to the data drive, while reading from all the other data drives, and writing to parity at essentially the same time. MUCH faster, but as i said, the trade-off is all drives need to be spun up. I only turn this on when doing the bulk copying like you are doing on setup. I believe there is a plugin to auto control this setting, based on how many drives are already spun up, but I haven't messed with it.

Originally Posted by glenner

I noticed when I copied files to a share and the cache was enabled, I was getting about 60MB/s write throughput at the cache. But if I write to the data disc directly I was getting 130MB+/s. I'm not sure why the cache is slower than the data disk since that seems to defeat the purpose. But I guess the cache pool in raid1 setup is actually slower than just writing to a data disk. I'll see what data disk write speeds I get when I enable the parity disk later.

I want to say I read that there were some performance related issues with btrfs cache pools that are fixed in the current beta, but honestly can't find that comment now.

[glenner]

Originally Posted by graywolf

Not sure if it was a plugin.
Tools tab => UnRaid OS section => Docker Safe New Perms

On my screen it is just to the right of Diagnostics and left of Hardware Profile

Hmm.. yeah I don't have it either on my system (unraid 6.3.5)

[glenner]

Originally Posted by stuckless

When sagetv starts up, it takes ownership of all media files and all sagetv intallation files that are configured via the "apps", "media" and "mediaext" path mappings. So any files that are placed there are already reset on every sagetv restart. If you have other files in other locations, then sagetv won't touch those, even if they are configured inside of sagetv.

Good to know. I'll try that out next time... I guess in my case I'm currently pointing /var/mediaext to /mnt/disks as that seemed to make sense to me: that's where my externally mounted disks might be that may contain media that I want to allow sage to see. In actually, I'm not importing anything in sage via /var/mediaext as I expect all my media to be local in unraid. But I may still mount disks there, so I'll assume sage won't try to chown everything under /mnt/disks to nobody.users...

In the meantime, I've created an extra regular non-root user using the unraid webui. I'll see how that goes for regular use, where non-root access is required.

[glenner]

Originally Posted by Fuzzy

If you are trying to get sage to map to a samba share that is NOT on you unraid server, you should have sage talk to that share directly via samba. Do so by browsing to it in sage. Go up in folders until you find Network, and then browse down to your server and share from there.

I actually tried this.... I went up to the top level /Network and it found my 2 valid workgroups (WORKGROUP and TORWIN1). I have all my Win PCs on a workgroup called TORWIN1. And the unraid box, and my Squeezebox Touch are on WORKGROUP evidently (interesting... both linux boxes).

But so if I click on one of my PCs (audi) under TORWIN1, I get "invalid directory".
See screencaps:

https://drive.google.com/file/d/0B-8...ew?usp=sharing
https://drive.google.com/file/d/0B-8...ew?usp=sharing

So this approach did not work for me. I tried making tweaking my windows shares settings. No good. Instead I SMB mounted my Windows shared drives on the unraid main panel, this actually requires a user/pass to access the windows share. That puts my external mounted disks under /mnt/disks and I mapped /var/mediaext to /mnt/disks so I can find my shares....

This is all kind of moot anyway... as transferring TBs over the network is like watching paint dry... and so I just hot swapped the drives directly into the unraid box and copied files over via SATA. Moved everything over in under a day.

[glenner]

Originally Posted by HuMan321

Hi, I think the LocalSystem is my issue on my unRaid SageTV setup. I cannot figure out how to log in as a user on unRaid - probably need an "idiots" guide...

Cannot browse to network folders - says invalid directory
Thanks,

I also get "invalid directory" when going through "network"... so not sure.

But so if you want to log in as a user on unraid, that's pretty easy... you just need to download putty or cygwin. Google for either. Putty is likely easier. cygwin is heavyweight unix on windows.

Then you either telnet or ssh to unraid. Using a cygwin window, you would just type:

ssh root@tower (and you will be prompted for your password).

The trick is what do you do there once logged in... and that's where the "linux for idiots" guide may come in handy. Or again, Google is your friend.

One word of caution, linux can be pretty unforgiving... ie. DO NOT type "rm -rf /*" at any time, as you will irreparably nuke your entire setup in a few seconds. So don't leave your terminal window unattended: if it takes 1000 monkeys a finite amount of time to type out Hamlet, they can bang out those 9 keystrokes pretty quick.

[glenner]

Originally Posted by Fuzzy

The 'parity hit' can be minimized by turning on 'Turbo Write' in unRAID. This is done on the Disk Settings page, by setting Tunable (md_write_method) to "reconstruct write", instead of the default "auto" which is synonymous with "read/modify/write" in current versions. Reconstruct write will require all pool drives to spin up to write to the array, but instead of doing the slower read data and parity, calculate new parity, then write data and parity, it writes to the data drive, while reading from all the other data drives, and writing to parity at essentially the same time. MUCH faster, but as i said, the trade-off is all drives need to be spun up. I only turn this on when doing the bulk copying like you are doing on setup. I believe there is a plugin to auto control this setting, based on how many drives are already spun up, but I haven't messed with it.I want to say I read that there were some performance related issues with btrfs cache pools that are fixed in the current beta, but honestly can't find that comment now.

Thanks. Turbo Write sounds like a useful tweak... I'll add that to my growing "to do list", which now also includes figuring out why my cache pool is slower than my SATA disk.

I guess my to do list looks as below now, since my basic unraid SageTV setup seems to be more or less online, though still pre-production (ie. not quite ready to replace my WinXP setup yet).

To Do List:

1. setup unraid server OS - done.
2. install/configure SageTV docker - done.
3. setup shares - done.
4. copy all media over to unraid shares - done.
5. copy over v6 Wiz.bin to unraid - done. this went surprisingly smooth. all my old recordings, and even my scheduled recording setup were ported over to unraid. I hope Wiz.bin does not depend on tuners. My old and new setups are using 2 different HDHR tuners. but so far my ported scheduled recordings seem to be running ok.
6. setup SageTV plugins done. so far have SageTV3 UI, SageMC, Phoenix3, Gemstone, Jetty Web Server, BMT installed... need to figure out what else I need - Is there a short list of "must have" plugins??
7. install SageTV placeshifter. done. somehow never set this up before, but clearly this is the way to connect to the server. If this had been called "SageTV placeshifter thin client" I might have set this up earlier! Not sure if I had a placeshifter license is the issue.
8. setup SageTV comskip. somehow never set this up on my V6 setup.... but want to get this going.
9. setup FanArt
10. figure out cache pool throughput issue.
11. setup parity disk
12. install other dockers (plex, logitech media center, windows vm, etc.)

...so enough to keep me busy indefinitely.

[glenner]

But so I did hit a snag on running the WebUI from the SageTV docker. I have installed the Jetty webserver plugin, also a first for me, as I wanted to get BMT working from the WebUI to create FanArt, which I was also missing in my v6 setup.

When I click WebUI, it goes to http://unraid:8080, and I get this:

Error 404 - Not Found.

No context on this server matched or handled this request.
Contexts known to this server are:
/ ---> org.mortbay.jetty.handler.ContextHandler@1e79d7a4{/,file:/opt/sagetv/server/jetty/static}
/apps ---> org.mortbay.jetty.webapp.WebAppContext@2e47ff82{/apps,/opt/sagetv/server/jetty/webapps/apps.war}
/bmt ---> org.mortbay.jetty.webapp.WebAppContext@e6f4d32{/bmt,/opt/sagetv/server/jetty/webapps/bmt.war}

Interestingly... I was still able to get to the BMT web interface by going here directly:

http://unraid:8080/bmt

That seems to work. And I've successfully been creating FanArt manually from here.

Another issue I'm having... something to add to my "to do list"... Is that if I have the Gemstone or Phoenix3 UIs loaded, it does not seem to generate the FanArt for me... even if I try to generate it or run BMT (run "BMT Metadata Scan" from settings). It just doesn't do it... I can dig into that later. But the only way I can create my FanArt has been through the BMT WebUI.

[stuckless]

Originally Posted by glenner

Good to know. I'll try that out next time... I guess in my case I'm currently pointing /var/mediaext to /mnt/disks as that seemed to make sense to me: that's where my externally mounted disks might be that may contain media that I want to allow sage to see. In actually, I'm not importing anything in sage via /var/mediaext as I expect all my media to be local in unraid. But I may still mount disks there, so I'll assume sage won't try to chown everything under /mnt/disks to nobody.users...

In the meantime, I've created an extra regular non-root user using the unraid webui. I'll see how that goes for regular use, where non-root access is required.

Yeah, i just checked the scripts. I actually don't modify permissions on mediaext stuff.. i do take ownership of /var/media/ (which is where sagetv generall puts recordings, etc). And the docker configuration can be configured to NOT try to take ownership as well.

[dgeezer]

When I click WebUI, it goes to http://unraid:8080, and I get this:

Try this instead -
http://unraid:8080/sage/Home

[stuckless]

Originally Posted by dgeezer

Try this instead -
http://unraid:8080/sage/Home

And you can quickly build your own landing page, if you want... see this snippet.

https://gist.github.com/stuckless/85...f7c676c9494ee5

[glenner]

Originally Posted by dgeezer

Try this instead -
http://unraid:8080/sage/Home

Ok... I tried this manually.. I get this:

HTTP ERROR: 404

Problem accessing /sage/Home. Reason:

    NOT_FOUND
Powered by Jetty://

Is http://unraid:8080 supposed to work out of the box after I install the Jetty plugin? Do I need to restart the docker after installing the plugin? Avoided doing that as Sage was recording shows.