Moving files from Windows to unRAID

[wayner]

I guess under unRAID I will be using a different FS so I can't just plug in my old data drives. What is the process to transfer? Just copy files over to the new unRAID drives? Assuming that I have .properties files set up then metadata should be good, correct? These files will be a combination of prior SageTV recordings and DVD-rips - plus music and photos.

[stuckless]

There are ways, in unRAID to add drives that are NOT a part of the storage pool... but, I don't recommend doing it, since, those drives will not benefit from the parity disk protection.

In setting up unRAID, you will likely have a large parity drive and a number of other drives, and I recommend creating "shares" and letting unRAID decide how to distribute the data across the disks... that way, as things fill up, you can just add more drives and unRAID can just start using those drives.

When I did mine... I started with a 3tb partity disk and a single 3TB drive. I then setup my shares for "media" and I copied a disk from my old setup into unRAID. Once that was done, I then added the drive to unRAID storage pool, and I got the next drive and did the same, until I was done.... It took some time to copy over all the media.

[wayner]

I know that unRAID uses a USB drive for the OS, but does it make sense to use a SSD for the partitions where SageTV and other applications are stored? I know that moving to an SSD years ago on Win dramatically improved performance.

[stuckless]

Originally Posted by wayner

I know that unRAID uses a USB drive for the OS, but does it make sense to use a SSD for the partitions where SageTV and other applications are stored? I know that moving to an SSD years ago on Win dramatically improved performance.

so unRAID allows this via it's "shares". When you create a "share" you can say which drives it can use, and whether or not it will use a "cache" drive, and whether or not it will ONLY use a cache drive. A Cache drive is generally 1 or more SSD drives.

So, if you wanted all your installed apps to be on SSD, you an just creats an "appdata" or "apps" share and then set it to be cache only, and then all the data for those apps would be on SDD.

For Docker, my docker images are all only SSD, but, docker images don't store their configurations in the image (or else you would lose it when docker images get updated), so typically when you are configuring a docker image (like sagetv), you have to tell it where on the unRAID system it can store it's configuration/appication files, and that share can also be a cache share, if you want.

[SomeWhatLost]

just to add to the confusion...
there is also a plugin for unRAID called unassigned drives (or something to that effect...)
it allows you to install a drive(s) that unRAID can see/use/etc but are not assigned to either the drive pool or the cache pool...
as I saw no reason to have my recordings "protected" by parity, I have my 2 recording drives "unassigned"... works well for me... anyway, just tossing out more options for you

[Taddeusz]

Originally Posted by SomeWhatLost

just to add to the confusion...
there is also a plugin for unRAID called unassigned drives (or something to that effect...)
it allows you to install a drive(s) that unRAID can see/use/etc but are not assigned to either the drive pool or the cache pool...
as I saw no reason to have my recordings "protected" by parity, I have my 2 recording drives "unassigned"... works well for me... anyway, just tossing out more options for you

I used the Unassigned Devices plugin to move my files from NTFS formatted drives to unRAID. It was a long process for me because the drives that I was moving files from I was also planning to use in the array. Mainly because after copying the files over I had to clear the drives for use in the array. It takes a couple hours to clear the drives before they can be added.

[KarylFStein]

Originally Posted by stuckless

so unRAID allows this via it's "shares". When you create a "share" you can say which drives it can use, and whether or not it will use a "cache" drive, and whether or not it will ONLY use a cache drive. A Cache drive is generally 1 or more SSD drives.

So, if you wanted all your installed apps to be on SSD, you an just creats an "appdata" or "apps" share and then set it to be cache only, and then all the data for those apps would be on SDD.

For Docker, my docker images are all only SSD, but, docker images don't store their configurations in the image (or else you would lose it when docker images get updated), so typically when you are configuring a docker image (like sagetv), you have to tell it where on the unRAID system it can store it's configuration/appication files, and that share can also be a cache share, if you want.

It seems like unRAID defaults seem to push VMs and docker data to the cache. I didn't do any special configuration and those things are on my cache SSDs. I guess if you don't set cache only then if you have space issues on the cache it'll move things to the pool?

[stuckless]

Originally Posted by KarylFStein

It seems like unRAID defaults seem to push VMs and docker data to the cache. I didn't do any special configuration and those things are on my cache SSDs. I guess if you don't set cache only then if you have space issues on the cache it'll move things to the pool?

If you don't set cache only, then it means all writes happen in the cache, and then transparently, unRAID will move it from the cache to the pool "later" (not sure when later is).

[EnterNoEscape]

Originally Posted by stuckless

If you don't set cache only, then it means all writes happen in the cache, and then transparently, unRAID will move it from the cache to the pool "later" (not sure when later is).

There's help on the page itself that tells you what they all do. Just click on Use cache disk:

[KarylFStein]

Just looked and the default for system/domains/appdata is Prefer. It writes to the cache disk unless full in which case it'll write to the array. BUT if space clears up it'll move back to the cache on the next mover invocation. Seemed sensable to me and keeps VMs / Docker stuff on the SSD as long as there is space. Only will fail on write if the cache disk is full. Anyway, all this was automatically done for me, but I also started with an SSD cache drive, a single parity drive and a single data drive, (since expanded to 2 cache, 1 parity and 3 data). If you don't start with a cache drive maybe it requires manual tweaking after.

Unassigned devices also lets you mount network shares. I did some testing of copying data from my Windows server that way. Faster to just mount locally, but can be used to get fairly static things moved over without causing an outage.

[stuckless]

Originally Posted by KarylFStein

Just looked and the default for system/domains/appdata is Prefer. It writes to the cache disk unless full in which case it'll write to the array. BUT if space clears up it'll move back to the cache on the next mover invocation. Seemed sensable to me and keeps VMs / Docker stuff on the SSD as long as there is space. Only will fail on write if the cache disk is full. Anyway, all this was automatically done for me, but I also started with an SSD cache drive, a single parity drive and a single data drive, (since expanded to 2 cache, 1 parity and 3 data). If you don't start with a cache drive maybe it requires manual tweaking after.

Unassigned devices also lets you mount network shares. I did some testing of copying data from my Windows server that way. Faster to just mount locally, but can be used to get fairly static things moved over without causing an outage.

I didn't start out with a cache drive, so likely, that is why, for me, I had to manaully tweak this stuff later.

[Fuzzy]

Regarding the moving of data into unRaid, I did the same as Taddeusz above. I went ahead and added all the drives of concern to unRAID, as well as a single new drive. I added only the new drive to the pool (didn't have a parity drive yet). I then installed the SageTV Docker, and let it create the recordings and import folders. I then installed the Unassigned Devices plugin, which allowed me to browse all the attached NTFS drives from my old windows server. I picked the largest NTFS, and moved the files to the appropriate folders in the pool, then formatted/cleared the drive, and added it to the pool. Then went on to the next largest NTFS, and so on. Did this a little at a time (I actually configured Sage to look at those unassigned drives as import folders, so sage still had read access to those recordings during this process). Eventually, I had all the drives in the pool, and all the sage content in the correct folders. I added a parity drive a couple days later after I received the new one (my previous 'new' drive was 3TB, and the parity has to be at least as big as the biggest singular drive in the array), and one installed, manually started a parity check to populate the parity drive.

[wayner]

Is there any reason not to just copy over the network from my old server to my new unRAID server. It will be much slower but if you aren't in a rush that isn't a big deal - just let it run overnight. I have about 7TB of data to move over.

[Fuzzy]

Originally Posted by wayner

Is there any reason not to just copy over the network from my old server to my new unRAID server. It will be much slower but if you aren't in a rush that isn't a big deal - just let it run overnight. I have about 7TB of data to move over.

That's fine, if you have twice as much hard drive as you have content. I was reusing the drives from my old server in my unRAID server.

For information, copying a single file from my desktop to my unraid tower over gigabit runs at about 70MB/sec. So your 7TB of data would take around 30 hours.

[wayner]

@Fuzzy - do you know what the speed was when you moved your files over using this methodology? When using the File Explorer in Win 10 my transfers start at about 110MBps, which uses pretty much the full network bandwidth of 1Gbps but then it slows down to about 10MBps. But I wonder what is the limiting factor, is it the parity drive? Note that my parity drive is still building right now. I think you can also stop the array while you are transferring files.

[Fuzzy]

No I was copying them in Midnight Commander so I didn't watch the transfer time.

[stuckless]

Like Fuzzy... when I do network copies... I see about 70mb/s consistent... i don't really see a burst and then a slowdown... it's just fairly consistent.

[stuckless]

I did some googling and found a few articles on performance tuning the unRAID system... I'm ok with my performance, but these might be of interst to those trying to get more IO bandwidth.

https://lime-technology.com/forum/in...?topic=31073.0

https://lime-technology.com/wiki/ind...ID_Performance

http://lime-technology.com/wiki/inde...ser_Benchmarks

[wayner]

Now that my array is built i am getting much faster transfers just using Windows File Explorer to transfer media files from one of my existing SageTV servers. I seem to be able to sustain close to 100MB/s for at least thirty minutes. That's probably good enough for me as that is about 0.36TB/hour.

The only issue is that occasionally you get this stupid dialog box about certain properties not able to transfer over to the new drive. You have to then confirm that. You don't want that to happen a few minutes into an overnight file transfer.

[Taddeusz]

Dang, so I have a few drives to replace if I want better performance. I ran that performance script and my slowest drive is an old WD Green 1TB. It averages 75MB/s and of course is only 5400 RPM. I have two other drives that average below 100MB/s that I will also be replacing as early as I can.