Moving files from Windows to unRAID

[Fuzzy]

I have no illusions on the speed of my mix of old drives, that said, they've worked fine for sage so far. They will of course be replaced eventually, but not yet.

[Taddeusz]

Originally Posted by Fuzzy

I have no illusions on the speed of my mix of old drives, that said, they've worked fine for sage so far. They will of course be replaced eventually, but not yet.

For me though, I've been experiencing some weird performance issues. These slower drives, particularly the green drives, are likely holding back array performance. I have 3 5400 RPM drives and one is an AV-GP model which I'm led to believe is not great for this scenario. It was working fine for holding relatively static media files in Windows but is probably not suited for being part of an unRAID array.

[Fuzzy]

well, for me, just about the bulk of my unRAID array is static media files. my mix of 5400 hitachi drives, AV-GP and WD purples, and a couple old seagates (all 5400) transfer plenty fast enough for my max 4 clients. Recording is going to a 400GB SSD cache drive, and then lazily written to the array, so array performance doesn't really matter there.

[Taddeusz]

Originally Posted by Fuzzy

well, for me, just about the bulk of my unRAID array is static media files. my mix of 5400 hitachi drives, AV-GP and WD purples, and a couple old seagates (all 5400) transfer plenty fast enough for my max 4 clients. Recording is going to a 400GB SSD cache drive, and then lazily written to the array, so array performance doesn't really matter there.

Yeah, my difference is that I don't have a cache drive. I also need to get a SAS expander so I can put all my drives through my 6GB SAS card. Right now I have a single drive on the onboard SATA.

[stanger89]

Originally Posted by Taddeusz

Right now I have a single drive on the onboard SATA.

That's not necessarily a problem, in fact I remember that was the recommended place to put drives who's performance matters (like the parity drive), since the onboard SATA usually/often have the most IO to the CPU rather than being limited by a PCI or PCIe slot.

Running everything through a SAS expander might actually make things worse.

[EnterNoEscape]

Originally Posted by stanger89

That's not necessarily a problem, in fact I remember that was the recommended place to put drives who's performance matters (like the parity drive), since the onboard SATA usually/often have the most IO to the CPU rather than being limited by a PCI or PCIe slot.

Running everything through a SAS expander might actually make things worse.

That depends on the topology of the motherboard. Some PCI-e slots have direct access to the CPU and others are routed through the same bus to the CPU that the onboard SATA ports use. If you try to max out the onboard SATA ports, you will quickly discover that the bus to the CPU is actually oversubscribed. In those situations, it actually makes sense to use a PCI-e card in one of the slots with direct access to the CPU. Generally you would need some very fast SSD's to see this happen, but it does happen and has happened to me a few times.

SAS expanders are often not the way to go if performance is of interest to you. You still have the same amount of bandwidth, but now you have more disks on the other side. This is another way to quickly find out where the limits are. It could work out fine for your use case or it could be an expensive mistake. This is coming from someone that has made a lot of expensive mistakes in hardware purchasing, but I think I've learned a lot more as a result.

[Taddeusz]

Originally Posted by EnterNoEscape

That depends on the topology of the motherboard. Some PCI-e slots have direct access to the CPU and others are routed through the same bus to the CPU that the onboard SATA ports use. If you try to max out the onboard SATA ports, you will quickly discover that the bus to the CPU is actually oversubscribed. In those situations, it actually makes sense to use a PCI-e card in one of the slots with direct access to the CPU. Generally you would need some very fast SSD's to see this happen, but it does happen and has happened to me a few times.

My unRAID server is a Core 2 era dual Xeon so the CPU has no PCIe lanes. That being said, the motherboard I have, a Supermicro X7DWE, has most of the PCIe slots going through the northbridge rather than the southbridge. Save for the one slot that goes through the southbridge they are all PCIe 2.0. So that should not be a bottleneck.

I probably should shuffle drives around a little to make sure that a 3Gb drive is on the onboard SATA since right now it's a 6Gb drive that could be a bit faster on the 6Gb SAS controller.

Originally Posted by EnterNoEscape

SAS expanders are often not the way to go if performance is of interest to you. You still have the same amount of bandwidth, but now you have more disks on the other side. This is another way to quickly find out where the limits are. It could work out fine for your use case or it could be an expensive mistake. This is coming from someone that has made a lot of expensive mistakes in hardware purchasing, but I think I've learned a lot more as a result.

I am aware of the consequences of using a SAS expander. I try not to make expensive mistakes though. I try not to spend a lot of money on stuff if I don't have to. Granted, this server hardware is relatively old, The initial outlay I think was about $200 for the 2 CPU's, original X7DVL-E motherboard, and RAM. I more recently exchanged the motherboard for the X7DWE that I got for $50. The expander I've been eyeing is the IBM 46M0997. I've seen them go for as little as $50. Not going to spend much more than that if I don't have to.

However, it seems that since the array performs as fast as the slowest drive I should either replace the slower 5400 RPM drives or put in a cache drive. Right now I don't have any SSD's larger than 120GB so it's going to be an investment to get something suitably larger.

[Fuzzy]

depending on your recording load, 120GB may be plenty for your cache drive. at 12mbps recording bitrate, 120GB can hold over 20 hours of media, should get most people through their peak recording times and then let the mover push it to the array during downtime.

[EnterNoEscape]

Originally Posted by Fuzzy

depending on your recording load, 120GB may be plenty for your cache drive. at 12mbps recording bitrate, 120GB can hold over 20 hours of media, should get most people through their peak recording times and then let the mover push it to the array during downtime.

I agree. If you have a few SSD's you're not sure what else to do with, you could also just pool them together using btrfs. It doesn't matter what the sizes are, you'll get the most space possible less the space needed for one copy redundancy. http://carfax.org.uk/btrfs-usage/

[stanger89]

Originally Posted by Taddeusz

I probably should shuffle drives around a little to make sure that a 3Gb drive is on the onboard SATA since right now it's a 6Gb drive that could be a bit faster on the 6Gb SAS controller.

That's unlikely to make a difference, I don't think there are any mechanical drives that can saturate 3Gb SATA/SAS.

Originally Posted by EnterNoEscape

I agree. If you have a few SSD's you're not sure what else to do with, you could also just pool them together using btrfs. It doesn't matter what the sizes are, you'll get the most space possible less the space needed for one copy redundancy. http://carfax.org.uk/btrfs-usage/

So it looks like you can't just put a bunch of arbitrary drives together, the usable space is limited by the smaller drives? eg, I've got a 240 (currently in my unRAID server), 160, and 40GB SSDs, if I were to add the 160 to my unRAID server for the cache pool, I'd have 80gB unavailable?

The nice thing is, it looks like if I put the 160, and 40GB drives in, I'd only have 40GB unavailable.

[EnterNoEscape]

Originally Posted by stanger89

So it looks like you can't just put a bunch of arbitrary drives together, the usable space is limited by the smaller drives? eg, I've got a 240 (currently in my unRAID server), 160, and 40GB SSDs, if I were to add the 160 to my unRAID server for the cache pool, I'd have 80gB unavailable?

The nice thing is, it looks like if I put the 160, and 40GB drives in, I'd only have 40GB unavailable.

If you added another 40GB drive, then you wouldn't lose that 40GB (240 + 160 + 40 + 40 = 240), but that only gets you to the point where you have redundancy and not more disk space. I had missed the part where you already had a 240GB drive. The calculator is the easiest way to see what would happen. Essentially you can't have one disk that's larger than the sum of all of the other disks without losing the difference. Every disk needs another disk(s) to copy it's data to.

[willieb]

I have an old ReadyNAS with about 3TB of files that I want to eventually move over to my unRAID server, but for now I would just like to add it as a recording directory. I can mount the ReadyNAS in unRAID using Unassigned Devices and see it in unRAID, but I can't see it in SageTV to add it as a Recording location. Do I need to add it somehow to the Docker Volume Mappings? In unRAID it is located in /mnt/disks/READYNAS, but in Sage /mnt/ is empty. Any thoughts? Thanks!

[stuckless]

Originally Posted by willieb

I have an old ReadyNAS with about 3TB of files that I want to eventually move over to my unRAID server, but for now I would just like to add it as a recording directory. I can mount the ReadyNAS in unRAID using Unassigned Devices and see it in unRAID, but I can't see it in SageTV to add it as a Recording location. Do I need to add it somehow to the Docker Volume Mappings? In unRAID it is located in /mnt/disks/READYNAS, but in Sage /mnt/ is empty. Any thoughts? Thanks!

You'll need to add your own path, something like,

Container Path: /readynas
Host Path: /mnt/disks/READYNAS

Then restart the sagetv container and then in the container, /readynas will be available.

[willieb]

Originally Posted by stuckless

You'll need to add your own path, something like,

Container Path: /readynas
Host Path: /mnt/disks/READYNAS

Then restart the sagetv container and then in the container, /readynas will be available.

Got it. Will try it out. I figured it was something like that. I am in the middle of moving muy TBs over to unRAID right now so I have sagetv shut down until I get it moved over. Thanks!

[willieb]

That did the trick. Easy. Thanks!

[atchoo]

If you're moving files from Windows try using RichCopy it works pretty well and fast network transfers. It also shows you progress and status, what has been copied, etc.

On Linux I like to use Rsync

rsync --recursive --progress --update SOURCE_PATH  DESTINATION_PATH