Just how "Fast" can network storage be?

[stanger89]

Some of you will probably know I'm a reasonably big fan of networked storage, keep all your files/data that the PC doesn't depend on (music, photos, videos, documents etc) out on networked storage, keeps it out of the PC in case something happens, makes it easily accessible from multiple machines, you can do whatever on the PC without worrying about your data availability to other machines.

I'm also a fan of Infrant's (now Netgear's) ReadyNAS line for how they're reliable, have a good feature set, etc.

And now with the advent of reasonably priced SSDs and their massive performance differential vs traditional discs for OS/scratch discs, but limited size, it seems to make more sense than ever to keep everything but scratch/transitory data on a network share, and have SSDs in the work PCs for local/scratch data.

However one thing I've become increasingly aware of is how my now old ReadyNAS X6 is rather long in the tooth and just doesn't have that great of performance anymore. Generally performance is acceptable, I can play BD rips off it, stream music, but that's really it. What prompted this thread is I'm trying to do a few things on my NAS at the moment, first is I'm trying to sort through some of my digital photos, while doing that I've got music playing, and in the background I'm trying to copy a VM image back onto a local disk. What's the problem? Well the pictures are far from "snappy" loading, and the music will hickup/glitch occasionally when I open a new picture, and the image is just not copying very fast. The other prompt is I've been looking to upgrade my Sage server for a while (I've got some extra hardware laying around that's screaming for a new home and to be put to use ).

So the question is, just how fast can network storage really be? And I don't just mean raw, sequential read/write, I mean in real use, how close, with GigE lan, can network storage come to the local equivalent? I'm looking for any real world tests/experiences the community here might have.

FWIW, I just threw together a quick experiment, I've got an OCZ SSD in my desktop as the OS drive plus a temp/scratch partition. I've also got IO meter so I thought this is the most objective way to do a comparison. So I've run IOmeter on the same parition, once direct to the drive letter, and once to the mapped drive of the same partition's network share. Here's what I've found:

Using Infrant's test iometer config (basically 256k sequential read/writes) I get:
Local Read - 220MB/sec
Local Write - 125MB/sec
Local Latency ~ 2ms (io response time)
"Network" Read - 210MB/sec
"Network" write - 125MB/sec
"Network" Latency ~ 2ms

So far so good, no appreciable difference. But like I said above, I'm curious about random access and simultaneous access. So here's 2k random read/write (stock 2k Rnd access specifications):

Local Read - 16.5MB/sec
Local Write - 2.9MB/sec
Local Latency - 0.12ms (0.65ms write)
"Network" Read - 14.4MB/sec
"Network" write - 2.8MB/sec
"Network" Latency ~ 0.13ms (0.70ms write)

Still looks pretty good, it's clear CIFS imparts some overhead, but it doesn't appear to be anything significant, that's good news too. But what about that simultaneous access, here's the stock 5/5 Seq Read/Write 256K access specification:

Local Read - 35MB/sec
Local Write - 35MB/sec
Local Latency - 3.4ms
"Network" Read - 35MB/sec
"Network" Write - 35MB/sec
"Network" Latency - 3.5ms

Again, more promising news. And now before you jump me I know what you're going to say, that's not a real "network" test because even though I'm using a network share, it doesn't actually go out on the network interface (that's why network above is in quotes ). The above I think just shows that CIFS itself isn't a big problem. Unfortunately I don't have a spare Core i7 machine or a spare SSD so I can't do a real apples/apples comparison, but what I can do is run iometer on my Sage server (Windows Server 2003, Athlon XP 1800) to the shared SSD and see what I get:

256K Sequential
Network Read - 45MB/sec
Network Write - 65MB/sec
Network Latency - 5.5ms (3.7ms write)

2K Random
Network Read - 5.5MB/sec
Network Write - 7.7MB/sec
Network Latency - 0.31ms (0.25ms)

5/5 Seq Read/Write 256K
Network Read - 26MB/sec
Network Write - 26MB/sec
Network Latency - 4.7ms

That's more what's expected, sequential throughput is way down, but 100MB/sec is about the theoretical max of GigE, and I think with the hardware I'm running, I only get about 600-700Mbps between those machines even with iperf. The random access is down by about half, but still WAY faster than a standard HDD, and the simultaneous access is only off by a quarter to a third. Just as an example I've got a 74GB WD Raptor (10k RPM) and in the same 2k Random test, it can only manage 0.35MB/sec Read (0.75MB/sec write) with 5.2ms (2.5ms write) latency.

So judging from the data, it seems that a GigE network doesn't affect disk performance much at all until you hit the throughput limits of the link. So it seems I really should be able to get close to "local" disk performance out of network storage.

So the question is, anyone out there really played with it and managed success as should be theoretically possible? Or do the above experiments miss some significant thing htat makes it impossible?

[mikejaner]

Very interesting writeup. I too am toying with a couple of technologies in the near future. I just ordered two Phenom rigs, to start my ESXi farm, and am toying with the iSCSI idea. I have an Ubuntu server with all of my RAID drives, and would rather just add more disks to it than order drives and RAID cards for my ESXi boxes, since it would be way cheaper. If performance as an iSCSI target is not good, then I will have to do the latter. SSD's are still way out of my pricerange for storage. They are just now getting into the range of making OS drives affordable for me.
I have seen recently that if I launch two Comskip processes from two different servers, while SageTV is trying to record to my Ubuntu Server, four different shows, and playback at the same time, I too see issues. Once I killed Comskip, these problems went away. It must do some pretty heavy thrashing over the network.

[paulbeers]

From server to server at my office, we have been able to saturate a gb connection (or close to it approx 900mbps), but that required sequential reads/writes to and from RAID 5 arrays. The real problem are mechanical hard drives are not fast enough to read/write by themselves. Even the fastest hard drives on the market are still only table to sustain around 80MB/s on consecutive reads and around 60-70MB/s writes. This means that in a standard server/client environment the clients become the limiting factor as the above numbers only equate to 480-640mpbs (since the generally only use single hard drives). In non-sequential writes, depending on the size of the file, it becomes very difficult to saturate the Gigabit connection especially in a home network environment. Even my RAID 5 can't saturate my gigabit connection if I do non-sequential writes (granted right now I am only using a 4 drive configuration as opposed to say the 20 drive configurations we use at work).

SSD's will/do change the above comments, but unfortunately due to price/size, I have yet to play with them much as size wins out to speed on most things at home and at the office (our server guys don't get called if it takes 30 seconds to copy a GB of data to the file servers, but they do get called if there isn't room!). I would love to build a huge RAID 5 array with SSD's to see what the performace would be like (I believe Anandtech.com or tomshardware.com did this very thing a few months back).

I guess what I am trying to say, from a home environment it would be difficult to saturate the GB connection. Hard drives begin to be the biggest bottle neck and not so much the connection speed. Saturating a 100mbps network is not difficult even with non-sequential read/writes, but until SSD's become the norm as opposed to the exception, I don't see gigabit networks being the bottleneck (and frankly I'm not pulling new cable throughout my house to accomplish 10gb).

[stanger89]

Originally Posted by mikejaner

Very interesting writeup.

It started out I was just going to ask the question if anyone had had "local" level performance from network storage, but kind of evolved from there

SSD's are still way out of my pricerange for storage. They are just now getting into the range of making OS drives affordable for me.

Yeah I didn't mean to imply that I'd use SSDs for the network storage, the primary reason for using it was it's what's in the computer I was doing the test on, but a secondary benefit is using an SSD for the test essentially eliminates the disk from the test, leaving only the network to impact performance.

I found it interesting that even over the network, asside from raw throughput the network wasn't a bottleneck.

Originally Posted by paulbeers

From server to server at my office, we have been able to saturate a gb connection (or close to it approx 900mbps), but that required sequential reads/writes to and from RAID 5 arrays. The real problem are mechanical hard drives are not fast enough to read/write by themselves. Even the fastest hard drives on the market are still only table to sustain around 80MB/s on consecutive reads and around 60-70MB/s writes. This means that in a standard server/client environment the clients become the limiting factor as the above numbers only equate to 480-640mpbs (since the generally only use single hard drives). In non-sequential writes, depending on the size of the file, it becomes very difficult to saturate the Gigabit connection especially in a home network environment. Even my RAID 5 can't saturate my gigabit connection if I do non-sequential writes (granted right now I am only using a 4 drive configuration as opposed to say the 20 drive configurations we use at work).

Oh, I completely understand that, and that wasn't really the question I was looking at, not raw performance. My curiosity was is there something inherent in network storage that would prevent you from having "local storage" type performance. Basically, does moving the storage to the network significantly reduce performance. It seems like the answer is no, and my performance "limitations" with my current network storage are due to the device hosting the storage and the storage itself and not the fact that it's networked.

I guess what I am trying to say, from a home environment it would be difficult to saturate the GB connection. Hard drives begin to be the biggest bottle neck and not so much the connection speed. Saturating a 100mbps network is not difficult even with non-sequential read/writes, but until SSD's become the norm as opposed to the exception, I don't see gigabit networks being the bottleneck (and frankly I'm not pulling new cable throughout my house to accomplish 10gb).

Yeah, and going in I understood that, at least from a throughput perspective. But I've never had really, really good results with network storage, performance wise. So I was curious if that was because it was networked, or because I typically use old/low power/slow machines for my network storage.

My investigation gives me hope that if/when I rebuild my Sage server, if I pay attention to building a fast storage subsystem, it will be essentially as fast for network computers as it would be if I were using it locally.

[Taddeusz]

I've found part of it has to do with the hosting OS. For quite some time I had been using FreeNAS to build my own. I was getting nowhere near gigabit speeds when transferring files to and from. At the time I assumed it was due to the limitations of the individual drives as I wasn't using any kind of RAID. After some experimentation and deciding to move my storage back to my Sage server I determined that the FreeBSD network stack just isn't very good.

With FreeBSD, according to the AddGadget Network Meter, I would NEVER get transfers above 45MB/s with FreeNAS. I would usually get around 25-35MB/s with it rarely going as high 45MB/s.

Now that I've moved my storage back to my SageTV server, which runs XP Pro SP3, transfers now peak at around 110MB/s with usual transfers in the 65-85MB/s range. When moving around 5-7GB files this can actually make a pretty huge difference in transfer times.

Both computers I'm using have Intel NIC's in them. One is an integrated Intel PHY and the other is a PCIe x1 NIC with only a single gigabit switch and CAT6 between the two. This is probably as near to ideal as I can get on a tight budget.

[mikejaner]

Originally Posted by Taddeusz

..I determined that the FreeBSD network stack just isn't very good....

From the description of your setup, you may be giving the windows network stack too much credit.

[jsin]

Originally Posted by stanger89

My investigation gives me hope that if/when I rebuild my Sage server, if I pay attention to building a fast storage subsystem, it will be essentially as fast for network computers as it would be if I were using it locally.

I've worked with my share of NAS systems, both personally and professionally, for last 20 years or so. Companies like NetApp are built on the premise that aggregate throughput can be much greater on NAS devices than on local devices - of course, aggregate throughput isn't nearly as important for personal use - but it's true that your NAS device can rival the speed of your local disk in many situations.

But as you already know, there are a lot of factors involved, such as the quality of your network switch and/or router, your basic Cat-5E cables & connectors, your computer's NIC, your NIC driver, your Operating System, your computer's CPU & amount of RAM, and even mundane things like what your other computers on your network are doing (and how many other computers there are).

Couple years ago I played with ReadyNAS NV and Buffalo Terrastation, but I ended up just building my own rackmount Linux servers with 8-drive, RAID-5 system. For SageTV, I'm using a WHS box with 8-drives. It is now my NAS device for all intents and purposes, and it works fast enough for Photos/Music/TV Shows/Videos. But I would never put my VM images on it - for that I use my Linux boxes.

I guess over the years I've learned that it's best to put the disks in the computer that will use it the most. And make sure you have good quality hardware. Then you can enjoy more and worry less.

But I still love to tinker with things.

[jsin]

Originally Posted by mikejaner

From the description of your setup, you may be giving the windows network stack too much credit.

Yeah, when one talks of the "network stack", one normally refers to the TCP/IP stack, which doesn't necessarily have a anything to do with reading/writing your files on disk (or network).

I think in general (i.e. un-tweaked/patched) Windows Vista networks better with other Windows systems, when compared to Linux/FreeBSD/Solaris systems using SAMBA.

[Taddeusz]

Originally Posted by mikejaner

From the description of your setup, you may be giving the windows network stack too much credit.

Not really. Same exact drives were moved from my FreeNAS server to my SageTV server.

Granted, my FreeNAS server was using an Intel PCI gigabit NIC. I had originally planned to use the onboard Realtek PCIe chip but discovered after trying to set it up that there were no pre-compiled FreeBSD drivers for it. Later on I acquired a pre-compiled driver for the onboard PCIe chip but soon figured out that performance using it was actually worse than the PCI Intel.

So no, I'm not giving the Windows stack too much credit. It really does perform better than the FreeBSD stack. Every comparison I've ever seen on the internet that compares network performance between Windows, Linux and FreeBSD seems to confirm it.

[panteragstk]

One thing I've been going back and forth on is whether or not a RAID-5 array or a server running WHS drive pool would be faster. Now it seems that it wouldn't matter. Am I right? I just need a reliable way to keep my files safe, and preferably be able to access them as quickly as possible.

[stanger89]

Originally Posted by panteragstk

One thing I've been going back and forth on is whether or not a RAID-5 array or a server running WHS drive pool would be faster. Now it seems that it wouldn't matter. Am I right?

If you're looking at my test results, I don't think you can say that, though it depends on what you mean by "faster". If you just mean raw throughput, you're right the RAID-5 array won't be the bottleneck, but depending on where the data lives in the WHS pool (ie if you're reading off only one drive) RAID should be faster.

Of course with WHS and Drive Extender with it's balancing, it's more complicated, I just found this review:
http://www.vistaforums.com/Forum/Topic11228-8-1.aspx

It appears performance plummets precipitously once you start going into storage balancing.

The other thing is if you look at some RAID-5 benchmarks, with a good controller a RAID-5 array is very fast, even for simultaneous users/random access.

[gplasky]

Originally Posted by stanger89

Of course with WHS and Drive Extender with it's balancing, it's more complicated, I just found this review:
http://www.vistaforums.com/Forum/Topic11228-8-1.aspx

It appears performance plummets precipitously once you start going into storage balancing.

The other thing to keep in mind was those benchmarks were performed pre-PowerPack 1. (PP1 was released on 7/20/2008.) It was still using the D drive as the landing zone. I would think those numbers would be better with the later powerpacks installed and not using D as an intermediate.

Gerry

[paulbeers]

Originally Posted by stanger89

My investigation gives me hope that if/when I rebuild my Sage server, if I pay attention to building a fast storage subsystem, it will be essentially as fast for network computers as it would be if I were using it locally.

You are correct. There is no reason that a network storage can't be as fast as local storage. Now that I understand what you are asking! Sorry maybe it was too early this morning?

One of the decisions we have been toying with at my office (and by toying I mean if we had the resources to do it all over again) is to rid ourselves of multiple file servers and instead going to a SAN. Basically we would build virtual servers (4-8 core Intel Xeon 55X0's) and then have all the storage in one place. This of course would be connected via fiber as we would bottleneck the our throughput between our SAN and the virtual servers. The point I am making is that if you have fast enough storage on a network server and a gb (or better) connection, you will not notice a difference (as stated the best hard drives now only pull off around 640mbps).

So I assume you are thinking about going with smaller SSD's and then using network storage as your "bulk" storage for your "client" pc's rather than having large hard drives in each one?

[stanger89]

Originally Posted by paulbeers

So I assume you are thinking about going with smaller SSD's and then using network storage as your "bulk" storage for your "client" pc's rather than having large hard drives in each one?

Yeah, something along those lines, though that makes my config sound bigger than it is

Basically I've got my desktop (Core i7+SSD, 60GB) that I use for general stuff, surfing, and any CPU intensive work I decide to play with. Also does my commercial detection.

Then there's my laptop (which I'm thinking of swapping the HDD for an SSD).

Finally on the "PC" front, is my SageTV server which is an old Athlon XP and originally held all my "mass" storage, a mix-match of left over drives plus an 8x250GB RAID-5 array (scary that I could replace that with a single HDD today and have space left over). I've also got a ReadyNAS X6 with 4x1.5TB drives.

The server and ReadyNAS have worked well when just used to hold static data. Their performance has never been blistering but for just playing media it's generally been fine.

But since getting the SSD for my desktop and having a mixed system with SSD and other drives, I've found that it's great except when something wakes the standard drive then the whole system grinds to a halt. Of course the other thing is it just seems "cleaner" to not have misc drives floating around my spare PCs. Anyway, since getting the SSD I've been wanting to get most storage out of my PCs, but to do that the storage has to be fast, at least close to as fast as using local drives.

[paulbeers]

I tried something basic like this at my house. Using 2 - 500GB drives in RAID 1 for our Pictures / Music / documents. It works great for my Sage Server and my desktops, but my wife and I spend most of our time on our laptops using wireless networking. Even using draft n, there becomes considerable latency from when a file is requested and when it is pulled up (large obviously depending on the size of the file). For anything hard wired there might be a fraction of a second additional latency, but it is virtually unnoticeable. The ironic part is that this solution would be perfect for laptops if the wireless networking didn't slow it down and, while I could hard wire my laptops, that kind of defeats their usablility.

[mikejaner]

http://www.tomshardware.com/reviews/...l#xtor=RSS-182

Interesting article about drive speeds. Worth reading..

[stanger89]

I didn't really read the whole thing, but what I saw wasn't surprising. The density of the recent, huge HDDs (1TB in teh test) is so dramatically higher than the older ones (320GB in the test) that for sequential reads/writes, even with slower rotational speed they're either just as fast, if not faster.

[paulbeers]

I can attest to that. My RAID 5 is using Samsung 5400rpm 1.5TB hard drives. Before creating the RAID, I tested them to see how they compared to my WD 320GB 7200rpm Blue drives in my main box at home and the consecutive read's were on par if not even faster. As stated, the higher the density, the faster the read so if you slow down the platter, but increase the density, it will even out.