Gigabit network realistic speed

[bialio]

So I've been looking at my network trying to benchmark how fast my computers are able to talk to each other, and I'm not sure I'm seeing very good results.

Basic network topology is :

                 router
                   |
            8 port gigabit switch
            | | | | | | | |
                          ---------8 port gigabit switch
                                    | | | | | | | |

All the computers in the house hang off of one of the 15 open gigabit ports.

So in theory a gigabit link can carry data at a rate of 1 billion bits per second, which is 125 million bytes per second, aka 125 MegaBytes / second.

I'm not seeing anywhere near 125 MB. Nor do I expect to. However, what I'm seeing is seems to be 20-25 MB/s. That seems awfully low. This is between two XP machines, both with 10/100/1000 NIC's on the motherboard, both connected at 1.0Gbs, doing a memory to memory transfer (takes the disk access time out of the equation).

Any ideas?

btl.

[GKusnick]

What are you using to measure throughput? I just tested four computers on two different gigabit links with Ixia Qcheck and got the following results:

A -> B: 500 Mbps
B -> A: 500 Mbps

C -> D: 350-400 Mbps
D -> C: 130-150 Mbps

I'm not sure what accounts for the asymmetry between C and D; maybe I've got a flaky cable somewhere. Gigabit is more sensitive to cabling issues, so if you're not getting the throughput you expect, that might be something to look at.

[Slipshod]

Could be a performance problem with the switch... We've seen some Dlink switches here at work that don't get much faster than that...

What are you using for the mem transfer, and what kind of NICs and switches are you using?

[MrD]

If you are not using jumbo frames then you will be limited.

Jumbo frame support seems to be inconsistent among hardware components.

[m1abrams]

Couple of things to keep in mind. TCP adds about 20% overhead, so the BEST possible throughput you can expect is ~100MB/s. Now then if you are using SMB(windows file sharing) for the test that adds more overhead. Note that smb can be tuned for better performance.

However none of the above is probably your bottleneck. Your bottleneck is most likely your harddrives. The transfer of a large file will only be as fast as the slowest hard drive in the equation can write. 20-30MB/s is probably not too far off for even a modern drives WRITE speed.

Edit sorry I see you are doing a memory to memory transfer. How are you performing this?

[bialio]

I tested using a small program that basically sends 100 MB of data over the TCP pipe and times how long it takes. You run it on both PC's and it spits out some numbers.

I'm not at home so can't find the exact name of the program. In general, it was taking 4 to 5 seconds to send 100MB - which comes down to 20-25 MB/s.

I've got the iXia tool set downloaded and am going to test with it on all my machines when I get a chance.

btl.

[sandor]

Originally Posted by MrD

If you are not using jumbo frames then you will be limited.

This is not true.


Jumbo Frames are not even part of the spec - the IEEE 802.3 working group has *not* reached a consensus on implementing jumbo frames.

on older hardware, frame size could cause problems, but not today - i can max out all of my hard drives on my consumer Dlink DIR 655 router, and can hit roughly 122 MB/s via 2 concurrent speed tests. This is without jumbo frames.


Furthermore, the slight increases in data transfer (as you are getting, theoretically, more data and less header in a transfer with jumbo frames) is quite minimal, and you are probably better served in making sure you have quality enet card.

Originally Posted by MrD

Jumbo frame support seems to be inconsistent among hardware components.

that is because jumbo frames are *not* part of the IEEE 802.3 standard, so each maker implements it how they see fit, and abide by no standard.

[sandor]

i would highly recommend using diskwriggler for throughput testing:

http://www.xdt.com.au/Resources/Downloads/


this setting works well for me-

$ ./diskwriggler -NTSC -C -t -n 2500 -o /Volumes/Disk-1

change the volume to whatever volume you want to test

[m1abrams]

Originally Posted by bialio

I tested using a small program that basically sends 100 MB of data over the TCP pipe and times how long it takes. You run it on both PC's and it spits out some numbers.

I'm not at home so can't find the exact name of the program. In general, it was taking 4 to 5 seconds to send 100MB - which comes down to 20-25 MB/s.

I've got the iXia tool set downloaded and am going to test with it on all my machines when I get a chance.

btl.

My question is, does this program in fact buffer the entire transfer in memory on both machines? If it does not then the harddrive would still be a factor. The sending machine would have to buffer the data to memory before starting its timer, and the receiving end needs to also buffer the entire data to memory and then send the final ack back to the sending and the sending stops the counter.

Stop watch timing on a 100MB file over gigabit is not really accurate.

[Slipshod]

Originally Posted by bialio

I tested using a small program that basically sends 100 MB of data over the TCP pipe and times how long it takes. You run it on both PC's and it spits out some numbers.

I'm not at home so can't find the exact name of the program. In general, it was taking 4 to 5 seconds to send 100MB - which comes down to 20-25 MB/s.

I've got the iXia tool set downloaded and am going to test with it on all my machines when I get a chance.

If the tool you're using is "iperf" you need to fiddle with the TCP settings (windows size, probably others) or increase the simultaneous transfers to get decent performance. By default it won't max your pipe.