180Mbps Limit on 1Gbps NICs

nitan · ‎05-08-2009

Has anyone seen this odd behavior before?

We have a L2 WAN that is rate-limited to 600Mbps, with no filtering or firewalling between the two sites. When transferring files via ftp between two windows 2003 server machines across this link, the throughput is limited to 180Mbps, every time. When using Windows 2008 servers, I am able to hit 550 to 580Mbps, but also hit the 180Mbps limit on occasion. Also, when using a Fluke Etherscope II, I am able to hit 580Mbps with no issues consistently.

Has anyone else seen this hard limit of 180Mbps before?

Any help would be greatly appreciated.

Nitan

JamesLuther · ‎05-08-2009

Hello Nitan,

This isn't a problem with your link but rather the way TCP works. Given a particular TCP window size and the latency on the link then there will be a maximum throughput limit. You are finding a difference between the OSes due to the way they handle TCP window sizes.

Also you will find that if you open several FTP sessions you'll use more of the link.

Line testers (ie fluke) will send a standard RFC2544 test pattern which will hammer the link with UDP packets.

Regards

nitan · ‎05-08-2009

James,

Exactly, multiple ftp sessions do break the 180Mbps limit. Do you have any resources I can check on setting window size or other NIC/OS parameters that may help?

Also, I found an interesting FAQ talking about â€œtcp_fast_at_modeâ€, which can be found at

http://netlab.caltech.edu/FAST/fastfaq.html

Here is the excerpt.

The alpha parameter is dynamically set during the connection to one of three values based on the network speed environment. Alpha tuning is not intended to perform congestion control, but simply select an appropriate alpha for each environment. The mode selection is done by detecting the maximum throughput pkts/sec achieved over the previous update period. All the pkt/sec threshold values are calculated assuming 1500 byte packets.

tcp_fast_at_mode Current alpha mode.

Mode 0 = low speed (default <15Mbps).

Mode 1 = medium speed (default 15-180Mbps).

Mode 2 = high speed (default >180Mbps).

Thanks again for you help,

Nitan

Joseph W. Doherty · ‎05-08-2009

As James notes, you're likely bumping into the differences for TCP between Windows Server 2003 vs. Windows Server 2008. The latter, along with Vista, has a newer stack.

For Windows 2003, you might see an improvement if current LAN connection is 100 Mbps and you connect it to gig (stack adjusts TCP receive window sized based on LAN bandwidth connection).

You can also change Windows 2003 TCP registry settings (specifically the TCP receive window size).

You might also try the CTCP hotfix if running Windows Server 2003, see: http://support.microsoft.com/kb/949316.

nitan · ‎05-08-2009

Joseph,

Great hotfix suggestio, I am forwarding it to my server guys as we speak.

Thanks,

Nitan