Not reaching WAN capacity

[ronshuster]

I have a 1GIG private network between two sites having 2811 on each side with a GIG interface facing the WAN and LAN. There is a significant amount of traffic going through the WAN but I am only seeing, using tools, an average of about 200MG going over the WAN. There is no traffic limitation or throttling configured on the WAN. Any idea why we cannot take advantage of the pipe more that now? Could this be a fragmentation issue? What is the best way to determine the root cause of this issue?

I have CAT56500's on the LAN's.

[ropethic]

this is probably due to latency or the application itself. In order to efficiently utilize a WAN link the Client & Server TCP window size must come close to Bandwidth Delay Product, BDP which is

Bandwidth * Delay = BDP

Sp even though you have 1G conenctions you need to see what the latency is.

If I had a 10M circuit with 80ms latency, my BDP is 800K. I need to send data segments with window size across the WAN to fully utilize the link.

Check what your latency is.

The other probelm could be due to the appliaction itself. Suppose you have again 100ms of latency. If the appliaction is "chatty", meaning request are sent and waiting on a response before another request gets sent. So for every 10 request - response sequences 1sec is injected into response time. This is not a problem across a LAN with less than 1ms of latency, but across a WAN, application delay is present.

Combine this with network latency, propagation delay and you have a less than optimal application response and throughput.

[Joseph W. Doherty]

Besides Robert's excellent points, I'm surprised you're getting 200 Mbps if you have the traffic passing through 2811s. What's the CPU load look like on the 2811s? Also, since 2811s, I thought, only came with built-in 100 Mbps ports, what did you do to support gig?

[ronshuster]

I see how latency can be a problem but the latency between the two sites is about 2ms which is very low. However I do see your point about the application(s) being a bottleneck.

In fact I just checked the port on the core switch (CAT6500-LAN) that is connected to the C2800 (WAN) and I see that the outgoing traffic to the WAN is the same as it is on the WAN, so it looks like the WAN is not the problem.

I guess now I need to figure out why the LAN is only sending this amount of traffic over the WAN. For example, sometime we send 5GIG files across the WAN but the core switch will only send an average of about 150MG or so to the WAN, this includes the rest of the other traffic. Now the question is on the LAN.

Is there any command\utility to further understand the traffic in the core switch. Will that require a sniffer?

[ropethic]

You could enable netflow to see what trafifc is ocming out of the interface or span the port to a sniffer, to check for MTU size, retransmisisons, etc.

Is there any rate limiting policies' ACls, etc?

Is the data getting harware or software (CPU) switched? Check CPU utilizations and sho process to see if any particular datagrams are hammering CPU.

Also, you can run a load generator and send traffic to other end and check throughput.

After hours of course.

I had a simialr problem a couple of years ago with a backup across a T3 taking 8 hours. A sniffer trace revealed a UNIX server IP settings to 512 byte frame sizes, so check MTU.

[ropethic]

also, I know it is GIG, but make sure 1000 full duplex appears on show interface.

[Giuseppe Larosa]

Hello Roni,

2 ms latency between the two sites : are they in the same town ?

the latency on the WAN can be still the limiting factor.

Most of file transfers use TCP and to take advantage of high speed high latency links the end user hosts, the servers themselves, need to use the TCP extended window, if they don't the WAN usage can be fair less.

the TCP window scale factor for extended TCP window is described in RFC 1323

http://www.ietf.org/rfc/rfc1323.txt

the different OSes can use this option but not all use it by default (it has to be negotiated at TCP session setup).

This has to be done by servers managers.

We had a complain about a transfer over 622 Mbps POS on over 700 kms and it sorted out they were using the original TCP window of 65535 bytes.

Be aware of what Joseph has written: if you are able to increase the WAN usage you are probably going to face performance issues on the 2811s if it is not already happening.

Hope to help

Giuseppe