Network performance question - TCP Window issue?

johnelliot6 · ‎04-29-2012

Hi Guys,

(Apologies if this is too off-topic for the cisco list)

Have 3 POPs A,B+C (A connected to B via 200Mb Eth, B connected to C via 250Mb Eth), all POPs are 7200's with 2960's and 3560's

Latency:

A -> B, 2 - 3 m/sec (No packet loss, minimal jitter)

B -> C, 12 - 13 m/sec (No packet loss, minimal jitter)

Performing wget or iperf I see the following (Have linux servers at each POP)

Pulling file from B->A (A's linux server is on 10/100 port) I get 9.59M/s (So basically 100Mbit)

Pulling file from C->B I get 8.30M/s, but it does fluctuate up+down considerably

Pulling file from C->A I get a flatline at 2.16M/s (Multiple sessions and all attain this speed, but no faster and doesnt fluctuate)

Im not seeing any excessive errors/drops nor any duplex issues on any of the switches - Is TCP Window size the cause of the huge speed difference I see from C->B vs C->A, even though the latency difference between the 2 is minimal?(i.e. 3m/sec?) or does the extra hop cause the issue?

thanks in advance for any advice.

lgijssel · ‎04-29-2012

You mention A's linux server is on a 10/100 port.

Are the other two on gig-ports perhaps?

Issues like this might be caused by buffer drops because of the differences in bandwidth.

Server-C to 7200 may be 1000Mb > 250Mb and B must adjust for 250 vs 200Mb/s.

How is the result when testing from A to C and what window sizes are you using?

regards,

Leo

johnelliot6 · ‎04-29-2012

Hi Leo - Yes the other 2 servers are on Gig ports

Think you might be on to something...in the other direction (C pulling from A) I am seeing 4.48M/s (Pretty much constant/no fluctuation).

Window sizes:

C:

cat /proc/sys/net/ipv4/tcp_wmem

4096 16384 3489792

cat /proc/sys/net/ipv4/tcp_rmem

4096 87380 3489792

A:

cat /proc/sys/net/ipv4/tcp_wmem

4096 16384 131072

cat /proc/sys/net/ipv4/tcp_rmem

4096 87380 174760

lgijssel · ‎04-29-2012

OK, an interesting test would be to change C to use 100Mb/s and repeat the tests.

(Just configure speed auto10/100 on the switchport)

The theory is that network devices generally keep pretty small buffers, at least too small to cater for a speed difference of 1000 to 250Mb/s. Please run a netstat -s on C before and after the test (no other services should be running ideally.

Then look for the increase in retransmitted segments and compare this the same output from A, this should provide a clue.

Perhaps you can achieve some improvement by increasing the number of big buffers in B but generally tcp should be able to adapt dynamically. I do not really understand the meaning of the output regarding the window sizes, after all I'm not a linux guru.

regards,

Leo