cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2918
Views
5
Helpful
17
Replies

Slow transfer (TCP) rate between sites

vincent-n
Level 3
Level 3

Hi all

Here's a scenario, I have two remote sites connected to HQ via an MPLS network. HQ has 200Mbps and the two sites both has 10Mbps links. Ping from HQ to HK (hong kong) is 180ms avg and ping from HQ to KL (kuala lumpur) is 121ms. Found that application responses at HK is a lot slower than KL. Did FTP transfer tests between HQ and the two sites and found that KL is able to achieve close to 450KByes/sec transfer (single session) while HK is doing ~180KBytes/sec at the most. Questions are:

1. I read about BDP (bandwidth delay product)http://en.wikipedia.org/wiki/Bandwidth_delay_product. IF my math is correct:

A. SIN = (10,000,000 * 0.121)/8 ~ 147.7KByes/sec transfer (on single sess)

B. HK = (10,000,000 * 0.18)/8 ~ 219KBytes/sec transfer (on single sess)

Anyone care to comment on th formula PLEASE !!!!! Something is wrong here.

Also, I could not find any references for IOS 'c2800nm-spservicesk9-mz.124-19b.bin' that would allow me to modify the TCP window size between clients on the LAN. I heard about the registry 'tweak' on MS Windows that allows larger TCP window but ....

1. I"m not sure if this is the underlying problems or not

2. Why is it that it only affect HK and not KL?

3. I'm using the same ISP but I know that the 'tail ends' are provided by different 3rd party vendor.

4. The config between KL and HL are almost identical (I used template)

5. The engineer at HK keeps on telling me that it's the TCP window size that cause the problem.

Any suggesion is greatly appreciated.

1 Accepted Solution

Accepted Solutions

Giuseppe Larosa
Hall of Fame
Hall of Fame

Hello Vincent,

when high RTT is involved an extended TCP window size can help.

see

http://www.ietf.org/rfc/rfc1323.txt

the native size of TCP window is a 16bit integer only 65535 octets can be sent before waiting for ACK.

To be noted that are the endpoints that need to negotiate an extended TCP window: the router work ends at OSI Layer3, so you cannot find a command on a router to enable usage of extended TCP window on end devices.

You rather need to tune TCP/IP stack.

see for Windows XP

http://www.psc.edu/networking/projects/tcptune/OStune/winxp/winxp_stepbystep.html

Note:

in data transmission the rates are expressed in bps or Mbps or Gbps

1Kbyte/s = 1024*8 bps

1Mbyte/s = 1024*1024*8 bps

and so on

Hope to help

Giuseppe

View solution in original post

17 Replies 17

Leo Laohoo
Hall of Fame
Hall of Fame

Q: Why is it that it only affect HK and not KL?

A: What is the result when you do a traceroute? Maybe there is network/fiber optic congestion going to HK. Try doing data transfer from your site to the HK & KUL routers (don't use the PC/server) and verify if the results are the same.

"The engineer at HK keeps on telling me that it's the TCP window size that cause the problem." < --- Do your own investigations.

Trace is looking good/correct from HQ to to SIN and HK. Nothing out of the usual hops and delays. I did TCP test from the routers using TTCPW and achieve the same results as the tests on PC. What I do find is that the tail end at HK office, I was able to achieve > 8Mbps transfer rate. Only when going between HK and SYD is when the xfer rate comes down to less than 0.5Mbps.

From HKG router (via the fiber optic) to SYD that's when the transfer goes down, is this correct?

yes, that's where the transfer rate quality declined. Funny thing is the ISP engineer is repoting that he's able to achieve ~ 6Mbps xfe rrate between SYD and HK using TTCPW PCs with the TCP window tweaked to accept maximum size.

Any line errors on your WAN interfaces? Ping and transfer SYD-HKG & HKG-SYD have the same result?

Gone through that already. No L1 problems and xfer speed between SYD-HK and HK-SYD is the same.

Giuseppe Larosa
Hall of Fame
Hall of Fame

Hello Vincent,

when high RTT is involved an extended TCP window size can help.

see

http://www.ietf.org/rfc/rfc1323.txt

the native size of TCP window is a 16bit integer only 65535 octets can be sent before waiting for ACK.

To be noted that are the endpoints that need to negotiate an extended TCP window: the router work ends at OSI Layer3, so you cannot find a command on a router to enable usage of extended TCP window on end devices.

You rather need to tune TCP/IP stack.

see for Windows XP

http://www.psc.edu/networking/projects/tcptune/OStune/winxp/winxp_stepbystep.html

Note:

in data transmission the rates are expressed in bps or Mbps or Gbps

1Kbyte/s = 1024*8 bps

1Mbyte/s = 1024*1024*8 bps

and so on

Hope to help

Giuseppe

Giuseppe

Thank you for your post. I was hoping that I do not have to resort to this answer. I have seen similar answers like yours on the web. XP default win size I think is 32KB and the max that you can set is 64K (65535). I heard that some people can actually goes higher than this (maybe this is what it means by GIANT packet at layer 2). Base on your answer, does it mean that companies will have to resort to changing default win size on their SOE (standard operating environment)? What a pain. I heard some people actually install WAN appliances just because of this reason. Please fell free to comment on any of my points. Thanks.

Hello Vincent,

the TCP extended window is a capability that is exchanged at TCP session setup between endpoints.

Actually I've seen other posts where someone said that actually Windows systems are passive on this option: they don't propose this option but that they agree on the other side proposal.

If this is true it should be enough to patch the servers only and not the user PCs.

For example most *nix based systems have the TCP extended window enabled by default.

Giant packets at layer2 are a different matter (related but different) and are handy in a campus to transfer huge amounts of data between servers.

Hope to help

Giuseppe

Joseph W. Doherty
Hall of Fame
Hall of Fame

"Anyone care to comment on th formula PLEASE !!!!! Something is wrong here."

BDP measure traffic "in flight", not "speed". Your calculations would calculate the receiving host's ideal receive window size for a single TCP flow. (In reference to one of your latter posts, XP default receive window varies based on host's interface bandwidth. Largest default value happens when host connected at gig. Receive window can be adjusted within registery. XP does support large receive windows [up to 2gig?]. BTW: Vista defaults to large windows.)

Although BDP being too small can certainly limit maximum TCP throughput, often a more common TCP performance killer is packet drops. Have you done any analysis of end-to-end TCP drop rates?

"Slower" application performance, except bulk data transfers, often has more causes than insufficient allowance for BDP. Again, packet drops can be critical, but transient queuing congestion and even just additional latency can be very noticable. Much more analysis is probably warranted, and even then, you might discover, it's just the additional distance based latency that's the root issue.

PS:

Not saying this is your problem, but an example case, had a problem on a MPLS network with 3rd party in mix. I noticed from analysis, 3rd party forgot to allow for extra 4 bytes of MPLS tag. (They also didn't notify fragmentation was needed.)

PPS:

BTW: You can adjust for BDP on your 2800, but that's only for the 2800 as a host, not for transit traffic.

Joseph

I think you've nailed the problem straight on the head with your analysis. I ran Ethereal (Wireshark) protocol analysis on FTP sessions and compare the differences between HQ-KL and HQ-HK. One thing I noticed when I do 'Analyze' & 'Expert Info' is that the HQ-HK sessions shows up hundreds of retransmission and duplicate packets.

I'm going to investigate a little bit more on tweaking the registry between 2 PCs and retest the FTP transfer test and see how I go. Will update everyone who looks at this post.

OK, here are the results:

- tweaking XP TCP windows size does make a diff. Before tweaking, xfer rate was around 0.7Mbits/sec on a single FTP session. After the tweak, it went up to 1Mbps/sec.

- Running multiple FTP sessions actually resulted in better xfer rate. I transfer 61.3MB of 10 files and was able to achieve 2.7Mbits/sec

- The application used to xfer the files also make a difference.

a. Windows file copy is the worse, estimated to take 33 minutes to xfer 61.3MB. Single 3.5MB took over 3 minutes to copy.

b. 10 x DOS FTP sessions for the 61.3MB took 187sec ~ 2.7Mbits/sec

c. 10 simultaneous sessions (61.3MB total) inside Filezilla took 127sec which is ~ 3.8Mbits/sec.

"Running multiple FTP sessions actually resulted in better xfer rate."

That's to be expected. Concurrent flows effectively ramp up bandwidth faster. If there are drops, faster recovery.

"a. Windows file copy is the worse"

Likely due to SMB. Newer version in Vista stack does much better.

PS:

Forgot to mention, another factor which can influcence bulk transfer performance is MTU. Insure you're able to send typical full size Ethernet packets end-to-end; also without fragmentation.

[edit]

PPS:

From your ealier post "One thing I noticed when I do 'Analyze' & 'Expert Info' is that the HQ-HK sessions shows up hundreds of retransmission and duplicate packets.", you really should try to improve this. Although you've found different applications and registry settings can improve bulk transfer performance, knowing this, alone, might not help resolve you users poor application performance.

ironman_xx
Level 1
Level 1

OK could you send the topology with visio and the current configuration.

and i will open conversation to solve the problem.

Regards

MOYAD BABIA

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco