Slow transfer (TCP) rate between sites

Answered Question
Feb 15th, 2009

Hi all

Here's a scenario, I have two remote sites connected to HQ via an MPLS network. HQ has 200Mbps and the two sites both has 10Mbps links. Ping from HQ to HK (hong kong) is 180ms avg and ping from HQ to KL (kuala lumpur) is 121ms. Found that application responses at HK is a lot slower than KL. Did FTP transfer tests between HQ and the two sites and found that KL is able to achieve close to 450KByes/sec transfer (single session) while HK is doing ~180KBytes/sec at the most. Questions are:

1. I read about BDP (bandwidth delay product)http://en.wikipedia.org/wiki/Bandwidth_delay_product. IF my math is correct:

A. SIN = (10,000,000 * 0.121)/8 ~ 147.7KByes/sec transfer (on single sess)

B. HK = (10,000,000 * 0.18)/8 ~ 219KBytes/sec transfer (on single sess)

Anyone care to comment on th formula PLEASE !!!!! Something is wrong here.

Also, I could not find any references for IOS 'c2800nm-spservicesk9-mz.124-19b.bin' that would allow me to modify the TCP window size between clients on the LAN. I heard about the registry 'tweak' on MS Windows that allows larger TCP window but ....

1. I"m not sure if this is the underlying problems or not

2. Why is it that it only affect HK and not KL?

3. I'm using the same ISP but I know that the 'tail ends' are provided by different 3rd party vendor.

4. The config between KL and HL are almost identical (I used template)

5. The engineer at HK keeps on telling me that it's the TCP window size that cause the problem.

Any suggesion is greatly appreciated.

I have this problem too.
0 votes
Correct Answer by Giuseppe Larosa about 7 years 11 months ago

Hello Vincent,

when high RTT is involved an extended TCP window size can help.

see

http://www.ietf.org/rfc/rfc1323.txt

the native size of TCP window is a 16bit integer only 65535 octets can be sent before waiting for ACK.

To be noted that are the endpoints that need to negotiate an extended TCP window: the router work ends at OSI Layer3, so you cannot find a command on a router to enable usage of extended TCP window on end devices.

You rather need to tune TCP/IP stack.

see for Windows XP

http://www.psc.edu/networking/projects/tcptune/OStune/winxp/winxp_stepbystep.html

Note:

in data transmission the rates are expressed in bps or Mbps or Gbps

1Kbyte/s = 1024*8 bps

1Mbyte/s = 1024*1024*8 bps

and so on

Hope to help

Giuseppe

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (2 ratings)
Loading.
Leo Laohoo Sun, 02/15/2009 - 20:30

Q: Why is it that it only affect HK and not KL?

A: What is the result when you do a traceroute? Maybe there is network/fiber optic congestion going to HK. Try doing data transfer from your site to the HK & KUL routers (don't use the PC/server) and verify if the results are the same.

"The engineer at HK keeps on telling me that it's the TCP window size that cause the problem." < --- Do your own investigations.

vincent-n Sun, 02/15/2009 - 20:46

Trace is looking good/correct from HQ to to SIN and HK. Nothing out of the usual hops and delays. I did TCP test from the routers using TTCPW and achieve the same results as the tests on PC. What I do find is that the tail end at HK office, I was able to achieve > 8Mbps transfer rate. Only when going between HK and SYD is when the xfer rate comes down to less than 0.5Mbps.

Leo Laohoo Sun, 02/15/2009 - 20:57

From HKG router (via the fiber optic) to SYD that's when the transfer goes down, is this correct?

vincent-n Sun, 02/15/2009 - 21:04

yes, that's where the transfer rate quality declined. Funny thing is the ISP engineer is repoting that he's able to achieve ~ 6Mbps xfe rrate between SYD and HK using TTCPW PCs with the TCP window tweaked to accept maximum size.

Leo Laohoo Sun, 02/15/2009 - 21:17

Any line errors on your WAN interfaces? Ping and transfer SYD-HKG & HKG-SYD have the same result?

vincent-n Sun, 02/15/2009 - 21:19

Gone through that already. No L1 problems and xfer speed between SYD-HK and HK-SYD is the same.

Correct Answer
Giuseppe Larosa Sun, 02/15/2009 - 23:56

Hello Vincent,

when high RTT is involved an extended TCP window size can help.

see

http://www.ietf.org/rfc/rfc1323.txt

the native size of TCP window is a 16bit integer only 65535 octets can be sent before waiting for ACK.

To be noted that are the endpoints that need to negotiate an extended TCP window: the router work ends at OSI Layer3, so you cannot find a command on a router to enable usage of extended TCP window on end devices.

You rather need to tune TCP/IP stack.

see for Windows XP

http://www.psc.edu/networking/projects/tcptune/OStune/winxp/winxp_stepbystep.html

Note:

in data transmission the rates are expressed in bps or Mbps or Gbps

1Kbyte/s = 1024*8 bps

1Mbyte/s = 1024*1024*8 bps

and so on

Hope to help

Giuseppe

vincent-n Mon, 02/16/2009 - 02:27

Giuseppe

Thank you for your post. I was hoping that I do not have to resort to this answer. I have seen similar answers like yours on the web. XP default win size I think is 32KB and the max that you can set is 64K (65535). I heard that some people can actually goes higher than this (maybe this is what it means by GIANT packet at layer 2). Base on your answer, does it mean that companies will have to resort to changing default win size on their SOE (standard operating environment)? What a pain. I heard some people actually install WAN appliances just because of this reason. Please fell free to comment on any of my points. Thanks.

Giuseppe Larosa Mon, 02/16/2009 - 02:55

Hello Vincent,

the TCP extended window is a capability that is exchanged at TCP session setup between endpoints.

Actually I've seen other posts where someone said that actually Windows systems are passive on this option: they don't propose this option but that they agree on the other side proposal.

If this is true it should be enough to patch the servers only and not the user PCs.

For example most *nix based systems have the TCP extended window enabled by default.

Giant packets at layer2 are a different matter (related but different) and are handy in a campus to transfer huge amounts of data between servers.

Hope to help

Giuseppe

Joseph W. Doherty Mon, 02/16/2009 - 05:20

"Anyone care to comment on th formula PLEASE !!!!! Something is wrong here."

BDP measure traffic "in flight", not "speed". Your calculations would calculate the receiving host's ideal receive window size for a single TCP flow. (In reference to one of your latter posts, XP default receive window varies based on host's interface bandwidth. Largest default value happens when host connected at gig. Receive window can be adjusted within registery. XP does support large receive windows [up to 2gig?]. BTW: Vista defaults to large windows.)

Although BDP being too small can certainly limit maximum TCP throughput, often a more common TCP performance killer is packet drops. Have you done any analysis of end-to-end TCP drop rates?

"Slower" application performance, except bulk data transfers, often has more causes than insufficient allowance for BDP. Again, packet drops can be critical, but transient queuing congestion and even just additional latency can be very noticable. Much more analysis is probably warranted, and even then, you might discover, it's just the additional distance based latency that's the root issue.

PS:

Not saying this is your problem, but an example case, had a problem on a MPLS network with 3rd party in mix. I noticed from analysis, 3rd party forgot to allow for extra 4 bytes of MPLS tag. (They also didn't notify fragmentation was needed.)

PPS:

BTW: You can adjust for BDP on your 2800, but that's only for the 2800 as a host, not for transit traffic.

vincent-n Mon, 02/16/2009 - 14:49

Joseph

I think you've nailed the problem straight on the head with your analysis. I ran Ethereal (Wireshark) protocol analysis on FTP sessions and compare the differences between HQ-KL and HQ-HK. One thing I noticed when I do 'Analyze' & 'Expert Info' is that the HQ-HK sessions shows up hundreds of retransmission and duplicate packets.

I'm going to investigate a little bit more on tweaking the registry between 2 PCs and retest the FTP transfer test and see how I go. Will update everyone who looks at this post.

vincent-n Mon, 02/16/2009 - 19:45

OK, here are the results:

- tweaking XP TCP windows size does make a diff. Before tweaking, xfer rate was around 0.7Mbits/sec on a single FTP session. After the tweak, it went up to 1Mbps/sec.

- Running multiple FTP sessions actually resulted in better xfer rate. I transfer 61.3MB of 10 files and was able to achieve 2.7Mbits/sec

- The application used to xfer the files also make a difference.

a. Windows file copy is the worse, estimated to take 33 minutes to xfer 61.3MB. Single 3.5MB took over 3 minutes to copy.

b. 10 x DOS FTP sessions for the 61.3MB took 187sec ~ 2.7Mbits/sec

c. 10 simultaneous sessions (61.3MB total) inside Filezilla took 127sec which is ~ 3.8Mbits/sec.

Joseph W. Doherty Tue, 02/17/2009 - 04:48

"Running multiple FTP sessions actually resulted in better xfer rate."

That's to be expected. Concurrent flows effectively ramp up bandwidth faster. If there are drops, faster recovery.

"a. Windows file copy is the worse"

Likely due to SMB. Newer version in Vista stack does much better.

PS:

Forgot to mention, another factor which can influcence bulk transfer performance is MTU. Insure you're able to send typical full size Ethernet packets end-to-end; also without fragmentation.

[edit]

PPS:

From your ealier post "One thing I noticed when I do 'Analyze' & 'Expert Info' is that the HQ-HK sessions shows up hundreds of retransmission and duplicate packets.", you really should try to improve this. Although you've found different applications and registry settings can improve bulk transfer performance, knowing this, alone, might not help resolve you users poor application performance.

ironman_xx Tue, 02/17/2009 - 06:35

OK could you send the topology with visio and the current configuration.

and i will open conversation to solve the problem.

Regards

MOYAD BABIA

vincent-n Thu, 02/19/2009 - 21:37

Ok, attached are the Visio diagram and a PDF version of the same diagram. Also attached are the configs. Thanks for your suggestions. Files are posted in two parts due to max 3 attachments per post.

vincent-n Thu, 02/19/2009 - 21:41

And here are the configs. Please note that the carrier's engineer was able to achieve more than 6Mbps transfer rate between 2 x Linux PCs located at SYD and HK POPs. His delays (in ms) is just the same as what I've got and we're running over the same circuit internationally. Like I mentioned before, I was able to achieve slightly higher results by tweaking the registry with the TCP settings but just simply would not be able to achieve similar result as the engineer.

Attachment: 
Joseph W. Doherty Fri, 02/20/2009 - 04:29

You need to run down the problem you were seeing in your Wireshark snifs.

Since the carrier engineer is seeing expected perforamance and you're not, issue might be between POP and your HK site.

As example, it took me two months to get a 1st tier carrier to find a problem that I believed was in their network (and it was). Turned out to be L2 issue caused by buggy firmware on a line card on one of their devices. Device was incorrectly dropping packets.

Have you tried performance tests between KL and HK?

If you're using some variant of TTCP, can it provide "verbose" stats, like packet retranmissions?

Actions

This Discussion