cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1354
Views
22
Helpful
29
Replies

Latency question

tonyp8581
Level 1
Level 1

Hi,

i need an expert opinion.

Where I work, we have a private MPLS network that connects over 25 remote sites to our IT center (Server Farm). We have a 100 Mb link between the IT center and the cloud . All other sites (Montreal and Quebec area) vary in speed.

The ISP that handles our MPLS network just install a new remote site situated in Toronto at 100 Mb. To confirm that were getting the promised throughput, I use a tool called Iperf. Simply, it's a Client-Server DOS application that sends packets and measure the throughput of the network.

Iperf is telling me the throughput between Toronto and IT center is around 30 Mb.

According to me, i should be hitting the 70 Mb mark. Speaking with my ISP, their telling me it's because of the distance (latency) that i'm getting this result (they can't do anything about it). I'm having a hard time believing this explanation. All my other sites including quebec city (running at 5 Mb) respect the speed that were paying for… the problem is with the Toronto site.

Starting from the IT center, tracing to Toronto, i go through 5 hops and quebec city I go through 4 hops. Can one more hope make that much of a difference ?

Pinging Toronto, I get a average of 23 ms. Pinging quebec city, I get a average of 17 ms.

I agree that the packet has to travel a longer distance, but I wasn't expecting that much of a low throughput.

Any opinions is greatly appreciated.

29 Replies 29

marikakis
Level 7
Level 7

Hello,

The distance is a factor, but not that much as you already expect. It is more important in interactive applications than it is in simple file transfers. In a simple file transfer you can reach pretty high rates almost regardless of distance, because once the network pipe is full (Imagine the network path as being filled with water. It takes some time to fill it, but as long as it is full, it doesn't matter how long the pipe is.), you keep receiving packets at high rates.

Try to make sure that you are measuring the throughput correctly before blaming the ISP. Are you using Iperf with TCP traffic or UDP? With TCP you will normally get slightly less throughput than with UDP. TCP needs you to keep the traffic running for some minutes in order to see your best throughput (The logic behind this is to minimize the initial cost in time and throughput to fill the pipe with traffic and give TCP the chance to increase its window). Also make sure that you have no congestion issues in the LANs where traffic sending/receiving PCs are located. In addition, make sure that those PCs are not doing much of other work, especially one that involves networking.

Kind Regards,

M.

marikakis
Level 7
Level 7

One more thing: The difference is quite possible to have to do with the time you let Iperf running. It doesn't take much time to reach 5 Mbps (as you did with another site), but it will take more for the applications to "trust" the capability of sending data at 100 Mbps.

Hi marikakis,

thx for your quick reply,

When i use TCP i get about 30Mb/s. UDP runs around 90 Mb/s.

Obviously, most applications use TCP.

i understand that UDP has less overhead, but i still considered that my link is not running at it's full capacity.

i will let iperf run a little longer.

i'll post the results.

thx

The difference between TCP and UDP is not just headers. It is much more. Your UDP performance is perfect and you cannot expect more from the network (Remember that you will not see 100 Mbps because of lower layer header overhead). It does show you the network performance and it is great! If you want to see even more from the network with TCP, try sending traffic from multiple PCs to multiple PCs.

here are my result:

sending 100 MB = 32 Mb/s throughput

sending 200 MB = 33 Mb/s throughput

sending 500 MB = 26 Mb/s throughput

right now i can't test it with multiple workstation.

i never expected to lose that much bandwidth when the packet has to travel over 500 Km.

Montreal and Quebec is around 200 Km and i don't lose any bandwidth.

I don't know if it's a urban legend, i heard sometimes ISP lower the bandwidth threshold (TCP).

Can they do that??

thx

Of course they can. And often their infrastructure is largely overbook and they have big problem to maintain the service levels sold to customers. You should really spend one day a done into a SP's noc to see all the things that go behind curtains, as you can image, not all are nice.

Hello,

I don't know about the urban legend that you mention.

What command did you use in the CLI for those experiments? How much time did you have those sessions running? This is not a lot of data that you have sent (not enough total MBytes for you to be able to reach high rates in 10 seconds, even if TCP wouldn't slow start). Remember, each experiment is different, you have to make each session run for some minutes, not a lot of successive sessions. It is not the same thing. Each session has the same bad luck at the beginning. There is an option in the CLI to make an individual TCP session run longer, but I do not remember it and I do not have Iperf downloaded right now. You could explore the CLI help to quickly find this out.

Also, consider setting TCP window size (iperf -s -w )according to: http://dast.nlanr.net/Guides/GettingStarted/TCP_window_size.html#computing

Kind Regards,

M.

Now that Paolo answered I see what you were getting at. I have been in SP NOC for a couple of years, but we were not that bad. Overbooking is done based on statistics, but we always tried to be able to satisfy the peak hour traffic (I know they still do). I mean, even though the theoretically sold could be higher, the real traffic would have to pass through for us to be able to survive the customer complaints and rumours such as those you mention. Sometimes it wouldn't be possible due to delays in circuit delivery or some other technical issue. Then again, it all depends in the competition between SPs in your area. Anyway, let's try to make sure we are doing our best to measure the performance before starting to spread any rumours.

I will try to extend the duration of the test to see if i get different result.

thx again for your help.

Tony

I let the test run for 15 minutes with window size set to 64Kb.

Transfer 1.36 GB : 19.6 Mb/s

as you can see, the throughput got worse.

so basically, right now i have to conclude that my ISP can't do anything about this issue, even if i disagree with their explanation...

i'll keep reading. if i find anything i'll post it..

if anybody has any suggestion, i would greatly appreciated.

thx again

Hello,

We had to do a great deal of Iperf experiments when we took a networking class about 3 years ago. I took a look in our reports (creepy :-) and I see in the graphs we plotted the following information that might interest you:

2 parallel connections from a single iperf client machine to a single iperf server machine can hardly go above 80 Mbps in 10 seconds with a window size less than 18 KByte. You need more than that to see good results and take into account that the experiment was in a 100 Mbps LAN with only us playing around. So, no question about how bad it can be in the WAN in only 10 seconds.

Start the server and then start 2 clients on the client machine that you are already using. Try to press enter in the client command windows as simultaneously as possible :-) (Still haven't found out how we can extend duration of individual session.) Remember to add to the end of the command the window option (20K or more): -w 20K

Kind Regards,

M.

p.s. Just saw your response. I will think about it, but at a first glance the results are pretty disappointing :-(

Hi,

can't believe it either. i have to lease a 100 Mb/s link to get back a 30Mb/s perfomance. wow...

FYI: i added the switch -t:seconds to extend my session.

thx for your help. If i find anything else i will post it.

May I ask something else? Just to make sure we are not troubleshooting Iperf instead of network performance. Have you tried a "heavy" file transfer from one site to the other over this connection? Do you see any performance issues in any of your activities besides the Iperf results?

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: