DS3 performance issue.

[GregLear57]

We have a newly provisioned P2P DS3 circuit between HQ in Virginia and a remote site in Colorado.  Users in Colorado are reporting slow performance.  Verizon has allegdedly tested the circuit end to end and says they see the full T3 bandwidth error free.  HQ has a 7206 and remote has a 3725. There is also another DS3 between the HQ 7206 and a 3725 at a remote site in Maryland.  The configuration on both circuits is identical and the 3725s are using the same firmware and IOS versions.

The initial testing was to copy a large file between Colorado and HQ, then copy the same file between Maryland and HQ.  It took 10 times longer to transfer the file from Colorado.  To rule out other network equipment, we connected a laptop to the HQ router and copied the IOS images from the 3725s to the laptop via TFTP.  Again the transfers where 10 times faster.

We swapped the circuits on the HQ end and the issue still pointed toward Colorado.  We placed a spare 3725 and T3 card on the Colorado end with no change.

The interfaces and controllers on both ends are error free and the CPU and memory utilization on both routers are fine. Pings between HQ and Colorado are consitently 52ms.

We now have laptops connected to the routers on each end to run iPerf test. The router on the far end is isolated from the network to prevent any other traffic.

Here are the results:

TCP - HQ to Colorado single data stream for 60 second gives us 693 Kbits/sec.

TCP - HQ to Colorado 15 data streams for 60 seconds gives us 5.45 Mbits/sec.

TCP - Colorado to HQ single data stream for 60 second gives us 3.17 Mbits/sec.

TCP - Colorado to HQ 15 data streams for 60 seconds gives us 30.0 Mbits/sec.

My first question is, what could cause the disparity in troughput depending on which way the data is being sent?

My second question has to do with the troughput on a single TCP session.  I realize that latency directly affects the amount of bandwidth that you can utilize, but would 52ms drop a DS3 down to 3 Mbits/sec?

[Joseph W. Doherty]

Disclaimer

The   Author of this posting offers the information contained within this   posting without consideration and with the reader's understanding that   there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not   be construed as rendering professional advice of any kind. Usage of  this  posting's information is solely at reader's own risk.

Liability Disclaimer

In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out  of the use or inability to use the posting's information even if  Author  has been advised of the possibility of such damage.

Posting

My first question is, what could cause the disparity in troughput depending on which way the data is being sent?

Errors in one direction might.  Can you push full line-rate in both direction using UDP?

My second question has to do with the troughput on a single TCP session.  I realize that latency directly affects the amount of bandwidth that you can utilize, but would 52ms drop a DS3 down to 3 Mbits/sec?

Depends on receiving host's TCP RWIN.  A 16 KB RWIN with 52 ms would be about 2.5 Mbps.

[GregLear57]

Thanks for the response Joseph>

We are not seeing interface or controller errors on either side of the circuit so I am still mystified at the throughput disparity.  I would assume based on the results of trying to TFTP the IOS from the Colorado router to the laptop at HQ that we can't get anything close to DS3 speeds using UDP in that direction.

I have tryed using the UDP option with iPerf but my results are not good and I'm not sure that what I am seeing is indicative of the problem with the circuit, a problem with the test tool or a Layer 8 problem.  I'm not on site right now so I can't provide examples, but if I run iPerf using UDP with the defaults I will sometimes get a normal result but more often get a message that 1 datagram was received out of order.  By default iPerf will default to using 1.05MBit of bandwidth for UDP test.  If I try to increase the bandwidth I always get the warning "did not receive ack of last datagram after 10 tries" and I never see any results on the server side and I actually have to kill the server because all of the sessions seem to hang.

According to iPerf the window size is 64Kb.  I'm assuming that is what it really is and I would expect to see around 10Mbps for a single session.

[Joseph W. Doherty]

Disclaimer

The    Author of this posting offers the information contained within this    posting without consideration and with the reader's understanding that    there's no implied or expressed suitability or fitness for any  purpose.   Information provided is for informational purposes only and  should not   be construed as rendering professional advice of any kind.  Usage of  this  posting's information is solely at reader's own risk.

Liability Disclaimer

In    no event shall Author be liable for any damages whatsoever  (including,   without limitation, damages for loss of use, data or  profit) arising  out  of the use or inability to use the posting's  information even if  Author  has been advised of the possibility of such  damage.

Posting

TFTP ACKs its packets, so it generally will be even slower than FTP across a link with higher latency.  What I had in mind was a UDP test flow that just "blasts" packets at any defined rate to confirm you can get full line rate.  What I use myself for this kind of test is the freebie PCATTCP and its UDP option, as it runs on Windows systems.  I haven't used IPerf, but you should be able to do similar (I would hope).

I agree for a p2p LL, I would expect errors on one side or the other if there were link errors, yet I can image, with today's virtual everything under the covers, the underlying technology might be working incorrectly and it doesn't show any physical errors on your equipment.

It once took me 3 months to convince a tier one verndor something was amiss on a WAN link with a gigEthernet hand-off.  "Nothing wrong on our equipment", until they finally found out-of-date firmware on a line card was causing very intermittent packet drops.

Your results, as described, imply some kind of issue.  Often the more difficult issue, though, is convincing the vendor.

Since the line is p2p, and you able to keep it out of service, you might also try some loopback tests.  My expertise doesn't extend to L1 testing, and your posted question likely won't pull responses for "how to", but if you're not familiar with doing those kinds of test, you might post a new question for that including what other tests you might request from your vendor to insure the link is working as expected (i.e. don't phrase a new posted question as a performance issue).

Ideally, a good initial test tool would be one that can send UDP packets at defined rate, but notes whether received matched what was sent.  On a link not being used for any other traffic, with a p2p LL, if you're at or below line rate, I would expect there should be no packet sequencing issues, jitter or loss.