T-1 latency (no errors and low traffic) issue

mbroberson1 · ‎07-01-2009

Trying to investigate a latency issue with a T-1.

Senario:

You have a full point-to-point T-1 to a remote site (approx 3 miles away). The circuit has 5% to 10% load max. When you perform a ping from the router at your main site to the branch office router you get very high 1000ms to 2000ms response times and averages of 200ms to 800ms. The circuit has no errors and you had the ISP run full testing on the circuit. If your equipment checks good then where is a next good place to start looking for the issue? This latency is even experienced in the early morning hours when the remote beanch is closed with virtually no load on the circuit.

Thanks,

Brandon

route2null · ‎07-01-2009

To clarify do you see the latency only when using ping or do you also experience it with application traffic.

pciaccio · ‎07-02-2009

To understand the issue you have can you please provide the output of the SHOW CONTROLLER T! command from both sides. Also can you please do a SHOW PROCESS CPU for both routers. This will assist us to understnad your issue a little better. Also do you have a configuration that you can provide to us from both routers...Thanks...

Joseph W. Doherty · ‎07-02-2009

"This latency is even experienced in the early morning hours when the remote beanch is closed with virtually no load on the circuit. "

If the branch is closed, why is there any load?

What you might try, while the branch is closed, is stop routing(?) across the link. While the ciruit is logically out-of-service, ping between the two end-points of the T-1 and see what latency is then.

Also while the branch is closed, bring the T-1 back into service, and use a traffic generator that can send UDP traffic at a fixed rate. Try sending at 25%, 50%, 75%, 99% and 125%. Check whether the other side is receiving the expected rate (i.e. 25, 50, 75, 99 and 100). Ping at the various load levels. Load both directions individually and together.

These tests will provide much more information whether the link is working as expected. I.e. Other side should receive 25% to 100% of packets, ping times should only go "wild" when you push at 100% or more.

mbroberson1 · ‎07-02-2009

Hi Joseph,

The ISP checked the headquarters to CO leg of the link and did find a wiring issue in the CO. The ISP did see errors in their testing that alarmed them to this, but it seems strange to me that there are no input/output errors on either HUB or headquarters router.

Thanks,

Brandon

pciaccio · ‎07-02-2009

Your interface may not show errors, however on the SHOW CONTROLLER T1 results may show errors. Look at the results from both sides...

mbroberson1 · ‎07-02-2009

Hear are the "sh controllers" from both sites. According to the ISP (AT&T) the issues was on the CO to headquarters link.

Thanks,

Brandon

patrickvanham · ‎07-02-2009

There is Frame Loss on both sides. This means you receive data but data is not sent back. The provider should be able to see LOS on their equipment coming from your routers. I'd advise loopback testing on the routers both in the branch and HQ, if possible for a 24 hour period during a weekend.

pciaccio · ‎07-02-2009

One thing I see along with the other responses is that one side (Branch) is set for Line clocking of LINE. The other side (Headquarter) is set for INTERNAL. If the HQ is set for Internal then that router is supplying the clock. This unit is probably not a good source of clock. The T-1 provider will almost always provide a good clock on their lines (unless ordered not to). I would recommend that you change the clock source on the HQ router to be LINE and not INTERNAL...

scottmac · ‎07-02-2009

Check the utilization of the router (not just traffic, the show proc, sh proc mem, etc) ... you could have a runaway process on one router or the other that's reducing the resources, or severe memory fragmentation (a reboot of the router would cure this, for a while).

Assuming you have integrated CSU/DSUs, the cable from teh back of the router to the "SmartJack" could be in bad shape, on either or both ends or can be the wrong type of cable, or be too long (or be too long for the cable type ... FYI: Cat{anything} is NOT certified for T1 applications, it's OK for short, but long runs are bad ... you need "T1 Cable" also called "Premesis Cable")

Those would be my first suspicions.

Good Luck