Latency variation

uthayaman · ‎04-21-2010

Hi,

We have around 8 branches connected over MPLS VPN(from service provider).We are routing voip traffic and web related traffic over this cloud and my call manger and other web application are hosted at one my main hub.

The strange thing is we are facing variable latency in the cloud.When i ping any branch from other branch there is a variation in the latency.

My service provider is giving me consistent ping response from his PE to PE.

I am pinging from CE(router WAN ip) to PE(Router WAN ip) and CE to CE.When we isolate LAN and ping CE of the other branch the ping response is stable.CE router CPU is not going beyond 3%.Bandwidth is also used only up to 10%.

I am facing this with all branch when i ping any other branch.

Can any one help me to fix the issue.

Reply from 172.21.0.10: bytes=32 time=128ms TTL=252
Reply from 172.21.0.10: bytes=32 time=63ms TTL=252
Reply from 172.21.0.10: bytes=32 time=140ms TTL=252
Reply from 172.21.0.10: bytes=32 time=206ms TTL=252
Reply from 172.21.0.10: bytes=32 time=155ms TTL=252
Reply from 172.21.0.10: bytes=32 time=34ms TTL=252
Reply from 172.21.0.10: bytes=32 time=37ms TTL=252
Reply from 172.21.0.10: bytes=32 time=106ms TTL=252
Reply from 172.21.0.10: bytes=32 time=271ms TTL=252
Reply from 172.21.0.10: bytes=32 time=133ms TTL=252
Reply from 172.21.0.10: bytes=32 time=33ms TTL=252
Reply from 172.21.0.10: bytes=32 time=124ms TTL=252
Reply from 172.21.0.10: bytes=32 time=84ms TTL=252
Reply from 172.21.0.10: bytes=32 time=68ms TTL=252
Reply from 172.21.0.10: bytes=32 time=96ms TTL=252

-uthay

Giuseppe Larosa · ‎04-21-2010

Hello Uthay,

have you implemented QoS with schedulers and LLQ low latency queue for VoIP traffic ?

Are the VoIP packets marked with DSCP EF?

in the SLA with ISP have you agreed on QoS different treatment for VoIP traffic also in the MPLS cloud?

the ping results can be worse then those of what VoIP packets experience.

In order to see if there are differences you should use extended ping and you should set TOS byte to 46 decimal

ping
Protocol [ip]:
Target IP address: x.x.x.x
Repeat count [5]:
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
>>>Type of service [0]: 46
Set DF bit in IP header? [no]:
Validate reply data? [no]:

if results for packets with DSCP marking the same as VoIP packets are good you are fine otherwise you have to work on it

Hope to help

Giuseppe

uthayaman · ‎04-22-2010

Thx for the reply...

We have not enabled any QOS in our network.Even we have requested to diable the QOS in mpls cloud when we faces thsi issue.

Want to share few more observation

We have two LANs one for voice and one for user PCs.When i shut down my user LAN on my router at the branches the ping response is fine.

ping
Protocol [ip]:
Target IP address: 172.21.0.26
Repeat count [5]: 1000
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]: 46
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 172.21.0.26, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (1000/1000), round-trip min/avg/max = 4/9/132 ms

I think i need to work on the QOS now.Can you refer me some document i can refer.

Giuseppe Larosa · ‎04-22-2010

Hello Uthay,

be aware that QoS tools and implementation are different on routers and switches

I suppose you have routers here is a link to CBWFQ and LLQ

http://www.cisco.com/en/US/docs/ios/qos/configuration/guide/config_wfq_ps6441_TSD_Products_Configuration_Guide_Chapter.html

Hope to help

Giuseppe

shailesh.h · ‎04-22-2010

Does this have any operational impact? Usually it is momentary characterstics goes away after some time. Mostly busy hours either in network / server or in local client system.
Please segregate your network in pieces (Requires little bit patience)

Local Site

Piece 1 -- user PC to local gateway - check the ping characterstics
Piece 2 -- local gateway to CE router - check the ping characterstics
Piece 3 -- CE router to PE Router (preferbly WAN interface if ping allowed) - check the ping characterstics

Remote Site

Piece 4 -- Server to local gateway - check the ping characterstics
Piece 5 -- local gateway to CE router - check the ping characterstics
Piece 6 -- CE router to PE Router (preferbly WAN interface if ping allowed) - check the ping characterstics

With this you can identify the problematic section and narrow down your pain area..

With regards,

Shailesh Hardas

Also share the output of traceroute will provide you where exactly the problem occur.

uthayaman · ‎04-22-2010

We are facing this almost every day.As mentioned if i ping local lan gateway from any PC/Server the latency is fine.I am facing the issue when i ping CE to CE with LAN connected on the router.

When my LAN-1/LAN-2 in connected on the routers Ping response to another WAN-2 from this router(WAN-1) gives latecny variation.

LAN-1---WAN-1------------------------cloud----------------------WAN-2---LAN-2.

As mentioned if i shut down my user LAN (Voice LAN on another vlan) everything works fine.

Since bandwidth is not highly utlized whether QOS plays any major role here.

Any how i will try to enable QOS as suggested....

shailesh.h · ‎04-22-2010

Of course, QoS will be recommended as you are using Voice in your network but problem you mentioned doesn't

seems related to congestion as you clearly stated that neither bandwidth utilized or CPU issue.

The only possibility to check the voice system (may be server / PBX) in depth for solution
Check local ping to Voice system as well

uthayaman · ‎04-22-2010

To figure out the issue i shut down my voice lan and tried, but the problem exist.So i think voice vlan is not creating the issue.I want to put up my thought around some kind of virus will cause this issue ?.

Need your suggestions....

shailesh.h · ‎04-22-2010

Yes, indeed you are having correct assumption but still need to factor which part of the segment is

impacted or impacting the network. The only method is method of isolation which i tried to explain in my previ

ous comment by doing step by step segment based ping test and traceroute output.

This will allow you to isolate the problematic segment and identify problematic system to traffic capturing tool..