Latency variation

Unanswered Question
Apr 21st, 2010
User Badges:

Hi,


We have around 8 branches connected over MPLS VPN(from service provider).We are routing voip traffic and web related traffic over this cloud and my call manger and other web application are hosted at one my main hub.


The strange thing is we are facing variable latency in the cloud.When i ping any branch from other branch there is a variation in the latency.

My service provider is giving me consistent ping response from his PE to PE.

I am pinging from CE(router WAN ip) to PE(Router WAN ip) and CE to CE.When we isolate LAN and ping CE of the other branch the ping response is stable.CE router CPU is not going beyond 3%.Bandwidth is also used only up to 10%.


I am facing this with all branch when i ping any other branch.


Can any one help me to fix the issue.


Reply from 172.21.0.10: bytes=32 time=128ms TTL=252
Reply from 172.21.0.10: bytes=32 time=63ms TTL=252
Reply from 172.21.0.10: bytes=32 time=140ms TTL=252
Reply from 172.21.0.10: bytes=32 time=206ms TTL=252
Reply from 172.21.0.10: bytes=32 time=155ms TTL=252
Reply from 172.21.0.10: bytes=32 time=34ms TTL=252
Reply from 172.21.0.10: bytes=32 time=37ms TTL=252
Reply from 172.21.0.10: bytes=32 time=106ms TTL=252
Reply from 172.21.0.10: bytes=32 time=271ms TTL=252
Reply from 172.21.0.10: bytes=32 time=133ms TTL=252
Reply from 172.21.0.10: bytes=32 time=33ms TTL=252
Reply from 172.21.0.10: bytes=32 time=124ms TTL=252
Reply from 172.21.0.10: bytes=32 time=84ms TTL=252
Reply from 172.21.0.10: bytes=32 time=68ms TTL=252
Reply from 172.21.0.10: bytes=32 time=96ms TTL=252


-uthay

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Giuseppe Larosa Wed, 04/21/2010 - 23:49
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Uthay,

have you implemented QoS with schedulers and LLQ low latency queue for VoIP traffic ?

Are the VoIP packets marked with DSCP EF?

in the SLA with ISP have you agreed on QoS different treatment for VoIP traffic also in the MPLS cloud?


the ping results can be worse then those of what VoIP packets experience.



In order to see if there are differences you should use extended ping and you should set TOS byte to 46 decimal



ping
Protocol [ip]:
Target IP address: x.x.x.x
Repeat count [5]:
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
>>>Type of service [0]: 46
Set DF bit in IP header? [no]:
Validate reply data? [no]:


if results for packets with DSCP marking the same as VoIP packets are good you are fine otherwise you have to work on it


Hope to help

Giuseppe

uthayaman Thu, 04/22/2010 - 00:36
User Badges:

Thx for the reply...


We have not enabled any QOS in our network.Even we have requested to diable the QOS in mpls cloud when we faces thsi issue.


Want to share few more observation


We have two LANs one for voice and one for user PCs.When i shut down my user LAN on my router at the branches the ping response is fine.


ping
Protocol [ip]:
Target IP address: 172.21.0.26
Repeat count [5]: 1000
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]: 46
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 172.21.0.26, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (1000/1000), round-trip min/avg/max = 4/9/132 ms


I think i need to work on the QOS now.Can you refer me some document i can refer.

shailesh.h Thu, 04/22/2010 - 04:19
User Badges:
  • Bronze, 100 points or more
  1. Does this have any operational impact? Usually it is momentary characterstics goes away after some time.  Mostly busy hours either in network / server or in local client system.
  2. Please segregate your network in pieces (Requires little bit patience)

          Local Site

    1. Piece 1 -- user PC to local gateway - check the ping characterstics
    2. Piece 2 -- local gateway to CE router - check the ping characterstics
    3. Piece 3 -- CE router to PE Router (preferbly WAN interface if ping allowed) - check the ping characterstics

          Remote Site

    1. Piece 4 -- Server to local gateway - check the ping characterstics
    2. Piece 5 -- local gateway to CE router - check the ping characterstics
    3. Piece 6 -- CE router to PE Router (preferbly WAN interface if ping allowed) - check the ping characterstics


    With this you can identify the problematic section and narrow down your pain area..


    With regards,


    Shailesh Hardas


    Also share the output of traceroute will provide you where exactly the problem occur.

    uthayaman Thu, 04/22/2010 - 05:10
    User Badges:

    We are facing this almost every day.As mentioned if i ping local lan gateway from any PC/Server the latency is fine.I am facing the issue when i ping CE to CE with LAN connected on the router.

    When my LAN-1/LAN-2 in connected on the routers Ping response to another WAN-2 from this router(WAN-1) gives latecny variation.


    LAN-1---WAN-1------------------------cloud----------------------WAN-2---LAN-2.


    As mentioned if i shut down my user LAN (Voice LAN on another vlan) everything works fine.




    Since bandwidth is not highly utlized whether QOS plays any major role here.


    Any how i will try to enable QOS as suggested....

    shailesh.h Thu, 04/22/2010 - 07:34
    User Badges:
    • Bronze, 100 points or more

    Of course, QoS will be recommended as you are using Voice in your network but problem you mentioned doesn't

    seems related to congestion as you clearly stated that neither bandwidth utilized or CPU issue.


    • The only possibility to check the voice system (may be server / PBX) in depth for solution
    • Check local ping to Voice system as well
    uthayaman Thu, 04/22/2010 - 10:56
    User Badges:

    To figure out the issue i shut down my voice lan and tried, but the problem exist.So i think voice vlan is not creating the issue.I want to put up my thought around some kind of virus will cause this issue ?.


    Need your suggestions....

    shailesh.h Thu, 04/22/2010 - 15:07
    User Badges:
    • Bronze, 100 points or more

    Yes, indeed you are having correct assumption but still need to factor which part of the segment is

    impacted or impacting the network. The only method is method of isolation which i tried to explain in my previ

    ous comment by doing step by step segment based ping test and traceroute output.


    This will allow you to isolate the problematic segment and identify problematic system to traffic capturing tool..

    Actions

    This Discussion

    Related Content