High latency issue in LAN....

Unanswered Question
Aug 10th, 2010

Hi All,

This is my 1st post in this forum.

We have different vlans for different processes in our organization & all the vlans are created on core switch (Cisco 3750G). When im pinging any VLAN's gateway in our LAN network im getting high latency. This has not cause any problem in our network so far but i want to resolve this issue as early as possible. I have checked config in our L3 switch but found no issue.. have attached snap shot of IT vlan gateway........

Network Architecture :--

Machine-->Access Switch-->Core Switch-->Cisco ASA-->Packet Shaper-->Radware-->Internet Router-->Internet Link

Any Soln. on this issue? Thanx in advance.

Gateway.JPG

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
vmiller Tue, 08/10/2010 - 07:29

First things first.

You need to run a traceroute that identifies every hop. At first glance it appears you have some form of asynchronous routing going on.

amit.rane Tue, 08/10/2010 - 07:36

Thanx for ur reply.

Im getting high latency only while pinging gateway, ping response to other devices is normal (1ms). I dont think trace route will help to troubleshoot this issue. Any other soln????

vmiller Tue, 08/10/2010 - 08:03

Where is the source device relative to the default gateway? If its on the same switch,

then what changed in the config ?

amit.rane Tue, 08/10/2010 - 08:19

Source machine is connected to access switch (Cisco 2960) and access switch is connected to core switch. I also tried pinging other ip in same vlan and got reply of 1ms.

vmiller Tue, 08/10/2010 - 10:28

Is the behavior the same from both the core and access switch ?

is this a new issue ? what, if anything has changed on your network.

robindisla0214 Mon, 10/24/2016 - 13:24

I have the same problem in my network the to the SVI interface is a 500ms and to other pc and other or the same vlan is 1ms the latency is only for the default-gateway from every vlan.

Attachment: 
ROGER FERNANDO ... Tue, 08/10/2010 - 10:41

Hola.

favor de confirmar si los puertos tanto del switch de acceso como el de core estan configurados como sigue:

switchport trunk encapsulation dot1q

switchport mode trunk

switchport trunk allowed vlan all

switchport trunkn native vlan 1

spanning-tree link type point-to-point

el switch de acceso debe tener un default-gateway:

default-gateway (ip address) --- esta ip address debe ser la ip address de la vlan 1 del switch core.

el switch core debe tener configurado:

ip routing

Consulta:

si desde el switch core haces ping al host conectado al switch de acceso tienes el mismo comportamiento en cuanto al ping ?

A la espera de tu respuesta.

Attn.

amit.rane Tue, 08/10/2010 - 11:32

@Roger --> didnt get u..use english lang

.

@Vincent --> its the same issue i have been facing this from last 2 days & haven't done any config changes on core switch in last 2 days.......also didnt find any issue with access switch....i guess its something related to core switch becoz when im pinging other devices in network im getting normal responsw 1ms.....

vmiller Tue, 08/10/2010 - 14:18

i don't understand secondary addressess that are in the same subnet.

your ping responses indicate the variance in time is from the primary address

for a given vlan. what happens when you try and reach the internet from a device

directly on the core switch ? or the next device upstream from the core /

jaison.dev Tue, 08/10/2010 - 21:31

Hi Amit,

Can you also check the

     ping response from the L3 to machine

     ping response to L3 in any other VLAN i.e. MGMT VLAN from an L2

Cheers,

DJ

vragotha Tue, 08/10/2010 - 23:01

It appears that you are pinging the SVI of VLAN 167 on the core switch.

Pinging the SVI (VLAN interface) means that it is handled by software and sent to the CPU for processing; hence the delay in ping response.

If you are seeing high ping response times when pinging devices in a VLAN, then you may have a problem in the network or the switch

amit.rane Wed, 08/11/2010 - 06:05

@Vincent -- we have 2 core switches which are working in single bundle that is the reason we have 2 gateways. im facing this issue only when im pinging primary gateway. Yes i tried connecting machine directly to core switch but faced same problem.

@Jaison -- 1) When i ping any machine in same network from L3 i get arnd 200ms response in between.                                                                       

                 2) Got same response when i tried pinging L3 from L2.

@Vijay -- Thanx for ur reply. im trying to ping gateway of my machine i.e interface ip of VLAN 167.

Scott Bradley Wed, 08/11/2010 - 09:49

Some useful commands to help.

You can check for  spanning tree state changes if redundancy is involved as far as  redundant l2 paths.  This could cause issues if happening often.

     

show spanning-tree detail | inc ieee|occur|from|is exec

You can check for interface input queue drops as well on the switches to see if interfaces are getting overrun which
might lead to lost packets.


sh int | in Input|line


This will show you all interfaces for comparison to problem vlan 167

vinod.agrahari Thu, 08/12/2010 - 04:16

Hi Amit,

Could you please check the speed and encapsulation of both the switches connected(Core and other) and check the CPU Utilization whenever you ping both the gateway.

Change the port configured as a trunk and gateways to check the physical problem port of the Switch

Regards

Vinod Agrahari

DouglasScott99 Thu, 08/12/2010 - 05:17

Let me re-state the "problem".

Ping Round Trip time from a PC to a "gateway"  (perhaps a Cisco 2960) alternates between 3ms and 200ms.

All other ping tests on network are 3ms or better.

"i guess its something related to core switch becoz when im pinging other devices in network im getting normal responsw 1ms"

The reason will be that the "core switch" is busy doing something else and takes a little time to get round to replying to the probe. On the face of it this does not seem very worrying to me. The 29xx switches do not have particularly fast CPUs and a slow ping response is not unexpected.

If this is a new behaviour then you might want to have a look to see what might be the cause. If so - check for high CPU on the 2960.

sh proc cpu is the place to start.

Note particularly that the 2960 does *not* use the CPU for switching traffic, only for managing itself. Traffic is switched in dedicated hardware. This is why I have few concerns.

You might I suppose want to collect the 1s cpu utilisation from the device with snmp.

As I have said, does not seem very worrying to me.

amit.rane Thu, 08/12/2010 - 09:10

@Vinod -- CPU utilization on L3 & L2 switches is normal (between 10-20 %). Also no physical prob. found on switch port.

@Douglas -- our core switch is of 3750G series. Im getting high latency only when i ping gateway of any vlans which are created on core switch. When i ping to other devices which are behind or after core switch i get proper response 1ms.

@Bradley-- NO issue found with the spanning tree. Core switch is a root switch in our network.

DouglasScott99 Thu, 08/12/2010 - 16:03

Hi,

"@Douglas -- our core switch is of 3750G series. Im getting high latency only when i ping gateway of any vlans which are created on core switch. When i ping to other devices which are behind or after core switch i get proper response 1ms."

I am saying that 200ms *IS* a proper response.

There is not necessarily any problem. If you check the cisco web site you will find NO indication of any expected ping response time.

Can you post the output from.

sh proc cpu.

By the way - I have already mentioned this but you have not responded with any data.

It also might be VERY interesting to have the results of SNMP probes of the 1s CPU load.

I have also already mentioned that.

HA! I have just remembered that BPDUs are sent every 2 seconds. Maybe you have some weird STP design with excesive BPDUs?

marcaccini Thu, 09/08/2011 - 12:22

I have to agree with DouglasScott99 on this one, you need to provide more information.

But I can say, that in my experience when you see an output like the one you included in the original post you have asymmetrical routing occuring.

The best thing to do here is to complete a number of traceroutes to the destination IP from your host. You will probably see that it takes two different paths. One path with the low ms response time and one path significantly longer.

It would be easier to troubleshoot your problem if you provided more details about the infrastructure design and results of these traceroutes.

glen.grant Thu, 09/08/2011 - 19:09

  Ping response is low on the priority list of things the switches do .  Do not use this as more than a rudimentary  test.  Ping to devices through the switch itself on the subnet and not a given L3 interface or SVI .

thphoenix Tue, 08/13/2013 - 07:18

The problem (which I have chased for years) is related to some sort of timing failure with the TCP/IP protocols..

The following seems to clear it up normally: (slight variation for Windows 2000, 8)

open an Administrator command prompt (press Windows key, type CMD and hold left Ctrl, left Shift and press enter, answer Yes)

type in the following commands: (Make note of your TCP/IP address and settings first if static.)

IPCONFIG /FLUSHDNS
ARP -D
GPUPDATE /FORCE
​NETSH INT IP RESET NULL
NETSH WINSOCK RESET
shutdown -r -t 0 -f

First, we clear DNS cache (important, and helpful)

Second, we clear ARP entries. (arp -a will list all connections, including MAC addresses. very handy to know)

Third, we force a PC/User group settings refresh (will happen at next boot)

Fourth, we reset the TCP/IP protocol settings to defaults. (This actually fixes some bottlenecking in protocol layers)

Fifth, we reset winsock (fixes timing issue, and both sections get resynced at boot. MUST BOOT at this point.

Last, we force an immediate reboot. (another handy command).

Once the system reboots, you are now in DHCP mode, and timings have been reset.

So why does this happen? After years of tracking how this gets caused, or better ways to fix it, the people who use video conferencing seem to make this fail more often, but I still can't figure out what is really happening yet.

This is the simple solution. If this fails, go into the protocol list for any adapter, click install, click have disk and point to "c:\windows\inf" and select TCP/IP. This is roughly the same as the procedure above, but changes other entries. Good for Win2k.

The last solution that fixes this when all else fails, go into Device Manager, remove the network adapter(s) [Leave drivers, Vista+], then immediately reboot.  -- on reboot, drivers are reinstalled, new UniqueID for device generated, and all settings reset, firewall settings added to fresh installation, and other strange issues fixed.

Sorry this is 2 years to late, but I am dealing with this a lot lately, and can't find a solution yet either.

-- Note, "Null" is the output filename, which is normally listed as c:\temp.txt, but some windows versions won't allow writing to the root, and can be replaced with c:\windows\temp\temp.txt, since this directory is likely to exist.

Good luck, and I do hope to figure this out one day.

Joseph W. Doherty Thu, 09/08/2011 - 17:12

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

In general, neither switches or routers are designed to provide highly accurate ping response values.  If your platform supports it, you might configure a SLA responder which should provide a more accurate ping response (if responding to a SLA request).

Actions

This Discussion