I have high latecy spkes on our 3560 switch. There is hardly any traffic on our network so I know it's not because of traffic. The switch latency all of the sudden shoots up to 100ms for no reason. What could be causing this problem?
A lot of things can cause latency:
A faulty LAN or WAN connection. faulty hardware, high CPU usage, bad cables, traffic shaping or some other form of QoS, etc...
Perhaps you can tell us a bit more about when and how you experience latency? Im assuming the 100ms you mention is the round-trip-time for a PING test. If so, what are you PINGing? What are the endpoints? Is the PING gpoing across a WAN or LAN link? Is it across a VPN tunnel? etc...you get the picture.
The switch is connected on my local area network. it has a fiber uplink to my core switch. There are no VLANs configured on this switch, it's not a faulty cable - there are no CRC errors, it's not faulty hardware of any sort, no errors anywhere on the switch, CPU is under 10% utilization and no QoS of any sort. Most of the time the connection is under 1ms, but about three times during the day it jumps to 100ms. I checked for traffic and it's the same amount as the rest of the time. The ping is round trip on my LAN link from my desktop. My desktop connects to the Core Switch - 6509SW - which connects to the 3560 distribution switch. I know it's not the 6509 since this problem is not present with the rest of the switches.
The route is from my desktop to the Core Switch, 6509 (which does not have any other problems with any other switch,) and to the 3560 switch that spikes with 100 ms delay.
From desktop to 6509 to 3560 switch.
Pinging a network device is not a good way to measure latency , you need to ping thru the switch to a client hung off the end to measure that . Network devices generally will put ping response low on the list of things to do when it trying to do other things.
Beyond Glen's mention that some devices respond to pings when they decide to, you write "There is hardly any traffic on our network so I know it's not because of traffic." Are you certain?
Reason I ask, many network tools or stats that measure/report traffic often only present average usage over some time period and might conceal microbursts. Unless you have packet captures showing what was on the wire at the same time as your ping, it's difficult to know traffic wasn't an issue.
I have SolarWinds monitoring the traffic and response time of each device. It evens shows traffic decreasing when latency rises on that same switch. I've replaced cables, checked all the hardware, actually replaced it with another 3560 and the same symptoms show. I've read on the internet that it may do to flow control. Is there a way to enable debugging on the switch so it detects when latency occurs?
Can't say for sure in this instance, but it is possible for traffic to decrease with increase of latency. Both could be a symptom of global synchronization of TCP drops. Or to quote Cisco "Global synchronization occurs as waves of congestion crest only to be followed by troughs during which the transmission link is not fully utilized. Global synchronization of TCP hosts, for example, can occur because packets are dropped all at once. Global synchronization manifests when multiple TCP hosts reduce their transmission rates in response to packet dropping, then increase their transmission rates once again when the congestion is reduced. " However, you should see packet drops.
No packet drops show any where in any of my devices, I wish it was that simple. This morning the switch showed 253 ms delay response time, this is really a concern, can anyone give me some direction of what I should troubleshoot. Could this be a malicious attack of some sort?
As Glen mentioned in his post, if you're pinging a network devices, they don't always repond promptly. Try either pinging through the device or using SLA tests with a responder. See if the high pings still occur.