VLANs losing connectivity

[jcrussell]

We have some Cisco 2960 switches running c2960s-universalk9-mz.122-55.SE7.bin.  We downgraded to this from c2960s-universalk9-mz.122-58.SE2.bin at the advice of our support provider when this issue first started.  Sometimes, a couple of VLANs on a switch will lost all connectivity to the rest of the network.  There are no errors in the log, spanning tree looks completely normal, etc.  The only thing that brings connectivity back is either rebooting the switch or deleting the affected VLANs from the switch and adding them back in.  Once either of those happens it is like everything is back to normal.  This has happened on both stacked and stand-alone 2960 switches.  Has anyone ever heard of this?  I'd like to hear any suggestions as to where to go next. 

[devils_advocate]

Can you provide a topology diagram and perhaps the config from whatever device is doing your intervlan routing?

[Dmytro Skotnikov]

Hello jcrussell,

I would suggest you to try the following approach. Identifie a particular "broken" VLAN ID, source and destination MAC addresses pair and investigate starting from there. You should be aware what are incoming and outgoing interfaces for this flow supposed to be.

! Check interfaces statistics

show interface status

show interfaces counters

show interfaces counters errors

! Verify STP

show spanning-tree vlan details

show spanning-tree interface

! Verify CAM table population

show mac address-table interface

show mac address-table dynamic address - from previous output

! Verify if the outgoing interface are programmed correctly

show platform forward

If still you won't be able to find out the reason for the issue, please, open a TAC service request.

Thank you

--

Best regards,

Dmitry Skotnikov

[jcrussell]

Thank you both very much.  A TAC case is already open and is making zero progress.  With that said, I am working on sanitizing the config from our core.  I am attaching a cheesy diagram of the setup, which is fairly simple.  The VLANs in question only exist on the switches depicted in the diagram, not on any others.

As far as the switches go, there are basically zero errors on any interfaces in the affected vlans:

GigabitEthernet1/0/1 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is c40a.cbXX.XXXX (bia c40a.cbXX.XXXX)

  Description: **REDACTED**

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:19, output 00:00:00, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 283

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 35000 bits/sec, 21 packets/sec

  5 minute output rate 47000 bits/sec, 23 packets/sec

     110821 packets input, 18547529 bytes, 0 no buffer

     Received 1077 broadcasts (855 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 855 multicast, 0 pause input

     0 input packets with dribble condition detected

     212861 packets output, 73744196 bytes, 0 underruns

     0 output errors, 0 collisions, 1 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

Syslogs show no spanning tree topology changes either.  That was my first thought when this happened.  Topology change times line up with the switch reboot times.  From the core:

VLAN0148 is executing the rstp compatible Spanning Tree protocol

  Bridge Identifier has priority 24576, sysid 148, address 588d.0941.ef00

  Configured hello time 2, max age 20, forward delay 15, transmit hold-count 6

  We are the root of the spanning tree

  Topology change flag not set, detected flag not set

  Number of topology changes 2 last change occurred 14:07:47 ago

          from Port-channel10

  Times:  hold 1, topology change 35, notification 2

          hello 2, max age 20, forward delay 15

  Timers: hello 0, topology change 0, notification 0, aging 300

Port 560 (Port-channel10) of VLAN0148 is designated forwarding

   Port path cost 3, Port priority 128, Port Identifier 128.560.

   Designated root has priority 24724, address 588d.0941.ef00

   Designated bridge has priority 24724, address 588d.0941.ef00

   Designated port id is 128.560, designated path cost 0

   Timers: message age 0, forward delay 0, hold 0

   Number of transitions to forwarding state: 1

   Link type is point-to-point by default

   BPDU: sent 25246, received 6