Could sustained high cpu utilization on a switch cause a port to drop out of blocking and start forwarding for random vlans? I know this is separate from the MSFC cpu but perhaps one of the edge switches became overloaded, started forwarding on a port where it should have been blocking caused the whole mess?

Every now and then someone does something insane in my lab which spikes everything but I have no CPU history for the switches in there to check it.

Just wondering...and grasping at straws since I really don't have anything common about vlan1 and vlan15.

Well...never mind. On of my edge switches had somehow had it's syslog server entry disabled so I didn't see any message from it in my central log file.

I logged into it and checked it and found this:

2007 Apr 11 17:05:26 %SYS-3-SYS_MEMERR:Null Cluster while getting next from address 0x30ae5770

2007 Apr 11 17:05:26 %SYS-3-SYS_MEMERR:Null Cluster while getting next from address 0x30ae5770

2007 Apr 11 17:05:26 %SYS-3-SYS_MEMERR:Null Cluster while getting next from address 0x30ae5770

2007 Apr 11 17:05:26 %SYS-3-SYS_MEMERR:Null Cluster while getting next from address 0x30ae5770

2007 Apr 11 17:05:26 %SYS-3-SYS_MEMERR:Null Cluster while getting next from address 0x30ae5770

2007 Apr 11 17:05:26 %SYS-3-SYS_MEMERR:No clusters left while allocating for address 0x409aec00

2007 Apr 11 17:05:26 %SYS-3-SYS_MEMERR:No clusters left while allocating for address 0x40a03500

2007 Apr 11 17:05:26 %SYS-3-SYS_MEMERR:Null Cluster while getting next from address 0x30ae5770

2007 Apr 11 17:05:30 %SYS-4-P2_WARN: 1/Astro(5/5) - timeout occurred

2007 Apr 11 17:05:56 %SYS-4-P2_WARN: 1/Astro(3/5) - timeout occurred

2007 Apr 11 17:05:57 %SYS-4-P2_WARN: 1/Astro(4/6) - timeout occurred

2007 Apr 11 17:06:01 %SYS-4-P2_WARN: 1/Host 00:02:fc:bd:0a:78 is flapping between port 1/2 and port 1/1

2007 Apr 11 17:06:12 %SYS-4-P2_WARN: 1/Astro(2/6) - timeout occurred

2007 Apr 11 17:06:14 %SYS-4-P2_WARN: 1/Host 00:02:fc:bd:0a:78 is flapping between port 1/2 and port 1/1

2007 Apr 11 17:06:16 %SYS-4-P2_WARN: 1/Host 08:00:3e:28:d5:e2 is flapping between port 1/1 and port 1/2

And on and on.

If this looks like a SUP failure. I'm wondering if I should just go ahead and pull the primary Supervisor to keep it from happening again until I can get a replacement.

Advice?

Thanks.

Not sure if it's a Sup failure. Null cluster errors can indicate the system is being overloaded.

Astro errors indicate the Sup and the linecard has lost connection (via a keepalive message it sends internally).

Then, you see two MAC addresses which are flapping. Your best option is to track down those MAC addresses to see where they are in the network.

It's of course possible that there is a hardware issue with this but by no means does it indicate a definite one.

A further way of testing is to schedule a maintenance window and reset the switch. When it powers on, it will do a diagnostic test of each linecard including the Sup. This should give a fairly good indication of whether there is a hardware fault with any of the modules.

Good luck,

Michael

Cisco TAC

Hi,

what if someone connected VLAN1 and VLAN15 together by a cable somewhere at your LAN perimeter?

Creating a common STP within those two VLANs could cause some loops (it takes STP some time to converge) and could make HSRP hello packets coming back to the originating MSFC with effects described.

Best regards,

Milan Kulik

I didn't post the entire log from the switch but there were over a thousand memory error messages in there all from the same memory address.

As well, there were mac address flap messages where a mac would be flapping between, for example, port 5/45 and 1/1. That indicates that the mac was seen first on a port that goes out into the switch's service area (where the users are), and then it was seen on one of the uplinks between the switch and network core. Unless someone ran a cable from the user's lan drop to the data center (impossible) I'm not sure how this could happen. Other than some crazy hardware fault.

In addition, I saw this behavior from mac addresses on three different vlans that this switch carries. So I saw a mac on vlan 15 flap between a user port and the uplink and another mac on vlan 6 do the same thing. I can't think of a physically feasible way that someone could have made connections in the user area that would explain this behavior.

If I'm missing something I would be grateful for any possible enlightenment. I'm no genius and welcome any education that the experts are willing to offer.

Thanks everyone.

I have run into this a few times. Each time it has been an issue with spanning tree. As suggested you should trace down the MAC addresses that are showing up in the logs. I will bet those trace down to access switchs. In our case it has been server enclosures that are dual homed to two different switches. One switch is in a forwarding state and the other is blocking. Not sure why, but something in the server enclosure goes crazy and suddenly both sides go into a forwarding state. This then causes a spanning-tree loop, CPU utilization spikes and you get the duplicate IP messages. Hope this helps.

Hi Allen,

We may have a similar problem; where an HSRP group in standby goes into active when the primary is still up and running. This occurs when we modify a VLAN trunk that is well down-stream from the core HSRP switches. We are fairly sure it is STP related, and possibly tied into a server that is dual homed. Besides mapping out the STP for the affected VLANs, where do you begin trouble-shooting?

thanks,

Jimmyc

We look for the duplicate IP address message in the logs. Trace down the MAC that shows up in these messages and that usually is the switch that started the issue. Sometimes it is the actual server MAC that shows up.