10-12-2010 08:06 AM - last edited on 03-25-2019 04:12 PM by ciscomoderator
Hi,
I am having a problem with one of my switches in my network. We have eight Cisco Catalyst 4948 switches all interconnected to each other. See Attachment. Root bridge sends and receive traffic from all the other switches. Recently, the Root Bridge has been running 100% CPU utilization. I have done some diagnosis and here are the output:
Extract:
Sh Processor CPU
46 0 1 0 0.00% 0.00% 0.00% 0 cpf_msg_rcvq_pro
47 0 2 0 0.00% 0.00% 0.00% 0 IP Host Track HA
48 0 2 0 0.00% 0.00% 0.00% 0 ARP HA
49 0 1 0 0.00% 0.00% 0.00% 0 IP Admission HA
50 0 1 0 0.00% 0.00% 0.00% 0 Network-rf Notif
51 42434672 69674961 609 7.35% 7.31% 7.70% 0 Cat4k Mgmt HiPri
52 742058684 122437240 6060 88.47% 89.25% 89.20% 0 Cat4k Mgmt LoPri
53 8588 444595 19 0.00% 0.00% 0.00% 0 Galios Reschedul
54 0 1 0 0.00% 0.00% 0.00% 0 IOS ACL Helper
55 0 10 0 0.00% 0.00% 0.00% 0 BACK CHECK
56 8 2 4000 0.00% 0.00% 0.00% 0 rf task
57 0 1 0 0.00% 0.00% 0.00% 0 RF High Priority
Sh Platform Health
K2AclCamMan Audit re 1.00 0.00 10 5 100 500 0 0 0 43:00
K2AclPolicerTableMan 1.00 0.00 10 2 100 500 0 0 0 2:49
K2L2 Address Table R 2.00 85.37 12 5 100 500 112 104 83 11319:09
K2L2 New Static Addr 2.00 0.00 10 0 100 500 0 0 0 0:00
K2L2 New Multicast A 2.00 0.00 10 5 100 500 0 0 0 0:00
K2L2 Dynamic Address 2.00 0.00 10 5 100 500 0 0 0 0:00
#sh platform cpu packet statistics
Packets Dropped In Hardware By CPU Subport (txQueueNotAvail)
CPU Subport TxQueue 0 TxQueue 1 TxQueue 2 TxQueue 3
------------ --------------- --------------- --------------- ---------------
0 0 0 0 478
RkiosSysPacketMan:
Packet allocation failures: 0
Packet Buffer(Software Common) allocation failures: 0
Packet Buffer(Software ESMP) allocation failures: 0
Packet Buffer(Software EOBC) allocation failures: 0
Packet Buffer(Software SupToSup) allocation failures: 0
IOS Packet Buffer Wrapper allocation failures: 0
Total packet queues 16
Packets Received by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
L2/L3Control 1593780 1 0 1 0
Host Learning 64349591 238 222 187 196
L3 Fwd Low 3 0 0 0 0
L2 Fwd Low 20011 0 0 0 0
L3 Rx Low 753653 2 0 0 0
Packets Dropped by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Host Learning 1593 0 0 0 0
Can anyone see from the output the cause of the high CPU utilization? I think the problem may be and what I can do to resolve it?
10-12-2010 08:52 AM
Hello Sam, Looking at the CPU queue stats, we have high number of traffic hitting "Host Learning" queue. In Cat4000/4500 platforms, if the switch receives a frame from unknown mac-address, it is sent to CPU. I would recommend you to monitor "show mac-address-table count" and make sure the count is stable. If not, then I would suspect spanning-tree TCNs occurring (as a result, Fast-Aging mac-addresses). Please check the STP Topology Change count under "show spanning-tree vlan X detail" If the mac-address-table looks stable, we need to sniff the packets hitting the CPU queue. CPU port sniffing: http://www.cisco.com/en/US/partner/docs/switches/lan/catalyst4500/12.2/54sg/configuration/guide/span.html#wp1039942 - Yogesh
10-13-2010 01:08 AM
Hi,
Thanks for your comment. I will monitor the the mac-address-table count throughout today. So far this is the output of show mac-address-table count on the switch: The Dynamic Unicast Address Count hovers between 221 and 215. Is that normal? The other switches in the topology show this value to be between 130 - 150.
sh mac-address-table count
MAC Entries for all vlans:
Dynamic Unicast Address Count: 221
Static Unicast Address (User-defined) Count: 0
Static Unicast Address (System-defined) Count: 1
Total Unicast MAC Addresses In Use: 222
Total Unicast MAC Addresses Available: 32768
Multicast MAC Address Count: 25
Total Multicast MAC Addresses Available: 32768
Below is the output of sh spanning-tree vlan 500 detail of one of the vlans. Whats the difference between the Times & Timers?
VLAN0500 is executing the rstp compatible Spanning Tree protocol
Bridge Identifier has priority 8192, sysid 500, address 001d.70c7.6f80
Configured hello time 2, max age 20, forward delay 15, transmit hold-count 6
We are the root of the spanning tree
Topology change flag not set, detected flag not set
Number of topology changes 4 last change occurred 1w5d ago
from Port-channel12
Times: hold 1, topology change 35, notification 2
hello 2, max age 20, forward delay 15
Timers: hello 0, topology change 0, notification 0, aging 300
10-13-2010 04:35 AM
Sam, Total mac-address count looks normal. Now, I would recommend you to sniff the CPU queue. http://www.cisco.com/en/US/partner/docs/switches/lan/catalyst4500/12.2/54sg/configuration/guide/span.html#wp1039942 - Yogesh
10-13-2010 06:01 AM
Yogesh,
What command do I use to sniff the traffic queues to the CPU?
10-13-2010 02:17 PM
The commands are listed in the link I provided earlier.
- Yogesh
10-13-2010 11:37 PM
Yogesh,
The link you provided does not work for me.
10-14-2010 05:32 AM
I apologize.
Please try this link:
- Yogesh
10-14-2010 06:26 AM
Hello,
Enable "terminal monitor" and look for C4K_EBM-4-HOSTFLAPPING messages. That would explain your processes consuming CPU.
Hope this helps.
10-18-2010 04:35 AM
I am a bit lost with this. I am sniffing all traffic to the CPU and sending the ouotput to a destination port on the switch. I am collection the output using wireshark (network anaylser). I am not sure what I am supposed to be looking for. the capture seems Ok to me. Can anyone help?
Sam
10-18-2010 04:58 AM
This doc might help a little more . Seeing the 4948 runs cat 4500 code this should all be valid including a built in cpu sniffer . Read thru this doc , it's fairly extensive on troubleshooting high cpu on 4500 series platforms.
http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml#tool2
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide