100% Utilization On Catalyst4948 Switch

Sam.Debrah · ‎10-12-2010

Hi,

I am having a problem with one of my switches in my network. We have eight Cisco Catalyst 4948 switches all interconnected to each other. See Attachment. Root bridge sends and receive traffic from all the other switches. Recently, the Root Bridge has been running 100% CPU utilization. I have done some diagnosis and here are the output:

Extract:

Sh Processor CPU

46           0         1          0 0.00% 0.00% 0.00%   0 cpf_msg_rcvq_pro
47           0         2          0 0.00% 0.00% 0.00%   0 IP Host Track HA
48           0         2          0 0.00% 0.00% 0.00%   0 ARP HA
49           0         1          0 0.00% 0.00% 0.00%   0 IP Admission HA
50           0         1          0 0.00% 0.00% 0.00%   0 Network-rf Notif
51    42434672 69674961        609 7.35% 7.31% 7.70%   0 Cat4k Mgmt HiPri
52   742058684 122437240       6060 88.47% 89.25% 89.20%   0 Cat4k Mgmt LoPri
53        8588    444595         19 0.00% 0.00% 0.00%   0 Galios Reschedul
54           0         1          0 0.00% 0.00% 0.00%   0 IOS ACL Helper
55           0        10          0 0.00% 0.00% 0.00%   0 BACK CHECK
56           8         2       4000 0.00% 0.00% 0.00%   0 rf task
57           0         1          0 0.00% 0.00% 0.00%   0 RF High Priority

Sh Platform Health

K2AclCamMan Audit re   1.00   0.00     10      5 100 500    0   0    0 43:00
K2AclPolicerTableMan   1.00   0.00     10      2 100 500    0   0    0 2:49
K2L2 Address Table R   2.00 85.37     12      5 100 500 112 104   83 11319:09
K2L2 New Static Addr   2.00   0.00     10      0 100 500    0   0    0 0:00
K2L2 New Multicast A   2.00   0.00     10      5 100 500    0   0    0 0:00
K2L2 Dynamic Address   2.00   0.00     10      5 100 500    0   0    0 0:00

#sh platform cpu packet statistics
Packets Dropped In Hardware By CPU Subport (txQueueNotAvail)

CPU Subport TxQueue 0 TxQueue 1 TxQueue 2 TxQueue 3
------------ --------------- --------------- --------------- ---------------
0 0 0 0 478

RkiosSysPacketMan:
Packet allocation failures: 0
Packet Buffer(Software Common) allocation failures: 0
Packet Buffer(Software ESMP) allocation failures: 0
Packet Buffer(Software EOBC) allocation failures: 0
Packet Buffer(Software SupToSup) allocation failures: 0
IOS Packet Buffer Wrapper allocation failures: 0

Total packet queues 16

Packets Received by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
L2/L3Control                   1593780         1         0         1          0
Host Learning                 64349591       238       222       187        196
L3 Fwd Low                           3         0         0         0          0
L2 Fwd Low                       20011         0         0         0          0
L3 Rx Low                       753653         2         0         0          0

Packets Dropped by Packet Queue

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Host Learning 1593 0 0 0 0

Can anyone see from the output the cause of the high CPU utilization? I think the problem may be and what I can do to resolve it?

Yogesh Ramdoss · ‎10-12-2010

Hello Sam, Looking at the CPU queue stats, we have high number of traffic hitting "Host Learning" queue. In Cat4000/4500 platforms, if the switch receives a frame from unknown mac-address, it is sent to CPU. I would recommend you to monitor "show mac-address-table count" and make sure the count is stable. If not, then I would suspect spanning-tree TCNs occurring (as a result, Fast-Aging mac-addresses). Please check the STP Topology Change count under "show spanning-tree vlan X detail" If the mac-address-table looks stable, we need to sniff the packets hitting the CPU queue. CPU port sniffing: http://www.cisco.com/en/US/partner/docs/switches/lan/catalyst4500/12.2/54sg/configuration/guide/span.html#wp1039942 - Yogesh

Sam.Debrah · ‎10-13-2010

Hi,

Thanks for your comment. I will monitor the the mac-address-table count throughout today. So far this is the output of show mac-address-table count on the switch: The Dynamic Unicast Address Count hovers between 221 and 215. Is that normal? The other switches in the topology show this value to be between 130 - 150.

sh mac-address-table count
MAC Entries for all vlans:
Dynamic Unicast Address Count:                  221
Static Unicast Address (User-defined) Count:    0
Static Unicast Address (System-defined) Count: 1
Total Unicast MAC Addresses In Use:             222
Total Unicast MAC Addresses Available:          32768
Multicast MAC Address Count:                    25
Total Multicast MAC Addresses Available:        32768

Below is the output of sh spanning-tree vlan 500 detail of one of the vlans. Whats the difference between the Times & Timers?

VLAN0500 is executing the rstp compatible Spanning Tree protocol
Bridge Identifier has priority 8192, sysid 500, address 001d.70c7.6f80
Configured hello time 2, max age 20, forward delay 15, transmit hold-count 6
We are the root of the spanning tree
Topology change flag not set, detected flag not set
Number of topology changes 4 last change occurred 1w5d ago
from Port-channel12
Times: hold 1, topology change 35, notification 2
hello 2, max age 20, forward delay 15
Timers: hello 0, topology change 0, notification 0, aging 300

Yogesh Ramdoss · ‎10-13-2010

Sam, Total mac-address count looks normal. Now, I would recommend you to sniff the CPU queue. http://www.cisco.com/en/US/partner/docs/switches/lan/catalyst4500/12.2/54sg/configuration/guide/span.html#wp1039942 - Yogesh

Sam.Debrah · ‎10-13-2010

Yogesh,

What command do I use to sniff the traffic queues to the CPU?

Yogesh Ramdoss · ‎10-13-2010

The commands are listed in the link I provided earlier.

- Yogesh

Sam.Debrah · ‎10-13-2010

Yogesh,

The link you provided does not work for me.

Yogesh Ramdoss · ‎10-14-2010

I apologize.

Please try this link:

http://www.cisco.com/en/US/docs/switches/lan/catalyst4500/12.2/52sg/configuration/guide/span.html#wp1039942

- Yogesh

jorge.calvo · ‎10-14-2010

Hello,

Enable "terminal monitor" and look for C4K_EBM-4-HOSTFLAPPING messages. That would explain your processes consuming CPU.

Hope this helps.

Sam.Debrah · ‎10-18-2010

I am a bit lost with this. I am sniffing all traffic to the CPU and sending the ouotput to a destination port on the switch. I am collection the output using wireshark (network anaylser). I am not sure what I am supposed to be looking for. the capture seems Ok to me. Can anyone help?

Sam

glen.grant · ‎10-18-2010

This doc might help a little more . Seeing the 4948 runs cat 4500 code this should all be valid including a built in cpu sniffer . Read thru this doc , it's fairly extensive on troubleshooting high cpu on 4500 series platforms.

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml#tool2