Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Community Member

100% Utilization On Catalyst4948 Switch

Hi,

I am having a problem with one of my switches in my network. We have eight Cisco Catalyst 4948 switches all interconnected to each other. See Attachment. Root bridge sends and receive traffic from all the other switches. Recently, the Root Bridge has been running 100% CPU utilization. I have done some diagnosis and here are the output:

Extract:

Sh Processor CPU

46           0         1          0  0.00%  0.00%  0.00%   0 cpf_msg_rcvq_pro
  47           0         2          0  0.00%  0.00%  0.00%   0 IP Host Track HA
  48           0         2          0  0.00%  0.00%  0.00%   0 ARP HA          
  49           0         1          0  0.00%  0.00%  0.00%   0 IP Admission HA 
  50           0         1          0  0.00%  0.00%  0.00%   0 Network-rf Notif
51    42434672  69674961        609  7.35%  7.31%  7.70%   0 Cat4k Mgmt HiPri
  52   742058684 122437240       6060 88.47% 89.25% 89.20%   0 Cat4k Mgmt LoPri

  53        8588    444595         19  0.00%  0.00%  0.00%   0 Galios Reschedul
  54           0         1          0  0.00%  0.00%  0.00%   0 IOS ACL Helper  
  55           0        10          0  0.00%  0.00%  0.00%   0 BACK CHECK      
  56           8         2       4000  0.00%  0.00%  0.00%   0 rf task         
  57           0         1          0  0.00%  0.00%  0.00%   0 RF High Priority

Sh Platform Health

K2AclCamMan Audit re   1.00   0.00     10      5  100  500    0   0    0  43:00
K2AclPolicerTableMan   1.00   0.00     10      2  100  500    0   0    0  2:49
K2L2 Address Table R   2.00  85.37     12      5  100  500  112 104   83  11319:09
K2L2 New Static Addr   2.00   0.00     10      0  100  500    0   0    0  0:00
K2L2 New Multicast A   2.00   0.00     10      5  100  500    0   0    0  0:00
K2L2 Dynamic Address   2.00   0.00     10      5  100  500    0   0    0  0:00

#sh platform cpu packet statistics
Packets Dropped In Hardware By CPU Subport (txQueueNotAvail)

CPU Subport  TxQueue 0       TxQueue 1       TxQueue 2       TxQueue 3
------------ --------------- --------------- --------------- ---------------
           0               0               0               0             478


RkiosSysPacketMan:
Packet allocation failures: 0
Packet Buffer(Software Common) allocation failures: 0
Packet Buffer(Software ESMP) allocation failures: 0
Packet Buffer(Software EOBC) allocation failures: 0
Packet Buffer(Software SupToSup) allocation failures: 0
IOS Packet Buffer Wrapper allocation failures: 0

Total packet queues 16

Packets Received by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
L2/L3Control                   1593780         1         0         1          0
Host Learning                 64349591       238       222       187        196
L3 Fwd Low                           3         0         0         0          0
L2 Fwd Low                       20011         0         0         0          0
L3 Rx Low                       753653         2         0         0          0

Packets Dropped by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Host Learning                     1593         0         0         0          0

Can anyone see from the output the cause of the high CPU utilization? I think the problem may be and what I can do to resolve it?

10 REPLIES
Cisco Employee

Re: 100% Utilization On Catalyst4948 Switch

Hello Sam,  Looking at the CPU queue stats, we have high number of traffic hitting "Host Learning" queue. In Cat4000/4500 platforms, if the switch receives a frame from unknown mac-address, it is sent to CPU. I would recommend you to monitor "show mac-address-table count" and make sure the count is stable. If not, then I would suspect spanning-tree TCNs occurring (as a result, Fast-Aging mac-addresses). Please check the STP Topology Change count under "show spanning-tree vlan X detail"  If the mac-address-table looks stable, we need to sniff the packets hitting the CPU queue. CPU port sniffing: http://www.cisco.com/en/US/partner/docs/switches/lan/catalyst4500/12.2/54sg/configuration/guide/span.html#wp1039942  - Yogesh

Community Member

Re: 100% Utilization On Catalyst4948 Switch

Hi,

Thanks for your comment. I will monitor the the mac-address-table count throughout today. So far this is the output of show mac-address-table count on the switch: The Dynamic Unicast Address Count hovers between 221 and 215. Is that normal? The other switches in the topology show this value to be between 130 - 150.

sh mac-address-table count
MAC Entries for all vlans:
Dynamic Unicast Address Count:                  221
Static Unicast Address (User-defined) Count:    0
Static Unicast Address (System-defined) Count:  1
Total Unicast MAC Addresses In Use:             222
Total Unicast MAC Addresses Available:          32768
Multicast MAC Address Count:                    25
Total Multicast MAC Addresses Available:        32768

Below is the output of sh spanning-tree vlan 500 detail of one of the vlans. Whats the difference between the Times & Timers?

VLAN0500 is executing the rstp compatible Spanning Tree protocol
  Bridge Identifier has priority 8192, sysid 500, address 001d.70c7.6f80
  Configured hello time 2, max age 20, forward delay 15, transmit hold-count 6
  We are the root of the spanning tree
  Topology change flag not set, detected flag not set
  Number of topology changes 4 last change occurred 1w5d ago
          from Port-channel12
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15
  Timers: hello 0, topology change 0, notification 0, aging 300

Cisco Employee

Re: 100% Utilization On Catalyst4948 Switch

Sam,  Total mac-address count looks normal. Now, I would recommend you to sniff the CPU queue.  http://www.cisco.com/en/US/partner/docs/switches/lan/catalyst4500/12.2/54sg/configuration/guide/span.html#wp1039942  - Yogesh

Community Member

Re: 100% Utilization On Catalyst4948 Switch

Yogesh,

What command do I use to sniff the traffic queues to the CPU?

Cisco Employee

Re: 100% Utilization On Catalyst4948 Switch

The commands are listed in the link I provided earlier.

- Yogesh

Community Member

Re: 100% Utilization On Catalyst4948 Switch

Yogesh,

The link you provided does not work for me.

Cisco Employee

Re: 100% Utilization On Catalyst4948 Switch

Bronze

Re: 100% Utilization On Catalyst4948 Switch

Hello,

Enable "terminal monitor" and look for C4K_EBM-4-HOSTFLAPPING messages. That would explain your processes consuming CPU.

Hope this helps.

Community Member

Re: 100% Utilization On Catalyst4948 Switch

I am a bit lost with this. I am sniffing all traffic to the CPU and sending the ouotput to a destination port on the switch. I am collection the output using wireshark (network anaylser). I am not sure what I am supposed to be looking for. the capture seems Ok to me. Can anyone help?

Sam

Purple

Re: 100% Utilization On Catalyst4948 Switch

   This doc might help a little more .  Seeing the 4948 runs cat 4500 code this should all be valid including a built in cpu sniffer .   Read thru this doc , it's fairly extensive on troubleshooting high cpu on 4500 series platforms.

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml#tool2

1819
Views
0
Helpful
10
Replies
CreatePlease to create content