×

Warning message

  • Cisco Support Forums is in Read Only mode while the site is being migrated.
  • Cisco Support Forums is in Read Only mode while the site is being migrated.

100% Utilization On Catalyst4948 Switch

Unanswered Question
Oct 12th, 2010
User Badges:

Hi,


I am having a problem with one of my switches in my network. We have eight Cisco Catalyst 4948 switches all interconnected to each other. See Attachment. Root bridge sends and receive traffic from all the other switches. Recently, the Root Bridge has been running 100% CPU utilization. I have done some diagnosis and here are the output:


Extract:


Sh Processor CPU


46           0         1          0  0.00%  0.00%  0.00%   0 cpf_msg_rcvq_pro
  47           0         2          0  0.00%  0.00%  0.00%   0 IP Host Track HA
  48           0         2          0  0.00%  0.00%  0.00%   0 ARP HA          
  49           0         1          0  0.00%  0.00%  0.00%   0 IP Admission HA 
  50           0         1          0  0.00%  0.00%  0.00%   0 Network-rf Notif
51    42434672  69674961        609  7.35%  7.31%  7.70%   0 Cat4k Mgmt HiPri
  52   742058684 122437240       6060 88.47% 89.25% 89.20%   0 Cat4k Mgmt LoPri

  53        8588    444595         19  0.00%  0.00%  0.00%   0 Galios Reschedul
  54           0         1          0  0.00%  0.00%  0.00%   0 IOS ACL Helper  
  55           0        10          0  0.00%  0.00%  0.00%   0 BACK CHECK      
  56           8         2       4000  0.00%  0.00%  0.00%   0 rf task         
  57           0         1          0  0.00%  0.00%  0.00%   0 RF High Priority



Sh Platform Health


K2AclCamMan Audit re   1.00   0.00     10      5  100  500    0   0    0  43:00
K2AclPolicerTableMan   1.00   0.00     10      2  100  500    0   0    0  2:49
K2L2 Address Table R   2.00  85.37     12      5  100  500  112 104   83  11319:09
K2L2 New Static Addr   2.00   0.00     10      0  100  500    0   0    0  0:00
K2L2 New Multicast A   2.00   0.00     10      5  100  500    0   0    0  0:00
K2L2 Dynamic Address   2.00   0.00     10      5  100  500    0   0    0  0:00





#sh platform cpu packet statistics
Packets Dropped In Hardware By CPU Subport (txQueueNotAvail)

CPU Subport  TxQueue 0       TxQueue 1       TxQueue 2       TxQueue 3
------------ --------------- --------------- --------------- ---------------
           0               0               0               0             478


RkiosSysPacketMan:
Packet allocation failures: 0
Packet Buffer(Software Common) allocation failures: 0
Packet Buffer(Software ESMP) allocation failures: 0
Packet Buffer(Software EOBC) allocation failures: 0
Packet Buffer(Software SupToSup) allocation failures: 0
IOS Packet Buffer Wrapper allocation failures: 0

Total packet queues 16

Packets Received by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
L2/L3Control                   1593780         1         0         1          0
Host Learning                 64349591       238       222       187        196
L3 Fwd Low                           3         0         0         0          0
L2 Fwd Low                       20011         0         0         0          0
L3 Rx Low                       753653         2         0         0          0

Packets Dropped by Packet Queue

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Host Learning                     1593         0         0         0          0



Can anyone see from the output the cause of the high CPU utilization? I think the problem may be and what I can do to resolve it?

Attachment: 
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Yogesh Ramdoss Tue, 10/12/2010 - 08:52
User Badges:
  • Cisco Employee,

Hello Sam,  Looking at the CPU queue stats, we have high number of traffic hitting "Host Learning" queue. In Cat4000/4500 platforms, if the switch receives a frame from unknown mac-address, it is sent to CPU. I would recommend you to monitor "show mac-address-table count" and make sure the count is stable. If not, then I would suspect spanning-tree TCNs occurring (as a result, Fast-Aging mac-addresses). Please check the STP Topology Change count under "show spanning-tree vlan X detail"  If the mac-address-table looks stable, we need to sniff the packets hitting the CPU queue. CPU port sniffing: http://www.cisco.com/en/US/partner/docs/switches/lan/catalyst4500/12.2/54sg/configuration/guide/span.html#wp1039942  - Yogesh

Sam.Debrah Wed, 10/13/2010 - 01:08
User Badges:

Hi,


Thanks for your comment. I will monitor the the mac-address-table count throughout today. So far this is the output of show mac-address-table count on the switch: The Dynamic Unicast Address Count hovers between 221 and 215. Is that normal? The other switches in the topology show this value to be between 130 - 150.


sh mac-address-table count
MAC Entries for all vlans:
Dynamic Unicast Address Count:                  221
Static Unicast Address (User-defined) Count:    0
Static Unicast Address (System-defined) Count:  1
Total Unicast MAC Addresses In Use:             222
Total Unicast MAC Addresses Available:          32768
Multicast MAC Address Count:                    25
Total Multicast MAC Addresses Available:        32768


Below is the output of sh spanning-tree vlan 500 detail of one of the vlans. Whats the difference between the Times & Timers?


VLAN0500 is executing the rstp compatible Spanning Tree protocol
  Bridge Identifier has priority 8192, sysid 500, address 001d.70c7.6f80
  Configured hello time 2, max age 20, forward delay 15, transmit hold-count 6
  We are the root of the spanning tree
  Topology change flag not set, detected flag not set
  Number of topology changes 4 last change occurred 1w5d ago
          from Port-channel12
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15
  Timers: hello 0, topology change 0, notification 0, aging 300

Sam.Debrah Wed, 10/13/2010 - 06:01
User Badges:

Yogesh,


What command do I use to sniff the traffic queues to the CPU?

Yogesh Ramdoss Wed, 10/13/2010 - 14:17
User Badges:
  • Cisco Employee,

The commands are listed in the link I provided earlier.


- Yogesh

Sam.Debrah Wed, 10/13/2010 - 23:37
User Badges:

Yogesh,


The link you provided does not work for me.

jorge.calvo Thu, 10/14/2010 - 06:26
User Badges:
  • Bronze, 100 points or more

Hello,


Enable "terminal monitor" and look for C4K_EBM-4-HOSTFLAPPING messages. That would explain your processes consuming CPU.


Hope this helps.

Sam.Debrah Mon, 10/18/2010 - 04:35
User Badges:

I am a bit lost with this. I am sniffing all traffic to the CPU and sending the ouotput to a destination port on the switch. I am collection the output using wireshark (network anaylser). I am not sure what I am supposed to be looking for. the capture seems Ok to me. Can anyone help?


Sam

glen.grant Mon, 10/18/2010 - 04:58
User Badges:
  • Purple, 4500 points or more

   This doc might help a little more .  Seeing the 4948 runs cat 4500 code this should all be valid including a built in cpu sniffer .   Read thru this doc , it's fairly extensive on troubleshooting high cpu on 4500 series platforms.


http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml#tool2

Actions

This Discussion