Hi everyone!
We're facing high cpu utilization for a Cisco Catalyst 3750X (5 switches in stack), running IOS v 15.0.(2)SE5. The cpu reaches 80-90 %
We don't know how to troubleshoot this issue, and i'm not sure if this could be related to many packets hitting the cpu, since we don't see an specific process involved.
I've extracted these outputs from the show controllers cpu-interface command:
#show controllers cpu-interface
ASIC Rxbiterr Rxunder Fwdctfix Txbuflos Rxbufloc Rxbufdrain
-------------------------------------------------------------------------
ASIC0 0 0 0 0 0 0
ASIC1 0 0 0 0 0 0
ASIC2 0 0 0 0 0 0
HOL Fix Counts
--------------
No Fixes: 0 Added: 0 In Use: 0 Both: 0
CPU Heartbeat Statistics
Tx Success Tx Fail 1st Thr 2nd Thr Unthr RetryCtMax
---------- ---------- ---------- ---------- ---------- ----------
11842375 0 0 0 0 1
Rx Delay
0 1 2 3 4
---------- ---------- ---------- ---------- ----------
11842375 0 0 0 0
AddlDelay AdvanceCnt
---------- ----------
0 0
Rx Retries by RetryCount
0 1 2 3 4 5 6
---------- ---------- ---------- ---------- ---------- ---------- ----------
11842375 0 0 0 0 0 0
7 8 9
---------- ---------- ----------
0 0 0
AddlRetry
----------
0
cpu-queue-frames retrieved dropped invalid hol-block stray
----------------- ---------- ---------- ---------- ---------- ----------
rpc 154609347 0 0 0 0
stp 851439 0 0 0 0
ipc 5960635 0 0 0 0
routing protocol 14454926 0 0 0 0
L2 protocol 267759 0 0 0 0
remote console 0 0 0 0 0
sw forwarding 754730 0 0 0 0
host 13102574 0 0 0 0
broadcast 13750681 0 0 0 0
cbt-to-spt 0 0 0 0 0
igmp snooping 2496882 0 0 0 0
icmp 1096079185 0 0 0 0
logging 16 0 0 0 0
rpf-fail 0 0 0 0 0
dstats 0 0 0 0 0
cpu heartbeat 11842375 0 0 0 0
cpu-queue static inuse static added
----------------- ------------ ------------
rpc 0 0
stp 0 0
ipc 0 0
routing protocol 0 0
L2 protocol 0 0
remote console 0 0
sw forwarding 0 0
host 0 0
broadcast 0 0
cbt-to-spt 0 0
igmp snooping 0 0
icmp 0 0
logging 0 0
rpf-fail 0 0
dstats 0 0
cpu heartbeat 0 0
#show controllers cpu-interface (10 sec later)
ASIC Rxbiterr Rxunder Fwdctfix Txbuflos Rxbufloc Rxbufdrain
-------------------------------------------------------------------------
ASIC0 0 0 0 0 0 0
ASIC1 0 0 0 0 0 0
ASIC2 0 0 0 0 0 0
HOL Fix Counts
--------------
No Fixes: 0 Added: 0 In Use: 0 Both: 0
CPU Heartbeat Statistics
Tx Success Tx Fail 1st Thr 2nd Thr Unthr RetryCtMax
---------- ---------- ---------- ---------- ---------- ----------
11844280 0 0 0 0 1
Rx Delay
0 1 2 3 4
---------- ---------- ---------- ---------- ----------
11844280 0 0 0 0
AddlDelay AdvanceCnt
---------- ----------
0 0
Rx Retries by RetryCount
0 1 2 3 4 5 6
---------- ---------- ---------- ---------- ---------- ---------- ----------
11844280 0 0 0 0 0 0
7 8 9
---------- ---------- ----------
0 0 0
AddlRetry
----------
0
cpu-queue-frames retrieved dropped invalid hol-block stray
----------------- ---------- ---------- ---------- ---------- ----------
rpc 154638939 0 0 0 0
stp 851588 0 0 0 0
ipc 5961725 0 0 0 0
routing protocol 14457558 0 0 0 0
L2 protocol 267791 0 0 0 0
remote console 0 0 0 0 0
sw forwarding 754903 0 0 0 0
host 13106529 0 0 0 0
broadcast 13755345 0 0 0 0
cbt-to-spt 0 0 0 0 0
igmp snooping 2497893 0 0 0 0
icmp 1096387074 0 0 0 0
logging 16 0 0 0 0
rpf-fail 0 0 0 0 0
dstats 0 0 0 0 0
cpu heartbeat 11844280 0 0 0 0
cpu-queue static inuse static added
----------------- ------------ ------------
rpc 0 0
stp 0 0
ipc 0 0
routing protocol 0 0
L2 protocol 0 0
remote console 0 0
sw forwarding 0 0
host 0 0
broadcast 0 0
cbt-to-spt 0 0
igmp snooping 0 0
icmp 0 0
logging 0 0
rpf-fail 0 0
dstats 0 0
cpu heartbeat 0 0
I see the icmp cpu-queue increasing steadily. Do you think this could be the reason of the high cpu utilization?
** I ran a debug platform cpu-queues icmp-q and I saw a lot of these outputs:
482873: Apr 14 11:51:18.295 GMT: ICMP-Q:Dropped redirect disabled on L3 IF: Loca
l Port Fwding L3If:Vlan1 L2If:GigabitEthernet1/1/1 DI:0xB4, LT:7, Vlan:1 SrcGP
N:49, SrcGID:49, ACLLogIdx:0x0, MacDA:2c3e.cf94.15c0, MacSA: b8ac.6f45.4510 IP
_SA:10.108.129.49 IP_DA:10.108.135.127 IP_Proto:6
TPFFD:EC000031_00000001_00B00040-000000B4_00220000_89A60008
482874: Apr 14 11:51:18.303 GMT: ICMP-Q:Dropped redirect disabled on L3 IF: Loca
l Port Fwding L3If:Vlan1 L2If:GigabitEthernet1/0/12 DI:0xB4, LT:7, Vlan:1 SrcG
PN:12, SrcGID:12, ACLLogIdx:0x0, MacDA:2c3e.cf94.15c0, MacSA: 000b.cd83.04f4 T
PFFD:EC000031_00000001_00B00046-000000B4_00220000_89A60008
483959: Apr 14 11:51:19.343 GMT: ICMP-Q:Dropped redirect disabled on L3 IF: Loca
l Port Fwding L3If:Vlan1 L2If:GigabitEthernet1/0/12 DI:0xB4, LT:7, Vlan:1 SrcG
PN:12, SrcGID:12, ACLLogIdx:0x0, MacDA:2c3e.cf94.15c0, MacSA: 000b.cd83.04f4 I
P_SA:10.108.129.8 IP_DA:10.108.168.216 IP_Proto:6
TPFFD:EC00000C_00000001_00B000D6-000000B4_00220000_89A60008
...
Thanks, and i appreciate any help.
