cpu @ 99%

[WILLIAM STEGMAN]

I have a 4510 that I recently put into production and its cpu is pegged at 99 constantly. A show proc cpu sort shows

Cat4k Mgmt LoPri to be the main culprit. A sh platform health reveals 2 processes that looked to be associated:

K5L3Unicast Adj Tabl 2.00 31.49 15 11 100 500 31 33 33 3822:10

K5 L2 Hardware Addre 2.00 59.05 20 14 100 500 71 56 50 5837:19

HBG-DataCenter-4510R(config)#do sh proc cpu sorted

CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

54 620006660 52990203 11700 89.28% 89.03% 88.48% 0 Cat4k Mgmt LoPri

53 61832668 111544261 554 8.95% 8.90% 9.13% 0 Cat4k Mgmt HiPri

120 2164724 5933214 364 0.23% 0.31% 0.31% 0 Spanning Tree

38 2478104 863775 2868 0.23% 0.25% 0.23% 0 IDB Work

114 904608 2828492 319 0.15% 0.13% 0.13% 0 IP Input

107 743744 2327469 319 0.15% 0.12% 0.08% 0 CDP Protocol

163 2880 1455 1979 0.07% 0.10% 0.41% 1 SSH Process

14 982968 4783695 205 0.07% 0.11% 0.11% 0 ARP Input

183 107260 1481823 72 0.07% 0.01% 0.00% 0 VLAN Manager

62 173040 282995 611 0.07% 0.02% 0.00% 0 Compute load avg

182 336500 18595895 18 0.07% 0.05% 0.07% 0 PM Callback

155 178620 5754473 31 0.07% 0.03% 0.02% 0 CEF: IPv4 proces

13 0 10 0 0.00% 0.00% 0.00% 0 ifIndex Receive

12 0 1 0 0.00% 0.00% 0.00% 0 Crash writer

15 0 1 0 0.00% 0.00% 0.00% 0 CEF MIB API

11 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun

10 96 5 19200 0.00% 0.00% 0.00% 0 PrstVbl

9 0 2 0 0.00% 0.00% 0.00% 0 Timers

8 0 181 0 0.00% 0.00% 0.00% 0 Pool Manager

20 4 397 10 0.00% 0.00% 0.00% 0 EEM ED Syslog

16 4 10 400 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

sh platform cpu pack stat all

Packets Dropped In Processing by CPU event

Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Unknown 0 0 0 0 0

Sa Miss 203154944 418 482 1648 953

Any ideas why this?

thank you,

Bill

[glen.grant · Accepted solution]

Take a look at this page and see if this helps with troubleshooting , if not a TAC case may need to be opened.

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml

[WILLIAM STEGMAN]

I found an oracle listener client using the command below. It's interfaces are bonded and somehow messing up the switch's cpu.

Switch#debug platform packet all receive buffer

platform packet debugging is on

Switch#show platform cpu packet buffered

Total Received Packets Buffered: 36

thank you Glen

[Lucien Avramov]

CSCeb21247 Cat4k Sup4 high cpu problem in Cat4k Mgmt LoPri process

[WILLIAM STEGMAN]

I don't think it's a bug, I'm running Version 12.2(53)SG. Do you know what these processes might be associated with?

I am running EIGRP with a couple distribution switches as neighbors.

K5L3Unicast Adj Tabl 2.00 31.49 15 11 100 500 31 33 33 3822:10

K5 L2 Hardware Addre 2.00 59.05 20 14 100 500 71 56 50 5837:19

[Lucien Avramov]

Ok now that I know your version, you are not impacted by this.

The process which has consumed the highest CPU util. was "K5 L2 Hardware Addre" and "K5L3Unicast Adj Tabl".

Process "K5 L2 Hardware Addre" is active, when new MAC addresses are learned and subsequently adjacency table has to be rewritten as well, hence "K5L3Unicast Adj Tabl" is active.

If a packet is received with an unknown source MAC address, a copy of that packet is forwarded to the CPU for MAC address learning. This is also happening, when a packet with an already known source MAC address is received on another interface. A copy is sent to the CPU and this MAC is re-learned on another interface.

In the phase of high CPU utilization, it could be observed that the traffic has been continuously received between a port channel and a gig interface for example.

I hope this clears out your questions.

[gskanth123]

Thanks all..I have not even known before

[boumendil]

Hi Lucien,

You mention that "In the phase of high CPU utilization, it could be observed that the traffic has been continuously received between a port channel and a gig interface for example.". Do you mean this is a known bug or a platform limitation.

I experience the same on multiple 4900M (either L2/L3 or pure L2 switch).

Thank you.

regards,

SB