98% CPU on New Core Switch 4507

kevinhum122 · ‎11-26-2013

I find that it's so abnormal CPU performance on 4507, the process "iosd" occupies lots of CPU resource, the details as below, anyone who can help me ?

Core_WS-C4507#sh proc cpu sorted

Core 0: CPU utilization for five seconds: 8%; one minute: 27%; five minutes: 37%

Core 1: CPU utilization for five seconds: 98%; one minute: 78%; five minutes: 68%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

8518 3252698 27999562 487 53.32 52.91 52.84 0 iosd

show platform cpu jobs

44 61597244 9018836 6829 69.69% 71.94% 57.99% K5L3Unicast Adj Tabl

61 17048546 29691925 574 29.65% 30.24% 24.31% K5CpuMan Review

daniel.dib · ‎11-26-2013

Refer to this document for troubleshooting CPU on Cat4500:

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml

Seems your RIB is getting updated a lot, do you have many routes that are very new? No flapping routing adjacencies? Anything else in the logs?

Daniel Dib
CCIE #37149

Daniel Dib
CCIE #37149
CCDE #20160011

Please rate helpful posts.

kevinhum122 · ‎11-27-2013

Hi Daniel,

We just replaced our core switch 3750 with 4507, all the configurations are copied by the original settings, and each interface with the command "logging event link-status", no abnormal is found ~~

Core_WS-C4507#sh platform health

%CPU %CPU RunTimeMax Priority Average %CPU Total

K5CpuMan Review 30.00 24.88 30 221 100 500 24 24 18 420:04

K5FlowStatsFifoSearc 25.00 0.00 2 0 100 500 0 0 0 0:00

%CPU Totals 368.50 122.15

Allocation ceiling Current allocation

------------------ ------------------

kbytes % in use kbytes % in use

Chassis 1 Linecard 1 5120.00 4% 227.95 100%

Chassis 1 Linecard 2 5120.00 45% 2310.53 100%

Chassis 1 Linecard 3 5120.00 10% 522.28 100%

Chassis 1 Linecard 4 5120.00 0% 0.00 0%

Chassis 1 Linecard 5 5120.00 45% 2310.53 100%

Chassis 1 Linecard 6 5120.00 0% 0.00 0%

Chassis 1 Linecard 7 5120.00 0% 0.00 0%

TSM objects ------------------ ------------------

PacketInfoItem 781.25 0% 1.32 0%

VbufNodes2400 80.50 0% 0.00 0%

VbufNodes1600 55.50 0% 3.46 0%

VbufNodes400 288.00 6% 18.00 96%

VbufNodes64 60.00 24% 15.23 96%

VbufNodes4200 68.37 6% 4.27 100%

Packet 2474.28 0% 0.21 0%

RkiosSysPacketBuf 281.25 0% 1.23 2%

IndexCache 800.78 0% 0.00 0%

K5InternalVlanIdMap 96.00 3% 3.00 100%

K5AclOpDescNode 10752.00 0% 0.00 0%

K5AclRetMapEntryNode 41.00 0% 0.00 0%

K5AclLabelListNode 768.00 0% 0.00 0%

K5AclIpv6PackedAddrH 1024.00 0% 0.00 0%

K5AdjGroups 960.00 3% 37.07 91%

IrmFibUnicastRpfList 8192.00 0% 62.09 92%

K5L3FwdEntrys 27040.00 0% 213.89 92%

K5L3FwdEntryAvlTree2 12480.00 0% 49.50 92%

K5L3FwdTreeEntrys 21840.00 0% 173.25 92%

K5L3FwdTreeEntryAvlT 49920.00 0% 49.35 92%

IrmMfibFastDropFlowM 576.00 0% 0.00 0%

K5QosTxQueSelTableBl 12.00 2% 0.32 100%

K5QosPolicerBlockNod 2.00 0% 0.00 0%

K5QosPolicerBlockMem 69.00 0% 0.00 0%

K5QosPolicerMemAlloc 448.00 0% 0.00 0%

K5QosFeatureInfoList 2048.00 0% 0.31 100%

K5QosLabelToFeatureE 1792.00 0% 0.05 100%

K5QosPathFeatureInfo 384.00 0% 0.01 100%

K5CpuPacketInfoItem 781.25 0% 0.00 0%

MatEntrys 18432.00 0% 20.81 100%

MatEntryTableIterato 1.00 0% 0.03 0%

RkiosL2MacVlanEntrie 64.00 0% 0.00 0%

RkiosL3Port 2732.90 0% 63.28 12%

PimPhyports 1820.31 41% 759.07 100%

PimPorts 1237.50 40% 496.03 100%

PimModules 296.00 1% 4.62 100%

PimSlots 18.00 2% 0.49 100%

PimChassis 8.25 50% 4.12 100%

PimQuack 1.64 13% 0.21 100%

EbmVlans 9152.00 1% 93.84 100%

EbmVlanGroupEntrys 6400.00 0% 8.78 95%

EbmPorts 487.50 39% 193.78 100%

EbmPortHostEntrys 448.00 0% 0.00 0%

EbmIeNodes 540.00 0% 5.27 100%

EbmPortVlanAclFeatur 3456.00 0% 0.00 0%

EbmPortVlanMap Alloc 64.00 0% 0.00 0%

EbmSortedHostTableIt 1.75 0% 0.05 0%

EbmSortedGroupTableI 1.75 0% 0.05 0%

EbmHostRedundancyMan 1082.81 0% 0.00 0%

EbmHostAgeRedundancy 1082.81 0% 0.00 0%

IrmVrfs 628.00 1% 7.35 100%

IrmFibLoadBalances 1280.00 0% 0.07 100%

IrmFibAdjs 1920.00 3% 74.49 91%

IrmPortMemMan 6783.43 0% 229.68 12%

IrmPortEtherAddrEntr 500.00 0% 0.00 0%

IrmFibEntries 12288.00 0% 98.76 92%

IrmMfibEntryMemMan 14336.00 0% 0.00 0%

IrmWccpMemMan 36.37 0% 0.00 0%

IrmWccpServiceGroupL 0.06 0% 0.00 0%

AclOp 2176.00 0% 0.14 100%

AclOpAceSet 3808.00 0% 0.16 100%

AclClassifier 1280.00 0% 1.56 100%

AclFeature 2755.59 0% 2.67 100%

Acl 1536.00 0% 3.18 100%

Ace24 10880.00 0% 7.26 100%

Ace48 17408.00 0% 4.75 100%

AclFlowLabelListNode 7616.00 0% 0.00 0%

AceActionDescStorage 1088.00 0% 0.00 0%

AclListNode 512.00 0% 0.32 100%

AceListNode 25600.00 0% 0.71 28%

AclClassifierActionL 2560.00 0% 1.36 100%

AclLayerFeatureListN 512.00 0% 0.05 28%

AclClassifierListNod 512.00 0% 0.00 0%

TableMapMan NameToTa 27.00 0% 0.00 0%

TableMapAllocator 128.00 0% 0.00 0%

FlatAcl 512.00 0% 0.35 4%

FlatAce24 11264.00 0% 3.60 19%

FlatAce48 17408.00 0% 0.00 0%

FlatAceActionListNod 307200.00 0% 4.12 29%

FlatAclOpSetStorage 3072.00 0% 0.00 0%

FlatAclCacheNode 4224.00 0% 0.12 100%

FlatAclListNode 256.00 0% 0.17 0%

QosFeatureClassifier 353.03 0% 0.00 0%

QosFeatureClassifier 706.06 0% 0.00 0%

QosClassifierActionL 9884.87 0% 0.00 0%

QosNestedClassifierA 15533.37 0% 0.21 100%

QosPortVlanAclFeatur 1224.00 0% 0.00 0%

QoS Policers 37000.00 0% 0.00 0%

Qos FlowFnf 7.81 0% 0.00 0%

KxAclPath 1024.00 0% 0.03 100%

KxAclPathListNode 1280.00 0% 0.00 0%

KxAclConstPathListNo 1280.00 0% 0.05 14%

RkNovaFfmTokenMapEnt 88342.00 0% 226.21 93%

RkNovaFfmInvalidToke 12288.00 0% 0.84 16%

RkNova L3 PBR RouteM 512.00 0% 0.00 0%

RkNova L3 PBR RouteM 1718.75 0% 0.00 0%

RkNova L3 PBR RouteM 156.25 0% 0.00 0%

RkNovaL3Vrf 10.00 2% 0.23 100%

RkNovaQosLayer3Targe 24.00 0% 0.00 0%

RkNovaQosPimPortIdTo 24.00 0% 0.00 0%

RkNovaQoSPolicers 3000.00 0% 0.00 0%

RkNovaQosQueues 104.25 0% 0.00 0%

ffmToRkNovaQosTableM 6.00 0% 0.00 0%

RkNovaQosTableMaps 2.00 0% 0.00 0%

RkNovaClassifClassMa 4096.00 0% 0.00 0%

RkNovaClassifClassMa 768.00 0% 0.00 0%

ffmToRkNovaClassifPo 96.00 0% 0.00 0%

RkNovaClassifPolicyM 575.71 0% 0.00 0%

FfmTokenToGenAclIdMa 700.00 0% 3.37 100%

RkNova Acl VlanMaps 512.00 0% 0.00 0%

RkNova Acl VlanMapEn 1093.75 0% 0.00 0%

RkNova ACL VlanMapEn 156.25 0% 0.00 0%

metadataIdToDefaultA 1.75 0% 0.00 0%

defaultAclToMetadata 1.75 0% 0.00 0%

FlowMetadataFlowSet 450.00 0% 0.00 0%

MacsecTransmitScMan 84.00 0% 0.00 0%

MacsecTransmitSaMan 304.00 0% 0.00 0%

MacsecReceiveScMan 168.00 0% 0.00 0%

MacsecReceiveSaMan 448.00 0% 0.00 0%

KxAclLabeledFlatAcl 3584.00 0% 0.10 100%

KxAclLabeledFlatAclE 3072.00 0% 0.09 100%

EbmVlanHostEntrys 3222.65 2% 84.49 91%

Rkios Fnf Records 56.00 0% 0.00 0%

Rkios Fnf Monitors 112.00 0% 0.00 0%

FlowTable 1.21 0% 0.00 0%

VsiBuffers(4096) 400.00 0% 0.00 0%

VsiBuffers(1024) 1500.00 21% 320.00 99%

VsiBuffers(128) 762.50 1% 43.37 28%

VsiBuffers(16) 146.87 0% 18.73 3%

VsiTransactions(1) 123.04 1% 9.02 22%

VsiTransactions(10) 266.60 6% 35.95 50%

VsiTransactions(18) 77.10 0% 17.62 0%

VsiTransactions(25) 88.59 0% 9.49 0%

VsiTransactions(80) 178.82 0% 20.43 0%

VsiTransactionRespon 31.71 3% 2.24 47%

VsiReqPool(s2w) 65.62 0% 1.68 33%

VsiReqPool(vli) 782.03 5% 134.92 29%

VsiReqPool(mdio22) 328.12 0% 112.50 0%

VsiReqPool(mdio45) 229.68 0% 0.37 0%

GalGbicEntrys 19.35 0% 0.00 0%

IrmMfibIntrfs 6144.00 0% 0.00 0%

Event Nodes 160.00 0% 0.00 0%

Event Nodes 160.00 0% 2.87 0%

K5L3FlcEntryAvlTree2 3225.58 2% 98.71 92%

K5FlowCachedFifoStat 19456.00 0% 0.00 0%

K5FlowQosFptPolicerE 2048.00 0% 0.00 0%

K5FlowQosFeatureInfo 1024.00 0% 0.00 0%

K5FlowQosLabelToFeat 896.00 0% 0.00 0%

K5FlowOperatorDescNo 612.00 0% 0.03 0%

K5AclLabelSignatureM 10368.00 0% 0.31 100%

K5AclLabelMapEntryPa 1408.00 0% 0.00 0%

TableMapMan NameToTa 27.00 0% 0.00 0%

TableMapAllocator 128.00 0% 0.00 0%

InpTosMarkTbl BlockA 14.00 1% 0.21 100%

InpCosMarkTbl BlockA 14.00 1% 0.21 100%

InpExpMarkTbl BlockA 14.00 1% 0.21 100%

OutTosMarkTbl BlockA 14.00 1% 0.21 100%

OutCosMarkTbl BlockA 14.00 1% 0.21 100%

OutExpMarkTbl BlockA 14.00 1% 0.21 100%

K5TxPacketInfo 384.00 0% 0.82 0%

K5TxPacket 320.00 0% 0.03 0%

K5FlowPathKeyListNod 384.00 0% 0.00 0%

K5FlowAclLabelInfo 5376.00 0% 0.00 0%

K5FlowPvLabelPathInf 896.00 0% 0.00 0%

TableMapMan NameToTa 27.00 0% 0.00 0%

TableMapAllocator 128.00 0% 0.00 0%

FlowTosMarkTbl Block 14.00 1% 0.21 100%

FlowCosMarkTbl Block 14.00 1% 0.21 100%

FlowExpMarkTbl Block 14.00 1% 0.21 100%

RkNova L3 WCCP Servi 186.87 0% 0.00 0%

CommandTables 48.00 13% 6.65 100%

------------------ ------------------

TSM totals 904065.96 0% 3877.56 82%

Core_WS-C4507#sh platform cpu jobs

44 98058602 14436329 6792 60.37% 70.35% 59.17% K5L3Unicast Adj Tabl

58 15055628 5369212 2804 14.88% 17.21% 14.51% K5 L2 Hardware Addre

61 25251009 43433128 581 22.78% 24.98% 19.90% K5CpuMan Review

Core_WS-C4507#show platform cpu packet statistics

Host Learning 233116896 2613 2777 2289 2159

L2 Control 7421246 28 17 16 5

Input ACL log, unreach 4416 0 0 0 0

L3 Glean 1261709 9 7 5 2

L3 Receive 155637 0 0 0 0

Ttl Expired 2319 0 0 0 0

Bfd 4803 0 0 0 0

Adj SameIf Fail 37012387 356 326 205 148

L2 router to CPU, 7 20658263 231 212 163 154

L3 Fwd 13 0 0 0 0

Could you give me some nice suggestions ? So appreciated in advance.