Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

98% CPU on New Core Switch 4507

I find that it's so abnormal CPU performance on 4507, the process "iosd" occupies lots of CPU resource, the details as below, anyone who can help me ?

Core_WS-C4507#sh proc cpu sorted

Core 0: CPU utilization for five seconds: 8%; one minute: 27%;  five minutes: 37%

Core 1: CPU utilization for five seconds: 98%; one minute: 78%;  five minutes: 68%

PID    Runtime(ms) Invoked  uSecs  5Sec     1Min     5Min     TTY   Process

8518   3252698     27999562 487    53.32    52.91    52.84    0     iosd         

show platform cpu jobs

44    61597244   9018836       6829 69.69% 71.94% 57.99% K5L3Unicast Adj Tabl

61    17048546  29691925        574 29.65% 30.24% 24.31% K5CpuMan Review 

Everyone's tags (1)
2 REPLIES
Silver

98% CPU on New Core Switch 4507

Refer to this document for troubleshooting CPU on Cat4500:

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml

Seems your RIB is getting updated a lot, do you have many routes that are very new? No flapping routing adjacencies? Anything else in the logs?

Daniel Dib
CCIE #37149

Daniel Dib CCIE #37149 Please rate helpful posts.
New Member

Re: 98% CPU on New Core Switch 4507

Hi Daniel,

    We just replaced our core switch 3750 with 4507, all the configurations are copied by the original settings, and each interface with the command "logging event link-status", no abnormal is found ~~ 

Core_WS-C4507#sh platform health

                     %CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total

K5CpuMan Review       30.00  24.88     30    221  100  500   24  24   18  420:04

K5FlowStatsFifoSearc  25.00   0.00      2      0  100  500    0   0    0  0:00

%CPU Totals          368.50 122.15

                       Allocation ceiling        Current allocation

                       ------------------        ------------------

                       kbytes    % in use        kbytes    % in use

Chassis 1 Linecard 1     5120.00       4%          227.95      100%

Chassis 1 Linecard 2     5120.00      45%         2310.53      100%

Chassis 1 Linecard 3     5120.00      10%          522.28      100%

Chassis 1 Linecard 4     5120.00       0%            0.00        0%

Chassis 1 Linecard 5     5120.00      45%         2310.53      100%

Chassis 1 Linecard 6     5120.00       0%            0.00        0%

Chassis 1 Linecard 7     5120.00       0%            0.00        0%

TSM objects            ------------------        ------------------

PacketInfoItem            781.25       0%            1.32        0%

VbufNodes2400              80.50       0%            0.00        0%

VbufNodes1600              55.50       0%            3.46        0%

VbufNodes400              288.00       6%           18.00       96%

VbufNodes64                60.00      24%           15.23       96%

VbufNodes4200              68.37       6%            4.27      100%

Packet                   2474.28       0%            0.21        0%

RkiosSysPacketBuf         281.25       0%            1.23        2%

IndexCache                800.78       0%            0.00        0%

K5InternalVlanIdMap        96.00       3%            3.00      100%

K5AclOpDescNode         10752.00       0%            0.00        0%

K5AclRetMapEntryNode       41.00       0%            0.00        0%

K5AclLabelListNode        768.00       0%            0.00        0%

K5AclIpv6PackedAddrH     1024.00       0%            0.00        0%

K5AdjGroups               960.00       3%           37.07       91%

IrmFibUnicastRpfList     8192.00       0%           62.09       92%

K5L3FwdEntrys           27040.00       0%          213.89       92%

K5L3FwdEntryAvlTree2    12480.00       0%           49.50       92%

K5L3FwdTreeEntrys       21840.00       0%          173.25       92%

K5L3FwdTreeEntryAvlT    49920.00       0%           49.35       92%

IrmMfibFastDropFlowM      576.00       0%            0.00        0%

K5QosTxQueSelTableBl       12.00       2%            0.32      100%

K5QosPolicerBlockNod        2.00       0%            0.00        0%

K5QosPolicerBlockMem       69.00       0%            0.00        0%

K5QosPolicerMemAlloc      448.00       0%            0.00        0%

K5QosFeatureInfoList     2048.00       0%            0.31      100%

K5QosLabelToFeatureE     1792.00       0%            0.05      100%

K5QosPathFeatureInfo      384.00       0%            0.01      100%

K5CpuPacketInfoItem       781.25       0%            0.00        0%

MatEntrys               18432.00       0%           20.81      100%

MatEntryTableIterato        1.00       0%            0.03        0%

RkiosL2MacVlanEntrie       64.00       0%            0.00        0%

RkiosL3Port              2732.90       0%           63.28       12%

PimPhyports              1820.31      41%          759.07      100%

PimPorts                 1237.50      40%          496.03      100%

PimModules                296.00       1%            4.62      100%

PimSlots                   18.00       2%            0.49      100%

PimChassis                  8.25      50%            4.12      100%

PimQuack                    1.64      13%            0.21      100%

EbmVlans                 9152.00       1%           93.84      100%

EbmVlanGroupEntrys       6400.00       0%            8.78       95%

EbmPorts                  487.50      39%          193.78      100%

EbmPortHostEntrys         448.00       0%            0.00        0%

EbmIeNodes                540.00       0%            5.27      100%

EbmPortVlanAclFeatur     3456.00       0%            0.00        0%

EbmPortVlanMap Alloc       64.00       0%            0.00        0%

EbmSortedHostTableIt        1.75       0%            0.05        0%

EbmSortedGroupTableI        1.75       0%            0.05        0%

EbmHostRedundancyMan     1082.81       0%            0.00        0%

EbmHostAgeRedundancy     1082.81       0%            0.00        0%

IrmVrfs                   628.00       1%            7.35      100%

IrmFibLoadBalances       1280.00       0%            0.07      100%

IrmFibAdjs               1920.00       3%           74.49       91%

IrmPortMemMan            6783.43       0%          229.68       12%

IrmPortEtherAddrEntr      500.00       0%            0.00        0%

IrmFibEntries           12288.00       0%           98.76       92%

IrmMfibEntryMemMan      14336.00       0%            0.00        0%

IrmWccpMemMan              36.37       0%            0.00        0%

IrmWccpServiceGroupL        0.06       0%            0.00        0%

AclOp                    2176.00       0%            0.14      100%

AclOpAceSet              3808.00       0%            0.16      100%

AclClassifier            1280.00       0%            1.56      100%

AclFeature               2755.59       0%            2.67      100%

Acl                      1536.00       0%            3.18      100%

Ace24                   10880.00       0%            7.26      100%

Ace48                   17408.00       0%            4.75      100%

AclFlowLabelListNode     7616.00       0%            0.00        0%

AceActionDescStorage     1088.00       0%            0.00        0%

AclListNode               512.00       0%            0.32      100%

AceListNode             25600.00       0%            0.71       28%

AclClassifierActionL     2560.00       0%            1.36      100%

AclLayerFeatureListN      512.00       0%            0.05       28%

AclClassifierListNod      512.00       0%            0.00        0%

TableMapMan NameToTa       27.00       0%            0.00        0%

TableMapAllocator         128.00       0%            0.00        0%

FlatAcl                   512.00       0%            0.35        4%

FlatAce24               11264.00       0%            3.60       19%

FlatAce48               17408.00       0%            0.00        0%

FlatAceActionListNod   307200.00       0%            4.12       29%

FlatAclOpSetStorage      3072.00       0%            0.00        0%

FlatAclCacheNode         4224.00       0%            0.12      100%

FlatAclListNode           256.00       0%            0.17        0%

QosFeatureClassifier      353.03       0%            0.00        0%

QosFeatureClassifier      706.06       0%            0.00        0%

QosClassifierActionL     9884.87       0%            0.00        0%

QosNestedClassifierA    15533.37       0%            0.21      100%

QosPortVlanAclFeatur     1224.00       0%            0.00        0%

QoS Policers            37000.00       0%            0.00        0%

Qos FlowFnf                 7.81       0%            0.00        0%

KxAclPath                1024.00       0%            0.03      100%

KxAclPathListNode        1280.00       0%            0.00        0%

KxAclConstPathListNo     1280.00       0%            0.05       14%

RkNovaFfmTokenMapEnt    88342.00       0%          226.21       93%

RkNovaFfmInvalidToke    12288.00       0%            0.84       16%

RkNova L3 PBR RouteM      512.00       0%            0.00        0%

RkNova L3 PBR RouteM     1718.75       0%            0.00        0%

RkNova L3 PBR RouteM      156.25       0%            0.00        0%

RkNovaL3Vrf                10.00       2%            0.23      100%

RkNovaQosLayer3Targe       24.00       0%            0.00        0%

RkNovaQosPimPortIdTo       24.00       0%            0.00        0%

RkNovaQoSPolicers        3000.00       0%            0.00        0%

RkNovaQosQueues           104.25       0%            0.00        0%

ffmToRkNovaQosTableM        6.00       0%            0.00        0%

RkNovaQosTableMaps          2.00       0%            0.00        0%

RkNovaClassifClassMa     4096.00       0%            0.00        0%

RkNovaClassifClassMa      768.00       0%            0.00        0%

ffmToRkNovaClassifPo       96.00       0%            0.00        0%

RkNovaClassifPolicyM      575.71       0%            0.00        0%

FfmTokenToGenAclIdMa      700.00       0%            3.37      100%

RkNova Acl VlanMaps       512.00       0%            0.00        0%

RkNova Acl VlanMapEn     1093.75       0%            0.00        0%

RkNova ACL VlanMapEn      156.25       0%            0.00        0%

metadataIdToDefaultA        1.75       0%            0.00        0%

defaultAclToMetadata        1.75       0%            0.00        0%

FlowMetadataFlowSet       450.00       0%            0.00        0%

MacsecTransmitScMan        84.00       0%            0.00        0%

MacsecTransmitSaMan       304.00       0%            0.00        0%

MacsecReceiveScMan        168.00       0%            0.00        0%

MacsecReceiveSaMan        448.00       0%            0.00        0%

KxAclLabeledFlatAcl      3584.00       0%            0.10      100%

KxAclLabeledFlatAclE     3072.00       0%            0.09      100%

EbmVlanHostEntrys        3222.65       2%           84.49       91%

Rkios Fnf Records          56.00       0%            0.00        0%

Rkios Fnf Monitors        112.00       0%            0.00        0%

FlowTable                   1.21       0%            0.00        0%

VsiBuffers(4096)          400.00       0%            0.00        0%

VsiBuffers(1024)         1500.00      21%          320.00       99%

VsiBuffers(128)           762.50       1%           43.37       28%

VsiBuffers(16)            146.87       0%           18.73        3%

VsiTransactions(1)        123.04       1%            9.02       22%

VsiTransactions(10)       266.60       6%           35.95       50%

VsiTransactions(18)        77.10       0%           17.62        0%

VsiTransactions(25)        88.59       0%            9.49        0%

VsiTransactions(80)       178.82       0%           20.43        0%

VsiTransactionRespon       31.71       3%            2.24       47%

VsiReqPool(s2w)            65.62       0%            1.68       33%

VsiReqPool(vli)           782.03       5%          134.92       29%

VsiReqPool(mdio22)        328.12       0%          112.50        0%

VsiReqPool(mdio45)        229.68       0%            0.37        0%

GalGbicEntrys              19.35       0%            0.00        0%

IrmMfibIntrfs            6144.00       0%            0.00        0%

Event Nodes               160.00       0%            0.00        0%

Event Nodes               160.00       0%            2.87        0%

K5L3FlcEntryAvlTree2     3225.58       2%           98.71       92%

K5FlowCachedFifoStat    19456.00       0%            0.00        0%

K5FlowQosFptPolicerE     2048.00       0%            0.00        0%

K5FlowQosFeatureInfo     1024.00       0%            0.00        0%

K5FlowQosLabelToFeat      896.00       0%            0.00        0%

K5FlowOperatorDescNo      612.00       0%            0.03        0%

K5AclLabelSignatureM    10368.00       0%            0.31      100%

K5AclLabelMapEntryPa     1408.00       0%            0.00        0%

TableMapMan NameToTa       27.00       0%            0.00        0%

TableMapAllocator         128.00       0%            0.00        0%

InpTosMarkTbl BlockA       14.00       1%            0.21      100%

InpCosMarkTbl BlockA       14.00       1%            0.21      100%

InpExpMarkTbl BlockA       14.00       1%            0.21      100%

OutTosMarkTbl BlockA       14.00       1%            0.21      100%

OutCosMarkTbl BlockA       14.00       1%            0.21      100%

OutExpMarkTbl BlockA       14.00       1%            0.21      100%

K5TxPacketInfo            384.00       0%            0.82        0%

K5TxPacket                320.00       0%            0.03        0%

K5FlowPathKeyListNod      384.00       0%            0.00        0%

K5FlowAclLabelInfo       5376.00       0%            0.00        0%

K5FlowPvLabelPathInf      896.00       0%            0.00        0%

TableMapMan NameToTa       27.00       0%            0.00        0%

TableMapAllocator         128.00       0%            0.00        0%

FlowTosMarkTbl Block       14.00       1%            0.21      100%

FlowCosMarkTbl Block       14.00       1%            0.21      100%

FlowExpMarkTbl Block       14.00       1%            0.21      100%

RkNova L3 WCCP Servi      186.87       0%            0.00        0%

CommandTables              48.00      13%            6.65      100%

                       ------------------        ------------------

TSM totals             904065.96       0%         3877.56       82%

Core_WS-C4507#sh platform cpu jobs

44    98058602  14436329       6792 60.37% 70.35% 59.17% K5L3Unicast Adj Tabl

58    15055628   5369212       2804 14.88% 17.21% 14.51% K5 L2 Hardware Addre

61    25251009  43433128        581 22.78% 24.98% 19.90% K5CpuMan Review   

Core_WS-C4507#show platform cpu packet statistics

Host Learning                233116896      2613      2777      2289       2159

L2 Control                     7421246        28        17        16          5

Input ACL log, unreach            4416         0         0         0          0

L3 Glean                       1261709         9         7         5          2

L3 Receive                      155637         0         0         0          0

Ttl Expired                       2319         0         0         0          0

Bfd                               4803         0         0         0          0

Adj SameIf Fail               37012387       356       326       205        148

L2 router to CPU, 7           20658263       231       212       163        154

L3 Fwd                              13         0         0         0          0

Could you give me some nice suggestions ? So appreciated in advance.   

463
Views
0
Helpful
2
Replies
CreatePlease login to create content