Re: N7K high CPU

Krzysztof Zaleski · ‎12-02-2011

Hello,

I have 4 N7K divided into 16 VDCs. Among all features, there are vPC, EIGRP, CTS, etc running. There are many L2 switches and routers connected to that N7K core. Since the implementation there is a high CPU load on all N7Ks, and lots of CoPP drops in class-default:

HQ1# show policy-map interface control-plane

Control Plane

[...]

class-map class-default (match-any)

set cos 0

police cir 100 kbps , bc 250 ms

module 1 :

conformed 28515934638 bytes; action: transmit

violated 811542237967 bytes; action: drop

module 3 :

conformed 27982500685 bytes; action: transmit

violated 192219772385 bytes; action: drop

module 8 :

conformed 13535911696 bytes; action: transmit

violated 34154857224 bytes; action: drop

module 10 :

conformed 28506039898 bytes; action: transmit

violated 249388190198 bytes; action: drop

HQ1# show proc cp his

1111122221 2132111122111222121112111113112111224332211112111

733469586876320137626337353209536663641440054245206499897922

100

90

80

70

60

50 #

40 #

30 #### # # # # # # #### #

20 # ###### # ## #### #### ### ### # # # # #############

10 ############################################################

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5

CPU% per second (last 60 seconds)

# = average CPU%

546573544485456655364576457577467742452445665774363547456569

289736633856876232605901545671670058018453073036938241712899

100 *

90 * *

80 * * * *

70 * * * * * * ** *** * ** * **

60 *** * ** *** * *** **** *** ** ** * * ****

50 ***** * ********* **************** * ******** * * *******

40 *********************************** ** *********************

30 ***********************************************************#

20 ############################################################

10 ############################################################

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5

CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%

11111 11 111 1 1111111111111111 1 1111 1 1

000007776790086675779998679000768699060000000000000000877899077000069090

000004604780023369944305965000861770080000000000000000288357050000087040

100 ***** *** **** * * **********##**** *** **** ** *

90 *#*** *** **** **** *** *******######*** *** **** ****

80 *##** * ***** * * **** ****** * *** ****#########*********** **** ****

70 ###***** ***** * *********************############*************##******

60 ####***********************************#############***********###******

50 ####**********************************##############***********###******

40 ####************************#*********##############***********###***#**

30 ####*******##**************##*********##############***********####**#*#

20 ########################################################################

10 ########################################################################

0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)

* = maximum CPU% # = average CPU%

How can I track what traffic is being that excessively dropped and making the CPU busy?

Best regards,

Krzysztof

Amit Singh · ‎12-02-2011

Krzysztof,

Please could you paste the output of " show proc cpu "

Cheers,

-amit singh

Krzysztof Zaleski · ‎12-02-2011

Here it is:

HQ# sh proc cp so

PID Runtime(ms) Invoked uSecs 1Sec Process

----- ----------- -------- ----- ------ -----------

2847 2074559 200667843 10 0.9% syslogd

2849 74018842 11939335 6199 0.9% platform

1 31946 866006 36 0.0% init

2 2 300 9 0.0% kthreadd

3 536 81347 6 0.0% migration/0

4 1402672 360216359 3 0.0% ksoftirqd/0

5 75017 324096 231 0.0% watchdog/0

6 401 61999 6 0.0% migration/1

7 1073567 319910877 3 0.0% ksoftirqd/1

8 902 267814 3 0.0% watchdog/1

9 31590 4533198 6 0.0% events/0

10 30685 4399335 6 0.0% events/1

11 0 20 21 0.0% khelper

12 12076 13267 910 0.0% kblockd/0

13 35787 36611 977 0.0% kblockd/1

14 0 2 0 0.0% kacpid

15 0 2 0 0.0% kacpi_notify

16 0 4 22 0.0% ksuspend_usbd

17 0 5 2 0.0% khubd

18 7641 148136 51 0.0% pdflush

19 43810 787986 55 0.0% pdflush

20 0 2 2 0.0% kswapd0

21 0 2 2 0.0% aio/0

22 0 2 2 0.0% aio/1

23 0 2 1 0.0% nfsiod

24 0 2 2 0.0% SerrLogKthread

25 55361 2569 21549 0.0% kide/0

26 1498896 1851140 809 0.0% kide/1

29 0 18 9 0.0% rpciod/0

30 0 3 8 0.0% rpciod/1

31 86412 9818230 8 0.0% kirqd

660 1849 613 3017 0.0% jffs2_gcd_mtd2

748 10468 142236 73 0.0% kjournald

753 23221 142174 163 0.0% kjournald

864 23475 142092 165 0.0% kjournald

1967 32976 178179 185 0.0% kjournald

2210 1 31 37 0.0% portmap

2219 3 5 672 0.0% rpc.statd

2234 0 2 4 0.0% lockd

2235 7 1180 6 0.0% nfsd

2236 7 1178 6 0.0% nfsd

2237 7 1180 6 0.0% nfsd

2238 8 1179 7 0.0% nfsd

2239 7 1178 6 0.0% nfsd

2240 7 1178 6 0.0% nfsd

2241 7 1180 6 0.0% nfsd

2242 7 1180 6 0.0% nfsd

2247 0 2 75 0.0% rpc.mountd

2267 2998696 10030928 298 0.0% sysmgr

2597 0 1 26 0.0% mping-thread

2649 3580 141130 25 0.0% redun_kthread

2845 18 81 229 0.0% xinetd

2846 101 753 134 0.0% tftpd

2848 119 89 1346 0.0% sdwrapd

2856 0 5 21 0.0% usd_mts_kthread

2869 11568 2116786 5 0.0% ls-notify-mts-t

2952 1937270 1479204 1309 0.0% cmond

2953 4210 956 4404 0.0% pfm_dummy

2958 14733 537796 27 0.0% vshd

2959 58 19 3104 0.0% smm

2960 51703 1619661 31 0.0% session-mgr

2961 71237 4242286 16 0.0% psshelper

2962 5039 70622 71 0.0% lmgrd

2963 52907 973456 54 0.0% licmgr

2964 5243 138720 37 0.0% fs-daemon

2965 8775 356542 24 0.0% feature-mgr

2966 1174 17921 65 0.0% confcheck

2967 24103 806876 29 0.0% capability

2968 71574 4241338 16 0.0% psshelper_gsvc

2978 26986 139757 193 0.0% cisco

2979 36714 430403 85 0.0% clis

2981 57248 1623333 35 0.0% port-profile

2982 37935 1415964 26 0.0% xmlma

2983 51704 1619536 31 0.0% vmm

2984 45994 1621709 28 0.0% vdc_mgr

2985 33336 2121305 15 0.0% ttyd

2986 12040 460281 26 0.0% sysinfo

2987 247 2978 83 0.0% sksd

2989 3166 87427 36 0.0% res_mgr

2990 100433 4266394 23 0.0% plugin

2991 281593 956868 294 0.0% plog_sup

2992 118 33 3597 0.0% npacl

2993 17603 541263 32 0.0% mvsh

2994 24 68 361 0.0% mping_server

2995 112283 865536 129 0.0% idehsd

2996 16817 543128 30 0.0% evms

2997 13486 410504 32 0.0% evmc

2998 42708917 15826398 2698 0.0% diagmgr

2999 38208 413588 92 0.0% crdcfg_server

3000 178 103 1730 0.0% core-dmon

3001 132327 1339104 98 0.0% cmpproxy

3002 87657 1471991 59 0.0% clk_mgr

3003 15605 848316 18 0.0% bios_daemon

3004 13258 530941 24 0.0% ascii-cfg

3005 78341 4243397 18 0.0% securityd

3006 74547 4241517 17 0.0% cert_enroll

3007 73854 4249325 17 0.0% aaa

3009 13184 531926 24 0.0% obfl

3014 80 22 3636 0.0% l3vm

3015 138 42 3293 0.0% urib

3017 23939763 9250506 2587 0.0% diagclient

3022 186064 1075309 173 0.0% aclmgr

3044 68 22 3118 0.0% adjmgr

3045 86 34 2556 0.0% u6rib

3051 129 23 5612 0.0% arp

3054 83 33 2532 0.0% icmpv6

3088 347381 632027 549 0.0% kjournald

3097 7 121 59 0.0% dhcpd

3098 36882 884709 41 0.0% cardclient

3100 313157 6124480 51 0.0% device_test

3101 38952 631133 61 0.0% xbar

3105 2058978 10298317 199 0.0% module

3107 149 98 1525 0.0% ExceptionLog

3108 38476 2124738 18 0.0% bootvar

3109 29141 806943 36 0.0% ifmgr

3113 234 61 3845 0.0% netstack

3133 130932 4250063 30 0.0% radius

3134 68393262 763173833 89 0.0% cfs

3135 21 31 695 0.0% ip_dummy

3136 21 31 679 0.0% ipv6_dummy

3137 51784 788243 65 0.0% ntp

3138 52196 1626305 32 0.0% otm

3139 21 31 682 0.0% pktmgr_dummy

3140 1087741 5399131 201 0.0% snmpd

3141 21 31 678 0.0% tcpudp_dummy

3142 268753 2371388 113 0.0% cdp

3143 135 760 178 0.0% dcos-xinetd

3144 211378 5803489 36 0.0% ntpd

3153 83726 4236617 19 0.0% callhome

3154 9118711 13014717 700 0.0% R2D2_usd

3155 715495 4115392 173 0.0% arbiter

3156 1600147 4593855 348 0.0% babycaesar

3157 5534689 7409603 746 0.0% sac_usd

3158 27884066 8092444 3445 0.0% sensor

3159 94356632 7430630 12698 0.0% xbar_driver_usd

3210 43185299 503282689 85 0.0% mts-sync-thr

3259 49744160 14142624 3517 0.0% oc_usd

3290 3351658 14674263 228 0.0% metropolis_usd

3354 16383 846862 19 0.0% sctpt_rx_thr

3355 0 1 5 0.0% sctpt_tx_thr

3387 0 2 7 0.0% kauditd

3399 23726 807581 29 0.0% u2

3400 198569 922879 215 0.0% spm

3401 51519 1620900 31 0.0% sal

3402 162 87 1870 0.0% rpm

3403 729986 2518050 289 0.0% pltfm_config

3404 75084 1684128 44 0.0% pixmc

3405 974954 4203012 231 0.0% pixm

3406 661070 1829195 361 0.0% pfstat

3407 374 238 1574 0.0% mrib

3408 49321 1621847 30 0.0% monitorc

3409 62753 1624401 38 0.0% monitor

3410 65158 1619636 40 0.0% mcm

3411 256 2729 93 0.0% m6rib

3412 1984630 5228426 379 0.0% l2pt

3413 113738811 170834093 665 0.0% l2fm

3414 30145 824023 36 0.0% eltm

3415 396391 1032987 383 0.0% copp

3416 387824 4917703 78 0.0% acllog

3418 185160 1222405 151 0.0% mfdm

3422 138076 1998698 69 0.0% ufdm

3424 230 4333 53 0.0% igmp

3425 218 3642 60 0.0% mcastfwd

3431 30195 817807 36 0.0% eth_port_channel

3432 27227 617739 44 0.0% vlan_mgr

3433 30714 432260 71 0.0% m2rib

3449 9754504 29329441 332 0.0% diag_port_lb

3456 2501 77484 32 0.0% eth_dstats

3458 67477 878905 76 0.0% ethpm

3459 1967847 1691399 1163 0.0% ipqosmgr

3465 5352838 38732178 138 0.0% stp

3485 24158 808779 29 0.0% port_client

3486 34851336 28839667 1208 0.0% statsclient

3487 22595 676715 33 0.0% xbar_client

3502 42270 4232044 9 0.0% wdpunch_thread

5341 8 51 161 0.0% gettylogin1

5541 23 30 771 0.0% login

7589 151 297 509 0.0% vsh

7676 4 8 529 0.0% vsh

7699 2 6 494 0.0% vdclogin

7702 16 15 1123 0.0% login

7703 442 1677 263 0.0% vsh

7986 6 12 546 0.0% more

16274 6 21 315 0.0% getty

18349 541 651 832 0.0% dcos_sshd

18352 142 179 793 0.0% vsh

18414 4 2 2113 0.0% more

18415 2 3 749 0.0% vsh

18416 41 22 1901 0.0% ps

19987 3062091 11691544 261 0.0% nfm

19995 1371823 3669210 373 0.0% lldp

20003 135771 930985 145 0.0% dhcp_snoop

20011 150947 1719326 87 0.0% bfd

20019 51214 1577299 32 0.0% bfd_app

20028 13036336 40225132 324 0.0% lacp

20036 10842459 9653444 1123 0.0% udld

20044 43021 610610 70 0.0% vtp

20057 944851 11982408 78 0.0% vpc

20069 18102 402645 44 0.0% interface-vlan

20081 15643 399966 39 0.0% hsrp_engine

20095 38951865 15145696 2571 0.0% dot1x

20104 41635630 18196374 2288 0.0% cts

20269 178349 5338768 33 0.0% ntpd

20678 179077 5372702 33 0.0% ntpd

21853 177129 5335598 33 0.0% ntpd

23071 247267 4832371 51 0.0% klogd

27855 103 38 2717 0.0% eigrp

CPU util : 8.0% user, 9.1% kernel, 82.9% idle

Please note that only processes from the requested vdc are shown above

Amit Singh · ‎12-02-2011

Hi Krzysztof,

That does not reveal much of the info.

Please could you run " Ethanlyzer" on the local sup inband interface to see what pakets are htting CPU and being dropped.

N7K(config)# ethanalyzer local interface inband

You can capture that on a file save it to the local bootflash and paste it here.

Cheers,

-amit singh

Krzysztof Zaleski · ‎12-05-2011

I captured some packets at different times, and among other, regular traffic, there is huge amount of this type of packets:

Frame 665 (96 bytes on wire, 64 bytes captured)

Arrival Time: Dec 5, 2011 12:06:48.908873000

[Time delta from previous captured frame: 0.000031000 seconds]

[Time delta from previous displayed frame: 0.000031000 seconds]

[Time since reference or first frame: 0.415769000 seconds]

Frame Number: 665

Frame Length: 96 bytes

Capture Length: 64 bytes

[Frame is marked: False]

[Protocols in frame: eth:llc:stp]

IEEE 802.3 Ethernet

Destination: 01:00:0c:cc:cc:cd (01:00:0c:cc:cc:cd)

Address: 01:00:0c:cc:cc:cd (01:00:0c:cc:cc:cd)

.... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)

.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)

Source: 00:26:f0:1c:00:00 (00:26:f0:1c:00:00)

Address: 00:26:f0:1c:00:00 (00:26:f0:1c:00:00)

.... ...0 .... .... .... .... = IG bit: Individual address (unicast)

.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)

Length: 50

Logical-Link Control

DSAP: SNAP (0xaa)

IG Bit: Individual

SSAP: SNAP (0xaa)

CR Bit: Command

Control field: U, func=UI (0x03)

000. 00.. = Command: Unnumbered Information (0x00)

.... ..11 = Frame type: Unnumbered frame (0x03)

Organization Code: Cisco (0x00000c)

PID: PVSTP+ (0x010b)

Spanning Tree Protocol

Protocol Identifier: Spanning Tree Protocol (0x0000)

Protocol Version Identifier: Spanning Tree (0)

BPDU Type: Configuration (0x00)

BPDU flags: 0x00

0... .... = Topology Change Acknowledgment: No

.... ...0 = Topology Change: No

Root Identifier: 4096 / 0 / 00:23:04:ee:be:01

Root Bridge Priority: 4096

Root Bridge System ID Extension: 0

Root Bridge System ID: 00:23:04:ee:be:01

Root Path Cost: 0

Bridge Identifier: 32768 / 0 / 00:23:04:ee:be:04

Bridge Priority: 32768

Bridge System ID Extension: 0

Bridge System ID: 00:23:04:ee:be:04

Port identifier: 0x901b

There is MST implemented in the core, on all N7Ks, but there are also 10-15 PVST+ switches, which are reported by MST port type as either Bound(PVST) or Pre-STD-Rx. I wonder if there could be some inconsistency, or so?

Krzysztof Zaleski · ‎12-06-2011

OK, problem solved. Do NOT, I repeat, DO NOT enable peer-switch on non-roon N7K peers :-)

edgarvarela · ‎08-27-2015

Hi Krzysztof,

I saw your post. Could you detail the solution problem ? I didn't understand. Is the peer-switch the cause of the problem ?

I hope you remeber...4 years is many time.....