Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. If you'd prefer to explore, try our test area to get started. And see here for current known issues.

New Member

N7K high CPU

Hello,

I have 4 N7K divided into 16 VDCs. Among all features, there are vPC, EIGRP, CTS, etc running. There are many L2 switches and routers connected to that N7K core. Since the implementation there is a high CPU load on all N7Ks, and lots of CoPP drops in class-default:

HQ1# show policy-map interface control-plane

Control Plane

[...]

    class-map class-default (match-any)

      set cos 0

      police cir 100 kbps , bc 250 ms

      module 1 :

        conformed 28515934638 bytes; action: transmit

        violated 811542237967 bytes; action: drop

      module 3 :

        conformed 27982500685 bytes; action: transmit

        violated 192219772385 bytes; action: drop

      module 8 :

        conformed 13535911696 bytes; action: transmit

        violated 34154857224 bytes; action: drop

      module 10 :

        conformed 28506039898 bytes; action: transmit

        violated 249388190198 bytes; action: drop

HQ1# show proc cp his

    1111122221 2132111122111222121112111113112111224332211112111

    733469586876320137626337353209536663641440054245206499897922

100

90

80

70

60

50                                                #

40                                                #

30      ####  # #      #    #      #     #        ####     #

20 #   ###### # ##  ####  #### ### ### # #  # # #############

10 ############################################################

    0....5....1....1....2....2....3....3....4....4....5....5....

              0    5    0    5    0    5    0    5    0    5

               CPU% per second (last 60 seconds)

                      # = average CPU%

    546573544485456655364576457577467742452445665774363547456569

    289736633856876232605901545671670058018453073036938241712899

100                                                            *

90           *                                                *

80           *               * *                              *

70   * *     *   *       *   * ** ***         * **      *    **

60   *** *   ** ***   * ***  **** ***        ** **  *   *  ****

50 ***** *  ********* ****************  *  ******** * * *******

40 *********************************** ** *********************

30 ***********************************************************#

20 ############################################################

10 ############################################################

    0....5....1....1....2....2....3....3....4....4....5....5....

              0    5    0    5    0    5    0    5    0    5

               CPU% per minute (last 60 minutes)

              * = maximum CPU%   # = average CPU%

    11111      11              111      1 1111111111111111      1  1111  1 1

    000007776790086675779998679000768699060000000000000000877899077000069090

    000004604780023369944305965000861770080000000000000000288357050000087040

100 *****     ***             ****    * * **********##****    ***  **** ** *

90 *#***     ***       ****  ****    *** *******######***    ***  **** ****

80 *##** *  *****  * * **** ****** * *** ****#########*********** **** ****

70 ###***** *****  * *********************############*************##******

60 ####***********************************#############***********###******

50 ####**********************************##############***********###******

40 ####************************#*********##############***********###***#**

30 ####*******##**************##*********##############***********####**#*#

20 ########################################################################

10 ########################################################################

    0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

              0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)

                  * = maximum CPU%   # = average CPU%

How can I track what traffic is being that excessively dropped and making the CPU busy?

Best regards,

Krzysztof

  • LAN Switching and Routing
6 REPLIES
Cisco Employee

Re: N7K high CPU

Krzysztof,

Please could you paste the output of " show proc cpu "

Cheers,

-amit singh

New Member

Re: N7K high CPU

Here it is:

HQ# sh proc cp so

PID    Runtime(ms)  Invoked   uSecs  1Sec    Process

-----  -----------  --------  -----  ------  -----------

2847      2074559  200667843     10    0.9%  syslogd

2849     74018842  11939335   6199    0.9%  platform

    1        31946    866006     36    0.0%  init

    2            2       300      9    0.0%  kthreadd

    3          536     81347      6    0.0%  migration/0

    4      1402672  360216359      3    0.0%  ksoftirqd/0

    5        75017    324096    231    0.0%  watchdog/0

    6          401     61999      6    0.0%  migration/1

    7      1073567  319910877      3    0.0%  ksoftirqd/1

    8          902    267814      3    0.0%  watchdog/1

    9        31590   4533198      6    0.0%  events/0

   10        30685   4399335      6    0.0%  events/1

   11            0        20     21    0.0%  khelper

   12        12076     13267    910    0.0%  kblockd/0

   13        35787     36611    977    0.0%  kblockd/1

   14            0         2      0    0.0%  kacpid

   15            0         2      0    0.0%  kacpi_notify

   16            0         4     22    0.0%  ksuspend_usbd

   17            0         5      2    0.0%  khubd

   18         7641    148136     51    0.0%  pdflush

   19        43810    787986     55    0.0%  pdflush

   20            0         2      2    0.0%  kswapd0

   21            0         2      2    0.0%  aio/0

   22            0         2      2    0.0%  aio/1

   23            0         2      1    0.0%  nfsiod

   24            0         2      2    0.0%  SerrLogKthread

   25        55361      2569  21549    0.0%  kide/0

   26      1498896   1851140    809    0.0%  kide/1

   29            0        18      9    0.0%  rpciod/0

   30            0         3      8    0.0%  rpciod/1

   31        86412   9818230      8    0.0%  kirqd

  660         1849       613   3017    0.0%  jffs2_gcd_mtd2

  748        10468    142236     73    0.0%  kjournald

  753        23221    142174    163    0.0%  kjournald

  864        23475    142092    165    0.0%  kjournald

1967        32976    178179    185    0.0%  kjournald

2210            1        31     37    0.0%  portmap

2219            3         5    672    0.0%  rpc.statd

2234            0         2      4    0.0%  lockd

2235            7      1180      6    0.0%  nfsd

2236            7      1178      6    0.0%  nfsd

2237            7      1180      6    0.0%  nfsd

2238            8      1179      7    0.0%  nfsd

2239            7      1178      6    0.0%  nfsd

2240            7      1178      6    0.0%  nfsd

2241            7      1180      6    0.0%  nfsd

2242            7      1180      6    0.0%  nfsd

2247            0         2     75    0.0%  rpc.mountd

2267      2998696  10030928    298    0.0%  sysmgr

2597            0         1     26    0.0%  mping-thread

2649         3580    141130     25    0.0%  redun_kthread

2845           18        81    229    0.0%  xinetd

2846          101       753    134    0.0%  tftpd

2848          119        89   1346    0.0%  sdwrapd

2856            0         5     21    0.0%  usd_mts_kthread

2869        11568   2116786      5    0.0%  ls-notify-mts-t

2952      1937270   1479204   1309    0.0%  cmond

2953         4210       956   4404    0.0%  pfm_dummy

2958        14733    537796     27    0.0%  vshd

2959           58        19   3104    0.0%  smm

2960        51703   1619661     31    0.0%  session-mgr

2961        71237   4242286     16    0.0%  psshelper

2962         5039     70622     71    0.0%  lmgrd

2963        52907    973456     54    0.0%  licmgr

2964         5243    138720     37    0.0%  fs-daemon

2965         8775    356542     24    0.0%  feature-mgr

2966         1174     17921     65    0.0%  confcheck

2967        24103    806876     29    0.0%  capability

2968        71574   4241338     16    0.0%  psshelper_gsvc

2978        26986    139757    193    0.0%  cisco

2979        36714    430403     85    0.0%  clis

2981        57248   1623333     35    0.0%  port-profile

2982        37935   1415964     26    0.0%  xmlma

2983        51704   1619536     31    0.0%  vmm

2984        45994   1621709     28    0.0%  vdc_mgr

2985        33336   2121305     15    0.0%  ttyd

2986        12040    460281     26    0.0%  sysinfo

2987          247      2978     83    0.0%  sksd

2989         3166     87427     36    0.0%  res_mgr

2990       100433   4266394     23    0.0%  plugin

2991       281593    956868    294    0.0%  plog_sup

2992          118        33   3597    0.0%  npacl

2993        17603    541263     32    0.0%  mvsh

2994           24        68    361    0.0%  mping_server

2995       112283    865536    129    0.0%  idehsd

2996        16817    543128     30    0.0%  evms

2997        13486    410504     32    0.0%  evmc

2998     42708917  15826398   2698    0.0%  diagmgr

2999        38208    413588     92    0.0%  crdcfg_server

3000          178       103   1730    0.0%  core-dmon

3001       132327   1339104     98    0.0%  cmpproxy

3002        87657   1471991     59    0.0%  clk_mgr

3003        15605    848316     18    0.0%  bios_daemon

3004        13258    530941     24    0.0%  ascii-cfg

3005        78341   4243397     18    0.0%  securityd

3006        74547   4241517     17    0.0%  cert_enroll

3007        73854   4249325     17    0.0%  aaa

3009        13184    531926     24    0.0%  obfl

3014           80        22   3636    0.0%  l3vm

3015          138        42   3293    0.0%  urib

3017     23939763   9250506   2587    0.0%  diagclient

3022       186064   1075309    173    0.0%  aclmgr

3044           68        22   3118    0.0%  adjmgr

3045           86        34   2556    0.0%  u6rib

3051          129        23   5612    0.0%  arp

3054           83        33   2532    0.0%  icmpv6

3088       347381    632027    549    0.0%  kjournald

3097            7       121     59    0.0%  dhcpd

3098        36882    884709     41    0.0%  cardclient

3100       313157   6124480     51    0.0%  device_test

3101        38952    631133     61    0.0%  xbar

3105      2058978  10298317    199    0.0%  module

3107          149        98   1525    0.0%  ExceptionLog

3108        38476   2124738     18    0.0%  bootvar

3109        29141    806943     36    0.0%  ifmgr

3113          234        61   3845    0.0%  netstack

3133       130932   4250063     30    0.0%  radius

3134     68393262  763173833     89    0.0%  cfs

3135           21        31    695    0.0%  ip_dummy

3136           21        31    679    0.0%  ipv6_dummy

3137        51784    788243     65    0.0%  ntp

3138        52196   1626305     32    0.0%  otm

3139           21        31    682    0.0%  pktmgr_dummy

3140      1087741   5399131    201    0.0%  snmpd

3141           21        31    678    0.0%  tcpudp_dummy

3142       268753   2371388    113    0.0%  cdp

3143          135       760    178    0.0%  dcos-xinetd

3144       211378   5803489     36    0.0%  ntpd

3153        83726   4236617     19    0.0%  callhome

3154      9118711  13014717    700    0.0%  R2D2_usd

3155       715495   4115392    173    0.0%  arbiter

3156      1600147   4593855    348    0.0%  babycaesar

3157      5534689   7409603    746    0.0%  sac_usd

3158     27884066   8092444   3445    0.0%  sensor

3159     94356632   7430630  12698    0.0%  xbar_driver_usd

3210     43185299  503282689     85    0.0%  mts-sync-thr

3259     49744160  14142624   3517    0.0%  oc_usd

3290      3351658  14674263    228    0.0%  metropolis_usd

3354        16383    846862     19    0.0%  sctpt_rx_thr

3355            0         1      5    0.0%  sctpt_tx_thr

3387            0         2      7    0.0%  kauditd

3399        23726    807581     29    0.0%  u2

3400       198569    922879    215    0.0%  spm

3401        51519   1620900     31    0.0%  sal

3402          162        87   1870    0.0%  rpm

3403       729986   2518050    289    0.0%  pltfm_config

3404        75084   1684128     44    0.0%  pixmc

3405       974954   4203012    231    0.0%  pixm

3406       661070   1829195    361    0.0%  pfstat

3407          374       238   1574    0.0%  mrib

3408        49321   1621847     30    0.0%  monitorc

3409        62753   1624401     38    0.0%  monitor

3410        65158   1619636     40    0.0%  mcm

3411          256      2729     93    0.0%  m6rib

3412      1984630   5228426    379    0.0%  l2pt

3413    113738811  170834093    665    0.0%  l2fm

3414        30145    824023     36    0.0%  eltm

3415       396391   1032987    383    0.0%  copp

3416       387824   4917703     78    0.0%  acllog

3418       185160   1222405    151    0.0%  mfdm

3422       138076   1998698     69    0.0%  ufdm

3424          230      4333     53    0.0%  igmp

3425          218      3642     60    0.0%  mcastfwd

3431        30195    817807     36    0.0%  eth_port_channel

3432        27227    617739     44    0.0%  vlan_mgr

3433        30714    432260     71    0.0%  m2rib

3449      9754504  29329441    332    0.0%  diag_port_lb

3456         2501     77484     32    0.0%  eth_dstats

3458        67477    878905     76    0.0%  ethpm

3459      1967847   1691399   1163    0.0%  ipqosmgr

3465      5352838  38732178    138    0.0%  stp

3485        24158    808779     29    0.0%  port_client

3486     34851336  28839667   1208    0.0%  statsclient

3487        22595    676715     33    0.0%  xbar_client

3502        42270   4232044      9    0.0%  wdpunch_thread

5341            8        51    161    0.0%  gettylogin1

5541           23        30    771    0.0%  login

7589          151       297    509    0.0%  vsh

7676            4         8    529    0.0%  vsh

7699            2         6    494    0.0%  vdclogin

7702           16        15   1123    0.0%  login

7703          442      1677    263    0.0%  vsh

7986            6        12    546    0.0%  more

16274            6        21    315    0.0%  getty

18349          541       651    832    0.0%  dcos_sshd

18352          142       179    793    0.0%  vsh

18414            4         2   2113    0.0%  more

18415            2         3    749    0.0%  vsh

18416           41        22   1901    0.0%  ps

19987      3062091  11691544    261    0.0%  nfm

19995      1371823   3669210    373    0.0%  lldp

20003       135771    930985    145    0.0%  dhcp_snoop

20011       150947   1719326     87    0.0%  bfd

20019        51214   1577299     32    0.0%  bfd_app

20028     13036336  40225132    324    0.0%  lacp

20036     10842459   9653444   1123    0.0%  udld

20044        43021    610610     70    0.0%  vtp

20057       944851  11982408     78    0.0%  vpc

20069        18102    402645     44    0.0%  interface-vlan

20081        15643    399966     39    0.0%  hsrp_engine

20095     38951865  15145696   2571    0.0%  dot1x

20104     41635630  18196374   2288    0.0%  cts

20269       178349   5338768     33    0.0%  ntpd

20678       179077   5372702     33    0.0%  ntpd

21853       177129   5335598     33    0.0%  ntpd

23071       247267   4832371     51    0.0%  klogd

27855          103        38   2717    0.0%  eigrp

CPU util  :   8.0% user,   9.1% kernel,   82.9% idle

Please note that only processes from the requested vdc are shown above

Cisco Employee

N7K high CPU

Hi Krzysztof,

That does not reveal much of the info.

Please could you run " Ethanlyzer" on the local sup inband interface to see what pakets are htting CPU and being dropped.

N7K(config)# ethanalyzer local interface inband 

You can capture that on a file save it to the local bootflash and paste it here.

Cheers,

-amit singh

New Member

N7K high CPU

I captured some packets at different times, and among other, regular traffic, there is huge amount of this type of packets:

Frame 665 (96 bytes on wire, 64 bytes captured)

    Arrival Time: Dec  5, 2011 12:06:48.908873000

    [Time delta from previous captured frame: 0.000031000 seconds]

    [Time delta from previous displayed frame: 0.000031000 seconds]

    [Time since reference or first frame: 0.415769000 seconds]

    Frame Number: 665

    Frame Length: 96 bytes

    Capture Length: 64 bytes

    [Frame is marked: False]

    [Protocols in frame: eth:llc:stp]

IEEE 802.3 Ethernet

    Destination: 01:00:0c:cc:cc:cd (01:00:0c:cc:cc:cd)

        Address: 01:00:0c:cc:cc:cd (01:00:0c:cc:cc:cd)

        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)

        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)

    Source: 00:26:f0:1c:00:00 (00:26:f0:1c:00:00)

        Address: 00:26:f0:1c:00:00 (00:26:f0:1c:00:00)

        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)

        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)

    Length: 50

Logical-Link Control

    DSAP: SNAP (0xaa)

    IG Bit: Individual

    SSAP: SNAP (0xaa)

    CR Bit: Command

    Control field: U, func=UI (0x03)

        000. 00.. = Command: Unnumbered Information (0x00)

        .... ..11 = Frame type: Unnumbered frame (0x03)

    Organization Code: Cisco (0x00000c)

    PID: PVSTP+ (0x010b)

Spanning Tree Protocol

    Protocol Identifier: Spanning Tree Protocol (0x0000)

    Protocol Version Identifier: Spanning Tree (0)

    BPDU Type: Configuration (0x00)

    BPDU flags: 0x00

        0... .... = Topology Change Acknowledgment: No

        .... ...0 = Topology Change: No

    Root Identifier: 4096 / 0 / 00:23:04:ee:be:01

        Root Bridge Priority: 4096

        Root Bridge System ID Extension: 0

        Root Bridge System ID: 00:23:04:ee:be:01

    Root Path Cost: 0

    Bridge Identifier: 32768 / 0 / 00:23:04:ee:be:04

        Bridge Priority: 32768

        Bridge System ID Extension: 0

        Bridge System ID: 00:23:04:ee:be:04

    Port identifier: 0x901b

There is MST implemented in the core, on all N7Ks, but there are also 10-15 PVST+ switches, which are reported by MST port type as either Bound(PVST) or Pre-STD-Rx. I wonder if there could be some inconsistency, or so?

New Member

Re: N7K high CPU

OK, problem solved. Do NOT, I repeat, DO NOT enable peer-switch on non-roon N7K peers :-)

New Member

Hi Krzysztof,  I saw your

Hi Krzysztof, 

 

I saw your post. Could you detail the solution problem ? I didn't understand. Is the peer-switch the cause of the problem ?

 

I hope you remeber...4 years is many time.....

 

1225
Views
0
Helpful
6
Replies
This widget could not be displayed.