Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Community Member

Cat 3750 Switch Crashed

Hi Team

We have Stack of  catalyst 3750 switches ( WS-C3750X-48T-S) running software

15.0(2)SE1.One of the switch in stack crashed and the users stopped getting connectivity.

Reloading this switch,retained the network connectivity.

The switch crashed with follwoing information

Aug  10 08:19:08.855: %SYS-2-NOBLOCK: may_suspend with blocking disabled. -Process= "hpm main process", ipl= 0, pid= 131

-Traceback= 4FC7E8z 26B9D54z 1C856E8z 2300D10z 2191200z 23E995Cz 242188Cz 2421994z 19C1794z 241786Cz 23E776Cz 21D9440z 21D9644z 22FDCB4z 1DEF740z 22F861Cz

Aug  10 08:19:08.855: %SYS-3-MGDTIMER: Timer has parent, timer link, timer = 491AE60. -Process= "hpm main process", ipl= 6, pid= 131

-Traceback= 4FC7E8z 1C3B5C0z 1C83328z 21D715Cz 26BF144z 26B9898z

Apr  4 08:19:14.845: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

i have seen few logs causing switch crashed related to  SYS MGDTIMER but those are not matching the release i am running.Is thhere any knon bug related to this in the current release ( i couldnt found the same in release notes as well in bug toolkit related to this symptoms).

Please suggest.

Thanks

Sameer

1 ACCEPTED SOLUTION

Accepted Solutions

Cat 3750 Switch Crashed

Hi Sameer,

I have verified throughly and I can confirm that the 15.0(2)SE1 version is also affected beacuse of the BUG CSCue83689.

And not neccessary that all the conditions mentioned in the BUG should be an exact match. If one of the conditions is matching then also it should be fine. The more important stuff is that when analyzing from the code perspective after decoding your crashinfo the BUG should be an exact match. And in this case it is an exact match.

You can be assured that the root cause for the switch crash is - Cisco BUG CSCue83689.

As I mentioned earlier this BUG is fixed from 15.0(2)SE3 version onwards.

***********Plz do rate this post if you found it helpful*************************

Thanks & Regards,

Vignesh R P

14 REPLIES
VIP Super Bronze

Re: Cat 3750 Switch Crashed

Hi,

Is the CPU also running high?

Is the switch running SSH?

What is the output of

sh process cpu sor

HTH

Community Member

Re: Cat 3750 Switch Crashed

Hi

Everything is working normal currently after the switch reload.During the issue as specific users were not getting connectivity,concentrated on the switch and after initial troubleshooting reloaded the switch.Attching crashinfo file for reference (please ignore date mismatch)...

REgards

Sameer

Community Member

Re: Cat 3750 Switch Crashed

Hi

Yes the switch is running SSH..(Misse in last post)

Community Member

Cat 3750 Switch Crashed

Hi reza

Looks like its matching the Bug CSCue83689 though its observed in 15.0.2(SE2) though not observed in 15.0.2 SE1 which is i am riunning currently..(we are running dot1x authentication )

Please suggest

Cat 3750 Switch Crashed

Hi Sameer,

I would require the output of show version from the device to have the crashinfo file decoded and further assist you in understanding the root cause of the crash. Kindly do the needful at the earliest.

Thanks & Regards,

Vignesh R P

Community Member

Cat 3750 Switch Crashed

Hi Vignesh

i am putting down the show version before we upgraded to 15.0.2SE1 which is current release..( i do not have device access currently)

------------------ show version ------------------

Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.0(1)SE3, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2012 by Cisco Systems, Inc.

Compiled Wed 30-May-12 13:41 by prod_rel_team

ROM: Bootstrap program is C3750E boot loader

BOOTLDR: C3750E Boot Loader (C3750X-HBOOT-M) Version 12.2(53r)SE2, RELEASE SOFTWARE (fc1)

Ozone-5A-115.97 uptime is 1 week, 3 days, 21 hours, 26 minutes

System returned to ROM by power-on

System image file is "flash:/c3750e-universalk9-mz.150-1.SE3/c3750e-universalk9-mz.150-1.SE3.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

License Level: ipbase

License Type: Permanent

Next reload license Level: ipbase

cisco WS-C3750X-48 (PowerPC405) processor (revision A0) with 262144K bytes of memory.

Processor board ID FDO1627V09D

Last reset from power-on

3 Virtual Ethernet interfaces

1 FastEthernet interface

312 Gigabit Ethernet interfaces

12 Ten Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

Community Member

Cat 3750 Switch Crashed

Any Luck Vignesh...

please let me know if you get any more details on this crash..

Thanks in advance..

Cat 3750 Switch Crashed


Hi Sameer,

The noblock message means that macsecdev_set_port_option called process_may_suspend but blocking was disabled. This is what caused the crash. And yes your finding is 100% right. The crash was caused due to the Cisco BUG CSCue83689

This BUG is fixed from 15.0(2)SE3 version onwards.

***********Plz do rate this post if you found it helpful*************************

Thanks & Regards,

Vignesh R P

Community Member

Cat 3750 Switch Crashed

Hi Vignesh

Regarding the bug you mentioned,the conditions mentioed in the bug are not matching (Thats why i mentioned n my previous mail as looks like)

  • its seen in 15.0(1)SE1 and 15.0(2)SE2 and we are running 15.0(2)SE1

  • The other condition says when vlan is assigned via ACS and switch is configured for 802.1x authentication...Though we do are  not using ACS,we have dot1x configured and radius server is doing the aunthentication.

can it be the case that the bug went unidentified in release 15.0.(2) SE1 ?

Cat 3750 Switch Crashed

Hi Sameer,

I have verified throughly and I can confirm that the 15.0(2)SE1 version is also affected beacuse of the BUG CSCue83689.

And not neccessary that all the conditions mentioned in the BUG should be an exact match. If one of the conditions is matching then also it should be fine. The more important stuff is that when analyzing from the code perspective after decoding your crashinfo the BUG should be an exact match. And in this case it is an exact match.

You can be assured that the root cause for the switch crash is - Cisco BUG CSCue83689.

As I mentioned earlier this BUG is fixed from 15.0(2)SE3 version onwards.

***********Plz do rate this post if you found it helpful*************************

Thanks & Regards,

Vignesh R P

Community Member

Cat 3750 Switch Crashed

Hi Vignesh

Thanks for your detailed response.

so as i mentioned in my prevoius post,this bug must have gone unidentified in 15.0.2SE1 and probably will reflect in coming days.

Even the bug condition says AT LEASE SEEN IN  so we are on right track and probably cisco will add 15.0.2 se1 also as affected release in coming days.

Regards

Sameer

VIP Super Bronze

Cat 3750 Switch Crashed

Hi Sam,

Is the CPU running high?

can you post

sh procce cpu sort

Reza

Community Member

Cat 3750 Switch Crashed

Hi Reza

The cpu is running normal ..

i can post the output of show proc cpu sort by tomorrow as switch access is not available currently .

Thanks

Sameer

Community Member

Cat 3750 Switch Crashed

Hi reza

Please find required details

------------------ show process cpu ------------------


CPU utilization for five seconds: 59%/2%; one minute: 61%; five minutes: 43%
PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
   1         748       11647         64  0.00%  0.00%  0.00%   0 Chunk Manager   
   2        1447       17500         82  0.00%  0.00%  0.00%   0 Load Meter      
   3          67          34       1970  0.00%  0.00%  0.00%   0 hulc_entropy_thr
   4       21379       34958        611  0.16%  0.02%  0.00%   0 Hulc Axiom Therm
   5     1703952       74294      22935  0.00%  2.39%  1.90%   0 Check heaps     
   6          51          64        796  0.00%  0.00%  0.00%   0 Pool Manager    
   7           0           1          0  0.00%  0.00%  0.00%   0 DiscardQ Backgro
   8           0           2          0  0.00%  0.00%  0.00%   0 Timers          
   9         109       12708          8  0.00%  0.00%  0.00%   0 WATCH_AFS       
  10           8           5       1600  0.00%  0.00%  0.00%   0 License Client N
  11      214729        1456     147478  0.00%  0.21%  0.18%   0 Licensing Auto U
  12         185          12      15416  0.00%  0.00%  0.00%   0 Image License br
  13       92888      135475        685  0.00%  0.10%  0.14%   0 ARP Input       
  14        1182       91089         12  0.00%  0.00%  0.00%   0 ARP Background  
  15           0           1          0  0.00%  0.00%  0.00%   0 CEF MIB API     
  16           0           1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT
  17           0           1          0  0.00%  0.00%  0.00%   0 Policy Manager  
  18         278          66       4212  0.00%  0.00%  0.00%   0 Entity MIB API  
  19           0           1          0  0.00%  0.00%  0.00%   0 IFS Agent Manage
  20         661       17450         37  0.00%  0.00%  0.00%   0 IPC Event Notifi
  21         633       84830          7  0.00%  0.00%  0.00%   0 IPC Mcast Pendin
  22          36        1459         24  0.00%  0.00%  0.00%   0 IPC Dynamic Cach
  23           0           1          0  0.00%  0.00%  0.00%   0 IPC Session Serv
  24           0           1          0  0.00%  0.00%  0.00%   0 IPC Zone Manager
  25         532       84830          6  0.00%  0.00%  0.00%   0 IPC Periodic Tim
  26         539       84829          6  0.00%  0.00%  0.00%   0 IPC Deferred Por
  27           0           1          0  0.00%  0.00%  0.00%   0 IPC Process leve
  28        4999       54073         92  0.00%  0.00%  0.00%   0 IPC Seat Manager
  29          42        5009          8  0.00%  0.00%  0.00%   0 IPC Check Queue 
  30          24         125        192  0.00%  0.00%  0.00%   0 IPC Seat RX Cont
  31           0           1          0  0.00%  0.00%  0.00%   0 IPC Seat TX Cont
  32          42        8772          4  0.00%  0.00%  0.00%   0 IPC Keep Alive M
  33         800       17497         45  0.00%  0.00%  0.00%   0 IPC Loadometer  
  34         284           9      31555  0.00%  0.00%  0.00%   0 PrstVbl         
  35           0           1          0  0.00%  0.00%  0.00%   0 Crash writer    
  36           0           1          0  0.00%  0.00%  0.00%   0 Exception contro
  37           8          11        727  0.00%  0.00%  0.00%   0 License IPC stat
  38           0           1          0  0.00%  0.00%  0.00%   0 License IPC serv
  39         347       87491          3  0.00%  0.00%  0.00%   0 GraphIt         
  40        2297         350       6562  0.00%  0.00%  0.00%   0 crypto sw pk pro
  41           0           2          0  0.00%  0.00%  0.00%   0 SMART           
  42           0           2          0  0.00%  0.00%  0.00%   0 XML Proxy Client
  43           0           1          0  0.00%  0.00%  0.00%   0 ARP Snoop       
  44        1047       86765         12  0.00%  0.00%  0.00%   0 Dynamic ARP Insp
  45           0           1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd  
  46      151005       29436       5129  0.00%  0.18%  0.17%   0 Net Background  
  47           0           1          0  0.00%  0.00%  0.00%   0 IDB Work        
  48         555        3229        171  0.00%  0.00%  0.00%   0 Logger          
  49        1190       86565         13  0.00%  0.00%  0.00%   0 TTY Background  
  50           0           4          0  0.00%  0.00%  0.00%   0 CTS CORE        
  51           0          28          0  0.00%  0.00%  0.00%   0 SXP CORE        
  52          51          30       1700  0.00%  0.00%  0.00%   0 IF-MGR control p
  53           9         470         19  0.00%  0.00%  0.00%   0 IF-MGR event pro
  54           0           1          0  0.00%  0.00%  0.00%   0 Inode Table Dest
  55           8           8       1000  0.00%  0.00%  0.00%   0 IP Admission HA 
  56         229        5902         38  0.00%  0.00%  0.00%   0 Net Input       
  57       92440       17505       5280  0.16%  0.10%  0.10%   0 Compute load avg
  58       52666        1511      34855  0.00%  0.03%  0.04%   0 Per-minute Jobs 
  59     1582374       87829      18016  1.28%  1.34%  1.28%   0 Per-Second Jobs 
  60           0           1          0  0.00%  0.00%  0.00%   0 AggMgr Process  
  61           0         135          0  0.00%  0.00%  0.00%   0 Transport Port A
  62        9730       25456        382  0.00%  0.00%  0.00%   0 HC Counter Timer
  63       15555       30091        516  0.00%  0.02%  0.00%   0 SFF8472         
  64           0         179          0  0.00%  0.00%  0.00%   0 EEM ED Identity 
  65           0         357          0  0.00%  0.00%  0.00%   0 EEM ED MAT      
  66         691        5245        131  0.00%  0.00%  0.00%   0 EEM ED ND       
  67          34          80        425  0.00%  0.00%  0.00%   0 USB Startup     
  68        5187     1660729          3  0.00%  0.01%  0.00%   0 DownWhenLooped  
  69           0           1          0  0.00%  0.00%  0.00%   0 HULC PBR Tcam Me
  70           0           1          0  0.00%  0.00%  0.00%   0 HULC ACL Tcam Me
  71           0           1          0  0.00%  0.00%  0.00%   0 Hulc Warm Upgrad
  72           0           1          0  0.00%  0.00%  0.00%   0 HRPC hdwl reques
  73           0           1          0  0.00%  0.00%  0.00%   0 Hulc EEM Process
  74           0           1          0  0.00%  0.00%  0.00%   0 HRPC lpip reques
  75           0           2          0  0.00%  0.00%  0.00%   0 HLPIP Sync Proce
  76           0           1          0  0.00%  0.00%  0.00%   0 HRPC Multi-FS Sy
  77          17          18        944  0.00%  0.00%  0.00%   0 HULC multifs pro
  78           0           1          0  0.00%  0.00%  0.00%   0 HRPC hnetwpol re
  79           0           1          0  0.00%  0.00%  0.00%   0 HRPC EnergyWise 
  80           0           1          0  0.00%  0.00%  0.00%   0 HRPC actual powe
  81        3971      349814         11  0.00%  0.01%  0.00%   0 HUSB Console    
  82           0           1          0  0.00%  0.00%  0.00%   0 HRPC ipadm reque
  83          16           2       8000  0.00%  0.00%  0.00%   0 SM RBCP Monitor 
  84         106        2917         36  0.00%  0.00%  0.00%   0 PSP Timer       
  85           0           1          0  0.00%  0.00%  0.00%   0 HULC QM Tcam Mem
  86       24043         141     170517  0.00%  0.00%  0.00%   0 X2 HAMM detect p
  87           0           1          0  0.00%  0.00%  0.00%   0 Madmax Process  
  88           0           1          0  0.00%  0.00%  0.00%   0 HRPC asic-stats 
  89      807025      679970       1186  1.12%  1.04%  0.82%   0 RedEarth I2C dri
  90     2479857      439297       5645  2.24%  2.45%  2.47%   0 RedEarth Tx Mana
  91       83156      339982        244  0.00%  0.06%  0.07%   0 RedEarth Rx Mana
  92         220       17501         12  0.00%  0.00%  0.00%   0 HULC Thermal Pro
  93           0           3          0  0.00%  0.00%  0.00%   0 HRPC hsm request
  94          59         210        280  0.00%  0.00%  0.00%   0 Stack Mgr       
  95        2671        1185       2254  0.00%  0.00%  0.00%   0 Stack Mgr Notifi
  96         269       31987          8  0.00%  0.00%  0.00%   0 Adjust Regions  
  97        9050       43633        207  0.00%  0.00%  0.00%   0 hrpc -> response
  98        7757      202700         38  0.00%  0.00%  0.00%   0 hrpc -> request 
  99       99518      312677        318  0.32%  0.33%  0.16%   0 hrpc <- response
100           0          11          0  0.00%  0.00%  0.00%   0 HRPC hcomp reque
101        4109      437174          9  0.00%  0.05%  0.01%   0 yeti2_emac_proce
102           0          43          0  0.00%  0.00%  0.00%   0 HULC Device Mana
103        2620        7475        350  0.00%  0.00%  0.00%   0 HRPC hdm non blo
104           0          12          0  0.00%  0.00%  0.00%   0 HRPC hdm blockin
105        1215       17413         69  0.00%  0.00%  0.00%   0 HIPC bkgrd proce
106           0           1          0  0.00%  0.00%  0.00%   0 RTTYS Process   
107          95         153        620  0.00%  0.00%  0.00%   0 Hulc Port-Securi
108         126        1136        110  0.00%  0.00%  0.00%   0 HRPC hpsecure re
109           0          13          0  0.00%  0.00%  0.00%   0 HRPC hfbm reques
110         167          28       5964  0.00%  0.00%  0.00%   0 hulc cfg mgr mas
111        6072       18021        336  0.00%  0.00%  0.00%   0 HRPC hlfm reques
112       79109     2161380         36  0.16%  0.16%  0.11%   0 HLFM address lea
113        3551       86570         41  0.00%  0.00%  0.00%   0 HLFM aging proce
114       25879     2182337         11  0.00%  0.02%  0.00%   0 HLFM address ret
115           0           1          0  0.00%  0.00%  0.00%   0 HRPC hrcmd reque
116          24         120        200  0.00%  0.00%  0.00%   0 HRPC emac reques
117           0          28          0  0.00%  0.00%  0.00%   0 HRPC hulc misc r
118         489       29097         16  0.00%  0.00%  0.00%   0 HVLAN main bkgrd
119           8           2       4000  0.00%  0.00%  0.00%   0 HVLAN Mapped Vla
120           8           2       4000  0.00%  0.00%  0.00%   0 Vlan shutdown Pr
121          41         262        156  0.00%  0.00%  0.00%   0 HRPC vlan reques
122           0           1          0  0.00%  0.00%  0.00%   0 HULC VLAN REF Ba
123          67        1457         45  0.00%  0.00%  0.00%   0 hulc_xps_process
124           0          11          0  0.00%  0.00%  0.00%   0 HRPC ilp request
125        6802        5819       1168  0.00%  0.00%  0.00%   0 Strider Tcam Mem
126         131         142        922  0.00%  0.00%  0.00%   0 HULC PM Vector P
127           0           1          0  0.00%  0.00%  0.00%   0 HPM Msg Retry Pr
128         605        5817        104  0.00%  0.00%  0.00%   0 MDFS LC Download
129       83360     1364667         61  0.00%  0.08%  0.08%   0 hpm main process
130         307         602        509  0.00%  0.00%  0.00%   0 HPM Stack Sync P
131        2923        6771        431  0.00%  0.00%  0.00%   0 HRPC pm request 
132           0           9          0  0.00%  0.00%  0.00%   0 HPM if_num mappi
133     1071101       87493      12242  1.28%  1.30%  1.29%   0 hpm counter proc
134      162962      349241        466  0.00%  0.15%  0.18%   0 HRPC pm-counters
135           0           1          0  0.00%  0.00%  0.00%   0 hpm vp events ca
136        1437       86716         16  0.00%  0.01%  0.00%   0 Hulc CTS Process
137           0           1          0  0.00%  0.00%  0.00%   0 HRPC cts request
138           0           1          0  0.00%  0.00%  0.00%   0 HULC Msg Retry P
139           0           1          0  0.00%  0.00%  0.00%   0 HRPC hcmp reques
140         159          43       3697  0.00%  0.00%  0.00%   0 HCEF ADJ Refresh
141           0           1          0  0.00%  0.00%  0.00%   0 hl3md_rpfq_thrl_
142         339       43580          7  0.00%  0.00%  0.00%   0 hl3mm_rp        
143           0           1          0  0.00%  0.00%  0.00%   0 HACL Queue Proce
144           0           1          0  0.00%  0.00%  0.00%   0 HRPC acl request
145          16          54        296  0.00%  0.00%  0.00%   0 HACL Acl Manager
146           0           1          0  0.00%  0.00%  0.00%   0 HRPC aim request
147          33          37        891  0.00%  0.00%  0.00%   0 HRPC axiom-mgmt 
148         313       43574          7  0.00%  0.00%  0.00%   0 Hulc Axiom Mgmt 
149           0           1          0  0.00%  0.00%  0.00%   0 HRPC backup inte
150         134         296        452  0.00%  0.00%  0.00%   0 OBFL TEMP obfl0 
151           0           1          0  0.00%  0.00%  0.00%   0 HRPC cdp request
152           0          11          0  0.00%  0.00%  0.00%   0 HULC CISP Proces
153         290        2103        137  0.00%  0.00%  0.00%   0 HRPC dot1x reque
154         182         151       1205  0.00%  0.00%  0.00%   0 Dot1X Msg Retry 
155        3656        6686        546  0.00%  0.00%  0.00%   0 HULC DOT1X Proce
156           0           1          0  0.00%  0.00%  0.00%   0 HRPC foxhound re
157           0           2          0  0.00%  0.00%  0.00%   0 Hulc Foxhound Pr
158           0           1          0  0.00%  0.00%  0.00%   0 HRPC lldp reques
159        2836          26     109076  0.00%  0.00%  0.00%   0 MACsec Process  
160           0           1          0  0.00%  0.00%  0.00%   0 HRPC macsec requ
161           8          11        727  0.00%  0.00%  0.00%   0 HRPC system mtu 
162           0          21          0  0.00%  0.00%  0.00%   0 HRPC rep request
163           8           9        888  0.00%  0.00%  0.00%   0 REP Helper Proc 
164           0           1          0  0.00%  0.00%  0.00%   0 HRPC sdm request
165           0           1          0  0.00%  0.00%  0.00%   0 SMI MSG Retry Pr
166           0           1          0  0.00%  0.00%  0.00%   0 HRPC Smart Insta
167      108979      429007        254  0.16%  0.25%  0.16%   0 Hulc Storm Contr
168           0           2          0  0.00%  0.00%  0.00%   0 HSTP Sync Proces
169           0           1          0  0.00%  0.00%  0.00%   0 HRPC stp_cli req
170          17          54        314  0.00%  0.00%  0.00%   0 HRPC stp_state_s
171           0           2          0  0.00%  0.00%  0.00%   0 S/W Bridge Proce
172           0          54          0  0.00%  0.00%  0.00%   0 HRPC hudld reque
173           0           1          0  0.00%  0.00%  0.00%   0 HRPC vqpc reques
174           0           1          0  0.00%  0.00%  0.00%   0 HRPC iec_load_ba
175        1042       86589         12  0.00%  0.00%  0.00%   0 IEC Load balance
176           0           1          0  0.00%  0.00%  0.00%   0 HRPC l2pt qnq rp
177           0          74          0  0.00%  0.00%  0.00%   0 HRPC hled reques
178    18776722     2175625       8630 20.38% 19.84% 19.74%   0 Hulc LED Process
179           0           1          0  0.00%  0.00%  0.00%   0 HRPC hl3mm reque
180        1204       60584         19  0.00%  0.00%  0.00%   0 hl3mm           
181        4743       64349         73  0.00%  0.00%  0.00%   0 HL3U bkgrd proce
182       10037      116433         86  0.00%  0.00%  0.00%   0 HRPC hl3u reques
183        1287       19962         64  0.00%  0.00%  0.00%   0 HIPV6 bkgrd proc
184       10197      116436         87  0.00%  0.00%  0.00%   0 HRPC IPv6 Unicas
185         218       19971         10  0.00%  0.00%  0.00%   0 HL3U PBR bkgrd p
186           9         730         12  0.00%  0.00%  0.00%   0 HL3U PBR n-h res
187           0           1          0  0.00%  0.00%  0.00%   0 HRPC obfl reques
188       19870        1492      13317 23.76% 21.80%  5.72%   1 SSH Process     
189           0           1          0  0.00%  0.00%  0.00%   0 HRPC dtp request
190           0           1          0  0.00%  0.00%  0.00%   0 HRPC hmedianet p
191           0           1          0  0.00%  0.00%  0.00%   0 HRPC show_forwar
192           9          39        230  0.00%  0.00%  0.00%   0 HRPC snmp reques
193         115          20       5750  0.00%  0.00%  0.00%   0 HULC SNMP Proces
194      339352       17485      19408  0.32%  0.32%  0.31%   0 HQM Stack Proces
195      448315      150344       2981  0.48%  0.48%  0.52%   0 HRPC qos request
196           0           1          0  0.00%  0.00%  0.00%   0 HRPC span reques
197          24          61        393  0.00%  0.00%  0.00%   0 HRPC system post
198          26          41        634  0.00%  0.00%  0.00%   0 Hulc Reload Mana
199         596        1254        475  0.00%  0.00%  0.00%   0 HRPC hrcli-event
201           0           2          0  0.00%  0.00%  0.00%   0 image mgr       
202           0           1          0  0.00%  0.00%  0.00%   0 HRFS OIR Proc   
203         182         578        314  0.00%  0.00%  0.00%   0 HL2MCM          
204          60          24       2500  0.00%  0.00%  0.00%   0 HL2MCM          
205        4726       29048        162  0.00%  0.00%  0.00%   0 Hulc SM10G Infra
206        2319        2962        782  0.00%  0.00%  0.00%   0 AAA Server      
207          34          68        500  0.00%  0.00%  0.00%   0 AAA ACCT Proc   
208           0           1          0  0.00%  0.00%  0.00%   0 ACCT Periodic Pr
209           0           3          0  0.00%  0.00%  0.00%   0 AAA System Acct 
210           0           1          0  0.00%  0.00%  0.00%   0 AUTH POLICY Fram
211       18493      124726        148  0.00%  0.05%  0.05%   0 Auth Manager    
212           0           1          0  0.00%  0.00%  0.00%   0 Auth-proxy AAA B
213           0         292          0  0.00%  0.00%  0.00%   0 IP Admin SM Proc
214       25813       13901       1856  0.00%  0.02%  0.00%   0 CDP Protocol    
215           0           1          0  0.00%  0.00%  0.00%   0 DiagCard3/-1    
216       11854      134346         88  0.00%  0.00%  0.00%   0 CEF: IPv4 proces
217           0           2          0  0.00%  0.00%  0.00%   0 CMD HANDLER     
218           0           2          0  0.00%  0.00%  0.00%   0 AAA Dictionary R
219           9         730         12  0.00%  0.00%  0.00%   0 DHCP Snooping   
220           0           1          0  0.00%  0.00%  0.00%   0 DHCP Snooping db
221         302        4179         72  0.00%  0.00%  0.00%   0 Dot1x Mgr Proces
222        5334       10145        525  0.00%  0.00%  0.00%   0 EAP Framework   
223           0           1          0  0.00%  0.00%  0.00%   0 EAP Test        
224           0           2          0  0.00%  0.00%  0.00%   0 EAPoUDP Process 
225           0           2          0  0.00%  0.00%  0.00%   0 CEF switching ba
226           0           9          0  0.00%  0.00%  0.00%   0 IP ARP Adjacency
227       26584     2161677         12  0.00%  0.04%  0.00%   0 IP ARP Retry Age
228       91461      431208        212  0.00%  0.37%  0.12%   0 IP Input        
229           0           1          0  0.00%  0.00%  0.00%   0 ICMP event handl
230        1865      167924         11  0.00%  0.00%  0.00%   0 IP ARP Track    
231         479       86839          5  0.00%  0.00%  0.00%   0 fnvRootHubPortSt
232           0           1          0  0.00%  0.00%  0.00%   0 tHUB            
233        4569      857954          5  0.00%  0.00%  0.00%   0 MDFS MFIB Proces
234           0           1          0  0.00%  0.00%  0.00%   0 ADJ NSF process 
235           0           1          0  0.00%  0.00%  0.00%   0 IPv6 ping proces
236           0           2          0  0.00%  0.00%  0.00%   0 KRB5 AAA        
237           0           2          0  0.00%  0.00%  0.00%   0 MKA             
238           0           1          0  0.00%  0.00%  0.00%   0 DiagCard2/-1    
239           0           2          0  0.00%  0.00%  0.00%   0 REP Topology cha
240           0           8          0  0.00%  0.00%  0.00%   0 SMI Director DB 
241         408        3920        104  0.00%  0.00%  0.00%   0 SMI CDP Update H
242           0           1          0  0.00%  0.00%  0.00%   0 SMI Backup Proce
243           0           2          0  0.00%  0.00%  0.00%   0 SMI IBC server p
244           0           1          0  0.00%  0.00%  0.00%   0 SMI IBC client p
245           0           2          0  0.00%  0.00%  0.00%   0 SMI IBC Download
246      187580      841581        222  0.00%  0.29%  0.20%   0 Spanning Tree   
247           8        1471          5  0.00%  0.00%  0.00%   0 Spanning Tree St
248           0           2          0  0.00%  0.00%  0.00%   0 CDP Forward Proc
249        2397         516       4645  0.00%  0.00%  0.00%   0 802.1x switch   
250           0           1          0  0.00%  0.00%  0.00%   0 802.1x Webauth F
251        1028        7174        143  0.00%  0.00%  0.00%   0 DTP Protocol    
252           0          11          0  0.00%  0.00%  0.00%   0 HRPC IPv6 Host r
253           0           2          0  0.00%  0.00%  0.00%   0 IPv6 Platform Ho
254           0           1          0  0.00%  0.00%  0.00%   0 axiom-module cra
255           0           1          0  0.00%  0.00%  0.00%   0 HRPC dai request
256      228139      562989        405  0.00%  0.20%  0.18%   0 HULC DAI Process
257           0           1          0  0.00%  0.00%  0.00%   0 HRPC power down 
258           0           1          0  0.00%  0.00%  0.00%   0 HRPC ip device t
259           0           1          0  0.00%  0.00%  0.00%   0 HRPC ip source g
260           0           1          0  0.00%  0.00%  0.00%   0 HULC IP Source g
261           0           1          0  0.00%  0.00%  0.00%   0 HRPC sisf reques
262           8           7       1142  0.00%  0.00%  0.00%   0 HULC SISF Proces
263           0           1          0  0.00%  0.00%  0.00%   0 HULC SISF Source
264       22821       86568        263  0.00%  0.02%  0.00%   0 PI MATM Aging Pr
265       31336      873528         35  0.00%  0.09%  0.02%   0 UDLD            
266          69        2918         23  0.00%  0.00%  0.00%   0 Port-Security   
267           0           2          0  0.00%  0.00%  0.00%   0 Switch Backup In
268           0           2          0  0.00%  0.00%  0.00%   0 IP Host Track Pr
269           0           1          0  0.00%  0.00%  0.00%   0 Link State Group
270          58        1456         39  0.00%  0.00%  0.00%   0 MMN bkgrd proces
271         140        8729         16  0.00%  0.00%  0.00%   0 Ethchnl         
272         814         251       3243  0.00%  0.00%  0.00%   0 VMATM Callback  
273        6667       63681        104  0.00%  0.00%  0.00%   0 IPC UDP Input   
274         377        1759        214  0.00%  0.00%  0.00%   0 CEF background p
275           0           1          0  0.00%  0.00%  0.00%   0 fib_fib_bfd_sb e
276           0           1          0  0.00%  0.00%  0.00%   0 IP IRDP         
277           9           1       9000  0.00%  0.00%  0.00%   0 CEF RF HULC Conv
278           0           6          0  0.00%  0.00%  0.00%   0 XDR background p
279        1560        2802        556  0.00%  0.00%  0.00%   0 XDR mcast       
280         108        8728         12  0.00%  0.00%  0.00%   0 IP ACL XDR LC Ba
281           0           9          0  0.00%  0.00%  0.00%   0 XDR RP Ping Back
282        3881        5666        684  0.00%  0.01%  0.00%   0 XDR mcast rcv   
283          66        1456         45  0.00%  0.00%  0.00%   0 XDR RP backgroun
284          59          16       3687  0.00%  0.00%  0.00%   0 IPC LC Message H
285           0           1          0  0.00%  0.00%  0.00%   0 XDR RP Test Back
286           0           4          0  0.00%  0.00%  0.00%   0 ADJ background  
287           0          17          0  0.00%  0.00%  0.00%   0 MDFS LC Process 
288           0           1          0  0.00%  0.00%  0.00%   0 Critical Auth   
289           0           2          0  0.00%  0.00%  0.00%   0 Routing Topology
290           0           2          0  0.00%  0.00%  0.00%   0 IP RIB Update   
291           0           2          0  0.00%  0.00%  0.00%   0 ADJ resolve proc
292           0           1          0  0.00%  0.00%  0.00%   0 IPv6 RIB Event H
293         285        5551         51  0.00%  0.00%  0.00%   0 TCP Timer       
294          34         113        300  0.00%  0.00%  0.00%   0 TCP Protocols   
295          25       86668          0  0.00%  0.00%  0.00%   0 Socket Timers   
296           8         295         27  0.00%  0.00%  0.00%   0 HTTP CORE       
297           0        1096          0  0.00%  0.00%  0.00%   0 Cluster L2      
298         256        8730         29  0.00%  0.00%  0.00%   0 Cluster RARP    
299         424       11909         35  0.00%  0.00%  0.00%   0 Cluster Base    
300        5761      173363         33  0.16%  0.03%  0.00%   0 DHCPD Receive   
301           0           2          0  0.00%  0.00%  0.00%   0 Dot1x Supplicant
302           0           2          0  0.00%  0.00%  0.00%   0 Dot1x Supplicant
303           0           2          0  0.00%  0.00%  0.00%   0 Dot1x Supplicant
304           0           3          0  0.00%  0.00%  0.00%   0 Flow Exporter Ti
305           0           2          0  0.00%  0.00%  0.00%   0 Multicast Offloa
306           0           1          0  0.00%  0.00%  0.00%   0 RARP Input      
307           0           2          0  0.00%  0.00%  0.00%   0 static          
308           0           1          0  0.00%  0.00%  0.00%   0 MAB Framework   
309         904        8733        103  0.00%  0.00%  0.00%   0 QoS stats proces
310           8         754         10  0.00%  0.00%  0.00%   0 DHCPD Timer     
311           0           2          0  0.00%  0.00%  0.00%   0 REP LSL Proc    
312           0           2          0  0.00%  0.00%  0.00%   0 REP BPA/EPA Proc
313           0           5          0  0.00%  0.00%  0.00%   0 SNMP Timers     
314           0           1          0  0.00%  0.00%  0.00%   0 HRPC dhcp snoopi
315           0           9          0  0.00%  0.00%  0.00%   0 HULC DHCP Snoopi
316        1416       10614        133  0.00%  0.00%  0.00%   0 HULC stack power
317        2299       34413         66  0.00%  0.00%  0.00%   0 HRPC stack power
318         150         164        914  0.00%  0.00%  0.00%   0 IGMPSN L2MCM    
319           0           1          0  0.00%  0.00%  0.00%   0 IGMPSN MRD      
320          82         130        630  0.00%  0.00%  0.00%   0 IGMPSN          
321           0           1          0  0.00%  0.00%  0.00%   0 IGMPQR          
322           8           3       2666  0.00%  0.00%  0.00%   0 L2TRACE SERVER  
323         140         161        869  0.00%  0.00%  0.00%   0 MLDSN L2MCM     
324           0           1          0  0.00%  0.00%  0.00%   0 MRD             
325           0           1          0  0.00%  0.00%  0.00%   0 MLD_SNOOP       
326          36         363         99  0.00%  0.00%  0.00%   0 HRPC hl2mcm igmp
327       17738      865912         20  0.00%  0.03%  0.00%   0 MDFS RP process 
328           0           1          0  0.00%  0.00%  0.00%   0 EM Background Pr
329           0           2          0  0.00%  0.00%  0.00%   0 AAA Cached Serve
330          26         355         73  0.00%  0.00%  0.00%   0 HRPC hl2mcm mlds
331           0           3          0  0.00%  0.00%  0.00%   0 Collection proce
332       21894          54     405444  0.00%  0.00%  0.00%   0 NIST rng proc   
333        1979      167923         11  0.00%  0.01%  0.00%   0 Timer Library   
334           0           2          0  0.00%  0.00%  0.00%   0 ENABLE AAA      
335           0           3          0  0.00%  0.00%  0.00%   0 LDAP process    
336           0           2          0  0.00%  0.00%  0.00%   0 Crypto Support  
337           0           1          0  0.00%  0.00%  0.00%   0 IPSECv6 PS Proc 
338           0           2          0  0.00%  0.00%  0.00%   0 LINE AAA        
339          43         141        304  0.00%  0.00%  0.00%   0 LOCAL AAA       
340          74         165        448  0.00%  0.00%  0.00%   0 EPM MAIN PROCESS
341           8          20        400  0.00%  0.00%  0.00%   0 OBFL Cfg Dispatc
342           0           2          0  0.00%  0.00%  0.00%   0 TPLUS           
343       13115        4365       3004  0.00%  0.01%  0.00%   0 HULC ISM FNF    
344       43315        2521      17181  0.00%  0.00%  0.00%   0 hulc running con
345         436          30      14533  0.00%  0.00%  0.00%   0 crypto engine pr
346        2545       87153         29  0.00%  0.00%  0.00%   0 Crypto IKEv2    
347           0           1          0  0.00%  0.00%  0.00%   0 IKEv2 AAA handle
348           0           1          0  0.00%  0.00%  0.00%   0 encrypt proc    
349           0           3          0  0.00%  0.00%  0.00%   0 CRYPTO MAP FREE 
350           0           1          0  0.00%  0.00%  0.00%   0 Crypto INT      
351           0           2          0  0.00%  0.00%  0.00%   0 Crypto IKE Dispa
352           8           3       2666  0.00%  0.00%  0.00%   0 Crypto IKMP     
353           0           1          0  0.00%  0.00%  0.00%   0 IPSEC manual key
354         349        4366         79  0.00%  0.00%  0.00%   0 IPSEC key engine
355           0           1          0  0.00%  0.00%  0.00%   0 Crypto ACL      
356           0           1          0  0.00%  0.00%  0.00%   0 Crypto PAS Proc 
357           0           1          0  0.00%  0.00%  0.00%   0 IPSec background
358         285         101       2821  0.00%  0.00%  0.00%   0 Crypto CA       
359           0           1          0  0.00%  0.00%  0.00%   0 Crypto PKI-CRL  
360           0           2          0  0.00%  0.00%  0.00%   0 MMON PROCESS    
361       24659     2720047          9  0.16%  0.01%  0.00%   0 MMON MENG       
362           0           1          0  0.00%  0.00%  0.00%   0 HRPC x_setup req
363           0           2          0  0.00%  0.00%  0.00%   0 REP Switch Helpe
364           0           1          0  0.00%  0.00%  0.00%   0 Licensing MIB pr
365           0          23          0  0.00%  0.00%  0.00%   0 VTP Trap Process
366           8         683         11  0.00%  0.00%  0.00%   0 ASP Process Crea
367         327         962        339  0.00%  0.00%  0.00%   0 AAA SEND STOP EV
368           0           1          0  0.00%  0.00%  0.00%   0 Test AAA Client 
369           0           2          0  0.00%  0.00%  0.00%   0 DHCP Security He
370           8           2       4000  0.00%  0.00%  0.00%   0 EEM ED Routing  
371         294        3215         91  0.00%  0.00%  0.00%   0 EEM ED Syslog   
372           0           1          0  0.00%  0.00%  0.00%   0 Syslog Traps    
373         431          49       8795  0.00%  0.00%  0.00%   0 OBFL UPTIME obfl
374          52        1456         35  0.00%  0.00%  0.00%   0 vlmap logger    
375           0           1          0  0.00%  0.00%  0.00%   0 HCD Process     
376           0           1          0  0.00%  0.00%  0.00%   0 HRPC cable diagn
377           0           1          0  0.00%  0.00%  0.00%   0 HRPC rsu request
378         323          93       3473  0.00%  0.00%  0.00%   0 SpanTree Helper 
379           0           2          0  0.00%  0.00%  0.00%   0 DiagCard1/-1    
380           0           1          0  0.00%  0.00%  0.00%   0 Online Diag EEM 
381       18304      785407         23  0.00%  0.06%  0.01%   0 PM Callback     
382          94         487        193  0.00%  0.00%  0.00%   0 HCMP sync proces
383          45        6994          6  0.00%  0.00%  0.00%   0 dhcp snooping sw
384          26         189        137  0.00%  0.00%  0.00%   0 SpanTree Flush  
385           9          69        130  0.00%  0.00%  0.00%   0 EEM Server      
386          17           2       8500  0.00%  0.00%  0.00%   0 Call Home proces
387           0           4          0  0.00%  0.00%  0.00%   0 CTS credentials 
388           0           2          0  0.00%  0.00%  0.00%   0 EEM Policy Direc
389           0           1          0  0.00%  0.00%  0.00%   0 HRPC eee request
390           0           2          0  0.00%  0.00%  0.00%   0 EEM ED CLI      
391           0           3          0  0.00%  0.00%  0.00%   0 EEM ED Config   
392           0           3          0  0.00%  0.00%  0.00%   0 EEM ED Counter  
393           8           3       2666  0.00%  0.00%  0.00%   0 EEM ED Env      
394           0           3          0  0.00%  0.00%  0.00%   0 EM ED GOLD      
395           0           3          0  0.00%  0.00%  0.00%   0 EEM ED Interface
396           0           3          0  0.00%  0.00%  0.00%   0 EEM ED IOSWD    
397           0           3          0  0.00%  0.00%  0.00%   0 EEM ED Ipsla    
398           0           3          0  0.00%  0.00%  0.00%   0 EEM ED None     
399           0           3          0  0.00%  0.00%  0.00%   0 EEM ED OIR      
400          76        1456         52  0.00%  0.00%  0.00%   0 Call Home Timer 
401           0           3          0  0.00%  0.00%  0.00%   0 EEM ED SNMP     
402           0           3          0  0.00%  0.00%  0.00%   0 EEM ED SNMP Obje
403           0           2          0  0.00%  0.00%  0.00%   0 EEM ED SNMP Noti
404           0           3          0  0.00%  0.00%  0.00%   0 EEM ED Test     
405          26        2705          9  0.00%  0.00%  0.00%   0 EEM ED Timer    
406        3470        1796       1932  0.00%  0.00%  0.00%   0 Syslog          
407           0          37          0  0.00%  0.00%  0.00%   0 RBM CORE        
408           0           2          0  0.00%  0.00%  0.00%   0 STP FAST TRANSIT
409           0          83          0  0.00%  0.00%  0.00%   0 CSRT RAPID TRANS
410          32           6       5333  0.00%  0.00%  0.00%   0 VLAN Manager    
411           9          11        818  0.00%  0.00%  0.00%   0 CEF RP IPC Backg
412         100           4      25000  0.00%  0.00%  0.00%   0 OBFL MSG obfl0  
413           0          27          0  0.00%  0.00%  0.00%   0 SSH Event handle
414        9903        5654       1751  0.00%  0.00%  0.00%   0 RADIUS          
415           0           1          0  0.00%  0.00%  0.00%   0 DiagCard4/-1    
416           0           1          0  0.00%  0.00%  0.00%   0 DiagCard5/-1    
417           0           1          0  0.00%  0.00%  0.00%   0 DiagCard6/-1    
418        1378        5563        247  0.00%  0.00%  0.00%   0 Dot1x Authentica
419        1610        6897        233  0.00%  0.00%  0.00%   0 Dot1x Authentica
420         923        1960        470  0.00%  0.00%  0.00%   0 IP SNMP         
421         152         980        155  0.00%  0.00%  0.00%   0 PDU DISPATCHER  
422         276         982        281  0.00%  0.00%  0.00%   0 SNMP ENGINE     
423           0           2          0  0.00%  0.00%  0.00%   0 IP SNMPV6       
424           0           1          0  0.00%  0.00%  0.00%   0 SNMP ConfCopyPro
425         764         111       6882  0.00%  0.00%  0.00%   0 SNMP Traps      
426          68        1456         46  0.00%  0.00%  0.00%   0 DHCPD Database  
427           0           1          0  0.00%  0.00%  0.00%   0 Connection Mgr  
428           0           1          0  0.00%  0.00%  0.00%   0 LICENSE AGENT   
429         408        1455        280  0.00%  0.00%  0.00%   0 RADIUS IO STATS 
430          58           5      11600  0.00%  0.00%  0.00%   0 OBFL ENV obfl0  
431          83        8746          9  0.00%  0.00%  0.00%   0 MACsec Device Pr
432       42288        1462      28924  0.00%  0.04%  0.00%   0 OBFL VOLT obfl0 
433         274        5798         47  0.00%  0.00%  0.00%   0 OBFL I/O Buffer 
435        1103       58791         18  0.00%  0.00%  0.00%   0 NTP            

------------------ show process cpu history ------------------

                                                                 
                                                                 
                           11111                                 
      599999999999999999999000009999944444333334444433333333333333
      922222111118888899999000009999944444555557777755555333334444
  100            ********************                          
   90  ******************************                          
   80  ******************************                          
   70  ******************************                          
   60 *******************************                          
   50 *******************************          *****           
   40 ***************************************************      
   30 **********************************************************
   20 **********************************************************
   10 **********************************************************
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)

                                                                 
                                                                 
      1                                                          
      045444454445455456454544444445544445554444455544444554444444
      014244939570204716609267997849984663119893502059796003393289
  100 *                                                        
   90 *                                                        
   80 *                                                        
   70 *                *                                       
   60 #                *           **                          
   50 # *   ****** *************** *** ******** ***********  * 
   40 #####*################################################*###
   30 ##########################################################
   20 ##########################################################
   10 ##########################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

                                                                             
                                                                             
             1       1                                                       
      555556505559555098555559                                               
      975760606569658093777769                                               
  100        *   *   **      *                                             
   90        *   *   **      *                                             
   80        *   *   ***     *                                             
   70        *   *   ***     *                                             
   60 ************************                                             
   50 ***************#********                                             
   40 ########################                                             
   30 ########################                                             
   20 ########################                                             
   10 ########################                                             
     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
               0    5    0    5    0    5    0    5    0    5    0    5    0 
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

1377
Views
5
Helpful
14
Replies
CreatePlease to create content