high cpu from process hulc running con

Unanswered Question
Nov 23rd, 2009
User Badges:

Strange problem on our 3750 stacks - when changing the config on many interfaces, the cpu utilization increases, to the point that sometimes my ssh connection to the switch is dropped, pings fail, etc.  E.g. if we do this command:

int range Fa1/0/1-48

  switchport access vlan 2

!


Happens consistently on all of our 3750 stacks.  Running 12.2(50)SE3 (ipbasek9).  Problem is worse if we run via script (i.e. send many commands one right after the other in rapid succession).

Thanks

Mike

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4 (5 ratings)
Loading.
Edison Ortiz Mon, 11/23/2009 - 15:12
User Badges:
  • Super Bronze, 10000 points or more
  • Hall of Fame,

    Founding Member

ippolito wrote:


Strange problem on our 3750 stacks - when changing the config on many interfaces, the cpu utilization increases, to the point that sometimes my ssh connection to the switch is dropped, pings fail, etc.  E.g. if we do this command:

int range Fa1/0/1-48

  switchport access vlan 2

!


Happens consistently on all of our 3750 stacks.  Running 12.2(50)SE3 (ipbasek9).  Problem is worse if we run via script (i.e. send many commands one right after the other in rapid succession).

Thanks

Mike

Normal behavior, please refer to this article:

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00807213f5.shtml#topic4


With that said, your session shouldn't drop. The problem started after upgrading to 50SE3?



Regards


Edison.

ippolito Tue, 11/24/2009 - 06:53
User Badges:

Thanks for the reply -- the problem has been there even in previous software versions.

Georg Pauwen Wed, 09/07/2016 - 04:12
User Badges:
  • Green, 3000 points or more
  • Cisco Designated VIP,

    2017 WAN

Mike,

a 'sh processes cpu sorted' should show you the process that is reponsible for the CPU utilization. Can you post the output of that exec command ?


ปลาวาฬทราย RMUT... Wed, 09/07/2016 - 20:17
User Badges:

Load for five secs: 83%/28%; one minute: 54%; five minutes: 56%
Time source is NTP, 10:06:05.267 bkk Thu Sep 8 2016

CPU utilization for five seconds: 83%/28%; one minute: 54%; five minutes: 56%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 184   894281414 624659521       1431 17.22% 16.79% 16.06%   0 Spanning Tree    
 280    17922411   8387590       2136 14.56%  1.19%  0.30%   0 SNMP ENGINE      
 121    98529688 193780974        508  9.39%  7.23%  8.03%   0 Hulc LED Process
  71    11311244 169972899         66  4.07%  1.87%  1.94%   0 HLFM address lea
   4    28817921   2308991      12480  2.97%  0.40%  0.29%   0 Check heaps      

Leo Laohoo Fri, 09/09/2016 - 06:02
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless


INFORMATION REMOVED

ปลาวาฬทราย RMUT... Wed, 09/07/2016 - 22:25
User Badges:

System image file is "flash:me340x-metroipaccessk9-mz.122-53.SE.bin"

cisco ME-3400G-12CS-A (PowerPC405) processor (revision C0) with 131072K bytes of memory.


Load for five secs: 99%/24%; one minute: 73%; five minutes: 67%
Time source is NTP, 12:18:40.984 bkk Thu Sep 8 2016

CPU utilization for five seconds: 99%/24%; one minute: 73%; five minutes: 67%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 276     4923536    391694      12569 39.83%  3.32%  0.69%   0 crypto sw pk pro
 184   898378679 625221974       1436 14.23% 15.65% 15.76%   0 Spanning Tree    
 121    99664001 194014411        513  9.11%  7.12%  7.86%   0 Hulc LED Process
  71    11785380 170188260         69  3.67%  3.28%  2.91%   0 HLFM address lea
 141    50818137   9203177       5521  0.95%  0.65%  0.64%   0 PI MATM Aging Pr
   4    28869836   2312567      12483  0.47%  0.29%  0.24%   0 Check heaps      
  86    19852030  78470032        252  0.47%  0.37%  0.37%   0 hpm main process
  58     8591248 300612692         28  0.15%  0.16%  0.15%   0 Fifo Error Detec
  89    12937781   6416237       2016  0.15%  0.17%  0.16%   0 hpm counter proc
 129     2725615   1417101       1923  0.15%  0.08%  0.06%   0 HQM Stack Proces
 102      510885  28670631         17  0.15%  0.05%  0.02%   0 Hulc Storm Contr

Leo Laohoo Fri, 09/09/2016 - 06:03
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless


INFORMATION REMOVED

ปลาวาฬทราย RMUT... Fri, 09/09/2016 - 00:24
User Badges:

No any encryption process.

Each time differently.

Load for five secs: 100%/3%; one minute: 40%; five minutes: 35%
Time source is NTP, 10:34:23.777 bkk Fri Sep 9 2016

CPU utilization for five seconds: 100%/3%; one minute: 40%; five minutes: 35%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 184   931861303 632212539       1473 15.97% 17.17% 16.98%   0 Spanning Tree    
 217   4048289402886936966          0  5.27%  4.23%  4.22%   0 REP LSL Hello PP
 121   103338118 196447140        526  2.71%  1.90%  1.56%   0 Hulc LED Process
 141    51567399   9329846       5527  0.63%  0.61%  0.60%   0 PI MATM Aging Pr
  86    20265101  79609166        254  0.15%  0.33%  0.31%   0 hpm main process
  89    13128497   6503740       2018  0.15%  0.20%  0.21%   0 hpm counter proc
  35     1295404   5950482        217  0.15%  0.08%  0.09%   0 Per-Second Jobs  

The IOS we won't change cause services impact if it is stable.

Leo Laohoo Fri, 09/09/2016 - 06:02
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

The IOS we won't change cause services impact if it is stable.

INFORMATION REMOVED

ปลาวาฬทราย RMUT... Fri, 09/09/2016 - 01:44
User Badges:

uptime is 9 weeks, 5 days, 5 hours, 21 minutes

I found the spanning-tree instance is full 128 instances and some VLANs spanning-tree disable, we plan to change to WS-C4900M which support 4,094 instances.

Thank you very much.

Leo Laohoo Fri, 09/09/2016 - 06:01
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Nice ... 4 points for helping out.  


You are on your own, mate.  :P

Actions

This Discussion