How to fix high CPU on 3650C

Unanswered Question
May 2nd, 2012

If you are searching due to finding out your 3650C has high CPU, you can try these tricks:

* add storm control

* upgrade to c3560c-universalk9-tar.122-55.EX3.tar

  (EX2 has an Hulc LED Process bug)

* If you are in area 0 with OSPF, turn it into an NSSA and reduce your routes!

Honestly, I am not sure what fixed my switch, but it was using a lot of CPU and rebooting itself.  It appears stable today (I did all three changes at once).  Attached is a graph showing CPU usage over time on the switch.

Oh, and the switch did not like the using archive...

archive download-sw /overwrite tftp://1.2.3.4/

c3560c-universalk9-tar.122-55.EX3.tar

So I did a

# copy tftp://1.2.3.4/

c3560c405ex-universalk9-mz.122-55.EX3/c3560c405ex-universalk9-mz.122-55.EX3.bin flash:

then

(config)# boot system

flash:/c3560c405ex-universalk9-mz.122-55.EX3.bin

This was on WS-C3560CG-8PC-S  

Old OS: 12.2(55)EX2

New OS: 12.2(55)EX3

Attachment: 
I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 0 (0 ratings)
Leo Laohoo Wed, 05/02/2012 - 16:39

Oh, and the switch did not like the using archive...

archive download-sw /overwrite tftp://1.2.3.4/c3560c-universalk9-tar.122-55.EX3.tar

Errrr ... I have >50 of these (WS-C3560CG-8PC) and they are all running 12.2(55)EX3 on a Layer 2. 

I upgraded a few (for testing) with the "archive download-sw /over /no-boot-set tftp://1.2.3.4/c3560c-universalk9-tar.122-55.EX3.tar" and I don't see any issues or "failure" of this command.


The rest of the 3560CG I've deployed had their IOS upgraded using Cisco ZeroTouch.  Again, I didn't see any issues.

No issues with the CPU either. 

crapsh@monkeybr... Wed, 05/02/2012 - 18:13

Maybe it was just the EX2 that came on the device.  Made it through the day, and CPU was low.

# show processes cpu sorted

Showed Hulc LED Process at 10% <--- not normal for the blinking LED to use 10% of the CPU. 

Leo Laohoo Wed, 05/02/2012 - 18:30
Maybe it was just the EX2 that came on the device.

I forgot to mention that all my 3560CG were initially running EX2.  I use ZeroTouch to upgrade all of them to EX3 plus slap a configuration template on them.

crapsh@monkeybr... Wed, 05/02/2012 - 21:10

Ah!  Found another one... will just update IOS this time (already disabled unused SFP port) and see if that gets the CPU lower than 47%.

zyz-white#sh interfaces gigabitEthernet 0/1 | include rate
  Queueing strategy: fifo
  5 minute input rate 24731000 bits/sec, 2758 packets/sec
  5 minute output rate 2924000 bits/sec, 1538 packets/sec
zyz-white#sh ver | include EX                             
Cisco IOS Software, C3560C Software (C3560c405ex-UNIVERSALK9-M), Version 12.2(55)EX2, RELEASE SOFTWARE (fc1)
BOOTLDR: C3560C Boot Loader (C3560C-HBOOT-M) Version 12.2(55r)EX1, RELEASE SOFTWARE (fc1)
System image file is "flash:/c3560c405ex-universalk9-mz.122-55.EX2/c3560c405ex-universalk9-mz.122-55.EX2.bin"
*    1 10    WS-C3560CG-8TC-S   12.2(55)EX2           C3560c405ex-UNIVERSALK9-M
zyz-white#sh processes cpu sorted                         
CPU utilization for five seconds: 45%/31%; one minute: 47%; five minutes: 47%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
 135   652688748 130171328       5014  7.39%  7.15%  7.09%   0 Hulc LED Process 
  80   199796523 163931519       1218  0.79%  1.20%  1.12%   0 HLFM address lea 
 196    25710741  79047519        325  0.29%  0.33%  0.35%   0 Spanning Tree    
  66    11734741   2447998       4793  0.19%  0.15%  0.14%   0 Adjust Regions   
  10    11917385  50175794        237  0.19%  0.26%  0.23%   0 ARP Input        
  82     1004611 164714671          6  0.09%  0.03%  0.00%   0 HLFM address ret 
 182     2636107  11078867        237  0.09%  0.03%  0.00%   0 ADJ resolve proc 
 211     2997298   6116886        490  0.09%  0.01%  0.00%   0 PI MATM Aging Pr 
   8    16351509    102001     160308  0.00%  0.14%  0.16%   0 Licensing Auto U 
   7           0         2          0  0.00%  0.00%  0.00%   0 License Client N 
crapsh@monkeybr... Fri, 05/04/2012 - 16:26

Updating OS didn't help that much.  Really, it took chaning the device to an NSSA. Looks like the 3560C is just barely a Layer 3 device...  Hopefully, this solves the packet loss we were seeing through the device.

Leo Laohoo Fri, 05/04/2012 - 17:53

Hey Rudy,

Just a stupid question:  Do you see the CPU utilization go down when you console/Telnet/SSH into the appliance?

Actions

Login or Register to take actions

This Discussion

Posted May 2, 2012 at 10:51 AM
Stats:
Replies:6 Avg. Rating:
Views:1161 Votes:0
Shares:0
Tags: cpu, hulc
+
Categories: Routers
+

Related Content

Discussions Leaderboard