How to fix high CPU on 3650C

Rudy R. · ‎05-02-2012

If you are searching due to finding out your 3650C has high CPU, you can try these tricks:

* add storm control

* upgrade to c3560c-universalk9-tar.122-55.EX3.tar

(EX2 has an Hulc LED Process bug)

* If you are in area 0 with OSPF, turn it into an NSSA and reduce your routes!

Honestly, I am not sure what fixed my switch, but it was using a lot of CPU and rebooting itself. It appears stable today (I did all three changes at once). Attached is a graph showing CPU usage over time on the switch.

Oh, and the switch did not like the using archive...

archive download-sw /overwrite tftp://1.2.3.4/

c3560c-universalk9-tar.122-55.EX3.tar

So I did a

# copy tftp://1.2.3.4/

c3560c405ex-universalk9-mz.122-55.EX3/c3560c405ex-universalk9-mz.122-55.EX3.bin flash:

then

(config)# boot system

flash:/c3560c405ex-universalk9-mz.122-55.EX3.bin

This was on WS-C3560CG-8PC-S

Old OS: 12.2(55)EX2

New OS: 12.2(55)EX3

Leo Laohoo · ‎05-02-2012

Oh, and the switch did not like the using archive...

archive download-sw /overwrite tftp://1.2.3.4/c3560c-universalk9-tar.122-55.EX3.tar

Errrr ... I have >50 of these (WS-C3560CG-8PC) and they are all running 12.2(55)EX3 on a Layer 2.

I upgraded a few (for testing) with the "archive download-sw /over /no-boot-set tftp://1.2.3.4/c3560c-universalk9-tar.122-55.EX3.tar" and I don't see any issues or "failure" of this command.

The rest of the 3560CG I've deployed had their IOS upgraded using Cisco ZeroTouch. Again, I didn't see any issues.

No issues with the CPU either.

Rudy R. · ‎05-02-2012

Maybe it was just the EX2 that came on the device. Made it through the day, and CPU was low.

# show processes cpu sorted

Showed Hulc LED Process at 10% <--- not normal for the blinking LED to use 10% of the CPU.

Leo Laohoo · ‎05-02-2012

Maybe it was just the EX2 that came on the device.

I forgot to mention that all my 3560CG were initially running EX2. I use ZeroTouch to upgrade all of them to EX3 plus slap a configuration template on them.

Rudy R. · ‎05-02-2012

Ah! Found another one... will just update IOS this time (already disabled unused SFP port) and see if that gets the CPU lower than 47%.

zyz-white#sh interfaces gigabitEthernet 0/1 | include rate
  Queueing strategy: fifo
  5 minute input rate 24731000 bits/sec, 2758 packets/sec
  5 minute output rate 2924000 bits/sec, 1538 packets/sec
zyz-white#sh ver | include EX                             
Cisco IOS Software, C3560C Software (C3560c405ex-UNIVERSALK9-M), Version 12.2(55)EX2, RELEASE SOFTWARE (fc1)
BOOTLDR: C3560C Boot Loader (C3560C-HBOOT-M) Version 12.2(55r)EX1, RELEASE SOFTWARE (fc1)
System image file is "flash:/c3560c405ex-universalk9-mz.122-55.EX2/c3560c405ex-universalk9-mz.122-55.EX2.bin"
*    1 10    WS-C3560CG-8TC-S   12.2(55)EX2           C3560c405ex-UNIVERSALK9-M
zyz-white#sh processes cpu sorted                         
CPU utilization for five seconds: 45%/31%; one minute: 47%; five minutes: 47%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
 135   652688748 130171328       5014  7.39%  7.15%  7.09%   0 Hulc LED Process 
  80   199796523 163931519       1218  0.79%  1.20%  1.12%   0 HLFM address lea 
 196    25710741  79047519        325  0.29%  0.33%  0.35%   0 Spanning Tree    
  66    11734741   2447998       4793  0.19%  0.15%  0.14%   0 Adjust Regions   
  10    11917385  50175794        237  0.19%  0.26%  0.23%   0 ARP Input        
  82     1004611 164714671          6  0.09%  0.03%  0.00%   0 HLFM address ret 
 182     2636107  11078867        237  0.09%  0.03%  0.00%   0 ADJ resolve proc 
 211     2997298   6116886        490  0.09%  0.01%  0.00%   0 PI MATM Aging Pr 
   8    16351509    102001     160308  0.00%  0.14%  0.16%   0 Licensing Auto U 
   7           0         2          0  0.00%  0.00%  0.00%   0 License Client N

Rudy R. · ‎05-04-2012

Updating OS didn't help that much. Really, it took chaning the device to an NSSA. Looks like the 3560C is just barely a Layer 3 device... Hopefully, this solves the packet loss we were seeing through the device.

Leo Laohoo · ‎05-04-2012

Hey Rudy,

Just a stupid question: Do you see the CPU utilization go down when you console/Telnet/SSH into the appliance?

GaneshSambandam0238 · ‎12-01-2019

Yes, i'm having the similar issue while i'm SSH the device utilization goes normal.

Georg Pauwen · ‎12-02-2019

Hello,

can you post the running config of your 3650 ? There are a few bugs related to specific features...