C3560-48PS - Spontaneously 100% CPU Utilization

RiesAlexander · ‎08-19-2010

Hello,

i have a problem on C3560-48PS Switch. It have spontaneously a CPU usage of 100%.

Version 12.2(53)SE2

Image C3560-IPBASEK9-M

Switch#show processes cpu history

     11111          11111                              1111111
    8000008888899999000008888899999666669999977777777772222200
100
90
80
70
60
50
40
30
20
10 **********************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

    1111111411111551311111111251111111111111111113121111111111
    4768767787677036189757767146768887676897767788876866679768
100
90
80
70
60
50        *     **           *
40        *     **           *                  *
30        *     ** *         *                  * *
20 ******#*****##*###******##****##**#**###**#####*#***##**#
10 ##########################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%
              1      1
    5557555555055555505548555855555555455555555553555555556565555555555555
    4414204230002132401095220100122040411321320027101311419360123311269472
100           *      *
90           *      *   *
80           *      *   *   *
70    *      *      *   *   *                            * *
60    *      *      *   *   *                            * *        ** *
50 ********************************** ********** ************************
40 **********************************************************************
30 **********************************************************************
20 ######################**#*###################*######################**
10 ######################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

When i look at the processes, i find "REP LSL Hello PP" at over 7,01%

Switch#show processes cpu sorted
CPU utilization for five seconds: 9%/0%; one minute: 14%; five minutes: 15%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
248         173     10068         17 0.95% 0.12% 0.02%   1 Virtual Exec
197    59494892 56599704       1051 0.79% 0.52% 0.53%   0 Spanning Tree
234    70914044 21489129       3299 0.79% 0.55% 0.56%   0 Marvell wk-a Pow
125    45757496 251358992        182 0.47% 0.38% 0.36%   0 Hulc LED Process
34    16809346   8573236       1960 0.31% 0.14% 0.12%   0 Per-Second Jobs
219   8201334892686047037          0 0.31% 6.39% 7.01%   0 REP LSL Hello PP
35     6436343    430927      14936 0.31% 0.04% 0.00%   0 Per-minute Jobs
211         754     13644         55 0.15% 0.01% 0.00%   0 TCP Timer
191     1109086   8875447        124 0.15% 0.03% 0.00%   0 IP Input
134     5563918   3448322       1613 0.15% 0.03% 0.01%   0 HRPC qos request
11           0         1          0 0.00% 0.00% 0.00%   0 CEF MIB API
10      993484   5008527        198 0.00% 0.00% 0.00%   0 ARP Input
13           0         2          0 0.00% 0.00% 0.00%   0 AAA high-capacit

I got this evaluation for one week. And I have the process for the 100% CPU load is not yet discovered.

With've noticed that the load to 8% - 9% decrease as soon as I connect via Telnet and stay looged in over 5 minutes.

Thanks in advance

MATTHIAS SCHAERER · ‎08-19-2010

Haven't done it myself yet but did you consider writing an EEM script that checks the CPU load periodically and if it goes beyond 90% or so sends you an email with the content of 'sh proc cpu sorted'?

I do not know how much the script adds to the CPU load. you might want to try this by changing the interval slowly observing the CPU load at the same time.

HTH

mat

Chad Peterson · ‎08-19-2010

EEM script would work.

Also, attach your config. As you see its a fast spike, it doesn't increase the avg utilization. It may be hard to catch, but mabey try removing some features and waiting to see if it happens.

RiesAlexander · ‎08-20-2010

Thanks for the Answers,

i find an interessting inI found an interesting information. And this problem is officially known, but unfortunately no solution.

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/release/12.2_55_se/release/notes/OL23054.html#wp1044235

Open Caveats

CSCth24278 (Catalyst 2960-S switch)

The CPU utilization on the switch remains high (50 to 60 percent) when the switch is not being accessed by a telnet or a console session. When you telnet or console into the switch, the CPU utilization goes down.

There is no workaround.

I have the same problem at 40x 3560 Devices with 12.2(53)SE2
After a downgrade to the stable 12.2(46), the problem has gone away. And i have enought Features for our Enviroment.

But the tip with the EEM I find really helpful and interessting for other troubleshootings

Thanks for the help