Cisco 3750x extrmly high CPU

JPCPIA130 · ‎08-04-2014

We are having some troubles with files opening very slow and outlook being extremely delayed so i started to look into our network ran a sh proc cpu his and here are the result are below. we have 2 VLans and no ACLs. I am seeing the Hulc LED Process uses any where from 15 - 40 % utilization at times. We have five 3750x 48 port stacked and when looking at the status of that it is running in full with no loops. All switches are just about fully populated. On out stack master we have a 3 sfp 1gig fiber modules with up links to 3 switches in out shop floor. all switches are running 12.2 (55) SE8. any help would be greatly appreciated.

3222222222222222222223333344444333332222222223333377777888
    1777779999988888777771111133333333339999888880000011111777
100
90                                                        ***
80                                                        ***
70                                                   ********
60                                                   ********
50                                                   ********
40                           *****                   ********
30 **********************************************************
20 **********************************************************
10 **********************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)
                                                11    1   1
    8998998999989989598989979989899899899899999800999904440994
    7783971913977908887956998629948798299480559900929903230997
100 ** ** * * ** * * * ** ** * * ** ** * *** #** ***   ***
90 *** ** ******* * ****** ** ******* ** ******#******   ***
80 **************** ***************************#******   ***
70 **************** ***************************#*****#   **#
60 ********************************************#**#**#   #*#
50 ********************************************##*#**#   ###*
40 ################*#######################*##########***###*
30 ##########################################################
20 ##########################################################
10 ##########################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%
    1
    0456464444456479474555555576664744856446664644465574554745545485754455
    0873509968834622545295572754619345845952139364831656005266174792937761
100 *
90 *              *                  *                           *
80 *              *          *       *               *           * *
70 *             ** *        * * * * *             *    *      * *
60 * ** *      * ** * **** ***** * * * *** *   * **    * *    * *   *
50 ******************************** ************ ************************
40 #*********************************************************************
30 ######################################################################
20 ######################################################################
10 ######################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

Leo Laohoo · ‎08-04-2014

Wow! I've never seen this behaviour on 12.2(55)SE8 before.

What's the uptime of the switch? Do you see any ports flapping?

JPCPIA130 · ‎08-05-2014

The stack up time is 12 weeks, 4 days, 22 hours, 19 minutes. I just noticed that the date/time was incorrect on the switch but this is what i am seeing when running show logging

070435: *May 28 17:44:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/27, changed state to down
070436: *May 28 17:44:02: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/27, changed state to down
070437: *May 28 17:44:04: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/27, changed state to up
070438: *May 28 17:44:05: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/27, changed state to up
070439: *May 28 17:47:20: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/16, changed state to down
070440: *May 28 17:47:24: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/16, changed state to down
070441: *May 28 17:47:26: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/16, changed state to up
070442: *May 28 17:47:27: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/16, changed state to up
070443: *May 28 17:47:50: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/16, changed state to down
070444: *May 28 17:47:51: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/16, changed state to down
070445: *May 28 17:47:53: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/16, changed state to up
070446: *May 28 17:47:54: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/16, changed state to up
070447: *May 28 17:50:25: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/42, changed state to down
070448: *May 28 17:50:28: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/42, changed state to down
070449: *May 28 17:50:30: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/42, changed state to up
070450: *May 28 17:50:31: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/42, changed state to up
070451: *May 28 17:50:56: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/42, changed state to down
070452: *May 28 17:50:57: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/42, changed state to down
070453: *May 28 17:50:59: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/42, changed state to up
070454: *May 28 17:51:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/42, changed state to up
070455: *May 28 17:52:46: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/38, changed state to down
070456: *May 28 17:52:49: %LINK-3-UPDOWN: Interface GigabitEthernet5/0/38, changed state to down
070457: *May 28 17:52:52: %LINK-3-UPDOWN: Interface GigabitEthernet5/0/38, changed state to up
070458: *May 28 17:52:53: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/38, changed state to up
070459: *May 28 17:53:25: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/38, changed state to down
070460: *May 28 17:53:26: %LINK-3-UPDOWN: Interface GigabitEthernet5/0/38, changed state to down
070461: *May 28 17:53:29: %LINK-3-UPDOWN: Interface GigabitEthernet5/0/38, changed state to up
070462: *May 28 17:53:30: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/38, changed state to up
070463: *May 28 18:00:40: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/22, changed state to down
070464: *May 28 18:00:42: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/22, changed state to up
070465: *May 28 18:00:45: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/22, changed state to down
070466: *May 28 18:00:47: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/22, changed state to up
070467: *May 28 18:24:30: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/46, changed state to down
070468: *May 28 18:24:31: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/46, changed state to down
070469: *May 28 18:24:33: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/46, changed state to up
070470: *May 28 18:24:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/46, changed state to up
070471: *Aug 5 08:50:00: %SYS-6-CLOCKUPDATE: System clock has been updated from 18:33:22 EDT Fri May 28 1993 to 08:50:00 EDT Tue Aug 5 2014, configured from console by on console.

070472: .Aug 5 08:55:31: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/16, changed state to down
070473: .Aug 5 08:55:33: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/16, changed state to up
070474: .Aug 5 08:55:35: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/16, changed state to down
070475: .Aug 5 08:55:36: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/16, changed state to down
070476: .Aug 5 08:55:39: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/16, changed state to up
070477: .Aug 5 08:55:40: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/16, changed state to up
070478: .Aug 5 08:55:56: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/16, changed state to down
070479: .Aug 5 08:55:57: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/16, changed state to down
070480: .Aug 5 08:56:00: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/16, changed state to up
070481: .Aug 5 08:56:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/16, changed state to up

JPCPIA130 · ‎08-05-2014

I was just looking at the inventory and all switches are are WS-C3750X-48 but SW1 in the stack, the stack master, is Version 2 all the rest are version 1. Could this cause any issues?

Leo Laohoo · ‎08-05-2014

No. Hardware revision doesn't play any role into why it's causing this.

Your "sh logs" output doesn't give any hint of something wrong. Your stack size good.

Hulc process is a service which determines if the interface is up/down. The uptime of 12 weeks is not much, however, I'd really be itching to ask you to reload the stack (if its possible) and see in a few days/weeks if the problem re-occurs.

Otherwise, I'd recommend you get TAC to get a closer look.

JPCPIA130 · ‎08-07-2014

On Tuesday we had a power outage for an extended period of time and i took the entire network off line. I will keep an eye on it but so far here is the cpu his and the show log is still filled with up/downs

    2233333222223333322222444442222222222222223333322222222222
    8822222777774444499999222227777788888999992222299999777778
100
90
80
70
60
50
40                       *****
30 **********************************************************
20 **********************************************************
10 **********************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

    4343433343444444334334433434434446344334343444343344343444
    2729199704000000890882266084292233933884615023707900921511
100
90
80
70
60                                  *
50                                  *                     *
40 ********* ******************************************** ***
30 ##########################################################
20 ##########################################################
10 ##########################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

    64547755444565555544555555444965458554455989
    39270713198152040775115324757212630947346962
100                                          *
90                              *           ***
80      *                       *    *      ***
70     **      *                *    *      ***
60 *   **      *    *    *      **   **    ****
50 ******** ***************************** *****
40 ********************************************
30 ############################################
20 ############################################
10 ############################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

Rajeev Sharma · ‎08-07-2014

Hey,

it could be bug CSCtn42790. Check it on:

https://tools.cisco.com/bugsearch/bug/CSCtn42790

Adding the snippet for you:

Symptom:
Hulc LED Process uses 15-30% CPU on Catalyst 3560X/3750X platforms.

Conditions:
The is seen in 12.2(53)SE releases or later.

Workaround:
This is an expected behavior and there is no workaround.

Further Problem Description:
The "Hulc LED" process does following tasks:
- Check Link status on every port
- If the switch supports POE, it checks to see if there is a Power Device (PD)
detected
- Check the status of the transceiver
- Update Fan status
- Set Main LED and ports LEDs
- Update both Power Supplies and RPS
- Check on system temperature status

HTH.

Regards,

RS.