2960 - HULC LED PROCESS

[CyberSecLead]

Dear All,

I have cisco WS-C2960S-48FPS-L stacked. Weekly twice, my PoE connections are dropped and when the device is restarted, everything starts working normal. This issue happens weekly once or twice.

Verified the power input and it's absolutely fine.

show power inline

Module   Available     Used     Remaining

          (Watts)     (Watts)    (Watts)

------   ---------   --------   ---------

1           740.0       91.5       648.5

2           740.0       60.2       679.8

3           740.0      144.3       595.7

4           740.0      157.1       582.9

Upon taking a look to the CPU Process, device is punted with 75 - 85 % at normal usage, with "Hulc LED Process" as 18% utilisation. CPU utilisation also peaks to 90 - 100% sometimes.

CPU utilization for five seconds: 78%/48%; one minute: 58%; five minutes: 50%

100 *              *                                                     

90 * **  *  *     *                                                     

80 *###############                                                     

70 ################                                                     

60 ################                                                     

50 ################                                                     

40 ################                                                     

30 ################                                                     

20 ################                                                     

10 ################                                                     

   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

             0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)

                  * = maximum CPU%   # = average CPU%

I can see that there is a bug id : CSCtg86211 and no work around for it.

Does anyone faced similar issue and any updates received from Cisco TAC ?

Any recommendations to overcome this issue !!

BR / Vimal.

[CyberSecLead]

Awaiting Cisco Experts to reply to the above subject.

Thanks / Vimal.

[lcubero]

Hi Vimal,

Did you have any answer from Cisco ? I am working on a project with 2960 and facing same issue. I would really appreciate if it has already been solved.

Regards,

Luis.

[Mark Pottage]

I am also seeing an issue with two stacks of 5 x WS-C2960X-48FP-S switches similar to this case.

Code in use:  c2960x-universalk9-mz.150-2.EX5.bin

At the moment there is nothing connected to the copper 48 ports in any of the switches.

Each stack has a LACP port-channel to a 2 x WS-C3850-48F-E core stack utilising ports x/0/51 and x/0/52 on each 2960X - total 10 ports in each port-channel (8 in use 2 Hot-standby)

The uplink ports are using GLC-Ts

There are 5 vlans (including Vlan1) on the switch.

sw02#show proc cpu | excl 0.00
CPU utilization for five seconds: 42%/0%; one minute: 42%; five minutes: 42%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
  91     1355993    26717170         50  0.29%  0.22%  0.19%   0 RedEarth Tx Mana
  92      886939    26742049         33  0.11%  0.12%  0.16%   0 RedEarth Rx Mana
 121      618726    15414683         40  0.11%  0.06%  0.06%   0 HLFM address lea
 127     2828875    11788935        239  0.41%  0.41%  0.42%   0 hpm main process
 132      637843     4108756        155  0.05%  0.07%  0.05%   0 HRPC pm-counters
 162   176218586    15422705      11425 25.07% 25.30% 25.31%   0 Hulc LED Process
 173     2274099      122941      18497  0.35%  0.31%  0.30%   0 HQM Stack Proces
 174     1720412     1228633       1400  0.23%  0.23%  0.23%   0 HRPC qos request
 183      844538      205684       4105  0.05%  0.10%  0.11%   0 Power RPS Proces
 207        2879       22868        125  0.11%  0.10%  0.06%   0 IP Input
 220      288284     1554997        185  0.11%  0.04%  0.05%   0 Spanning Tree
 237     1356330     6170406        219  0.11%  0.07%  0.14%   0 UDLD
 293      324727     1359786        238  0.05%  0.04%  0.05%   0 Marvell wk-a Pow
 372         120         123        975  0.05%  0.07%  0.01%   1 Virtual Exec

sw03#sh proc cpu | exc 0.00
CPU utilization for five seconds: 42%/0%; one minute: 46%; five minutes: 45%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
  63         507        1733        292  0.05%  0.06%  0.03%   0 Per-Second Jobs
  91        5409       72997         74  0.35%  0.23%  0.23%   0 RedEarth Tx Mana
  92        4526       73169         61  0.11%  0.19%  0.23%   0 RedEarth Rx Mana
 129       11881       29115        408  0.54%  0.42%  0.42%   0 hpm main process
 133        1259        1674        752  0.23%  0.16%  0.12%   0 hpm counter proc
 163      494386       41535      11902 25.38% 25.35% 25.30%   0 Hulc LED Process
 174        6130         374      16390  0.29%  0.31%  0.29%   0 HQM Stack Proces
 175        4446        3249       1368  0.29%  0.25%  0.24%   0 HRPC qos request
 183        2237         559       4001  0.11%  0.16%  0.13%   0 Power RPS Proces
 206         894       41778         21  0.05%  0.03%  0.03%   0 IP ARP Retry Age
 237        5298       16703        317  0.11%  0.14%  0.26%   0 UDLD
 251        4045         452       8949  0.29%  2.68%  0.98%   1 Virtual Exec
 291        1183        3706        319  0.05%  0.05%  0.05%   0 Marvell wk-a Pow

Any advice from Cisco would be much appreciated.

Mark

[chris phillips]

Seeing the same issue here with 2960x 48 port poe's running 15.02a no devices connected to the stack.

Cisco, are you looking to address this at any stage?

[Sam Almeida]

Do you have cisco stackwise stack power cables connected between the stacks? If so ensure they are connected correctly.

DATA Stack Cable:
Sw1  Stack 1 >>>>  SW2 Stack 2
Sw1  Stack 2 >>>>  SW2 Stack 1

Stackwise Power cable
Sw1  Stack 1 >>>>  SW2 Stack 1
Sw1  Stack 2 >>>>  SW2 Stack 2

[chris phillips]

sadly nothing as exotic or costly as that, just a stack of 3 2960-x poe switches.

further research shows that Cisco don't think this is an issue. HULC process will be low priority and even if running 100% other interrupts will take precedence. Seems HULC is an amalgamation of other processes in one.

The biggest issue here is environmental as excess power and therefore heat is generated when not needed.

[Sam Almeida]

You mention  no devices are connected. Admin shut all the ports on the switch (except management) and verify if cpu goes down

[Leo Laohoo]

Weekly twice, my PoE connections are dropped and when the device is restarted,

If your switch stacks rebooted without any intervention, then check each switch stack for the crashinfo file(s) or subdirectories and post the three LATEST files.

Also, please state the IOS version. 

[CyberSecLead]

Hi,

IOS code running is  c2960s-universalk9-mz.122-53.SE2. I haven't mentioned that the device got rebooted, instead all the PoE connections dropped. Example, Ip phones and AP's were in registering phase for a long time, until a reboot is needed to re-establish the connections back.

Any suggestions.

Regards / Vimal.

[tmomani]

Hi,

Its is totally expected fir the HULC LED to have up to 30% of the cpu utilization, as thi process handles ports changes (state, poe..etc).

The high cpu u r facing is under interrups (X%/Y%) where X is the total utilization and Y is the interrupt value.

The interrupt means that the switch is forwarding alot of traffic. Would u plz send a full outpu of "sh proc cpu sorted | ex 0.00".

Based on the output, if no "ip input or stp or any otger special process is in place" then it would be a HW limitation for this platform, then u would rather offload this switch(load balance traffic to multiple devices) or maybe use a more powerful switch.

Cheers,

Tareq

Sent from Cisco Technical Support iPhone App

[CyberSecLead]

Hi, please find attached to the "show tech". There is not much process involved in "input" "stp" etc.

Regards / Vimal.

[tmomani]

Hello,

TCAM utilization is OK, lets identify the fastest growing CPU Queue, and then we will capture traffic punted to that Q.

when the CPU is High,

take "show controllers cpu-interface" command 3 - 4 times seperated by 2 seconds interval

identify the fastest growing counter, and based on that , capture traffic using "debug platform cpu-queuequeue", but please notice that the output is very large, so  enable safe debugging before

+ increase the buffer log by "SW(config)#logging buffer 100000"

+ SW(config)#no logging console

+ SW#terminal no monitor

refer to below link as a reference,

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

let me know on how this goes,

Tareq

[Leo Laohoo]

IOS code running is  c2960s-universalk9-mz.122-53.SE2.

I read your reply post and didn't bother to read any further..

Do NOT use this IOS, 12.2(53)SE.  I know I wouldn't.

IOS I would personally recommend are the following:

1. 12.2(55)SE5 or 12.2(55)SE6; or
2. 15.0(1)SE3; or
3. 15.0(2)SE

Read the Release Notes first before making a decision.