3750 - High Sustained CPU - HULC DAI Process

jorchard · ‎03-26-2010

Hello.

We have a couple of 3750 switch stacks that have been sustaining cpu of 99%. They've been this way for more than 24 hours, and not sure what is causing the issue. We tried doing a reload of the switch this morning but the cpu came right back up to 99% after the reload. Here is the sh proc cpu:

Br-327-3750-Admin#sh proc cpu sorted
CPU utilization for five seconds: 100%/10%; one minute: 99%; five minutes: 99%
PID Runtime(ms)   Invoked       uSecs   5Sec   1Min   5Min TTY Process
170   212128678 162676677       1303 65.23% 65.81% 65.76%   0 HULC DAI Process
77    31557079 244150005         129 10.36% 10.26% 10.21%   0 HLFM address lea
135    56832130 196397793        289 2.55% 2.93% 2.98%   0 Hulc LED Process
237     56318985 18360226       3067 0.63% 0.42% 0.46%   0 Marvell wk-a Pow
205    67890494 168277229        403 0.63% 0.46% 0.47%   0 Spanning Tree
101    24887317 89295971        278 0.63% 0.50% 0.44%   0 HRPC pm-counters
63    12360872 437007936         28 0.63% 0.27% 0.24%   0 Fifo Error Detec
155    20428352   8320260       2455 0.47% 0.35% 0.32%    0 PI MATM Aging Pr
175     3584095   4908931        730 0.31% 0.05%   0.04%   0 Port-Security
144    44080904 20086076       2194 0.31% 0.24%   0.26%   0 HRPC qos request
192    33579091 49757829        674 0.31%   0.45% 0.43%   0 CDP Protocol
67    51142477 163092690        313 0.31%   0.36% 0.35%   0 hrpc <- response
100    22326991 17338923       1287   0.31% 0.27% 0.29%   0 hpm counter proc
78      684742   8321467          82 0.15% 0.03% 0.00%   0 HLFM aging proce
153     8629798   15205139        567 0.15% 0.06% 0.01%   0 HL2MCM
199      964475    5870158        164 0.15% 0.02% 0.00%   0 IP Input
66     1431581   55879102         25 0.15% 0.01% 0.00%   0 hrpc -> request
291     14162665 18379505        770 0.15% 0.13% 0.14%   0 SNMP ENGINE
143      3628971   1681067       2158 0.15% 0.14% 0.15%   0 HQM Stack Proces
137     1990866 16641527        119 0.15% 0.02% 0.00%   0 HRPC hl3u reques
42     3021743   1687781       1790 0.15% 0.11% 0.10%   0 Compute load avg
91     2887841 24044166        120 0.15% 0.04% 0.01%    0 HRPC ilp request
174      409825   8421668         48 0.15% 0.11%   0.14%   0 UDLD
185      411968 82459173          4 0.15% 0.07% 0.05%   0 MDFS MFIB Proces
24         218        27       8074 0.00% 0.00% 0.00%    0 PrstVbl
23           0         1          0 0.00% 0.00% 0.00%   0 IPC Session Serv
...

Here is our sh ver:

Br-375-3750#sh ver
Cisco IOS Software, C3750 Software (C3750-IPBASE-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 25-Sep-09 02:53 by sasyamal
Image text-base: 0x01000000, data-base: 0x02A00000

ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

Br-375-3750 uptime is 1 hour, 27 minutes
System returned to ROM by power-on
System restarted at 11:15:54 UTC Fri Mar 26 2010
System image file is "flash:c3750-ipbase-mz.122-52.SE/c3750-ipbase-mz.122-52.SE.bin"

Any thoughts on what might be causing this issue?

Thanks.

Reza Sharifi · ‎03-26-2010

Hi Jorchard,

Check your config to see if you have a default route configured pointing to an interface instead the IP address as seen below:

ip route 0.0.0.0 0.0.0.0 Fastethernet0/0

If yes change the default route to point to on IP address and check the CPU utilization again.

Here is the document for more info:

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00807213f5.shtml

HTH

Reza

alex_bsb_by · ‎10-07-2021

I had the same problem with disabled DAI. In logs I discovered: Host X.X.X (MAC address) in vlan 15 is flapping between port Gi0/21 and port Gi0/14. This problem continue not a long time. That's why I didn't have time to try to disable one of the interfaces (Gi0/21 or Gi0/14). Mabye it will be helpful for someone