Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

High CPU usage - 3750 X stack

Support Community

We recently configured a stack of four 48 port 3750-x switches . We are noticing high CPU usage. "Hulc LED process" seems pretty high.

This has coincided with VMware servers getting slow and non-responsive at times, perhaps a coincidence, not sure.

Below I provided some outputs that might help to diagnose it

Thanks

John

System image file is "flash:/c3750e-ipbasek9-mz.122-58.SE2/c3750e-ipbasek9-mz.122-58.SE2.bin"

Show inventory output

NAME: "1", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S   , VID: V02  ,

NAME: "Switch 1 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC   , VID: V02L ,

NAME: "2", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S   , VID: V02 

NAME: "Switch 2 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC   , VID: V02D ,

NAME: "3", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S   , VID: V02 

NAME: "Switch 3 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC   , VID: V02L ,

NAME: "4", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S   , VID: V02 

NAME: "Switch 4 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC   , VID: V02L ,

SWITCH#sh processes cpu sorted

CPU utilization for five seconds: 61%/5%; one minute: 50%; five minutes: 49%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

168   260466386    44948517       5794 14.53% 13.98% 13.70%   0 Hulc LED Process

231    97586088    27253906       3580  4.95%  4.73%  4.64%   0 Spanning Tree

213    63106121   154928892        407  4.15%  3.89%  3.91%   0 IP Input

284    70113217    34537588       2030  3.51%  3.98%  4.17%   0 RARP Input

   4     6663412      421278      15817  3.03%  0.43%  0.32%   0 Check heaps

374     9872291    10805181        913  3.03%  0.77%  0.62%   0 IP SNMP

376    11142951     5370604       2074  3.03%  0.73%  0.66%   0 SNMP ENGINE

  12    35389011    32152175       1100  2.87%  2.08%  2.20%   0 ARP Input

128    34962407     3622140       9652  2.07%  1.69%  1.63%   0 hpm counter proc

  85    49034286     8536062       5744  1.91%  2.44%  2.44%   0 RedEarth Tx Mana

107    25127806    46459053        540  1.27%  1.10%  0.93%   0 HLFM address lea

174        2412        1714       1407  0.95%  0.39%  0.25%   1 SSH Process

220     6423643    12634764        508  0.79%  0.70%  0.56%   0 ADJ resolve proc

181     6913179     2890070       2392  0.63%  0.31%  0.36%   0 HRPC qos request

375     1681949     5000777        336  0.47%  0.08%  0.07%   0 PDU DISPATCHER

  84    10180707    12623537        806  0.47%  0.30%  0.37%   0 RedEarth I2C dri

        1

      666666096996666666666666659666667666666666666666666766676666666656666666

      249363098992351145264823289455360612252332233522344115537230141392553343

  100       ** **               *

   90       ** **               *

   80       ** **               *

   70   * * *****  *   * * *    * ** ***   *       *     * ****         **

   60 **********************************************************************

   50 ######################################################################

   40 ######################################################################

   30 ######################################################################

   20 ######################################################################

   10 ######################################################################

     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..

               0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)

                  * = maximum CPU%   # = average CPU%

      455555555554444444444555554444455555555555555555555444444444

      922222111118888866666000009999911111555554444422222444448888

  100

   90

   80

   70

   60                                     *****

   50 ***************************************************     **

   40 **********************************************************

   30 **********************************************************

   20 **********************************************************

   10 **********************************************************

     0....5....1....1....2....2....3....3....4....4....5....5....6

               0    5    0    5    0    5    0    5    0    5    0

               CPU% per second (last 60 seconds)

      565756555555555555555555555555555556555555555555565555565556

      518841757869248569271526666733778330496833777819929379701861

  100

   90

   80    *

   70    *

   60 **** *******  **** * * *****  ***  * ***  **** **** **** *

   50 ##########################################################

   40 ##########################################################

   30 ##########################################################

   20 ##########################################################

   10 ##########################################################

     0....5....1....1....2....2....3....3....4....4....5....5....6

               0    5    0    5    0    5    0    5    0    5    0

               CPU% per minute (last 60 minutes)

              * = maximum CPU%   # = average CPU%

37 REPLIES

Re: High CPU usage - 3750 X stack

What about the configs?

Sent from Cisco Technical Support iPad App

Purple

Re: High CPU usage - 3750 X stack

New Member

High CPU usage - 3750 X stack

Thanks for your replies.

Here is another output and an excerpt from the config:

switch# show platform tcam utilization

CAM Utilization for ASIC# 0                      Max            Used

                                             Masks/Values    Masks/values

Unicast mac addresses:                       6364/6364       2029/2029

IPv4 IGMP groups + multicast routes:         1120/1120          1/1

IPv4 unicast directly-connected routes:      6144/6144       1232/1232

IPv4 unicast indirectly-connected routes:    2048/2048        104/104

IPv4 policy based routing aces:               452/452          12/12

IPv4 qos aces:                                512/512          21/21

IPv4 security aces:                           964/964         956/956

clock timezone CST -6 0

clock summer-time CDT recurring

switch 1 provision ws-c3750x-48

switch 2 provision ws-c3750x-48

switch 3 provision ws-c3750x-48

switch 4 provision ws-c3750x-48

system mtu routing 1500

ip routing

!

!

ip domain-name domain.com

vtp domain vtpdomain

vtp mode transparent

spanning-tree mode pvst

spanning-tree extend system-id

!

!

!

!

vlan internal allocation policy ascending

!

vlan 2

name Inside

!

vlan 3

name extranet

!

vlan 4

name DMZ

!

vlan 5

name Internet

!

vlan 60,169,210,230-231,240-242,300-301,500-501,503-506,777

!

ip ssh version 2

!

!

!

!

!

interface Port-channel2

switchport trunk encapsulation dot1q

switchport mode trunk

!

interface FastEthernet0

no ip address

no ip route-cache cef

no ip route-cache

interface Vlan2

description Inside

no ip address

!

interface Vlan3

description Extranet

no ip address

!

interface Vlan4

description DMZ

no ip address

!

interface Vlan5

description Internet

no ip address

!

interface Vlan46

no ip address

!

interface Vlan60

description vLAN

no ip address

!

interface Vlan90

no ip address

!

interface Vlan169

ip address 169.254.254.254 255.255.255.0

ip access-group ip169 in

ip access-group ip169out out

!

interface Vlan210

description Management Network

ip address 10.210.254.254 255.255.0.0

ip access-group ip210 in

ip access-group ip210out out

!

interface Vlan230

description Infrastructure Network

ip address 10.230.254.254 255.255.0.0

ip access-group ip230 in

ip access-group ip230out out

!

interface Vlan231

description Services Network

ip address 10.231.254.254 255.255.0.0

ip access-group ip231 in

ip access-group ip231out out

!

interface Vlan240

description VLAN

ip address 10.240.254.254 255.255.0.0

!

interface Vlan241

ip address 10.241.254.254 255.255.0.0

ip access-group ip241 in

ip access-group ip241out out

!

interface Vlan242

description Test VLAN

ip address 10.242.254.254 255.255.0.0

ip access-group ip242 in

ip access-group ip242out out

!

interface Vlan300

description Isolation Network

ip address 192.168.3.151 255.255.255.0

ip access-group ip300 in

ip access-group ip300out out

!

interface Vlan301

description Customer Network

ip address 192.168.4.254 255.255.255.0

ip access-group ip301 in

ip access-group ip301out out

!

interface Vlan500

description Contractor Network

ip address 172.16.1.254 255.255.255.0

ip access-group ip500 in

ip access-group ip500out out

!

interface Vlan501

description Customer

ip address 172.16.2.254 255.255.255.0

ip access-group ip501 in

ip access-group ip501out out

!

interface Vlan503

description vlan

ip address 172.16.3.254 255.255.255.0

ip access-group ip503 in

!

interface Vlan504

description vlan

ip address 172.16.4.254 255.255.255.0

ip access-group ip504 in

ip access-group ip504out out

!

interface Vlan505

description vlan

ip address 172.16.5.254 255.255.255.0

ip access-group ip505 in

ip access-group ip505out out

!

interface Vlan506

description vlan

ip address 172.16.6.254 255.255.255.0

ip access-group ip506 in

ip access-group ip506out out

!

interface Vlan777

description TestVLAN

ip address 172.16.177.254 255.255.255.0

ip access-group ip777 in

ip access-group ip777out out

!

ip http server

******THERE ARE ABOUT 950 LINES OF ACLs ENTRIES LIKE THE ONE BELOW FOR DIFFERENT VLANS******

ip access-list extended ip501

permit tcp any eq 3389 any

permit tcp any host 10.250.80.13

permit udp any host 10.250.80.13

permit tcp any host 10.250.11.1

permit tcp any host 10.250.11.6

permit udp any host 10.250.11.1

permit udp any host 10.250.11.6

permit tcp any host 10.250.80.21

permit udp any host 10.250.80.21

permit tcp any host 10.250.100.60

permit udp any host 10.250.100.60

permit icmp any host 10.250.100.60

permit udp any host 10.250.72.2

permit tcp any host 10.250.72.2

permit tcp any host 10.250.72.3

permit udp any host 10.250.72.3

permit icmp any host 10.250.11.1

permit icmp any host 10.250.11.6

permit tcp host 172.16.2.252 eq www 192.168.2.0 0.0.0.255

permit icmp host 172.16.2.252 192.168.2.0 0.0.0.255

permit icmp host 172.16.2.251 192.168.2.0 0.0.0.255

permit tcp host 172.16.2.251 192.168.2.0 0.0.0.255

permit tcp any host 10.250.80.25

permit udp any host 10.250.80.25

permit tcp any host 10.250.80.32

permit tcp 172.16.2.0 0.0.0.255 199.7.0.0 0.0.255.255 eq www

permit icmp any 10.210.0.0 0.0.255.255

permit tcp any 10.210.0.0 0.0.255.255

permit udp any 10.210.0.0 0.0.255.255

permit tcp any host 192.168.3.40

ip access-list extended ip501out

deny   tcp any eq 3389 any

permit tcp any any

permit icmp any any

permit udp any host 10.250.100.60

permit udp any host 10.250.11.1

permit udp any host 10.250.11.6

permit udp any any

ip sla enable reaction-alerts

logging esm config

logging 10.1.1.32

snmp-server community ERT99

snmp-server community FTR88 RW

!

!

line con 0

logging synchronous

line vty 0 4

access-class restrict-vty in

exec-timeout 30 0

logging synchronous

login local

transport input ssh

line vty 5 15

access-class restrict-vty in

exec-timeout 30 0

logging synchronous

login local

transport input ssh

!

ntp server 10.1.1.25

end

Re: High CPU usage - 3750 X stack

Well I guess we know why you have high CPU now. 1,000 aces looks like a lot.

I would start by making sure each acl is optimized. Highest used generic entries first (e.g., permit tcp any 172.10.0.0/16) followed by the more specific entries for specific hosts. Acls are processed top down, first match, so having the most frequently used ace first lowers processing requirements.

I don't understand the requirements, but I have to believe some of those aces are unnecessary. Why put in 3 allows (tcp, udp, icmp) for all inbound to a specific host? Would private Vlans be a better fit?

Sent from Cisco Technical Support iPad App

Cisco Employee

Re: High CPU usage - 3750 X stack

Hello,

The "Hulc LED" process does following tasks:

- Check Link status on every port
- If the switch supports POE, it checks to see if there is a Power Device (PD)
detected
- Check the status of the transceiver
- Update Fan status
- Set Main LED and ports LEDs
- Update both Power Supplies and RPS
- Check on system temperature status

The Catalyst 3750-X switches have a CPU utilization level that is higher than the previous models of the Catalyst 3750 switches. This is normal behavior. One can expect that the overall CPU utilization will be between 25 – 35% with Hulc Led process taking about 15%. Plugging PoE devices into all ports on a 3750-X switch will cause the CPU utilization to increase. Since almost all of the packet forwarding is done in hardware and not by the CPU a CPU utilization of 35 – 45% should not be cause for concern.

In your case that is Higher - but Hulc Led is not the problem as I said above. Seeing this process taking about 15% in production is normal. You need to concentrate to other small processes which together add to performance.

Jeff shared a good idea - your QoS and ACL table are almost full so it is worth to optimize those.

Also the level of interrupts is about 10 percent:

CPU utilization for five seconds: 61%/5%  <<<<<<<<<<< 5 % here are the interrupts due to traffic sent to switch itself

213    63106121   154928892        407  4.15%  3.89%  3.91%   0 IP Input <<<<<< IP Input is also due to interrupts

So 10% of CPU is taken by traffic coming to the switch. You can use the tools below to understnad what packets are sent to CPU and locate the sources doing it and stop eventually:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html#wp1000983

Let us know if it helps.

*Please also rate replies if were helpful.

Nik

Re: High CPU usage - 3750 X stack

New Member

High CPU usage - 3750 X stack

Thanks to all for your replies.

Jeff

I was aware of the many ACLs however we used to have the same ACLs in a previous 3750G stack about 2 weeks ago and we never had this issue. I agree I need to optimize them and do somehing because it is reaching its max before the CPU starts processing them but I am not certain this is what is causing the issue.

Nikolay,

I am trying to understand "interrupts" with the analysis of the outputs I posted. Here is another output deom the link you provided. Please post your thoughts if you can.

This switch also serves as a gateway(L3 role)  for many systems. Would it make sense to offload that responsability from this switch and let an actual router do it?

Thanks

Johnny

show controllers cpu-interface

ASIC    Rxbiterr   Rxunder    Fwdctfix   Txbuflos   Rxbufloc   Rxbufdrain

-------------------------------------------------------------------------

ASIC0     0          0          0          0          0          0

ASIC1     0          0          0          0          0          0

ASIC2     0          0          0          0          0          0

HOL Fix Counts

--------------

No Fixes:          0 Added:          0 In Use:          0 Both:          0

CPU Heartbeat Statistics

Tx Success Tx Fail    1st Thr    2nd Thr    Unthr      RetryCtMax

---------- ---------- ---------- ---------- ---------- ----------

  37139562          0          0          0          0          1

Rx Delay

         0          1          2          3          4

---------- ---------- ---------- ---------- ----------

  37139562          0          0          0          0

AddlDelay AdvanceCnt

---------- ----------

         0          0

Rx Retries by RetryCount

         0          1          2          3          4          5          6

---------- ---------- ---------- ---------- ---------- ---------- ----------

  37139562          0          0          0          0          0          0

         7          8          9

---------- ---------- ----------

         0          0          0

AddlRetry

----------

         0

cpu-queue-frames  retrieved  dropped    invalid    hol-block  stray

----------------- ---------- ---------- ---------- ---------- ----------

rpc               104077409  0          0          0          0

stp               19189469   0          0          0          0

ipc               11093838   0          0          0          0

routing protocol  141021559  0          0          0          0

L2 protocol       230347     0          0          0          0

remote console    17         0          0          0          0

sw forwarding     257436702  0          0          0          0

host              21146276   0          0          0          0

broadcast         332154608  0          0          0          0

cbt-to-spt        0          0          0          0          0

igmp snooping     2796987    0          0          0          0

icmp              90752156   0          0          0          0

logging           0          0          0          0          0

rpf-fail          0          0          0          0          0

dstats            0          0          0          0          0

cpu heartbeat     37139562   0          0          0          0

cpu-queue         static inuse static added

----------------- ------------ ------------

rpc               0            0

stp               0            0

ipc               0            0

routing protocol  0            0

L2 protocol       0            0

remote console    0            0

sw forwarding     0            0

host              0            0

broadcast         0            0

cbt-to-spt        0            0

igmp snooping     0            0

icmp              0            0

logging           0            0

rpf-fail          0            0

dstats            0            0

cpu heartbeat     0            0

Supervisor ASIC receive-queue parameters

----------------------------------------

queue 0 maxrecevsize 7E0 pakhead 5505A88 paktail 54655A8

queue 1 maxrecevsize 7E0 pakhead 5689164 paktail 5687F54

queue 2 maxrecevsize 7E0 pakhead 5547AA4 paktail 554719C

queue 3 maxrecevsize 7E0 pakhead 5DC233C paktail 5DBA4CC

queue 4 maxrecevsize 7E0 pakhead 56A7198 paktail 56A7AA0

queue 5 maxrecevsize 7E0 pakhead 5D61304 paktail 5D72F80

queue 6 maxrecevsize 7E0 pakhead 5D856D4 paktail 5D989E4

queue 7 maxrecevsize 7E0 pakhead 5BDE29C paktail 5BDC784

queue 8 maxrecevsize 7E0 pakhead 5CC00A8 paktail 5CB3574

queue 9 maxrecevsize 7E0 pakhead 59DD86C paktail 59DD86C

queue A maxrecevsize 7E0 pakhead 59BF43C paktail 59C13D8

queue B maxrecevsize 7E0 pakhead 5DD18A0 paktail 5DCE6F4

queue C maxrecevsize 7E0 pakhead 59E9CBC paktail 5A049B8

queue D maxrecevsize 7E0 pakhead 59D8EA0 paktail 59DD25C

queue E maxrecevsize 0 pakhead 0 paktail 0

queue F maxrecevsize 7E0 pakhead 59A7080 paktail 59A6BFC

Supervisor ASIC exception status

--------------------------------

Receive overrun    00000000   Transmit overrun 00000000

FrameSignatureErr  00000000   MicInitialize    00000002

BadFrameErr        00000000   LenExceededErr   00000000

BadJumboSegments   00000000

Supervisor ASIC Mic Registers

------------------------------

MicDirectPollInfo               80000200

MicIndicationsReceived          00000000

MicInterruptsReceived           00000009

MicPcsInfo                      0000001F

MicPlbMasterConfiguration       00000000

MicRxFifosAvailable             00000000

MicRxFifosReady                 0000BFFF

MicTimeOutPeriod:       FrameTOPeriod: 00000EA6 DirectTOPeriod: 00004000

MicTransmFramesCopied           00000003

MicTxFifosAvailable             0000000E

MicConfiguration:       Conf flag: 00000110     Interrupt Flag: 00000008

MicReceiveFifoAssignmen Queue 0 - 7: 33333333   Queue 8 - 15:33333333

MicReceiveFramesReady:  FrameAvailable: 00000181        frameAvaiMask: 00000000

MicException:

        Exception_flag  00000000

        Message-1       00000000

        Message-2       00000000

        Message-3       00000000

MicIntRxFifo:

        ReadPtr         000005C0        WritePtr        000005C0

        WHeadPtr        000005C0        TxFifoDepth     C0000800

MicIntTxFifo:

        ReadPtr         00000728        WritePtr        00000728

        WHeadPtr        00000728        TxFifoDepth     C0000800

MicDecodeInfo:

Fifo0:  address:        03FF4000 asic_num:      00000100

Fifo1:  address:        03FF4400 asic_num:      00000101

MicTransmitFifoInfo:

Fifo0:   StartPtrs:     0E2CE800        ReadPtr:        0E2CEBE8

        WritePtrs:      0E2CEBE8        Fifo_Flag:      8A800800

        Weights:        001E001E

Fifo1:   StartPtrs:     0E02D000        ReadPtr:        0E02D138

        WritePtrs:      0E02D138        Fifo_Flag:      89800400

        Weights:        000A000A

MicReceiveFifoInfo:

Fifo0:  StartPtr:       0E4AF000        ReadPtr:        0E4AF2A8

        WritePtrs:      0E4AF308        Fifo_Flag:      8B000FA0

        writeHeaderPtr: 0E4AF308

Fifo1:  StartPtr:       0E78C000        ReadPtr:        0E78C2E8

        WritePtrs:      0E78C2E8        Fifo_Flag:      89800400

        writeHeaderPtr: 0E78C2E8

Fifo2:  StartPtr:       0E744800        ReadPtr:        0E744A70

        WritePtrs:      0E744A70        Fifo_Flag:      89800400

        writeHeaderPtr: 0E744A70

Fifo3:  StartPtr:       0EBD1000        ReadPtr:        0EBD13B8

        WritePtrs:      0EBD13B8        Fifo_Flag:      89800400

        writeHeaderPtr: 0EBD13B8

Fifo4:  StartPtr:       0E7D3800        ReadPtr:        0E7D3A58

        WritePtrs:      0E7D3A58        Fifo_Flag:      89800400

        writeHeaderPtr: 0E7D3A58

Fifo5:  StartPtr:       0EB40600        ReadPtr:        0EB40688

        WritePtrs:      0EB40688        Fifo_Flag:      88800200

        writeHeaderPtr: 0EB40688

Fifo6:  StartPtr:       0EB87400        ReadPtr:        0EB874F0

        WritePtrs:      0EB874F0        Fifo_Flag:      89800400

        writeHeaderPtr: 0EB874F0

Fifo7:  StartPtr:       0E880000        ReadPtr:        0E880E20

        WritePtrs:      0E881520        Fifo_Flag:      8C001900

        writeHeaderPtr: 0E881520

Fifo8:  StartPtr:       0EB1A600        ReadPtr:        0EB1A770

        WritePtrs:      0EB1A780        Fifo_Flag:      880001F0

        writeHeaderPtr: 0EB1A780

Fifo9:  StartPtr:       0E2E0CD8        ReadPtr:        0E2E0CD8

        WritePtrs:      0E2E0CD8        Fifo_Flag:      82800008

        writeHeaderPtr: 0E2E0CD8

Fifo10: StartPtr:       0E81D000        ReadPtr:        0E81D1D8

        WritePtrs:      0E81D1D8        Fifo_Flag:      88800200

        writeHeaderPtr: 0E81D1D8

Fifo11: StartPtr:       0E4AEF00        ReadPtr:        0E4AEF60

        WritePtrs:      0E4AEF60        Fifo_Flag:      86800080

        writeHeaderPtr: 0E4AEF60

Fifo12: StartPtr:       0E84A000        ReadPtr:        0E84A300

        WritePtrs:      0E84A000        Fifo_Flag:      89000100

        writeHeaderPtr: 0E84A000

Fifo13: StartPtr:       0E4AEE00        ReadPtr:        0E4AEE00

        WritePtrs:      0E4AEE00        Fifo_Flag:      86800080

        writeHeaderPtr: 0E4AEE00

Fifo14: StartPtr:       00000000        ReadPtr:        00000000

        WritePtrs:      00000000        Fifo_Flag:      00800000

        writeHeaderPtr: 00000000

Fifo15: StartPtr:       0E02CEC0        ReadPtr:        0E02CED0

        WritePtrs:      0E02CED0        Fifo_Flag:      84800020

        writeHeaderPtr: 0E02CED0

===========================================================

Complete Board Id:0x00B2

===========================================================

Super Bronze

Re: High CPU usage - 3750 X stack

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

I would start by making sure each acl is optimized.  Highest used generic entries first (e.g., permit tcp any 172.10.0.0/16) followed by the more specific entries for specific hosts.  Acls are processed top down, first match, so having the most frequently used ace first lowers processing requirements.

Platforms like the 3750-X should be processing ACLs in hardware, so the "software" processed ACL optimizations don't normally apply.  However, 3750 series have limited TCAM resources and it's possible to overflow the TCAM allocation for them when you have lots of ACLs.  When you do, CPU will rise and performance will slow.  TCAM resources can be somewhat "tuned" by your selection of the SDM template.

Hall of Fame Super Gold

High CPU usage - 3750 X stack

Unless you really, really, really have a need to run 12.2(58)SE2, I'd recommend you go down to 12.2(55)SE5. 

Starting next week, I'm going to be testing the 15.0(1)SE3 IOS.

Bronze

Re: High CPU usage - 3750 X stack

Tried 12.2(55)SE7, 12.2(58)SE2, and 15.0(2)SE2 with universal image, and all gave 30-40% CPU, even when all nonconnected ports are disabled.  This is just on a single WS-C3750X-48T-S V02, no routing, no ACLs, Rapid-PVST with 2 VLANs.

Interesting thing is I have a pair of WS-C3750X-48T-S, stacked, running 12.2(55)SE3.  They are at 15% CPU.  I've gone through the configs and can't spot any difference.  This is a real mystery.

Hall of Fame Super Gold

Re: High CPU usage - 3750 X stack

Thanks Johnny.

Just found a small bug in 3750E/X running 15.0(2)SE2.  CPU didn't spike, but boy did it went nuts!  I had tracebacks galore and the response to the switch was slow.  In the end, it all boils down to the usual culprit:  SNMP traps.

I removed ALL SNMP traps and the tracebacks stops and I was able to gain control of the switch.

Using 12.2(55)SE7 in all my 3750/G/E/X fleet.  No issue and CPU is not that high either.  Wierd. 

Bronze

Re: High CPU usage - 3750 X stack

Yeah I've been running 12.2(55)SE7 on the 3560s with great success.  Even with PoE and ports bouncing, cpu never gets above 10% and SSH/SNMP never hang.  I avoid SNMP traps at all costs and rely only on syslogs. 

There is definitely something "unique" to the 3750X and 2960S platforms pertaining to the Hulc LED Process. Just wish there was more work being done on Cisco's side to investigate.  

New Member

High CPU usage - 3750 X stack

Glad to see this post in the thread.  I just flew over a thousand miles to help a consistent long time customer whom is "all of a sudden" having issues with performance.   3750X devices with universal images running at 40% CPU utilization and higher with NOTHING plugged in or configured.  I erased config to kill it and have vlan 1 to tftp my preferred 12.2(55)SE5 "safe harbor" image and I cannot even get it on there, well I can but cannot surpass 4 packets per second !!  I've never seen anything like this in 21 years of being a Cisco Engineer.  

  No clue what triggered it but no seen enough today to push my "safe" release version to all devices being left behind after I forklift network and upgrade with Nexus 6009's and more 3750X's just orderd.

  Please post if you find results, I'm taking last device (the bad one mentioned above) down to 12.2(55)SE5 and buring all with universal images tonight.

Dave

Hall of Fame Super Gold

Re: High CPU usage - 3750 X stack

Sorry, Dave.  I just saw this.

In regards to 3560- and 3750-series, I have made the following conclusion:

1.  If you are running Layer 2 and no 802.1X, you can choose either 12.2(55)SE8 or 15.0(2)SE4; and

2.  If you are running Layer 3 and/or 802.1X, then use 12.2(55)SE8

In regards to 2960-series switches (2960/G/S), I am currently running 15.0(2)SE4 and I don't see any problems.  My 2960-series switches have PoE and 802.1X but very limited SNMP traps enabled.

If anyone reading this is experiencing high or abnormal CPU on 2960-series (2960/G/S) and running 15.0(2)SE4, then check your SNMP enabled traps.  DO NOT enable ALL SNMP traps.

New Member

High CPU usage - 3750 X stack

  I replaced the universal image with my 12.2(55)SE8 non universal image or ip base in most instances with this customer and a few others.   In both customer networks the high cpu issue immediately went away by applying the new image with no loss in features whatsoever.    As stated above its a known issue and documented and its likely the links he posted will take you there to review them I'm thinking.  I had done that but already had my image replaced and watched cpu fall and never go above 11% since both alone and in stacks with up to 7 switches thus far.   Haven't had an issue since I replaced the images the very night I posted that about 2 months ago, I suggest this image and version which is very version used for 3850 IOS-XE images compiled for FCS as 15.x

New Member

Re:High CPU usage - 3750 X stack

Hi ,

I dont know if it's relevant in this context but just want to share the info that we had a 2950 in the infrastructure which we removed due to High CPU Utilization . Later only we came to know that around 30 %to 50% at idle state was acceptable CPU Utilization for 2950 Switches.

So I guess its same with your 3750 X stack showing High CPU Utilization due to the LED process .

Anup Sasikumar

Sent from Cisco Technical Support Android App

Regards, Anup Don't forget to rate if you found this helpful !
New Member

High CPU usage - 3750 X stack

Thanks to all for your replies.

Jeff

I was aware of the many ACLs however we used to have the same ACLs in a previous 3750G stack about 2 weeks ago and we never had this issue. I agree I need to optimize them and do somehing because it is reaching its max before the CPU starts processing them but I am not certain this is what is causing the issue.

Nikolay,

I am trying to understand "interrupts" with the analysis of the outputs I posted. Here is another output deom the link you provided. Please post your thoughts if you can.

This switch also serves as a gateway(L3 role)  for many systems. Would it make sense to offload that responsability from this switch and let an actual router do it?

Thanks

Johnny

show controllers cpu-interface

ASIC    Rxbiterr   Rxunder    Fwdctfix   Txbuflos   Rxbufloc   Rxbufdrain

-------------------------------------------------------------------------

ASIC0     0          0          0          0          0          0

ASIC1     0          0          0          0          0          0

ASIC2     0          0          0          0          0          0

HOL Fix Counts

--------------

No Fixes:          0 Added:          0 In Use:          0 Both:          0

CPU Heartbeat Statistics

Tx Success Tx Fail    1st Thr    2nd Thr    Unthr      RetryCtMax

---------- ---------- ---------- ---------- ---------- ----------

  37139562          0          0          0          0          1

Rx Delay

         0          1          2          3          4

---------- ---------- ---------- ---------- ----------

  37139562          0          0          0          0

AddlDelay AdvanceCnt

---------- ----------

         0          0

Rx Retries by RetryCount

         0          1          2          3          4          5          6

---------- ---------- ---------- ---------- ---------- ---------- ----------

  37139562          0          0          0          0          0          0

         7          8          9

---------- ---------- ----------

         0          0          0

AddlRetry

----------

         0

cpu-queue-frames  retrieved  dropped    invalid    hol-block  stray

----------------- ---------- ---------- ---------- ---------- ----------

rpc               104077409  0          0          0          0

stp               19189469   0          0          0          0

ipc               11093838   0          0          0          0

routing protocol  141021559  0          0          0          0

L2 protocol       230347     0          0          0          0

remote console    17         0          0          0          0

sw forwarding     257436702  0          0          0          0

host              21146276   0          0          0          0

broadcast         332154608  0          0          0          0

cbt-to-spt        0          0          0          0          0

igmp snooping     2796987    0          0          0          0

icmp              90752156   0          0          0          0

logging           0          0          0          0          0

rpf-fail          0          0          0          0          0

dstats            0          0          0          0          0

cpu heartbeat     37139562   0          0          0          0

cpu-queue         static inuse static added

----------------- ------------ ------------

rpc               0            0

stp               0            0

ipc               0            0

routing protocol  0            0

L2 protocol       0            0

remote console    0            0

sw forwarding     0            0

host              0            0

broadcast         0            0

cbt-to-spt        0            0

igmp snooping     0            0

icmp              0            0

logging           0            0

rpf-fail          0            0

dstats            0            0

cpu heartbeat     0            0

Supervisor ASIC receive-queue parameters

----------------------------------------

queue 0 maxrecevsize 7E0 pakhead 5505A88 paktail 54655A8

queue 1 maxrecevsize 7E0 pakhead 5689164 paktail 5687F54

queue 2 maxrecevsize 7E0 pakhead 5547AA4 paktail 554719C

queue 3 maxrecevsize 7E0 pakhead 5DC233C paktail 5DBA4CC

queue 4 maxrecevsize 7E0 pakhead 56A7198 paktail 56A7AA0

queue 5 maxrecevsize 7E0 pakhead 5D61304 paktail 5D72F80

queue 6 maxrecevsize 7E0 pakhead 5D856D4 paktail 5D989E4

queue 7 maxrecevsize 7E0 pakhead 5BDE29C paktail 5BDC784

queue 8 maxrecevsize 7E0 pakhead 5CC00A8 paktail 5CB3574

queue 9 maxrecevsize 7E0 pakhead 59DD86C paktail 59DD86C

queue A maxrecevsize 7E0 pakhead 59BF43C paktail 59C13D8

queue B maxrecevsize 7E0 pakhead 5DD18A0 paktail 5DCE6F4

queue C maxrecevsize 7E0 pakhead 59E9CBC paktail 5A049B8

queue D maxrecevsize 7E0 pakhead 59D8EA0 paktail 59DD25C

queue E maxrecevsize 0 pakhead 0 paktail 0

queue F maxrecevsize 7E0 pakhead 59A7080 paktail 59A6BFC

Supervisor ASIC exception status

--------------------------------

Receive overrun    00000000   Transmit overrun 00000000

FrameSignatureErr  00000000   MicInitialize    00000002

BadFrameErr        00000000   LenExceededErr   00000000

BadJumboSegments   00000000

Supervisor ASIC Mic Registers

------------------------------

MicDirectPollInfo               80000200

MicIndicationsReceived          00000000

MicInterruptsReceived           00000009

MicPcsInfo                      0000001F

MicPlbMasterConfiguration       00000000

MicRxFifosAvailable             00000000

MicRxFifosReady                 0000BFFF

MicTimeOutPeriod:       FrameTOPeriod: 00000EA6 DirectTOPeriod: 00004000

MicTransmFramesCopied           00000003

MicTxFifosAvailable             0000000E

MicConfiguration:       Conf flag: 00000110     Interrupt Flag: 00000008

MicReceiveFifoAssignmen Queue 0 - 7: 33333333   Queue 8 - 15:33333333

MicReceiveFramesReady:  FrameAvailable: 00000181        frameAvaiMask: 00000000

MicException:

        Exception_flag  00000000

        Message-1       00000000

        Message-2       00000000

        Message-3       00000000

MicIntRxFifo:

        ReadPtr         000005C0        WritePtr        000005C0

        WHeadPtr        000005C0        TxFifoDepth     C0000800

MicIntTxFifo:

        ReadPtr         00000728        WritePtr        00000728

        WHeadPtr        00000728        TxFifoDepth     C0000800

MicDecodeInfo:

Fifo0:  address:        03FF4000 asic_num:      00000100

Fifo1:  address:        03FF4400 asic_num:      00000101

MicTransmitFifoInfo:

Fifo0:   StartPtrs:     0E2CE800        ReadPtr:        0E2CEBE8

        WritePtrs:      0E2CEBE8        Fifo_Flag:      8A800800

        Weights:        001E001E

Fifo1:   StartPtrs:     0E02D000        ReadPtr:        0E02D138

        WritePtrs:      0E02D138        Fifo_Flag:      89800400

        Weights:        000A000A

MicReceiveFifoInfo:

Fifo0:  StartPtr:       0E4AF000        ReadPtr:        0E4AF2A8

        WritePtrs:      0E4AF308        Fifo_Flag:      8B000FA0

        writeHeaderPtr: 0E4AF308

Fifo1:  StartPtr:       0E78C000        ReadPtr:        0E78C2E8

        WritePtrs:      0E78C2E8        Fifo_Flag:      89800400

        writeHeaderPtr: 0E78C2E8

Fifo2:  StartPtr:       0E744800        ReadPtr:        0E744A70

        WritePtrs:      0E744A70        Fifo_Flag:      89800400

        writeHeaderPtr: 0E744A70

Fifo3:  StartPtr:       0EBD1000        ReadPtr:        0EBD13B8

        WritePtrs:      0EBD13B8        Fifo_Flag:      89800400

        writeHeaderPtr: 0EBD13B8

Fifo4:  StartPtr:       0E7D3800        ReadPtr:        0E7D3A58

        WritePtrs:      0E7D3A58        Fifo_Flag:      89800400

        writeHeaderPtr: 0E7D3A58

Fifo5:  StartPtr:       0EB40600        ReadPtr:        0EB40688

        WritePtrs:      0EB40688        Fifo_Flag:      88800200

        writeHeaderPtr: 0EB40688

Fifo6:  StartPtr:       0EB87400        ReadPtr:        0EB874F0

        WritePtrs:      0EB874F0        Fifo_Flag:      89800400

        writeHeaderPtr: 0EB874F0

Fifo7:  StartPtr:       0E880000        ReadPtr:        0E880E20

        WritePtrs:      0E881520        Fifo_Flag:      8C001900

        writeHeaderPtr: 0E881520

Fifo8:  StartPtr:       0EB1A600        ReadPtr:        0EB1A770

        WritePtrs:      0EB1A780        Fifo_Flag:      880001F0

        writeHeaderPtr: 0EB1A780

Fifo9:  StartPtr:       0E2E0CD8        ReadPtr:        0E2E0CD8

        WritePtrs:      0E2E0CD8        Fifo_Flag:      82800008

        writeHeaderPtr: 0E2E0CD8

Fifo10: StartPtr:       0E81D000        ReadPtr:        0E81D1D8

        WritePtrs:      0E81D1D8        Fifo_Flag:      88800200

        writeHeaderPtr: 0E81D1D8

Fifo11: StartPtr:       0E4AEF00        ReadPtr:        0E4AEF60

        WritePtrs:      0E4AEF60        Fifo_Flag:      86800080

        writeHeaderPtr: 0E4AEF60

Fifo12: StartPtr:       0E84A000        ReadPtr:        0E84A300

        WritePtrs:      0E84A000        Fifo_Flag:      89000100

        writeHeaderPtr: 0E84A000

Fifo13: StartPtr:       0E4AEE00        ReadPtr:        0E4AEE00

        WritePtrs:      0E4AEE00        Fifo_Flag:      86800080

        writeHeaderPtr: 0E4AEE00

Fifo14: StartPtr:       00000000        ReadPtr:        00000000

        WritePtrs:      00000000        Fifo_Flag:      00800000

        writeHeaderPtr: 00000000

Fifo15: StartPtr:       0E02CEC0        ReadPtr:        0E02CED0

        WritePtrs:      0E02CED0        Fifo_Flag:      84800020

        writeHeaderPtr: 0E02CED0

===========================================================

Complete Board Id:0x00B2

===========================================================

Bronze

Re: High CPU usage - 3750 X stack

I think your ACL issue is a red herring.  You're not the only one seeing the 30% CPU and higher.

Now the part that really drives me nuts is I have a stack that's only at 14%.  Check this out:

CPU utilization for five seconds: 21%/0%; one minute: 14%; five minutes: 14%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

162         302        77       3922  4.15%  0.33%  0.06%   1 SSH Process     

  74   243161037  39366660       6176  3.03%  2.85%  2.84%   0 RedEarth Tx Mana

   4    96582240   4319109      22361  2.07%  1.40%  1.16%   0 Check heaps     

156   118927698 191802387        620  1.91%  1.35%  1.35%   0 Hulc LED Process

116   143864159  16784938       8571  1.75%  1.68%  1.66%   0 hpm counter proc

186     9436800   8137819       1159  0.79%  0.13%  0.11%   0 CDP Protocol    

Its configuration and software version is the same as a stack right next to it that's at 29% CPU.  Go figure.  I'm rolling out more 3750Xs at a new data center next week and it will be interesting to see how they behave.

New Member

High CPU usage - 3750 X stack

Hello !

The WS-C3750X-48T-E "12.2(58)SE2 C3750E-UNIVERSALK9-M" Today I met the same question, high cpu usage, but I just ran EIGRP,VLAN, the following information is output:

3750#show proc c

CPU utilization for five seconds: 27%/1%; one minute: 28%; five minutes: 30%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

   1           0          20          0  0.00%  0.00%  0.00%   0 Chunk Manager   

   2           0         669          0  0.00%  0.00%  0.00%   0 Load Meter      

   3           0          28          0  0.00%  0.00%  0.00%   0 DHCPD Timer     

   4        9437         693      13617  0.00%  0.21%  0.23%   0 Check heaps     

   5           0           1          0  0.00%  0.00%  0.00%   0 Pool Manager  

  85       84718       15847       5345  2.40%  2.37%  2.36%   0 RedEarth Tx Mana

107       48443       94138        514  3.20%  2.19%  2.43%   0 HLFM address lea

169      126798       83456       1519 13.27% 10.35%  7.59%   0 Hulc LED Process  "This is highset"

New Member

High CPU usage - 3750 X stack

Hi, we had the same issue on these switches where ther HULC process went through the roof,

This was down to there being unused interfaces on the switches where left in a not connect state.

Shutdown all unused interfaces on the switch and if its like what we experienced it should solve your problem.

Ross

New Member

High CPU usage - 3750 X stack

Thank you very much, I will try!

Bronze

High CPU usage - 3750 X stack

Ross - what version of IOS are you running?  I've already tried shutting down all unused ports and rebooting, but to no avail. 

Re:High CPU usage - 3750 X stack

R u using route map if u r then u will also b using acls in that just remove deny statement from route map ace

Sent from Cisco Technical Support Android App

Jawad
New Member

High CPU usage - 3750 X stack

Guys,

What is the bottom line with regards to the software you will put in a 3750X switch? I got 5 cisco 3750X switches and I am about to put them into production (no stack config, just stand-alone) and I'd like to know if I am better safe with the

12.2(55)SE5 version Dave Genton suggested?

My switches came with IOS version (C3750E-UNIVERSALK9-M), Version 15.0(2)SE. I quickly looked this version up and it appears to have few big bugs already plus, I also see the CPU utilization at 22-26% with nothing connected in the ports. I also have couple of the switches with ALL the ports manually SHUT and I still the this CPU %.

Please advise.

Thank you in advance

Hall of Fame Super Gold

High CPU usage - 3750 X stack

If all of your 3750X can run 12.2(55)SE8, then use this version as it's considered to be the most stable.

If you need features only found in 15.0 then go with 15.0(2)SE4.  Do not even attempt to use 15.2(1)E.

New Member

High CPU usage - 3750 X stack

I just tried following this exact advice (I loaded IOS version c3750e-ipbasek9-mz.150-2.SE4.bin) but the switch went into an infinite loop.

Did I use the wrong image file (instead of c3750e-universalk9-mz.150-2.SE.bin) ??

All my switches are C3750X platform with license level IP Base.

Hall of Fame Super Gold

High CPU usage - 3750 X stack

Post the entire sequence.  If the switch crashed, I want to see the error messages.

New Member

High CPU usage - 3750 X stack

I wasnt recording the output and it is all gone by now.... I quickly performed a recovery procedure and went back to the version that the switch came with --- IOS version (C3750E-UNIVERSALK9-M), Version 15.0(2)SE

For some reason I am suspecting is the the IP Base version I tried. But before I try the Universal version of that image (c3750e-universalk9-mz.150-2.SE.bin), I wanted to ask you guys for help.

I did read the pre-requisites of the c3750e-ipbasek9-mz.150-2.SE4.bin and it stated that I needed 256/64 in the switch which I do have but for some reason I think I might not be seeing something else that perhaps states that I can't use the ip-base flavor of the image....

What is your take on this?

Hall of Fame Super Gold

High CPU usage - 3750 X stack

I'm not totally sure what's going on.

Try to install the IOS on a switch without any config and any network module. 

40786
Views
20
Helpful
37
Replies
CreatePlease login to create content