3750 Stack CPU 99% - IP routing 50% of cpu

Unanswered Question
Mar 10th, 2008

hi,

ive a stack of 8 3750 Switches, since 2 weeks i have trouble with the CPU. he goes up to 99%.

when i restart the stack after 2-3 days he goes up again.

ip routing is about 50% cpu

--------------------------------------

CPU utilization for five seconds: 99%/28%; one minute: 88%; five minutes: 69%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

174 5042657 4383110 1150 44.81% 41.48% 32.67% 0 IP Input

117 336631 6621662 50 5.74% 3.91% 2.20% 0 Hulc LED Process

176 1529913 11173357 136 5.42% 4.16% 2.89% 0 Spanning Tree

65 290732 8241263 35 3.18% 2.79% 1.84% 0 HLFM address lea

51 229343 14774690 15 1.75% 0.99% 0.60% 0 Fifo Error Detec

156 737906 6332176 116 1.43% 1.53% 1.04% 0 LLDP Protocol

85 133749 441829 302 0.47% 0.40% 0.26% 0 hpm counter proc

86 829447 4169684 198 0.47% 0.38% 0.33% 0 HRPC pm-counters

126 1999665 887387 2253 0.47% 0.63% 0.68% 0 HRPC qos request

30 146123 173878 840 0.31% 0.09% 0.06% 0 Net Background

37 501722 61340 8179 0.31% 0.26% 0.20% 0 Compute load avg

9 589049 746053 789 0.15% 0.20% 0.22% 0 ARP Input

55 375259 1278169 293 0.15% 0.25% 0.18% 0 hrpc <- response

81 59753 2606018 22 0.15% 0.22% 0.15% 0 hpm main process

98 519 354 1466 0.15% 0.29% 0.11% 1 SSH Process

119 62384 774492 80 0.15% 0.02% 0.00% 0 HRPC hl3u reques

138 49777 277019 179 0.15% 0.31% 0.19% 0 PI MATM Aging Pr

211 14562 2897412 5 0.15% 0.03% 0.00% 0 MDFS RP process

239 42361 431344 98 0.15% 0.17% 0.11% 0 CEF: IPv4 proces

266 8141 279888 29 0.15% 0.03% 0.00% 0 IGMP Input

1 4525 83670 54 0.00% 0.00% 0.00% 0 Chunk Manager

--------------------------------------

show ip interfaces

Vlan<ID> is up, line protocol is up

Internet address is <IP-ADDRESS>/24

Broadcast address is 255.255.255.255

Address determined by non-volatile memory

MTU is 1500 bytes

Helper addresses are <IP-ADDRESS>

<IP-ADDRESS>

Directed broadcast forwarding is disabled

Outgoing access list is not set

Inbound access list is not set

Proxy ARP is enabled

Local Proxy ARP is disabled

Security level is default

Split horizon is enabled

ICMP redirects are always sent

ICMP unreachables are always sent

ICMP mask replies are never sent

IP fast switching is enabled

IP CEF switching is enabled

IP CEF switching turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is enabled

IP route-cache flags are Fast, CEF

Router Discovery is disabled

IP output packet accounting is disabled

IP access violation accounting is disabled

TCP/IP header compression is disabled

RTP/IP header compression is disabled

Probe proxy name replies are disabled

Policy routing is disabled

Network address translation is enabled, interface in domain outside

WCCP Redirect outbound is disabled

WCCP Redirect inbound is disabled

WCCP Redirect exclude is disabled

BGP Policy Mapping is disabled

Input features: NAT Outside

Output features: Post-routing NAT Outside

Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(37)SE1, RELEASE SOFTWARE (fc1)

strange is, if i count all processes cpu %, this arent 100%. where is the rest of the cpu?

what can it be? i have no ideas...

thx...

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Joseph W. Doherty Mon, 03/10/2008 - 05:29

"strange is, if i count all processes cpu %, this arent 100%. where is the rest of the cpu?"

When you add up process CPU percentages, they should be (about) the sum of the difference between total CPU and interrupt CPU. In your post: "CPU utilization for five seconds: 99%/28%; one minute: 88%; five minutes: 69%", 99% - 28% = 71%.

Of concern in your process CPU, is the amount for IP Input. See if anything in this link helps: http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af3.shtml and for all software processes: http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af6.shtml

vaisharm Mon, 03/10/2008 - 05:47

Hi,

Are you seeing any errors in the logs? Capture the following outputs atleast 3-4 times a day:

-show log

-show proc mem

-show proc cpu

You might notice some pattern. Also, what was changed (IOS/traffic/config) on this switch or in the topology recently?

~Vaibhav

Actions

This Discussion