7513(RSP2)router high cpu utilization(99%)

[zesum]

We have a high cpu utilization problem on our Internet 7513 router.

ios : rsp-jsv-mz.121-20.bin

interface use distributed CEF

Help me. please... T.T

------------------ show process cpu ------------------

CPU utilization for five seconds: 99%/97%; one minute: 99%; five minutes: 99%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

1 14340 1802506 7 0.00% 0.00% 0.00% 0 Load Meter

2 169512 35626500 4 0.00% 0.02% 0.00% 0 CEF IPC Backgrou

3 16545528 1109470 14913 0.00% 0.22% 0.15% 0 Check heaps

4 0 1 0 0.00% 0.00% 0.00% 0 Chunk Manager

5 6592 5030 1310 0.00% 0.00% 0.00% 0 Pool Manager

6 0 2 0 0.00% 0.00% 0.00% 0 Timers

7 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun

8 0 1 0 0.00% 0.00% 0.00% 0 OIR Handler

9 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager

10 274696 8977305 30 0.00% 0.00% 0.00% 0 IPC Periodic Tim

11 182736 8977305 20 0.00% 0.00% 0.00% 0 IPC Deferred Por

12 12751492 127646600 99 0.08% 0.09% 0.09% 0 IPC Seat Manager

13 3090548 4663415 662 0.00% 0.01% 0.00% 0 ARP Input

14 43772 2245769 19 0.00% 0.00% 0.00% 0 HC Counter Timer

15 36 334 107 0.00% 0.00% 0.00% 0 DDR Timers

16 0 2 0 0.00% 0.00% 0.00% 0 Dialer event

17 0 1 0 0.00% 0.00% 0.00% 0 Entity MIB API

18 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect

19 0 1 0 0.00% 0.00% 0.00% 0 Microcode Loader

20 0 2 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

21 291912 2108017 138 0.00% 0.00% 0.00% 0 Net Background

22 40 1289 31 0.00% 0.00% 0.00% 0 Logger

23 277868 8977299 30 0.00% 0.00% 0.00% 0 TTY Background

24 10311952 8977305 1148 0.00% 0.00% 0.00% 0 Per-Second Jobs

25 72 580 124 0.00% 0.00% 0.00% 0 Inode Table Dest

26 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Refr

27 0 1 0 0.00% 0.00% 0.00% 0 stuckinfo_proces

28 0 1 0 0.00% 0.00% 0.00% 0 IP Crashinfo Inp

29 131556 9014473 14 0.00% 0.00% 0.00% 0 DSX3MIB ll handl

30 0 2 0 0.00% 0.00% 0.00% 0 VSI Master

31 1700560 8977306 189 0.08% 0.02% 0.00% 0 RSP Background

32 0 1 0 0.00% 0.00% 0.00% 0 Memory Scanner

33 7012 150252 46 0.00% 0.00% 0.00% 0 Slave Time

34 0 1 0 0.00% 0.00% 0.00% 0 Slave IPC OIR

35 358272 396 904727 0.00% 0.00% 0.05% 2 Virtual Exec

36 60512 2549980 23 0.00% 0.00% 0.00% 0 Chassis Daemon

37 8512 150260 56 0.00% 0.00% 0.00% 0 RSP Chassis Back

38 0 1 0 0.00% 0.00% 0.00% 0 MIP Mailbox

39 0 1 0 0.00% 0.00% 0.00% 0 vcq_proc

40 0 1 0 0.00% 0.00% 0.00% 0 CT3 Mailbox

41 0 1 0 0.00% 0.00% 0.00% 0 CE3 Mailbox

42 6486376 129502902 50 0.08% 0.06% 0.06% 0 IPC CBus process

43 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM Input

44 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM TIMER

45 27566712 37112358 742 0.08% 0.10% 0.12% 0 IP Input

46 0 1 0 0.00% 0.00% 0.00% 0 DSS Process

47 424 15007 28 0.00% 0.00% 0.00% 0 MOP Protocols

48 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana

49 576 150314 3 0.00% 0.00% 0.00% 0 LDP Background

50 0 1 0 0.00% 0.00% 0.00% 0 frr_tunnel

51 2826764 160401 17623 0.00% 0.02% 0.00% 0 IP Background

52 420040 305166 1376 0.00% 0.00% 0.00% 0 IP RIB Update

53 0 1 0 0.00% 0.00% 0.00% 0 SNMP Timers

54 0 1 0 0.00% 0.00% 0.00% 0 PPP IP Add Route

55 24156 1013504 23 0.00% 0.00% 0.00% 0 TCP Timer

56 1192 899 1325 0.00% 0.00% 0.00% 0 TCP Protocols

57 0 1 0 0.00% 0.00% 0.00% 0 Probe Input

58 0 1 0 0.00% 0.00% 0.00% 0 RARP Input

59 0 1 0 0.00% 0.00% 0.00% 0 HTTP Timer

60 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers

61 16 13 1230 0.00% 0.00% 0.00% 0 DHCPD Receive

62 959060 151078 6348 0.00% 0.00% 0.00% 0 IP Cache Ager

63 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall

64 4 2 2000 0.00% 0.00% 0.00% 0 X.25 Background

65 11700 150260 77 0.00% 0.00% 0.00% 0 TCP Intercept Ti

66 0 2 0 0.00% 0.00% 0.00% 0 SPX Input

67 84928 150312 565 0.00% 0.00% 0.00% 0 Adj Manager

68 1046232 13183072 79 0.00% 0.01% 0.00% 0 CEF process

69 286732 44670692 6 0.00% 0.01% 0.00% 0 MDFS RP process

70 4 2 2000 0.00% 0.00% 0.00% 0 TC-ATM Proc

71 0 2 0 0.00% 0.00% 0.00% 0 Tag Input

72 50772 506 100339 0.47% 0.81% 0.39% 0 Exec

73 4 1 4000 0.00% 0.00% 0.00% 0 SYSMGT Events

74 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps

75 28516 17939497 1 0.00% 0.00% 0.00% 0 cbus utilization

76 585928 4034288 145 0.00% 0.00% 0.00% 0 Net Input

77 27340 1802507 15 0.00% 0.00% 0.00% 0 Compute load avg

78 2366268 150259 15747 0.00% 0.01% 0.00% 0 Per-minute Jobs

79 21834992 7233025 3018 0.00% 0.07% 0.20% 0 IP SNMP

80 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro

81 4 3 1333 0.00% 0.00% 0.00% 0 SNMP Traps

82 0 2 0 0.00% 0.00% 0.00% 0 IP Flow Backgrou

83 2381796 19318529 123 0.00% 0.07% 0.06% 0 IP-EIGRP Hello

85 0 9 0 0.00% 0.00% 0.00% 0 TCP Listener

86 47412 9017589 5 0.00% 0.00% 0.00% 0 NTP

87 20304 475045 42 0.00% 0.00% 0.00% 0 CEF Scanner

89 344 75153 4 0.00% 0.00% 0.00% 0 DHCPD Timer

90 38716 2554184 15 0.00% 0.00% 0.00% 0 DHCPD Database

91 2557308 9695069 263 0.00% 0.00% 0.00% 0 IP-EIGRP Router

92 4984536 23393200 213 0.00% 0.05% 0.07% 0 BGP Router

93 874020 1616454 540 0.00% 0.00% 0.00% 0 BGP I/O

94 192078900 1222840 157083 0.00% 0.66% 0.84% 0 BGP Scanner

thanks for reading..

[Georg Pauwen]

Hello,

do you have a CCO logon ? I checked the bug toolkit and there are a couple of bugs in 12.1-20 which might apply to you, but since I haven´t seen your config I cannot say for sure. Check out this link for the bugs related to high CPU utilization in version 12.1-20:

http://www.cisco.com/cgi-bin/Support/Bugtool/bugnav2.pl?swver1=&fset1=&severity1=&resultsperpage1=&target1=&train1=&swver=12.1&target=20&train=%3F&mdf_label=Cisco+IOS+Releases&rneflag=1&swver2=1009&target2=1321&train2=%3F&fset=%3F&keyw=cpu&operator=a...

You might also have hit the Blaster or Nachi worm, which causes high CPU utilization as well. Check out these links:

For the Nachi worm:

http://www.cisco.com/en/US/products/sw/voicesw/ps556/products_tech_note09186a00801b143a.shtml

For the Blaster worm:

http://www.cisco.com/en/US/products/sw/voicesw/ps556/products_tech_note09186a00801aedd6.shtml

And last but not least, since there doesn´t seem to be a specific process responsible for the CPU utilization, it might be due to high interrupts. Check out this document to troubleshoot high CPU utilization on 7500 routers due to interrupts:

http://www.cisco.com/en/US/partner/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml#inapprop_swpath

Lots of documents...

Regards,

Georg

[tbaranski]

It is indeed due to interrupts, which generally means inappropriate switching path (i.e., not CEF) and/or too many packets. If CEF is enabled and being used for most packets (look at 'show interface switching' or 'show cef not-cef-switched' to confirm) then there may be worms of some sort on your network causing a lot of excess traffic.

[ruwhite]

I would chek the output of show alignment, and if the numbers there are not high and increasing, then it's definitely traffic. I can't say this is high, since I don't know the "normal" traffic load on the router, but if it is high, then I'd agree that it's probably some sort of worm on your network/etc.

:-)

Russ.W

[zesum]

I can't check the output of show alignment..T.T

7513router#sh alignment

No alignment data has been recorded.

No spurious memory references have been recorded.

7513router#

[rajesh444]

Hi,

Could you check if you have CEF enabled on the router. CEF is less CPU-intensive than fast/process switching.

If it is not enabled the output would be as follows:

Router#sh ip cef

%CEF not running

CEF can be enabled using the global config mode:

Router(config)#ip cef

CEF Overview:

http://www.cisco.com/en/US/partner/products/sw/iosswrel/ps1835/products_configuration_guide_chapter09186a00800ca7cb.html

Configuring CEF:

http://www.cisco.com/en/US/partner/products/sw/iosswrel/ps1835/products_configuration_guide_chapter09186a00800ca7cc.html

Hope this helps,

Rajesh

[ruwhite]

Actually, that just means there are no alignment errors or spurious accesses, which means it's pure switching traffic taking up the processor time. It's just a straight traffic issue, which implies either a worm of some type, or perhaps a lot of stuff being done in the CEF path, or perhaps just a lot of traffic.

What sort of features do you have configured on the interfaces on this router? How much traffic load is it handling?

Russ.W

[zesum]

7513router all cef switching..use..

sh conf

...

boot system disk0:rsp-isv-mz.121-20.bin

boot system disk0:rsp-jsv-mz.122-15.T5.bin

enable secret xxxx

...

ip cef distributed

....

7513router#sh ip int

GigabitEthernet3/0/0 is up, line protocol is up

Directed broadcast forwarding is disabled

Multicast reserved groups joined: 224.0.0.10

Proxy ARP is disabled

Security level is default

Split horizon is enabled

ICMP redirects are never sent

ICMP unreachables are never sent

ICMP mask replies are never sent

IP fast switching is enabled

IP fast switching on the same interface is disabled

IP Flow switching is enabled

IP CEF switching is enabled

IP Distributed switching is enabled

IP Flow switching turbo vector

IP Flow CEF switching turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is disabled

IP route-cache flags are Fast, Flow, Distributed, CEF

...

Serial8/1/0 is up, line protocol is up

Directed broadcast forwarding is disabled

Multicast reserved groups joined: 224.0.0.10

Proxy ARP is disabled

Security level is default

Split horizon is enabled

ICMP redirects are never sent

ICMP unreachables are never sent

ICMP mask replies are never sent

IP fast switching is enabled

IP fast switching on the same interface is enabled

IP Flow switching is enabled

IP CEF switching is enabled

IP Distributed switching is enabled

IP Flow switching turbo vector

IP Flow CEF switching turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is disabled

IP route-cache flags are Fast, Flow, Distributed, CEF

...

Serial8/1/1 is administratively down, line protocol is down

Internet protocol processing disabled

...

FastEthernet10/0/0 is up, line protocol is up

Directed broadcast forwarding is disabled

Proxy ARP is disabled

Security level is default

Split horizon is enabled

ICMP redirects are never sent

ICMP unreachables are never sent

ICMP mask replies are never sent

IP fast switching is enabled

IP fast switching on the same interface is disabled

IP Flow switching is enabled

IP CEF switching is enabled

IP Distributed switching is enabled

IP Flow switching turbo vector

IP Flow CEF switching turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is disabled

IP route-cache flags are Fast, Flow, Distributed, CEF

...

FastEthernet10/1/0 is up, line protocol is up

Directed broadcast forwarding is disabled

Proxy ARP is disabled

Security level is default

Split horizon is enabled

ICMP redirects are never sent

ICMP unreachables are never sent

ICMP mask replies are never sent

IP fast switching is enabled

IP fast switching on the same interface is disabled

IP Flow switching is enabled

IP CEF switching is enabled

IP Distributed switching is enabled

IP Flow switching turbo vector

IP Flow CEF switching turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is disabled

IP route-cache flags are Fast, Flow, Distributed, CEF

...

Serial12/0 is up, line protocol is up

Directed broadcast forwarding is disabled

Multicast reserved groups joined: 224.0.0.10

Proxy ARP is disabled

Security level is default

Split horizon is enabled

ICMP redirects are never sent

ICMP unreachables are never sent

ICMP mask replies are never sent

IP fast switching is enabled

IP fast switching on the same interface is enabled

IP Flow switching is enabled

IP CEF switching is enabled

IP Flow switching turbo vector

IP Flow CEF switching turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is disabled

IP route-cache flags are Fast, Flow, CEF, No Distributed

...

Serial12/1 is up, line protocol is up

Directed broadcast forwarding is disabled

Proxy ARP is disabled

Security level is default

Split horizon is enabled

ICMP redirects are never sent

ICMP unreachables are never sent

ICMP mask replies are never sent

IP fast switching is enabled

IP fast switching on the same interface is enabled

IP Flow switching is disabled

IP CEF switching is enabled

IP Feature Fast switching turbo vector

IP Feature CEF switching turbo vector

IP multicast fast switching is disabled

IP multicast distributed fast switching is disabled

IP route-cache flags are Fast, CEF, No Distributed

...

Serial12/2 is up, line protocol is up

Directed broadcast forwarding is disabled

Multicast reserved groups joined: 224.0.0.10

Outgoing access list is no_icmp

Inbound access list is not set

Proxy ARP is disabled

Security level is default

Split horizon is enabled

ICMP redirects are never sent

ICMP unreachables are never sent

ICMP mask replies are never sent

IP fast switching is enabled

IP fast switching on the same interface is enabled

IP Flow switching is enabled

IP CEF switching is enabled

IP Flow switching turbo vector

IP Flow CEF switching turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is disabled

IP route-cache flags are Fast, Flow, CEF, No Distributed

...