Cisco-6509, High Cpu utilization.

Unanswered Question
Mar 12th, 2009
User Badges:

Hi,


We are experiencing very high cpu utilization on our 6509 switch, is there any possibility we can fix the issue ?


sh proc cpu

CPU utilization for five seconds: 77%; one minute: 75%; five minutes: 60%

PID 5Sec 1Min 5Min Process

1 0.4% 4.4% 4.3% kernel

3 0.0% 0.0% 0.0% qdelogger

4 0.0% 0.0% 0.0% devc-pty

5 0.0% 0.0% 0.0% devc-mistral.proc

6 0.0% 0.0% 0.0% pipe

7 0.0% 0.0% 0.0% dumper.proc

4104 0.0% 0.0% 0.0% pcmcia_driver.proc

4105 0.0% 0.0% 0.0% bflash_driver.proc

12298 0.0% 0.0% 0.0% mqueue

12299 0.0% 0.0% 0.0% flashfs_hes.proc

12300 0.0% 0.0% 0.0% dfs_bootdisk.proc

12301 0.0% 0.0% 0.0% ldcache.proc

12302 0.0% 0.0% 0.0% watchdog.proc

12303 0.0% 0.0% 0.0% syslogd.proc

12304 0.0% 0.0% 0.0% name_svr.proc

12305 0.0% 0.0% 0.0% wdsysmon.proc

12306 0.0% 0.0% 0.0% sysmgr.proc

16386 0.0% 0.0% 0.0% chkptd.proc

16403 0.0% 0.0% 0.0% sysmgr.proc

16404 0.0% 0.0% 0.0% syslog_dev.proc

16405 0.0% 0.0% 0.0% itrace_exec.proc

PID 5Sec 1Min 5Min Process

16406 0.0% 0.0% 0.0% packet.proc

16407 0.0% 0.0% 0.0% installer.proc

16408 45.0% 42.0% 33.8% ios-base

16409 0.0% 0.0% 0.0% fh_fd_oir.proc

16410 0.0% 0.0% 0.0% fh_metric_dir.proc

16411 0.0% 0.0% 0.0% fh_fd_snmp.proc

16412 0.0% 0.0% 0.0% fh_fd_none.proc

16413 0.0% 0.0% 0.0% fh_fd_intf.proc

16414 0.0% 0.0% 0.0% fh_fd_gold.proc

16415 0.0% 0.0% 0.0% fh_fd_timer.proc

16416 0.0% 0.0% 0.0% fh_fd_ioswd.proc

16417 0.0% 0.0% 0.0% fh_fd_counter.proc

16418 0.0% 0.0% 0.0% fh_fd_rf.proc

16419 0.0% 0.0% 0.0% fh_fd_cli.proc

16420 0.0% 0.0% 0.0% fh_server.proc

16421 0.0% 0.0% 0.0% fh_policy_dir.proc

16422 27.1% 25.2% 18.9% tcp.proc

16423 0.0% 0.0% 0.0% ipfs_daemon.proc

16424 0.4% 0.4% 0.4% raw_ip.proc

16425 0.0% 0.0% 0.0% inetd.proc

16426 2.2% 1.9% 1.4% udp.proc

16427 0.2% 0.4% 0.4% iprouting.iosproc

16428 0.4% 0.2% 0.2% cdp2.iosproc

PID 5Sec 1Min 5Min Process

765997 0.0% 0.0% 0.0% tftp_fs.proc




  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Giuseppe Larosa Fri, 03/13/2009 - 01:04
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Sumit,


may you post as an attachment file the following?


sh proc cpu sorted 1min


that allows to see the processes that use most resources


sh proc cpu history that provides whole cpu usage over time as histograms


and also allows to understand if there are recurring peaks in usage or it is steady high




Hope to help

Giuseppe





sumitsept3 Fri, 03/13/2009 - 01:52
User Badges:

sh proc cpu

CPU utilization for five seconds: 86%; one minute: 88%; five minutes: 90%

PID 5Sec 1Min 5Min Process

1 0.0% 2.7% 3.8% kernel

3 0.0% 0.0% 0.0% qdelogger

4 0.0% 0.0% 0.0% devc-pty

5 0.0% 0.0% 0.0% devc-mistral.proc

6 0.0% 0.0% 0.0% pipe

7 0.0% 0.0% 0.0% dumper.proc

4104 0.0% 0.0% 0.0% pcmcia_driver.proc

4105 0.0% 0.0% 0.0% bflash_driver.proc

12298 0.0% 0.0% 0.0% mqueue

12299 0.0% 0.0% 0.0% flashfs_hes.proc

12300 0.0% 0.0% 0.0% dfs_bootdisk.proc

12301 0.0% 0.0% 0.0% ldcache.proc

12302 0.0% 0.0% 0.0% watchdog.proc

12303 0.0% 0.0% 0.0% syslogd.proc

12304 0.0% 0.0% 0.0% name_svr.proc

12305 0.0% 0.0% 0.0% wdsysmon.proc

12306 0.0% 0.0% 0.0% sysmgr.proc

16386 0.0% 0.0% 0.0% chkptd.proc

16403 0.0% 0.0% 0.0% sysmgr.proc

16404 0.0% 0.0% 0.0% syslog_dev.proc

16405 0.0% 0.0% 0.0% itrace_exec.proc

PID 5Sec 1Min 5Min Process

16406 0.0% 0.0% 0.0% packet.proc

16407 0.0% 0.0% 0.0% installer.proc

16408 48.1% 50.4% 51.8% ios-base

16409 0.0% 0.0% 0.0% fh_fd_oir.proc

16410 0.0% 0.0% 0.0% fh_metric_dir.proc

16411 0.0% 0.0% 0.0% fh_fd_snmp.proc

16412 0.0% 0.0% 0.0% fh_fd_none.proc

16413 0.0% 0.0% 0.0% fh_fd_intf.proc

16414 0.0% 0.0% 0.0% fh_fd_gold.proc

16415 0.0% 0.0% 0.0% fh_fd_timer.proc

16416 0.0% 0.0% 0.0% fh_fd_ioswd.proc

16417 0.0% 0.0% 0.0% fh_fd_counter.proc

16418 0.0% 0.0% 0.0% fh_fd_rf.proc

16419 0.0% 0.0% 0.0% fh_fd_cli.proc

16420 0.0% 0.0% 0.0% fh_server.proc

16421 0.0% 0.0% 0.0% fh_policy_dir.proc

16422 33.0% 31.0% 29.6% tcp.proc

16423 0.0% 0.0% 0.0% ipfs_daemon.proc

16424 0.2% 0.6% 0.5% raw_ip.proc

16425 0.0% 0.0% 0.0% inetd.proc

16426 2.5% 2.4% 2.5% udp.proc

16427 1.5% 0.5% 0.5% iprouting.iosproc

16428 0.4% 0.3% 0.2% cdp2.iosproc






show proc cpu his


8889899999998888899999988888888889889898889998888898888899

8891800102009859901110099998998885783709980008899919989800

100 *

90 **********************************************************

80 **********************************************************

70 **********************************************************

60 **********************************************************

50 **********************************************************

40 **********************************************************

30 **********************************************************

20 **********************************************************

10 **********************************************************

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5


CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%



Giuseppe Larosa Fri, 03/13/2009 - 02:29
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Sumit,

two processes are causing the high cpu usage:


CPU utilization for five seconds: 86%; one minute: 88%; five minutes: 90%

PID 5Sec 1Min 5Min Process


16408 48.1% 50.4% 51.8% ios-base


16422 33.0% 31.0% 29.6% tcp.proc


please provide IOS image name and type of cpu you may be hitting a bug or the device is under some form of attack with malformed TCP packets or it may be some misconfiguration.


Have you added / changed the configuration recently ?

There is a companion device colocated, if yes how does it behave ?


if you have a registered account with enough privileges you can try to use


https://www.cisco.com/pcgi-bin/Support/OutputInterpreter/home.pl


it will ask you to post

sh ver

sh proc cpu


This is the output interpreter feedback:


-----------------------------------------------


SHOW PROCESS CPU NOTIFICATIONS (if any)


INFO: Total CPU Utilization is comprised of process and interrupt percentages.

Total CPU Utilization: 86%

Process Utilization: %

Interrupt Utilization: %

These values are found on the first line of the output:

CPU utilization for five seconds: x%/y%; one minute: a%; five minutes: b%

Total CPU Utilization: x%

Process Utilization: (x - y)%

Interrupt Utilization: y%

Process Utilization is the difference between the Total and Interrupt (x and y).

The one and five minute utilizations are exponentially decayed averages (rather

than an arithmetic average), therefore recent values have more influence on the

calculated average.


ERROR: Total CPU Utilization is at 90% for the past 5 minutes, which is very high

(>90%).

NOTE: This is an exponentially decayed average rather than an arithmetic average,

therefore recent events have a greater effect than past events.

This can cause the following symptoms:

- Input queue drops

- Slow performance

- Slow response in Telnet or unable to Telnet to the router

- Slow response on the console

- Slow or no response to ping

- Router doesn't send routing updates

The following processes are causing excessive CPU usage:

PID CPU Time Process

16408 50.4% ios-base

16422 31.0% tcp.proc

TRY THIS: If there is no indication of any problem in logged messages, then the

problem could possibly be caused by a bug in the IOS. Using the Bug ToolKit, run

a search for the specified process to see if any bugs have been reported. If this

is not caused by a bug, this device may be overloaded. Investigate upgrading this

device or moving some CPU intensive tasks to a second device.

INFO: If you need help from a Customer Support Engineer in the Cisco TAC, capture

the 'show tech-support' command output (from enable mode) before contacting Cisco

TAC. Also, if the high CPU utilization is caused by a process, please capture the

'show stacks {pid}' command output (where pid is the process ID of the process

causing the high CPU utilization). If the problem is caused by a bug in IOS, please

relay the bug ID to the Cisco Customer Support Engineer handling the case.

NOTE:

- Make sure all debugging commands in your router are turned off by issuing the

undebug all or no debug all command.





Hope to help

Giuseppe


Giuseppe Larosa Fri, 03/13/2009 - 02:43
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Sumit,


I made a search for release 12.2(18)SXF16 with search key high cpu on bug toolkit


CSCek78237

High CPU on ATM PA Helper process on PA-A3-T3

Info Information on this bug

Fixed 2

CSCdz83100

Multicast pkts should not be policy routed in CEF

Info Information on this bug

Fixed 2

CSCsa51770

Configuration of RSPAN on 12.2(18)SXD3 causes high CPU.

Info Information on this bug

Fixed 3

CSCsv40527

High CPU in IP RIB when LI enabled

Info Information on this bug

Fixed 3



Hope to help

Giuseppe



sumitsept3 Fri, 03/13/2009 - 22:12
User Badges:

Thanks for all your support , i am still trying to fix it at the earliest.


We are using



s72033-advipservicesk9_wan-vz.122-18.SXF6.bin




Giuseppe Larosa Sat, 03/14/2009 - 04:06
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Sumit,

all the bugs I've listed could apply to yours image.


if you like you can send a filtered version of your configuration.


Be aware that some problems come out from having two devices competing for something like to be HSRP active for the same group.

So if you have a pair of C6509 look at what happens at the other device.


I would suggest you to open a TAC service request if you hadn't made any configuration change on the device or in some neighboring device yesterday or the day before yesterday.


Hope to help

Giuseppe


Actions

This Discussion