CPU process utilization is 90% with no process consuming more th

Palaniswamy P · ‎04-24-2012

Hi ,

In Cisco 7206 VXR (NPE-G2) router , the cpu utilization is at an 80-90% always , but none of the process is consuming not more than 1%.In the show stacks output we are observing network interface interrupt is called very frequently. so what does network interface interrupt is about.

Logs for the reference:

show proce cpu sorted

CPU utilization for five seconds: 88%/88%; one minute: 89%; five minutes: 89%

PID Runtime(uS) Invoked uSecs 5Sec 1Min 5Min TTY Process

1 0 72 0 0.00% 0.00% 0.00% 0 Chunk Manager

2 20020000 17159 1166 0.00% 0.02% 0.01% 0 Load Meter

3 0 1 0 0.00% 0.00% 0.00% 0 cpf_process_tpQ

4 0 1 0 0.00% 0.00% 0.00% 0 EDDRI_MAIN

5 471180000 27537 17110 0.00% 0.04% 0.03% 0 Check heaps

6 8000 21 380 0.00% 0.00% 0.00% 0 Pool Manager

7 0 1 0 0.00% 0.00% 0.00% 0 DiscardQ Backgro

8 0 2 0 0.00% 0.00% 0.00% 0 Timers

9 0 2 0 0.00% 0.00% 0.00% 0 ATM AutoVC Perio

10 0 2 0 0.00% 0.00% 0.00% 0 ATM VC Auto Crea

----------output truncated------------------------------------

ABC#sh stacks

Minimum process stacks:

Free/Size Name

11068/12000 MRIB IPv4 Init Process

11480/12000 MRIB IPv6 Init Process

5552/6000 Inspect Init Msg

59324/60000 script background loader

58924/60000 EEM Auto Registration Proc

5324/6000 CDP Protocol

5548/6000 SASL MAIN

2528/3000 allegro libretto init

11068/12000 Router Init

3336/12000 Init

4360/6000 DIB error message

5508/6000 BGP Accepter

5040/6000 BGP Open

5340/6000 RADIUS INITCONFIG

34148/36000 TCP Command

5492/6000 URPF stats

2548/3000 Rom Random Update Process

5240/6000 Delayed Reload

6016/12000 Virtual Exec

Interrupt level stacks:

Level Called Unused/Size Name

1 2845320824 7344/9000 Network interfaces

2 0 9000/9000 DMA level Interrupt

3 321 8780/9000 PA Management Int Handler

4 2904 8892/9000 Console Uart

5 0 9000/9000 OIR/Error Interrupt

7 21453943 9000/9000 4ms Timer Interrupt

Regards,

Palani

Joseph W. Doherty · ‎04-25-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

It likely is packet forwarding in the "fast path"; which doesn't show as a process. If you're pushing around a gig or so of aggregate bandwidth through the 7200, this would be expected.

Palaniswamy P · ‎04-26-2012

Hi joseph,

But the traffic is less than 60-70mbps on gig interface always . CEF is enabled. a Cisco 7200 router is capable of handling that much traffic.

Regards,

Palani

Giuseppe Larosa · ‎04-27-2012

Hello Palani,

follow the guidelines of the following document to further troubleshoot your issue

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

in some cases the traffic volume is not high as you noted, but packets are processed in a slow path.

Hope to help

Giuseppe

Joseph W. Doherty · ‎04-27-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

But the traffic is less than 60-70mbps on gig interface always . CEF is enabled. a Cisco 7200 router is capable of handling that much traffic.

Agreed, a -G2 should easily handle only 60-70 Mbps of traffic in "fast path".

As Giuseppe noted, "slow path" could consume much more CPU at a much lower transfer rate, but then you would expect to see the CPU being consumed by IP Input process.

When I looked at your original post, one oddity that did catch my eye, your 5 second CPU usage was 88%/88%. Usually, at best, as such high usage, you see interrupt usage within 1 or 3 percent of total CPU, not exactly the same, as some CPU is consumed by background processes. Your posted process does show some of this, but numbers like .02% and .04%. So, I'm wonder whether this might be some display bug.

Palaniswamy P · ‎04-27-2012

Hi ,

As per my assumption, the cpu utilization is due to interrupts. which we can see in the output of show stacks where the network interface interrupt is called very offen and called in high numbers i searched a lot about netowork interface field not much help. Suspecting on the bug side upgraded the ios to latest release and done the reload too..

Logs:

ABC# sh stacks

Minimum process stacks:

Free/Size Name

11068/12000 MRIB IPv4 Init Process

11480/12000 MRIB IPv6 Init Process

5552/6000 Inspect Init Msg

59324/60000 script background loader

58924/60000 EEM Auto Registration Proc

5324/6000 CDP Protocol

5548/6000 SASL MAIN

2528/3000 allegro libretto init

11068/12000 Router Init

3336/12000 Init

4360/6000 DIB error message

5508/6000 BGP Accepter

5040/6000 BGP Open

5340/6000 RADIUS INITCONFIG

34148/36000 TCP Command

5492/6000 URPF stats

2548/3000 Rom Random Update Process

5240/6000 Delayed Reload

6016/12000 Virtual Exec

Interrupt level stacks:

Level Called Unused/Size Name

1 3215499403 7344/9000 Network interfaces

2 0 9000/9000 DMA level Interrupt

3 327 8780/9000 PA Management Int Handler

4 2914 8892/9000 Console Uart

5 0 9000/9000 OIR/Error Interrupt

7 21981998 9000/9000 4ms Timer Interrupt

ABC# sh stacks

Minimum process stacks:

Free/Size Name

11068/12000 MRIB IPv4 Init Process

11480/12000 MRIB IPv6 Init Process

5552/6000 Inspect Init Msg

59324/60000 script background loader

58924/60000 EEM Auto Registration Proc

5324/6000 CDP Protocol

5548/6000 SASL MAIN

2528/3000 allegro libretto init

11068/12000 Router Init

3336/12000 Init

4360/6000 DIB error message

5508/6000 BGP Accepter

5040/6000 BGP Open

5340/6000 RADIUS INITCONFIG

34148/36000 TCP Command

5492/6000 URPF stats

2548/3000 Rom Random Update Process

5240/6000 Delayed Reload

6016/12000 Virtual Exec

Interrupt level stacks:

Level Called Unused/Size Name

1 3215732279 7344/9000 Network interfaces

2 0 9000/9000 DMA level Interrupt

3 327 8780/9000 PA Management Int Handler

4 2914 8892/9000 Console Uart

5 0 9000/9000 OIR/Error Interrupt

7 21982331 9000/9000 4ms Timer Interrupt

Regards,

Palani

Joseph W. Doherty · ‎04-27-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

As per my assumption, the cpu utilization is due to interrupts. which we can see in the output of show stacks where the network interface interrupt is called very offen and called in high numbers i searched a lot about netowork interface field not much help. Suspecting on the bug side upgraded the ios to latest release and done the reload too..

What's prior and reloaded IOS versions? Same CPU stats?

Suggest you open a TAC case.

Palaniswamy P · ‎04-27-2012

Hi Doherty,

Yes the cpu stats remain the same( 80-90%) after the ugrade.

c7200p-adventerprisek9-mz.150-1.M8.bin

c7200p-adventerprisek9-mz.124-24.T7.bin

c7200p-adventerprisek9-mz.124-24.T2.bin

Regards,

Palani

Joseph W. Doherty · ‎04-27-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

If the total traffic passing through your -G2 is only 60-70 Mb(its)ps (not one interface, i.e. sum all ingress or all egress traffic across all interfaces), then, again, suggest you open a TAC case.

diyaatout · ‎04-28-2012

Try to undebug all , may be it is the issue

Sent from Cisco Technical Support iPhone App

Palaniswamy P · ‎04-28-2012

Hi ,

Thanks all for the valuable suggestions. will check with tac.

Regards,

Palani

kitlim888 · ‎05-01-2013

Hi Palani,

Did you manage to solve your problem? I am facing the same issue with high interrupts 90%-100% running voip on 7206vxr NPE-G2 at about 90Mbps. The pps was only 170k which is way below the published 2mil pps.

Any suggestions on this is very much appreciated.

Regards

Kit

Daniele Giordano · ‎04-16-2016

Hi Experts, same situation, same issue.

Do you have found a solution?

C7206VXR with NPE-G2 with hig cpu interrupts running 100Mbps of VoIP traffic.

I noticed also a strange 4ms Timer Interrupt in show stacks.

router#sh stacks

Interrupt level stacks:
Level Called Unused/Size Name
1 1297587812 6596/9000 Network interfaces
2 0 9000/9000 DMA level Interrupt
3 8 8772/9000 PA Management Int Handler
4 3309 8892/9000 Console Uart
5 0 9000/9000 OIR/Error Interrupt
7 31070716 9000/9000 4ms Timer Interrupt

Can you explain me what is this process?

Regards.

CPU process utilization is 90% with no process consuming more than 1%