Router cpu hits %95+ out of nowhere

[husycisco]

2651XM IOS version 124-25b.bin (Latest). show processes cpu sort shows only a %10 usage by IP input process, but CPU usage is %90+ . I cant see the process that is hogging cpu. What can be causing this?

[Lucien Avramov]

please post sh proc cpu sort | e 0.0

[husycisco]

Lucien,

You can see the output in attachment. Direct paste looks scrambled thats why i took screenshot.

I couldnt locate the process that is hogging the cpu.

Btw, some background about the issue here

http://forums.cisco.com/eforum/servlet/NetProf?page=netprof&forum=Network%20Infrastructure&topic=LAN%2C%20Switching%20and%20Routing&topicID=.ee71a04&fromOutline=&CommCmd=MB%3Fcmd%3Ddisplay_location%26location%3D.2cd4b23e

Regards

[husycisco]

Lucien,

I believe you have the necessary resources&contacts to sort this out. Here is some more info

I did cpu profiling according to following article

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml#pxf_punts

Thanks to Cisco, they wrote very well about cpu profiling, but conclusion? How can I locate that interrupt thread which is hogging the CPU by using these outputs? Attached is the output of profiling, I highleted some high values but dont know what to do. I really need help on this to determine the issue to decide router upgrade or interface upgrade or whatever.

Thanks

[Lucien Avramov]

In both outputs you attached, the cpu per engine does not exceed 15%. I dont see any evidence of what is causing. this looks normal.

So you have then a high cpu from times to times? Do you have any snmp activity? Anything showing in the logs (show logging)?

[husycisco]

The CPU profile shows that the issue is not cpu process related, it is cpu interrupt related. Thats why it doesnt appear under show cpu processes. The latest attachment i added has cpu profile output. Can you interpret it?

Thanks

[Lucien Avramov]

This is quite complicated and other information must be collected for the interpretation.

Please open a TAC case.

[Edison Ortiz]

Your other thread is very helpful as it contains more information than this thread.

Your problem is simple, you are running too many services on this router.

You are doing router-on-a-stick for 26 Vlans and NAT and perhaps your bandwidth demands have also increased.

You need to redesign your network and move your router-on-a-stick to a L3 switch for inter-vlan routing done in hardware and if your bandwidth requirements have increased or you've upgraded your WAN circuit, you may need to move to a better router - 2851 ISR for instance.

Tip: As stated on the other thread, when you don't see the process that is taking the CPU from the 'show proc cpu' it means the router is doing interrupt - in other words, the router itself is overloaded - not a service running on the router.

Regards

Edison.

[husycisco]

Edison,

Problem is not that simple actually :)Called TAC and they said 2651XM is EOL and they cant support it. Great.

Configured a NETASQ 120 UTM device with subinterfaces, let it handle routing and become DHCP server. At Cisco, removed all subinterfaces, dhcp server and let it do only NAT and guess what, CPU is still at % 99 but one difference. Interrupts used to be about % 80 and process about % 15-20 before NETASQ, but now interrupts are at %50, and IP Input process is at % 45-50 (Used to be % 15 or so) although CEF is enabled everywhere.

When I disable IPS, CPU is still at % 95+ but now interrupts are about % 70.

Cisco was NATing (overload) for 16 vlans. I decreased it to 8, cleared the table and still same, decreased to 4.. same, and i decreased to 1 only and CPU came down to % 67. Dynamic translations never exceed 3000 that i decreased timer from 3000 to 1800, i also removed the timer modifications etc etc

Finally, I removed that god damn device, configured NETASQ to do NAT also and everythng works great. Cpu utilization is % 3-4. I removed Cisco to analyze further in my lab tomorrow. Suggestions are welcome.

Thanks

[Edison Ortiz]

Based on your feedback, the NAT translation may be causing the spike on the CPU. Can you modify your topology to do NAT on the NETASQ appliance and leave this router just for routing?

Regards

Edison