high cpu only when traffic is high

narainarun · ‎09-07-2010

Hi,

I am facing a weird problem in 7206VXR-NPEG1 router. The router has a STM1 interface to WAN facing side and onboard gigi interfaces on LAN side.

CPU of the router goes only high during peak production hours. CPU is constanly in 70% and maximum CPU goes till 92% when the traffic of STM1 is around 50-55Mbps and above. When the traffic of STM1 drops below 40Mbps then CPU also drops down to 50% and during night time when the traffic of STM1 drops around 20Mbps then CPU comes down to 20-30%. It infers the CPU is heavily based on the traffic flowing through it and not on any specific protocols or processes, it is only due to interrupts.

show process cpu shows most of the CPU is due to the interrupt and no single process is holding high CPU. Of course I could see BGP (scanner and event) and OSPF is showing as top contributor (around 4%) of CPU to the total 80% which I think is very negligible. The IOS that I have has BGP Next Hop tracking feature enabled so CPU due to BGP scanner is not an issue.

I also made sure that CEF is enabled and all the points discussed in this doc: http://www1.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml#pxf_punts but still of no help to me.

Can somebody point me what would be my next step in finding out the root cause of the issue.

Is CPU Profiling the way now for me to find out the root cause due to interrupts?

Is this could be related to the STM1 interface on hardware. (though it sounds weird, i am thinking of all possibilities).

any help is much appreciated...

thanks

Arun

Giuseppe Larosa · ‎09-07-2010

Hello Arun,

high cpu usage with WAN Port adapters on C7206VXR with NPE-G1 and NPE-G2 have been reported in the forums.

your case looks like worse then others as 55 Mbps on the STM-1 are enough to reach 92% of cpu usage.

have you configured additional features on the STM-1 link ?

What IOS image is running on the router?

It might be caused by a bug

post a show ver | inc image

You can use cpu profiling, but probably it is better use show interface switching command to see if traffic sent and received on the STM-1 link is process switched or processed by CEF.

You have already found a relationship with traffic volume on STM-1, so cpu profiling may be of little value in this case.

Hope to help

Giuseppe

narainarun · ‎09-07-2010

Hi Guiseppe,

Thanks for your feedback.

The router is running "c7200-adventerprisek9-mz.122-33.SRC6.bin".

The config in POS interface is below:

ip address x.x.x.x x.x.x.x

no ip proxy-arp

ip pim sparse-mode

ip ospf message-digest-key 1 md5 7 xxxxxxxxxxxxx

ip ospf cost 50

load-interval 30

mpls traffic-eng tunnels

mpls ip

no cdp enable

service-policy output Service_Queues

end

Since POS interface is MPLS enabled, show interface switching does not show much increase in MPLS process but still I could see IP process counters are increasing. Let me do a bug scrub for this image. I have another POS interface in the same router which is not used, can i try shifting the link to that interface and see (as I said though it sounds weird, i m running out of options and want to figure out any hardware related issues either PA card or on the chassis slot).

thanks

Arun