Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

7206VXR NPE-300

Anyone seen this hardware's CPU hit in the 90% range with only 50Mbps traffic, ~50-70Kpps? Router has 1 - POS interface and 1Gbps interface in use. All traffic is legitimate VoIP and GRE.

15 REPLIES

Re: 7206VXR NPE-300

you do "sh proc cpu" to see which process is actually hitting CPU.

alternatively, has there been any new config changes recently. maybe new policy-maps were added, new tunnels.

HTH

Sam

Re: 7206VXR NPE-300

Hi Joe,

I've seen this only when CEF was disabled or not used by the router for any mean (the case that i've seen that the router was short in memory to built the FIB table - full internet routing table), but agree with Sam, lets see the show processes cpu sorted output.

HTH,

Mohammed Mahmoud.

Hall of Fame Super Gold

Re: 7206VXR NPE-300

Beside the shop proc cpu and the check on CEF as others suggested, there is also to see a show interfaces at peak hours.

If the traffic volumes you mentioned is bidirectional, you are approaching the point where an NPE upgrade is due.

New Member

Re: 7206VXR NPE-300

Here's some more information. Thanks for the input everyone.

1. eBGP but only accepting default.

2. No ACL's, NAT, PBR, VPN, etc.

3. Traffic is around 25Mbps bidirectional, peaking around 30Mbps birectional.

4. CEF is enabled globally.

5. Show proc cpu doesn't indicate anything out of hand. I believe this is because it doesn't show interrupts in this summary.

6. 90% of traffic is VoIP (very small packets!) This is what I am thinking is causing the CPU to spike under a lot less PPS then this NPE should be able to handle but wanted to hear from others with similiar experiences with this router and NPE.

New Member

Re: 7206VXR NPE-300

A couple more additional info items for this problem:

The only change recently was adding 2 additional VoIP gateways to the network increasing the VoIP traffic another 10Mbps bidirectional.

Netflow indicates legitimate traffic and no malicious, out of the ordinary flows.

Super Bronze

Re: 7206VXR NPE-300

According to this listing: http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf

a NPE-300 should be good for about 180 Mbps.

What I would wonder about is GRE, especially if GRE is fragmenting packets.

What's your interrupt % vs. CPU %? If there's a significant spread, what's the top software processes using? ("show proc cpu sorted")

Re: 7206VXR NPE-300

Good point Joseph !

I have seen same issues on 7600 SUP720-3BXL, it was MTU mismatch on both end of a VLAN forcing defrag. CPU was over 90% but not listed in show proc cpu just as described in previous posts.

HTH

Sam

New Member

Re: 7206VXR NPE-300

Sorry, I should have clarified. GRE is not terminated on the router it is simply routing the GRE traffic through the network.

Interrupt/CPU are just about the same.

Could excessive ARP traffic/incomplete entries and CEF Stalled adjacencies cause that much more CPU cycles?

I'm still waiting to get remote access to the client's switching environment to review their switch, VLAN, and port configurations.

Thanks again for the input.

Super Bronze

Re: 7206VXR NPE-300

Agree, GRE shouldn't be an issue if tunnel not terminated on router.

Have you run through all the performance tuning problem areas and guidelines?

e.g.

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a7306.shtml

http://www.cisco.com/en/US/products/sw/iosswrel/ps1828/products_tech_note09186a00800a65d1.shtml

In the above, especially "show cef drop" and "show interfaces switching"?

Have you looked at "show buffers" to confirm you're not making/dropping them?

You mentioned netflow. You're also using netflow caching? Defined on all interfaces?

What IOS are you using? (If I remember correctly, later versions complain when run on a NPE-300.)

"Could excessive ARP traffic/incomplete entries and CEF Stalled adjacencies cause that much more CPU cycles?"

Unsure, but interrupt to CEF might certainly slow things down. If you haven't already, you might look at: http://www.cisco.com/en/US/tech/tk827/tk831/technologies_tech_note09186a0080094303.shtml

New Member

Re: 7206VXR NPE-300

BTW, that's the same PDF I looked at while I was scratching my head trying to figure out what could cause the CPU to be so high with this amount of traffic. I did some research with others in puck.nether.net and found a few others complaining about poor PPS and throughput with high CPU.

Could the large amount of small packets be the culprit. It makes sense but wanted to get some feedback from others in a large VoIP traffic environment.

Re: 7206VXR NPE-300

yes small packets will kill your router, no doubts!

it also correlates with the addition of your Voice gateways.

Sam

Hall of Fame Super Silver

Re: 7206VXR NPE-300

Joe

I would like to go back to your question about:

Could excessive ARP traffic/incomplete entries and CEF Stalled adjacencies cause that much more CPU cycles?

Is it possible that you have a static route (especially a static default route) that points to an outbound interface rather than pointing to a next hop address? I have seen static routes to an interface (especially a LAN interface) cause excessive ARP traffic and lots of incomplete entries (and though I have not seen it, I suspect that stalled CEF adjacencies might go along). If there is a static route to an outbound interface couls you change it to specify a next hop address and let us know if the behavior changes any?

HTH

Rick

New Member

Re: 7206VXR NPE-300

all static routes point to the next hop ip address, no interfaces

i'm trying to get the client to order NPE-G1 to replace the NPE-300 but client is not convinced its the increased traffic + small packet issue.

:( i can't win this one

Hall of Fame Super Gold

Re: 7206VXR NPE-300

Try showing them the date NPE-300 went EOS(2002) and the traffic volumes in use at that time. He probably has replaced already PCs and servers twice since then, why not the poor old router ?

Another thing that I like to do in these cases, is saying, fine you take the responsibility of leaving things like that, sign here, and I will pass the note up the chain.

On these terms none has signed anything else than a PO :)

Re: 7206VXR NPE-300

I have experience with 7200's (similar setup with yours)

and have seen how small packets affect CPU usage

in various circumstances (either voice or attack).

I remember 100Kpps easily knocking down NPE-400 (higher than NPE-300).

It is not unreasonable for 70Kpps to be too much for NPE-300.

In your case the bandwidth number is almost irrelevant,

since you do have high pps rate.

The packet switching capability is very important.

I don't remember the theoretical numbers, but it doesn't matter.

Those are usually determined by optimistic, quick-and-dirty calculations

based on router internal bus frequencies using an "average" size for packets,

and do not hold when you have lots of small packets.

Hope you win this one :-),

M.

p.s. I suppose you have taken care of balancing the "bandwidth points"

corresponding to your PA's across the internal router buses.

Anyway, I still believe 70Kpps can easily be killing your router.

365
Views
7
Helpful
15
Replies