7206VXR NPE-300

Unanswered Question
cisco_lad2004 Sat, 09/08/2007 - 00:05
User Badges:
  • Gold, 750 points or more

you do "sh proc cpu" to see which process is actually hitting CPU.


alternatively, has there been any new config changes recently. maybe new policy-maps were added, new tunnels.


HTH


Sam

mohammedmahmoud Sat, 09/08/2007 - 00:29
User Badges:
  • Green, 3000 points or more

Hi Joe,


I've seen this only when CEF was disabled or not used by the router for any mean (the case that i've seen that the router was short in memory to built the FIB table - full internet routing table), but agree with Sam, lets see the show processes cpu sorted output.


HTH,

Mohammed Mahmoud.


paolo bevilacqua Sat, 09/08/2007 - 01:59
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    Founding Member

Beside the shop proc cpu and the check on CEF as others suggested, there is also to see a show interfaces at peak hours.


If the traffic volumes you mentioned is bidirectional, you are approaching the point where an NPE upgrade is due.

Here's some more information. Thanks for the input everyone.


1. eBGP but only accepting default.

2. No ACL's, NAT, PBR, VPN, etc.

3. Traffic is around 25Mbps bidirectional, peaking around 30Mbps birectional.

4. CEF is enabled globally.

5. Show proc cpu doesn't indicate anything out of hand. I believe this is because it doesn't show interrupts in this summary.

6. 90% of traffic is VoIP (very small packets!) This is what I am thinking is causing the CPU to spike under a lot less PPS then this NPE should be able to handle but wanted to hear from others with similiar experiences with this router and NPE.


Joseph W. Doherty Sat, 09/08/2007 - 12:50
User Badges:
  • Super Bronze, 10000 points or more

According to this listing: http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf

a NPE-300 should be good for about 180 Mbps.


What I would wonder about is GRE, especially if GRE is fragmenting packets.


What's your interrupt % vs. CPU %? If there's a significant spread, what's the top software processes using? ("show proc cpu sorted")

cisco_lad2004 Sat, 09/08/2007 - 22:20
User Badges:
  • Gold, 750 points or more

Good point Joseph !


I have seen same issues on 7600 SUP720-3BXL, it was MTU mismatch on both end of a VLAN forcing defrag. CPU was over 90% but not listed in show proc cpu just as described in previous posts.


HTH


Sam

Sorry, I should have clarified. GRE is not terminated on the router it is simply routing the GRE traffic through the network.


Interrupt/CPU are just about the same.


Could excessive ARP traffic/incomplete entries and CEF Stalled adjacencies cause that much more CPU cycles?


I'm still waiting to get remote access to the client's switching environment to review their switch, VLAN, and port configurations.


Thanks again for the input.

Joseph W. Doherty Sun, 09/09/2007 - 05:09
User Badges:
  • Super Bronze, 10000 points or more

Agree, GRE shouldn't be an issue if tunnel not terminated on router.


Have you run through all the performance tuning problem areas and guidelines?


e.g.

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a7306.shtml

http://www.cisco.com/en/US/products/sw/iosswrel/ps1828/products_tech_note09186a00800a65d1.shtml


In the above, especially "show cef drop" and "show interfaces switching"?


Have you looked at "show buffers" to confirm you're not making/dropping them?


You mentioned netflow. You're also using netflow caching? Defined on all interfaces?


What IOS are you using? (If I remember correctly, later versions complain when run on a NPE-300.)


"Could excessive ARP traffic/incomplete entries and CEF Stalled adjacencies cause that much more CPU cycles?"


Unsure, but interrupt to CEF might certainly slow things down. If you haven't already, you might look at: http://www.cisco.com/en/US/tech/tk827/tk831/technologies_tech_note09186a0080094303.shtml

BTW, that's the same PDF I looked at while I was scratching my head trying to figure out what could cause the CPU to be so high with this amount of traffic. I did some research with others in puck.nether.net and found a few others complaining about poor PPS and throughput with high CPU.


Could the large amount of small packets be the culprit. It makes sense but wanted to get some feedback from others in a large VoIP traffic environment.

cisco_lad2004 Sat, 09/08/2007 - 22:41
User Badges:
  • Gold, 750 points or more

yes small packets will kill your router, no doubts!

it also correlates with the addition of your Voice gateways.


Sam

Richard Burts Sun, 09/09/2007 - 12:47
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Joe


I would like to go back to your question about:

Could excessive ARP traffic/incomplete entries and CEF Stalled adjacencies cause that much more CPU cycles?


Is it possible that you have a static route (especially a static default route) that points to an outbound interface rather than pointing to a next hop address? I have seen static routes to an interface (especially a LAN interface) cause excessive ARP traffic and lots of incomplete entries (and though I have not seen it, I suspect that stalled CEF adjacencies might go along). If there is a static route to an outbound interface couls you change it to specify a next hop address and let us know if the behavior changes any?


HTH


Rick

paolo bevilacqua Mon, 09/10/2007 - 01:36
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    Founding Member

Try showing them the date NPE-300 went EOS(2002) and the traffic volumes in use at that time. He probably has replaced already PCs and servers twice since then, why not the poor old router ?


Another thing that I like to do in these cases, is saying, fine you take the responsibility of leaving things like that, sign here, and I will pass the note up the chain.

On these terms none has signed anything else than a PO :)

marikakis Mon, 09/10/2007 - 03:55
User Badges:
  • Gold, 750 points or more

I have experience with 7200's (similar setup with yours)

and have seen how small packets affect CPU usage

in various circumstances (either voice or attack).

I remember 100Kpps easily knocking down NPE-400 (higher than NPE-300).

It is not unreasonable for 70Kpps to be too much for NPE-300.

In your case the bandwidth number is almost irrelevant,

since you do have high pps rate.

The packet switching capability is very important.

I don't remember the theoretical numbers, but it doesn't matter.

Those are usually determined by optimistic, quick-and-dirty calculations

based on router internal bus frequencies using an "average" size for packets,

and do not hold when you have lots of small packets.


Hope you win this one :-),

M.


p.s. I suppose you have taken care of balancing the "bandwidth points"

corresponding to your PA's across the internal router buses.

Anyway, I still believe 70Kpps can easily be killing your router.

Actions

This Discussion