2610xm capacity problem

Unanswered Question
Jun 19th, 2008
User Badges:


The document says 2610XM CPU have ability of handling 25000packets/sec.

However, in my experiment system, when the CPU usage reaches 90%, I can only observe, by "show interfaces", 5 minutes input rate 1965000 bits/sec, 2611 packets/sec and 5 minutes output rate 2425000 bits/sec, 2584 packets/sec. Those two numbers add up to 5195 packets/sec. Why is this number so far away from the CPU capacity?

Here is an example output of

router#show process cpu sorted

CPU utilization for five seconds: 88%/83%; one minute: 85%; five minutes: 72%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

108 58973 690 85468 3.44% 0.51% 0.41% 66 Virtual Exec

92 1018454 880694 1156 0.24% 0.23% 0.22% 0 DHCPD Receive

55 324588 13672455 23 0.24% 0.19% 0.18% 0 ACCT Periodic Pr

58 559925 72597 7712 0.24% 0.06% 0.02% 0 IP Input

42 12253 88420 138 0.16% 0.02% 0.00% 0 Compute load avg

3 70017 687863 101 0.08% 0.05% 0.05% 0 CEF process

132 25099 860025 29 0.08% 0.01% 0.00% 0 IP NAT Ager

8 260 14745 17 0.00% 0.00% 0.00% 0 Environmental mo

7 0 1 0 0.00% 0.00% 0.00% 0 Crash writer

10 26992 459637 58 0.00% 0.01% 0.00% 0 ARP Background

6 0 2 0 0.00% 0.00% 0.00% 0 Timers

5 58687 1822 32210 0.00% 0.00% 0.00% 0 Pool Manager

13 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

4 2026197 78869 25690 0.00% 0.41% 0.33% 0 Check heaps

2 721693 88418 8162 0.00% 0.03% 0.02% 0 Load Meter

16 8 2 4000 0.00% 0.00% 0.00% 0 Entity MIB API

When I stop sending requests, the CPU usage is 0%.

By the way, is it possible because of memory shortage as someone told me? But I checked the memory usage, the process memory usage is 20% (total 84MB), and the packet memory usage is 45% (total 3.2MB). And those numbers are stable during the experiment.



  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (2 ratings)
sshumakov Thu, 06/19/2008 - 23:01
User Badges:

You have read this notes?

Router Switching Performance in Packets Per Second (PPS) Numbers are given with 64 byte packet size, IP only, and are only an indication of raw switching performance.

These are testing numbers, usually with FE to FE or POS to POS, no services enabled. As you add ACL's, encryption, compression, etc - performance will decline significantly from the given numbers, unless it is a hardware-assisted platform, such as the 7600 or 12000, which process QoS, ACL's, and other features in hardware (or when a hardware assist is installed, for instance an AIM-VPN in a 3745 will offload the encryption from the CPU). Every situation is different - please simulate the true environment to get applicable performance values.

Knowing the performance for a specific router platform is not a good indication of how well a specific feature will perform. If a feature is supported in the CEF path, for instance, and we know the feature-free CEF throughput in

a specific configuration, then we only know the platform's "never-to-exceed" performance but we do not know the actual performance of any given feature, which will always be less.

All numbers are for IP packets only - no IPX/AT/DEC, etc. - Mbps calculated by pps * 64bytes * 8bits/byte

insert7992 Thu, 06/19/2008 - 23:38
User Badges:

Thanks a lot. This note is really helpful to me.

So, this means the CPU usage does indicate that my router is overloaded in my experiment, right? There is no other possible reason such as server side incorrect configuration, router memory problem, DLY(delay of interface), BW(bandwidth of interface) etc? And is it getting a better router the only way to solve my problem?



sshumakov Fri, 06/20/2008 - 00:07
User Badges:

Could you show me a results of command "show interfaces switching"?

CEF is enabled on your cisco?

insert7992 Fri, 06/20/2008 - 00:21
User Badges:

Thanks for your reply.

Sorry, sshumakov, my experiment system, located in my lab, is seperated from internet. Thus, I currently have no access to it. I will try to get into lab tomorrow noon.

And yes, ip cef is enabled on my router.


sshumakov Fri, 06/20/2008 - 00:33
User Badges:

Just "process switching" also may be a reason of low performance.

insert7992 Fri, 06/20/2008 - 00:47
User Badges:

OK, but is there a way to trace and measure the effect of "process switching"? show processes? I have an attachment here, could you please check it for me?

Thanks again.


sshumakov Fri, 06/20/2008 - 01:42
User Badges:

To identify "process switching" you need analyze results of command "show interfaces switching"

insert7992 Fri, 06/20/2008 - 11:46
User Badges:

Hi, sshumakov,

I attached my show interfaces switching output to this post.

I found another interesting question: in the document, it says fast/cef switching is 10.24Mb/s, that is 1.28MBytes/sec. Is this the max(or ideal) routering speed of 2610XM? If so, I have already observed this number in my experiment--the maximum sending speed(not include the receiving) of my server was 1.2MB/sec! Does this mean I'v already get the best performance out of my router?

By the way, the sending speed and the receiving speed on my server (or on the router interfaces) are not added up to get the over all routering speed, right? Are those two speed concurrent or seperated?



Joseph W. Doherty Fri, 06/20/2008 - 15:48
User Badges:
  • Super Bronze, 10000 points or more

Several things to note. First, when you take the 5 minute packet averages, I wonder if you let the test run long enough for the average to represent the peak rate. I especially wonder because your CPU histories snapshots tend to show a much lower 5 minute CPU rate vs. the 1 minute and 5 second rates. When your 5 minute CPU rate matches about the 5 second rate, then you can probably use the 5 minute interface averages. (If you don't want to wait that long, decrease your load interval on the interface to 30 seconds.)

Second, realize packet forwarding rates tend to decline as packet size increases. Usually the decline in forwarding rate is more than offset by effective transmission rate of the larger packets.

Third, as another poster noted, the documented rates are for very ideal conditions. For instance, notice in your original post about 5% CPU is being consumed outside of the interrupt path, most of this by virtual exec, but even though IP Input is only using .24%, on a 261x(XM) process switching is 13 times slower. (Your switching stats also confirms some process switching.)

insert7992 Wed, 06/25/2008 - 18:19
User Badges:

Thanks to all of you.

It turned out that the router was the bottleneck. We borrow a 7204-NPE-G1 router today. Without any other changes to the experiment system, the performance boost up more than 3 times of the original while the CPU running at 10%!

Thanks to all of you again for your warm hearted help.



This Discussion