11-11-2008 06:05 AM - edited 03-04-2019 12:17 AM
Hi,
during an FTP from Fa0/1 to Fa0/0 on a 2611XM the CPU goes up to 94%/92%. During this time ICMP packets through Fa0/0 per-packet load shared S0/0:0 and S0/0:1 get dropped. I do not yet know where the ICMP gets lost but if anyone has an idea what pushes up the CPU that might be helpful. Thanks. Mat
I add some config and the "sh proc cpu" command:
router01#sh proc cpu sorted
CPU utilization for five seconds: 94%/92%; one minute: 67%; five minutes: 35%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
71 4760 496 9596 0.69% 0.70% 0.41% 66 Virtual Exec
37 5928 6030 983 0.30% 0.55% 0.35% 0 IP Input
64 2028 391 5186 0.15% 0.30% 0.16% 0 TPLUS
77 1024 36203 28 0.15% 0.13% 0.06% 0 PPP Events
74 892 11718 76 0.07% 0.08% 0.04% 0 Mwheel Process
75 140 1662 84 0.07% 0.01% 0.00% 0 PIM Process
73 476 1334 356 0.07% 0.01% 0.00% 0 IGMP Input
95 2544 1218 2088 0.07% 0.00% 0.00% 0 OSPF Router
...
interface FastEthernet0/0
ip address 10.100.0.1 255.255.255.0
no ip proxy-arp
duplex auto
speed auto
no cdp enable
!
interface Serial0/0:0
bandwidth 2048000
ip address 192.168.255.1 255.255.255.252
ip load-sharing per-packet
ip pim sparse-mode
encapsulation ppp
service-policy output PM-CRITICAL-TRAFFIC
!
interface FastEthernet0/1
ip address 10.101.0.1 255.255.255.0
ip pim sparse-mode
duplex auto
speed auto
no cdp enable
!
interface Serial0/1:0
bandwidth 2048000
ip address 192.168.255.5 255.255.255.252
ip load-sharing per-packet
ip pim sparse-mode
encapsulation ppp
service-policy output PM-CRITICAL-TRAFFIC
11-11-2008 06:43 AM
Hi,
I would turn on CEF, if it isn't already. Have a look at the service policy, which can contribute to CPU load depending on which kind of traffic you have.
Can you also show the output of "show proc cpu history" and "show interface" preferably with a load interval of 30 sec?
So far there is no consistent picture allowing to spot the culprit.
Regards,
Martin
11-11-2008 06:52 AM
Hi Martin,
I do not have the outputs ready. They first need to be captured (maybe tomorrow). To answer your questions: CEF is switched on and the service-policy in my opinion does play a subordinate role because of the fact that the main traffic is from FastEthernet to FastEthernet. From the top of my mind the traffic on the Serial interfaces does not go higher than 300k and on the FastEthernets no higher than 30M.
So much for now and thanks for your reply. I'll try to get more info tomorrow.
Regards,
Mat
11-11-2008 06:59 AM
Mat,
Your CPU output indicates typical interrupt traffic which often is caused by high utilization in links.
Your router is rated at 10Mbps at best /regardless the interface speed/ so if you go from a 100Mbps interface to another 100Mbps interface, the router itself will be the bottleneck.
If the FTP software allows it, you must set the maximum speed on this transfer to 5Mbps or so.
For more information on router performance, please visit this URL and click under 'Router Performance'.
http://www.cisco.com/web/partners/tools/quickreference/index.html
HTH,
__
Edison.
Please rate helpful posts
11-11-2008 07:21 AM
Thanks Edison,
this is indeed rather limiting (how in the world do I see 30Mb on the interfaces if it only switches 10Mb???). I try to figure out a QoS way of securing the ICMP traffic because that one is important for a management application. And for mid-term I try to sell some new platform.
Thanks for your reply.
Mat
11-11-2008 07:38 AM
Hi,
Please note that the performance data assumes 64 Byte packets, which is not the case for your FTP traffic. Assuming 1500 Bytes instead of 64 Bytes you could get up to 240 Mbps under optimum conditions, which might not be realistic in your case.
For the time being you could simply fix the interface speed of the FastEthernet to 10 Mbps, which should solve the issue until you get a more powerful platform. If your connected equipment does not allow that you could configure a shaper on the FE. By trying various rates you likely could adjust the shaper to what the router can take without causing issues to ICMP or any other traffic.
Regards,
Martin
11-11-2008 07:43 AM
(Edison, hope you don't mind if I answer this question.)
The 2611XM has a Fast/CEF switching rate of 20 Kpps, which is specified as providing 10.24 Mbps for 64 byte packets. The packet size is key. As packet size increases, device PPS rates usually decrease but often not at the same rate as the effective increase of bandwidth. Assuming your FTP is using max size typical Ethernet (1500), a 3x effective bandwidth increase is certainly possible.
11-11-2008 07:32 AM
The high interrupt CPU rate is to be expected when you max out the router pushing its maximum forwarding rate.
Not 100% certain, but usage of CEF per-packet load sharing isn't all that common, so it wouldn't surprise me that it exhibits an issue when the router's CPU is running nearly 100%. (Other issues may also arise when CPU is so busy.)
Somethings you might try:
o stop using per-packet CEF
o run Ethernet ports at 10 Mbps
o implement an inbound policer
PS:
Small software routers like the 2600 series are really intended for low bandwidth WAN routing, not LAN routing. A much better solution might be to "front" the 2600 with a small L3 switch for LAN routing. The 8 port 3560 is very attractive for this role. This should also provide a higher FTP transfer rate too.
11-11-2008 07:50 AM
Hi Joseph,
The per-packet CEF is due to the two WAN links that need to be used evenly (the switching would lead to one link being used and the other one being standby). However I doubt (as you) that the load-sharing is the culprit.
I already have tried a policing configuration, however we did not rate it down to lower than 10Mb so therefore we were still over the router's switching performance. The next thing we try is to police the FTP down to 5Mbps to see if it changes the behavior. We also think of redesigning the LAN-LAN routing part over switches.
Thanks for your reply and hints.
Mat
11-11-2008 08:04 AM
If you have multiple flows, I believe, CEF should use both links. CEF per packet is really for splitting single flows and can cause issues with TCP. If ordinary CEF doesn't load balance enough, if possible, I would then look toward using MLPPP, although that too will contribute to your CPU load, although for just WAN traffic, it shouldn't be too significant.
One concern I do have using a policer, although I didn't mention it, is whether it would reduce the CPU load a significant amount since the policer itself will consume CPU cycles. I hope it might use less CPU than complete forwarding of the traffic, but that might not be true. Setting the interfaces down to 10 Mbps, as also suggested by Martin, provides a hardware policer. Although you've only seen the issue with FTP, any high volume LAN flow, or flows, will likely run out your CPU, so you wouldn't want to police just FTP.
Another advantage of a L3 switch in front of the WAN router, even the latest ISRs, or even 7200s, can struggle with 100 Mbps or gig Ethernet, so the combination of L3 switch and WAN router, I believe, is very attractive until you get into the chassis solutions like the 6500/7600.
11-11-2008 08:25 AM
Hi Joseph,
Thanks for your reply. I really need to do some testing of the CEF load sharing one day. I thought about MLPPP already but as I do not have any document explaining the load-sharing for it I am a bit cautios on using it right away.
For the CPU cycles with the policer I have less doubts, since all Cisco documentation asks for CEF togehter with CBWFQ/LLQ. So I assume that some CEF accounting task is co-used by QoS to keep the performance high. But again I lack the documentation - it's just wild guessing.
For the isolation of the FTP you are right. But as the environment is pretty static and the daily FTP is troubling everybody this might fit the momentary needs. It would be really neat to have a sort of "per-flow-Policer" based on user defined criteria. But let's stay realistic, we'll try out the hints that we have gotten already and I try to update the thread.
Thanks for your detailed thoughts. I appreciate this very much.
Mat
11-11-2008 08:41 AM
Much of the newer IOS technologies depend on CEF being active; it's not just some accounting task. There are whitepapers on Cisco's site that explain it further, it's an interesting technology.
MLPPP, in brief, will also send packets down multiple links. Not 100% certain, but it might better load balance than CEF per-packet, or at least I believe it can. It guarantees packets given to multiple links are provided in the same sequence before being forwarded from the receiving device. It can even split large packets and send the parts across multiple links concurrently (where it reassembles them on the receiving device).
PS:
One last note, although setting your interface down to 10 Mbps should keep from overloading your router's CPU, you might then encounter switch congestion and without L2 QoS, might have performance issues with other traffic, similar or worst to what you're seeing now. A general inbound policer, or FTP inbound policer, might be the best thing to try first.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide