High CPU on 2611XM 12.3(26) during FTP

[MATTHIAS SCHAERER]

Hi,

during an FTP from Fa0/1 to Fa0/0 on a 2611XM the CPU goes up to 94%/92%. During this time ICMP packets through Fa0/0 per-packet load shared S0/0:0 and S0/0:1 get dropped. I do not yet know where the ICMP gets lost but if anyone has an idea what pushes up the CPU that might be helpful. Thanks. Mat

I add some config and the "sh proc cpu" command:

router01#sh proc cpu sorted

CPU utilization for five seconds: 94%/92%; one minute: 67%; five minutes: 35%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

71 4760 496 9596 0.69% 0.70% 0.41% 66 Virtual Exec

37 5928 6030 983 0.30% 0.55% 0.35% 0 IP Input

64 2028 391 5186 0.15% 0.30% 0.16% 0 TPLUS

77 1024 36203 28 0.15% 0.13% 0.06% 0 PPP Events

74 892 11718 76 0.07% 0.08% 0.04% 0 Mwheel Process

75 140 1662 84 0.07% 0.01% 0.00% 0 PIM Process

73 476 1334 356 0.07% 0.01% 0.00% 0 IGMP Input

95 2544 1218 2088 0.07% 0.00% 0.00% 0 OSPF Router

...

interface FastEthernet0/0

ip address 10.100.0.1 255.255.255.0

no ip proxy-arp

duplex auto

speed auto

no cdp enable

!

interface Serial0/0:0

bandwidth 2048000

ip address 192.168.255.1 255.255.255.252

ip load-sharing per-packet

ip pim sparse-mode

encapsulation ppp

service-policy output PM-CRITICAL-TRAFFIC

!

interface FastEthernet0/1

ip address 10.101.0.1 255.255.255.0

ip pim sparse-mode

duplex auto

speed auto

no cdp enable

!

interface Serial0/1:0

bandwidth 2048000

ip address 192.168.255.5 255.255.255.252

ip load-sharing per-packet

ip pim sparse-mode

encapsulation ppp

service-policy output PM-CRITICAL-TRAFFIC

[mheusing]

Hi,

I would turn on CEF, if it isn't already. Have a look at the service policy, which can contribute to CPU load depending on which kind of traffic you have.

Can you also show the output of "show proc cpu history" and "show interface" preferably with a load interval of 30 sec?

So far there is no consistent picture allowing to spot the culprit.

Regards,

Martin

[MATTHIAS SCHAERER]

Hi Martin,

I do not have the outputs ready. They first need to be captured (maybe tomorrow). To answer your questions: CEF is switched on and the service-policy in my opinion does play a subordinate role because of the fact that the main traffic is from FastEthernet to FastEthernet. From the top of my mind the traffic on the Serial interfaces does not go higher than 300k and on the FastEthernets no higher than 30M.

So much for now and thanks for your reply. I'll try to get more info tomorrow.

Regards,

Mat

[Edison Ortiz]

Mat,

Your CPU output indicates typical interrupt traffic which often is caused by high utilization in links.

Your router is rated at 10Mbps at best /regardless the interface speed/ so if you go from a 100Mbps interface to another 100Mbps interface, the router itself will be the bottleneck.

If the FTP software allows it, you must set the maximum speed on this transfer to 5Mbps or so.

For more information on router performance, please visit this URL and click under 'Router Performance'.

http://www.cisco.com/web/partners/tools/quickreference/index.html

HTH,

__

Edison.

Please rate helpful posts

[MATTHIAS SCHAERER]

Thanks Edison,

this is indeed rather limiting (how in the world do I see 30Mb on the interfaces if it only switches 10Mb???). I try to figure out a QoS way of securing the ICMP traffic because that one is important for a management application. And for mid-term I try to sell some new platform.

Thanks for your reply.

Mat

[mheusing]

Hi,

Please note that the performance data assumes 64 Byte packets, which is not the case for your FTP traffic. Assuming 1500 Bytes instead of 64 Bytes you could get up to 240 Mbps under optimum conditions, which might not be realistic in your case.

For the time being you could simply fix the interface speed of the FastEthernet to 10 Mbps, which should solve the issue until you get a more powerful platform. If your connected equipment does not allow that you could configure a shaper on the FE. By trying various rates you likely could adjust the shaper to what the router can take without causing issues to ICMP or any other traffic.

Regards,

Martin

[Joseph W. Doherty]

(Edison, hope you don't mind if I answer this question.)

The 2611XM has a Fast/CEF switching rate of 20 Kpps, which is specified as providing 10.24 Mbps for 64 byte packets. The packet size is key. As packet size increases, device PPS rates usually decrease but often not at the same rate as the effective increase of bandwidth. Assuming your FTP is using max size typical Ethernet (1500), a 3x effective bandwidth increase is certainly possible.

[Joseph W. Doherty]

The high interrupt CPU rate is to be expected when you max out the router pushing its maximum forwarding rate.

Not 100% certain, but usage of CEF per-packet load sharing isn't all that common, so it wouldn't surprise me that it exhibits an issue when the router's CPU is running nearly 100%. (Other issues may also arise when CPU is so busy.)

Somethings you might try:

o stop using per-packet CEF

o run Ethernet ports at 10 Mbps

o implement an inbound policer

PS:

Small software routers like the 2600 series are really intended for low bandwidth WAN routing, not LAN routing. A much better solution might be to "front" the 2600 with a small L3 switch for LAN routing. The 8 port 3560 is very attractive for this role. This should also provide a higher FTP transfer rate too.

[MATTHIAS SCHAERER]

Hi Joseph,

The per-packet CEF is due to the two WAN links that need to be used evenly (the switching would lead to one link being used and the other one being standby). However I doubt (as you) that the load-sharing is the culprit.

I already have tried a policing configuration, however we did not rate it down to lower than 10Mb so therefore we were still over the router's switching performance. The next thing we try is to police the FTP down to 5Mbps to see if it changes the behavior. We also think of redesigning the LAN-LAN routing part over switches.

Thanks for your reply and hints.

Mat

[Joseph W. Doherty]

If you have multiple flows, I believe, CEF should use both links. CEF per packet is really for splitting single flows and can cause issues with TCP. If ordinary CEF doesn't load balance enough, if possible, I would then look toward using MLPPP, although that too will contribute to your CPU load, although for just WAN traffic, it shouldn't be too significant.

One concern I do have using a policer, although I didn't mention it, is whether it would reduce the CPU load a significant amount since the policer itself will consume CPU cycles. I hope it might use less CPU than complete forwarding of the traffic, but that might not be true. Setting the interfaces down to 10 Mbps, as also suggested by Martin, provides a hardware policer. Although you've only seen the issue with FTP, any high volume LAN flow, or flows, will likely run out your CPU, so you wouldn't want to police just FTP.

Another advantage of a L3 switch in front of the WAN router, even the latest ISRs, or even 7200s, can struggle with 100 Mbps or gig Ethernet, so the combination of L3 switch and WAN router, I believe, is very attractive until you get into the chassis solutions like the 6500/7600.

[MATTHIAS SCHAERER]

Hi Joseph,

Thanks for your reply. I really need to do some testing of the CEF load sharing one day. I thought about MLPPP already but as I do not have any document explaining the load-sharing for it I am a bit cautios on using it right away.

For the CPU cycles with the policer I have less doubts, since all Cisco documentation asks for CEF togehter with CBWFQ/LLQ. So I assume that some CEF accounting task is co-used by QoS to keep the performance high. But again I lack the documentation - it's just wild guessing.

For the isolation of the FTP you are right. But as the environment is pretty static and the daily FTP is troubling everybody this might fit the momentary needs. It would be really neat to have a sort of "per-flow-Policer" based on user defined criteria. But let's stay realistic, we'll try out the hints that we have gotten already and I try to update the thread.

Thanks for your detailed thoughts. I appreciate this very much.

Mat

[Joseph W. Doherty]

Much of the newer IOS technologies depend on CEF being active; it's not just some accounting task. There are whitepapers on Cisco's site that explain it further, it's an interesting technology.

MLPPP, in brief, will also send packets down multiple links. Not 100% certain, but it might better load balance than CEF per-packet, or at least I believe it can. It guarantees packets given to multiple links are provided in the same sequence before being forwarded from the receiving device. It can even split large packets and send the parts across multiple links concurrently (where it reassembles them on the receiving device).

PS:

One last note, although setting your interface down to 10 Mbps should keep from overloading your router's CPU, you might then encounter switch congestion and without L2 QoS, might have performance issues with other traffic, similar or worst to what you're seeing now. A general inbound policer, or FTP inbound policer, might be the best thing to try first.