High CPU on 2811 but no visible process that causes it

kasper123 · ‎02-20-2014

We have a cisco 2811 that we use to terminate 2 VPN's and to get access to internet.

Several times a day there is high CPU on the router but I can't seem to be able to find what causes it.

Router#show processes cpu sorted

CPU utilization for five seconds: 98%/94%; one minute: 94%; five minutes: 88%

PID Runtime(uS) Invoked uSecs 5Sec 1Min 5Min TTY Process

118 3747800704 3426462 2347 2.13% 2.71% 2.37% 0 IP Input

126 3060000 175 17485 0.45% 0.03% 0.04% 514 SSH Process

22 1275008000 4030571 316 0.22% 0.24% 0.22% 0 ARP Input

274 310512000 786648 394 0.22% 0.19% 0.18% 0 IP NAT Ager

...

These are the top processes. The CPU is at 98% but there is no process using more then 3%. This processes usage is the same even when the CPU load is normal.

Any ideas how to troubleshoot this?

Regards.

Joseph W. Doherty · ‎02-20-2014

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

The "missing" CPU is "interrupt" CPU, which generally is devoted to forwarding traffic. What's the aggregate of all ingress interface bandwidth? My experience, the 2811 will max out with about 40 to 50 Mbps, aggregate.

If your bandwidth consumption is much lower, I also noticed you mentioned VPN. That will consume more CPU per packet forwarded expecially if there's any fragmentation being done. (The latter can greatly add to the load of the router.) Are you using tcp mss-adjust?

kasper123 · ‎02-20-2014

Hi Joseph,

there are several WAN links terminating at the router. Two of them are dedicated for the VPN's and there is a third (an ADSL PPPoE connection) that is 50Mbps and that is the default route.

And yes we are using tcp adjust-mss on the LAN facing interface.

ip tcp adjust-mss 1452

Is there something we can do to mitigate the problem?

Joseph W. Doherty · ‎02-20-2014

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Well the 1452 would adjust for the PPPoE overhead, but VPN tunnels have even more overhead. Do you adjust for them too?

There other tunnel interface options that sometime can help a little too, but if you're dealing with any full size UDP packets (or similar traffic), they will need to be fragmented, and there's nothing you can do unless you adjust the MTU of the host.

BTW, a 50 Mbps ADSL, alone, is, again in my experience, enough to overwhelm a 2811.

If you have enough traffic, especially traffic that requires fragmentation, the only real solution is a "faster" router.