High CPU usage on Cisco 1812 router (dual wan)

jesper_petersen · ‎07-07-2010

Hello folks,

I have recently observed heavy CPU usage on one of our Cisco 1812W routers. I have been unable to find the cause of this, since the "sh proc cpu" does not show which process uses the CPU. The two command outputs below are taken at the same time. The IOS IS:

Cisco IOS Software, C181X Software (C181X-ADVIPSERVICESK9-M), Version 12.4(15)T1, RELEASE SOFTWARE (fc2)

Any help in finding the cause will be greatly appreciated.

I have attached the config for the router (some line have been removed). I have also attached a text file containing the output of "sh proc cpu his" and "sh proc cpu" (taken at the time of the high cpu utilization).

Please note that this router has a rather (for us at least) unusual setup where is has two physical WAN connection and I use PBR to route packets out of the correct interface based on the destination IP address. I suspect that this is the cause somehow - maybe I configured it incorrectly?

Thanks in advance

-- JRP

Brandon Lynch · ‎07-10-2010

Hey Jesper,

The CPU outputs attached do not show any instances of sustained high CPU -

    813121112211121297756889231222112222128369575896811212211133211232121111
    257814147435799111778960843454420347520771994505867537245100125438534844
100
90                 *    ***                 *   ** *
80 *               * * ***              * * * ** *
70 *               *** ****              * ** * ****
60 *               ********              * **** ****
50 *               ********              * *********
40 * *             ********              ***********
30 * *     *    * ********** *      * *********** * *    **    **
20 *****   ** *************** *** ******************** ** * *** ****** *
10 ****************########****************########************************
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
             0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

When reviewing the CPU utilization graphs, we generally want to focus on the '#' signs as this indicates where the CPU was sustained at a given point. As you can see from the graph, sustained utilization is generally at or less than 10%. Is your concern the spikes denoted by the '*' signs? Or has the CPU shown sustained highs since this output?

The 'show proc cpu' output you attached shows a point at which the CPU was higher -

Pff#sh proc cpu
CPU utilization for five seconds: 92%/89%; one minute: 56%; five minutes: 19%

I suspect that this may have been during one of the spikes, though, given the graph above. Spikes in CPU can often be attributed to a number of things including traffic increases and specific processes kicking off. There are 2 types of CPU utilization: process-level and interrupt-level. In order to tell, we have to look at the overall reading at a given point. In your case, we see this as 92%/89%. The first number is the total CPU while the second number is how much of the CPU is interrupt-driven. The difference between these two numbers will be what you find for the total accumulation of the CPU usage by the processes in the table.

For yours, it's almost all interrupt. Interrupt-driven CPU is often associated with two things: traffic and features configured. When a packet enters the device, an interrupt is thrown to tell the CPU that we have a packet in the Rx ring of the interface that needs to be serviced. Once the CPU is able to respond to this request, we go into an interrupt routine. Within this routine, we must perform a CEF look-up (on software-switched platforms) along with any feature processing that is configured on the interface. For that reason, increased traffic means increased interrupts and more features means more interrupt-level processing. This is likely not a problem with the configuration but is simply the box performing normal routines for the traffic flowing through it. Please let me know if you've seen sustained CPU at any given time as denoted by the '#' signs in the CPU history.

Hope this helps.

Brandon