Cisco 3745 high cpu usage

[IgorHamzic]

Hi. For some time on one of ours Cisco 3745 routers we have been having a very high CPU usage around 70%. I know this is very high and wondered if you could help me to find the cause. Below is the output from the sh proc cpu | exclude 0.00%__0.00%__0.00% command.

CPU utilization for five seconds: 60%/25%; one minute: 69%; five minutes: 72%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

4 22848424 2914987 7838 0.00% 0.05% 0.05% 0 Check heaps

22 3151572 24641326 127 0.00% 0.02% 0.00% 0 Per-Second Jobs

35 6275548 425586 14745 0.00% 0.01% 0.00% 0 Per-minute Jobs

47 3490448562875681874 0 0.49% 0.74% 0.66% 0 IP Input

73 357226642514347994 0 0.08% 0.04% 0.05% 0 Socket Timers

115 9046748 21118779 428 0.08% 0.03% 0.02% 0 SAA Event Proces

126 310612044 34928714 8892 32.42% 25.41% 26.10% 0 FRF9 manager

131 10119524 139770053 72 0.00% 0.02% 0.03% 0 IP-EIGRP Hello

138 11579950442514347994 0 1.39% 2.08% 2.15% 0 Rtt Responder

149 5092728 74037445 68 0.00% 0.01% 0.00% 0 IP-EIGRP Router

The router has 128 MBs of memory and IOS version c3745-is-mz.122-13.T1.bin.

Any help is greatly appreciated.

[Joseph W. Doherty]

Even with an increase of the input queue size, I would still expect you'll see drops. The question is whether the percentage of drops decreased.

Going back to the CPU issue, likely caused by frame-relay compression. If you can, disable it and see what happens to your CPU utilization.

[paolo bevilacqua]

Can you get a L3 switch to handle FE to FE traffic? There is an 8 port model of the 3560 that have an unsurpassed price / performance ratio.

The router would then be relieved of all issues with FE and have the resources to handle WAN with any feature you want on it.

hope this helps, please rate post if it does!

[IgorHamzic]

I did remove frame-relay compression on the serial sub-interface but didn't get any lower CPU usage. In fact I started seeing drops on the serial interface when I removed it but they stopped when I returned the compression.

I put the remote office doing most traffic on a different router and the rate of the drops decreased very much.

And did you mean I should put a L3 switch in front of the router and let it handle all of the remote offices and then pass on that information to the router?

[paolo bevilacqua]

Yes, use a L3 switch for LAN traffic and have the router use only a LAn interface and the serial ones. There are also L3 switches in form of network modules that you can use in the 3745, but a 3560-8 should be much cheaper.

[IgorHamzic]

So in effect I should have something like this:

remote offices---L3 switch---router

And the configuration something like this:

-on L3 switch IP addresses I now have on the subinterfaces of the router which will handle all of the metroethernet traffic to remote offices and the connection to the rest of the network

- on the router just the serial sub intefaces and connection to the L3 switch

Something like that?

[IgorHamzic]

Hi all. After a period of inactivity I have done some further test on the router in question.

I have managed to eliminate the drops on the interface but the problem with high CPU still persists.

Here is the output of the sh proc cpu | exclude 0.00%__0.00%__0.00% command:

CPU utilization for five seconds: 79%/77%; one minute: 82%; five minutes: 81%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

3 44740 2493 17946 0.00% 0.00% 0.39% 162 Virtual Exec

4 33567892 3562379 9422 0.00% 0.09% 0.06% 0 Check heaps

5 131252 448630 292 0.00% 0.06% 0.03% 0 Pool Manager

47 4052790723095525684 0 1.30% 0.47% 0.47% 0 IP Input

73 394495802726608548 0 0.00% 0.03% 0.00% 0 Socket Timers

115 13248812 22562505 587 0.16% 0.04% 0.03% 0 SAA Event Proces

131 14046112 164918057 85 0.08% 0.03% 0.02% 0 IP-EIGRP Hello

138 13506041962726608548 0 0.40% 1.87% 1.90% 0 Rtt Responder

141 2287760 282632683 8 0.08% 0.00% 0.00% 0 fastblk backgrou

I'm losing this battle as I have removed the highest using office from this router, eliminated the drops but there are still times when the processor maxed out due to interrupts as you can see from the show command.

IP CEF is enabled on all fastethernet interfaces and their associated subifs. I'm running out of ideas on this one.

BTW good advice in previous posts.

[paolo bevilacqua]

Hi,

what is the traffic volume over all interfaces when you take the show proc cpu ?

Considering that the 3745 is rated for a max of 225 Kpps (see attached), at 80% cpu you could be around 150 Kpps, nothing out of ordinary for sustained LAN to LAN traffic.

Because router performance limitations, the suggestion of using a L3 switch for inter-VLAN routing.

[paolo bevilacqua]

Missing attachment.

Hope this helps, please rate post if it does!

[IgorHamzic]

The PDF was really helpful but even when there is a high CPU usage on the router I can only see about 180 packets/second in input and about 174 packets/packet in output when I hit show interface command. Doesn't seem nowhere close to 150000 packets/second you mentioned.

[paolo bevilacqua]

Ok, if you are positive about the low traffic, then is something else.

Unfortunately it is difficult to diagnose what.

I've seen routers spike CPU but most of the time it was due to some rogue traffic that could be "seen" with regular show commands. Please check again the router's counter against the ones of the connected switch. There is a small chances that some high traffic is not being counted by the routers.

Also if possible at all, could you reload the router while it exhibits high CPU? If when it comes back with low CPU, that could point to some kind of strange bug.

Going forward, ultimately you might need to 'span' a port from the switch to a network analyzer like wireshark, to find out what is really going on there.

Thanks again for the nice rating and good luck!

[IgorHamzic]

I have plugged in wireshark on the switch in front of the router and for now have seen a lot of UDP traffic between different routers with source ports above 50000 and destination ports around 14000 and vice versa.

Any idea what these might be?

[paolo bevilacqua]

Easily that is some kind of P2P. Any more detail on the packet ? Can you confirm you see source and destination address same as routers, that seems strange.

[IgorHamzic]

Yes it's the addresses of my routers and my on central location and one of my routers on the remote location plus I see the address of the switch on the remote location as the router is configured as a router on a stick with the ISP link first going in the router and then from switch into the router.

It strikes me as really odd that routers and switches should be talking to each other using UDP and such high ports.

About the packet what would you like to know so I can copy paste it from Wireshark?

[Joseph W. Doherty]

Enough traffic will load down the interrupt CPU %. Can you estimate the total traffic flow going through the box?

Otherwise, the delta of 2% looks great between the total CPU and interrupt CPU.

[IgorHamzic]

I think I might have found the guilty party for the problem.

I have found the following on a router:

rtr responder

rtr responder type udpEcho port 14400

rtr responder type udpEcho port 14401

rtr responder type udpEcho port 14402

rtr responder type udpEcho port 14403

rtr 17

type jitter dest-ipaddr x.x.x.x dest-port 14388 num-packets 50

request-data-size 172

frequency 12

hours-of-statistics-kept 25

rtr schedule 17 start-time now life forever

rtr 21

type jitter dest-ipaddr y.y.y.y dest-port 14402 num-packets 3000

request-data-size 172

frequency 70

The UDP ports seem to match as far as I can tell(it's past 3 AM here). I'm not familiar with these commands but as far as I can tell they send UDP packets with specific ports to measure jitter. The ip addresses match the IP addresses of my remote office router and switch. There are similar configurations on the router and switch in the remote office. Could this be the problem behind high cpu usage?