Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

Cisco 3745 high cpu usage

Hi. For some time on one of ours Cisco 3745 routers we have been having a very high CPU usage around 70%. I know this is very high and wondered if you could help me to find the cause. Below is the output from the sh proc cpu | exclude 0.00%__0.00%__0.00% command.

CPU utilization for five seconds: 60%/25%; one minute: 69%; five minutes: 72%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

4 22848424 2914987 7838 0.00% 0.05% 0.05% 0 Check heaps

22 3151572 24641326 127 0.00% 0.02% 0.00% 0 Per-Second Jobs

35 6275548 425586 14745 0.00% 0.01% 0.00% 0 Per-minute Jobs

47 3490448562875681874 0 0.49% 0.74% 0.66% 0 IP Input

73 357226642514347994 0 0.08% 0.04% 0.05% 0 Socket Timers

115 9046748 21118779 428 0.08% 0.03% 0.02% 0 SAA Event Proces

126 310612044 34928714 8892 32.42% 25.41% 26.10% 0 FRF9 manager

131 10119524 139770053 72 0.00% 0.02% 0.03% 0 IP-EIGRP Hello

138 11579950442514347994 0 1.39% 2.08% 2.15% 0 Rtt Responder

149 5092728 74037445 68 0.00% 0.01% 0.00% 0 IP-EIGRP Router

The router has 128 MBs of memory and IOS version c3745-is-mz.122-13.T1.bin.

Any help is greatly appreciated.

1 ACCEPTED SOLUTION

Accepted Solutions
Super Bronze

Re: Cisco 3745 high cpu usage

Quite a difference, especially if it holds!

If no one is actually using RTR/SAA, deactivation might make sense. Or, check the parameters and have the tests sample much, much less frequently.

35 REPLIES
Hall of Fame Super Gold

Re: Cisco 3745 high cpu usage

Hi,

verify that "ip cef" is configured.

Send output of "show interface summary" if it happens again.

Silver

Re: Cisco 3745 high cpu usage

" 126 310612044 34928714 8892 32.42% 25.41% 26.10% 0 FRF9 manager "

Do you have configure QOS ??? Is it fuctioning properly?

Second, Put sniffer and checkout top ten talkers.

Regards,

Dharmesh Purohit

Super Bronze

Re: Cisco 3745 high cpu usage

"126 310612044 34928714 8892 32.42% 25.41% 26.10% 0 FRF9 manager"

Are you doing frame-relay payload compression? If so, and if the compression is being done in software, as it might be in this case, expect it to consume CPU especially as the actual data rate increases.

If you want to eliminate this CPU consumption, and if it is caused by frame-relay payload compression, can stop doing the compression or some hardware modules will off-load some types of compression. (Without research, don't know if any are available or would do so for 3745.)

New Member

Re: Cisco 3745 high cpu usage

Hi.Sorry for the late reply but I've been out of action so to speak since I last posted. So here is more information:

1. ip cef is configured

2. QoS is configured for some frame-relay sub-ifs, configured by previous administrator and never had any problems with them

3. frame relay compression is enabled for 1 frame relay sub-if

Output of show interface summary:

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

---------------------------------------------------------------------

* FastEthernet0/0 0 505245 0 0 3466000 498 949000 448 0

* Serial0/0 0 688 0 2753634 15000 51 0 0 0

* FastEthernet0/1 0 438139 0 0 1114000 480 2472000 524 0

* Virtual-Access1 0 0 0 0 0 0 0 0 0

But something Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

---------------------------------------------------------------------

* FastEthernet0/0 0 505245 0 0 3466000 498 949000 448 0

* Serial0/0 0 688 0 2753634 15000 51 0 0 0

* FastEthernet0/1 0 438139 0 0 1114000 480 2472000 524 0

* Virtual-Access1 0 0 0 0 0 0 0 0 0

IQD seems very high on serial and fastethernet interfaces.

But I've been thinking of something else also. On the same router we have several subifs on Fastethernet 0/1 interface that we use to connect to several remote locations. Could the problem be in multiple subifs on FastEthernet and Serial(for frame relay) interfaces. There are 7 subifs on each interface.Is that perhaps too much subinterfaces?

Hall of Fame Super Gold

Re: Cisco 3745 high cpu usage

Please send output of "show interface summary" at the time high CPU occurs.

New Member

Re: Cisco 3745 high cpu usage

Here it is. The one here is about 15 minutes newer than the one above.

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

---------------------------------------------------------------------

* FastEthernet0/0 1 505245 0 0 3530000 534 1283000 504 0

* Serial0/0 0 688 0 2753634 14000 51 1000 2 0

* FastEthernet0/1 35 439826 0 0 2249000 631 1948000 590 0

* Virtual-Access1 0 0 0 0 0 0 0 0 0

The high processor time is almost constant through the day but it doesn't seem to affect network performance at this moment.

Hall of Fame Super Gold

Re: Cisco 3745 high cpu usage

Hi,

doesn't seem like you have a very high traffic but you are experiencing many input drops. S0/0 appears to be severely congested with only 14 kbps on output. Do you have FR shaping on it ?

This router needs to be looked interactively, something doesn't seem right.

Do you have CEF enabled ?

New Member

Re: Cisco 3745 high cpu usage

Yes ip cef is enabled. The serial interface isn't active as much. It's main purpose is as backup to some remote sites(that we usually have access to over fastethernet sub interfaces) over frame relay and it will be phased out in the near future.

Hall of Fame Super Gold

Re: Cisco 3745 high cpu usage

Ok, then it's a FE to FE issue. Please monitor if drops are increasing, it can be a buffers issue.

New Member

Re: Cisco 3745 high cpu usage

I have been following the situation for some time with following results. I have entered the show interface summary command and have seen that sometimes when there are many packets in input queue FastEthernet 0/1 that there are dropped packets. Here are the outputs of 3 consecutive show interface summary commands:

*: interface is up

IHQ: pkts in input hold queue IQD: pkts dropped from input queue

OHQ: pkts in output hold queue OQD: pkts dropped from output queue

RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec)

TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec)

TRTL: throttle count

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

---------------------------------------------------------------------

* FastEthernet0/0 2 505245 0 0 1140000 153 715000 137 0

* Serial0/0 3 688 0 2753634 14000 51 0 0 0

* FastEthernet0/1 76 471420 0 0 406000 248 2855000 393 0

* Virtual-Access1 0 0 0 0 0 0 0 0 0

NOTE:No separate counters are maintained for subinterfaces

Hence Details of subinterface are not shown

*: interface is up

IHQ: pkts in input hold queue IQD: pkts dropped from input queue

OHQ: pkts in output hold queue OQD: pkts dropped from output queue

RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec)

TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec)

TRTL: throttle count

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

---------------------------------------------------------------------

* FastEthernet0/0 1 505245 0 0 1330000 171 708000 145 0

* Serial0/0 0 688 0 2753634 15000 52 0 0 0

* FastEthernet0/1 18 471429 0 0 428000 313 4353000 529 0

* Virtual-Access1 0 0 0 0 0 0 0 0 0

NOTE:No separate counters are maintained for subinterfaces

Hence Details of subinterface are not shown

*: interface is up

IHQ: pkts in input hold queue IQD: pkts dropped from input queue

OHQ: pkts in output hold queue OQD: pkts dropped from output queue

RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec)

TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec)

TRTL: throttle count

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

---------------------------------------------------------------------

* FastEthernet0/0 0 505245 0 0 1330000 171 708000 145 0

* Serial0/0 1 688 0 2753634 15000 52 0 0 0

* FastEthernet0/1 4 471433 0 0 428000 313 4353000 529 0

* Virtual-Access1 0 0 0 0 0 0 0 0 0

NOTE:No separate counters are maintained for subinterfaces

Hence Details of subinterface are not shown

Any advice if it is a buffers issue? What could I check to confirm it?

Super Bronze

Re: Cisco 3745 high cpu usage

You could review "Troubleshooting Input Queue Drops and Output Queue Drops", http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml#topic2

You might also try increasing (e.g. doubling) your ethernet inbound queue sizes.

New Member

Re: Cisco 3745 high cpu usage

One other thing I noticed. I have been pinging the router at random intervals and noticed that sometime the ping reply is around 300 ms and that drops happen on FE0/1 at those times.

Could that be also caused by queue size?

I'll try to double the input size to 150 with the hold-queue command as the default for the interface is 75 and see if it will resolve anything.

Super Bronze

Re: Cisco 3745 high cpu usage

Possibly the delay in pings is caused by the box being busy doing whatever also is causing the input queue to fill. Routers respond to pings when they get around to it.

Increasing any FIFO queue also adds latency. So we're trying to reduce drops without dramatically increasing latency. Increasing the queue size by 75 would add roughly about 9 or 10 ms, for 1500 byte size packets at 100 Mbps. So, best to try a bit and see what happens.

New Member

Re: Cisco 3745 high cpu usage

Sorry for a very late reply. I did increase the input queue by 75 but I'm still seeing input queue drops. In fact the CPU usage has gone up about 3 percent in this time also with no configuration changes.

And from time to time I also see this error in the logs: %ERROR: delta cannot be less than 0.

I think I may know what the problem is though.The regional offices are generating more and more traffic toward our central site.I'll try to split them over 2 routers and see what happens.

If you have anymore advice I'll try them before splitting the traffic so please post them.

Super Bronze

Re: Cisco 3745 high cpu usage

Even with an increase of the input queue size, I would still expect you'll see drops. The question is whether the percentage of drops decreased.

Going back to the CPU issue, likely caused by frame-relay compression. If you can, disable it and see what happens to your CPU utilization.

Hall of Fame Super Gold

Re: Cisco 3745 high cpu usage

Can you get a L3 switch to handle FE to FE traffic? There is an 8 port model of the 3560 that have an unsurpassed price / performance ratio.

The router would then be relieved of all issues with FE and have the resources to handle WAN with any feature you want on it.

hope this helps, please rate post if it does!

New Member

Re: Cisco 3745 high cpu usage

I did remove frame-relay compression on the serial sub-interface but didn't get any lower CPU usage. In fact I started seeing drops on the serial interface when I removed it but they stopped when I returned the compression.

I put the remote office doing most traffic on a different router and the rate of the drops decreased very much.

And did you mean I should put a L3 switch in front of the router and let it handle all of the remote offices and then pass on that information to the router?

Hall of Fame Super Gold

Re: Cisco 3745 high cpu usage

Yes, use a L3 switch for LAN traffic and have the router use only a LAn interface and the serial ones. There are also L3 switches in form of network modules that you can use in the 3745, but a 3560-8 should be much cheaper.

New Member

Re: Cisco 3745 high cpu usage

So in effect I should have something like this:

remote offices---L3 switch---router

And the configuration something like this:

-on L3 switch IP addresses I now have on the subinterfaces of the router which will handle all of the metroethernet traffic to remote offices and the connection to the rest of the network

- on the router just the serial sub intefaces and connection to the L3 switch

Something like that?

New Member

Re: Cisco 3745 high cpu usage

Hi all. After a period of inactivity I have done some further test on the router in question.

I have managed to eliminate the drops on the interface but the problem with high CPU still persists.

Here is the output of the sh proc cpu | exclude 0.00%__0.00%__0.00% command:

CPU utilization for five seconds: 79%/77%; one minute: 82%; five minutes: 81%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

3 44740 2493 17946 0.00% 0.00% 0.39% 162 Virtual Exec

4 33567892 3562379 9422 0.00% 0.09% 0.06% 0 Check heaps

5 131252 448630 292 0.00% 0.06% 0.03% 0 Pool Manager

47 4052790723095525684 0 1.30% 0.47% 0.47% 0 IP Input

73 394495802726608548 0 0.00% 0.03% 0.00% 0 Socket Timers

115 13248812 22562505 587 0.16% 0.04% 0.03% 0 SAA Event Proces

131 14046112 164918057 85 0.08% 0.03% 0.02% 0 IP-EIGRP Hello

138 13506041962726608548 0 0.40% 1.87% 1.90% 0 Rtt Responder

141 2287760 282632683 8 0.08% 0.00% 0.00% 0 fastblk backgrou

I'm losing this battle as I have removed the highest using office from this router, eliminated the drops but there are still times when the processor maxed out due to interrupts as you can see from the show command.

IP CEF is enabled on all fastethernet interfaces and their associated subifs. I'm running out of ideas on this one.

BTW good advice in previous posts.

Hall of Fame Super Gold

Re: Cisco 3745 high cpu usage

Hi,

what is the traffic volume over all interfaces when you take the show proc cpu ?

Considering that the 3745 is rated for a max of 225 Kpps (see attached), at 80% cpu you could be around 150 Kpps, nothing out of ordinary for sustained LAN to LAN traffic.

Because router performance limitations, the suggestion of using a L3 switch for inter-VLAN routing.

Hall of Fame Super Gold

Re: Cisco 3745 high cpu usage

Missing attachment.

Hope this helps, please rate post if it does!

New Member

Re: Cisco 3745 high cpu usage

The PDF was really helpful but even when there is a high CPU usage on the router I can only see about 180 packets/second in input and about 174 packets/packet in output when I hit show interface command. Doesn't seem nowhere close to 150000 packets/second you mentioned.

Hall of Fame Super Gold

Re: Cisco 3745 high cpu usage

Ok, if you are positive about the low traffic, then is something else.

Unfortunately it is difficult to diagnose what.

I've seen routers spike CPU but most of the time it was due to some rogue traffic that could be "seen" with regular show commands. Please check again the router's counter against the ones of the connected switch. There is a small chances that some high traffic is not being counted by the routers.

Also if possible at all, could you reload the router while it exhibits high CPU? If when it comes back with low CPU, that could point to some kind of strange bug.

Going forward, ultimately you might need to 'span' a port from the switch to a network analyzer like wireshark, to find out what is really going on there.

Thanks again for the nice rating and good luck!

New Member

Re: Cisco 3745 high cpu usage

I have plugged in wireshark on the switch in front of the router and for now have seen a lot of UDP traffic between different routers with source ports above 50000 and destination ports around 14000 and vice versa.

Any idea what these might be?

Hall of Fame Super Gold

Re: Cisco 3745 high cpu usage

Easily that is some kind of P2P. Any more detail on the packet ? Can you confirm you see source and destination address same as routers, that seems strange.

New Member

Re: Cisco 3745 high cpu usage

Yes it's the addresses of my routers and my on central location and one of my routers on the remote location plus I see the address of the switch on the remote location as the router is configured as a router on a stick with the ISP link first going in the router and then from switch into the router.

It strikes me as really odd that routers and switches should be talking to each other using UDP and such high ports.

About the packet what would you like to know so I can copy paste it from Wireshark?

Super Bronze

Re: Cisco 3745 high cpu usage

Enough traffic will load down the interrupt CPU %. Can you estimate the total traffic flow going through the box?

Otherwise, the delta of 2% looks great between the total CPU and interrupt CPU.

New Member

Re: Cisco 3745 high cpu usage

I think I might have found the guilty party for the problem.

I have found the following on a router:

rtr responder

rtr responder type udpEcho port 14400

rtr responder type udpEcho port 14401

rtr responder type udpEcho port 14402

rtr responder type udpEcho port 14403

rtr 17

type jitter dest-ipaddr x.x.x.x dest-port 14388 num-packets 50

request-data-size 172

frequency 12

hours-of-statistics-kept 25

rtr schedule 17 start-time now life forever

rtr 21

type jitter dest-ipaddr y.y.y.y dest-port 14402 num-packets 3000

request-data-size 172

frequency 70

The UDP ports seem to match as far as I can tell(it's past 3 AM here). I'm not familiar with these commands but as far as I can tell they send UDP packets with specific ports to measure jitter. The ip addresses match the IP addresses of my remote office router and switch. There are similar configurations on the router and switch in the remote office. Could this be the problem behind high cpu usage?

1051
Views
35
Helpful
35
Replies