cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2364
Views
35
Helpful
35
Replies

Cisco 3745 high cpu usage

IgorHamzic
Level 1
Level 1

Hi. For some time on one of ours Cisco 3745 routers we have been having a very high CPU usage around 70%. I know this is very high and wondered if you could help me to find the cause. Below is the output from the sh proc cpu | exclude 0.00%__0.00%__0.00% command.

CPU utilization for five seconds: 60%/25%; one minute: 69%; five minutes: 72%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

4 22848424 2914987 7838 0.00% 0.05% 0.05% 0 Check heaps

22 3151572 24641326 127 0.00% 0.02% 0.00% 0 Per-Second Jobs

35 6275548 425586 14745 0.00% 0.01% 0.00% 0 Per-minute Jobs

47 3490448562875681874 0 0.49% 0.74% 0.66% 0 IP Input

73 357226642514347994 0 0.08% 0.04% 0.05% 0 Socket Timers

115 9046748 21118779 428 0.08% 0.03% 0.02% 0 SAA Event Proces

126 310612044 34928714 8892 32.42% 25.41% 26.10% 0 FRF9 manager

131 10119524 139770053 72 0.00% 0.02% 0.03% 0 IP-EIGRP Hello

138 11579950442514347994 0 1.39% 2.08% 2.15% 0 Rtt Responder

149 5092728 74037445 68 0.00% 0.01% 0.00% 0 IP-EIGRP Router

The router has 128 MBs of memory and IOS version c3745-is-mz.122-13.T1.bin.

Any help is greatly appreciated.

1 Accepted Solution

Accepted Solutions

Quite a difference, especially if it holds!

If no one is actually using RTR/SAA, deactivation might make sense. Or, check the parameters and have the tests sample much, much less frequently.

View solution in original post

35 Replies 35

paolo bevilacqua
Hall of Fame
Hall of Fame

Hi,

verify that "ip cef" is configured.

Send output of "show interface summary" if it happens again.

purohit_810
Level 5
Level 5

" 126 310612044 34928714 8892 32.42% 25.41% 26.10% 0 FRF9 manager "

Do you have configure QOS ??? Is it fuctioning properly?

Second, Put sniffer and checkout top ten talkers.

Regards,

Dharmesh Purohit

Joseph W. Doherty
Hall of Fame
Hall of Fame

"126 310612044 34928714 8892 32.42% 25.41% 26.10% 0 FRF9 manager"

Are you doing frame-relay payload compression? If so, and if the compression is being done in software, as it might be in this case, expect it to consume CPU especially as the actual data rate increases.

If you want to eliminate this CPU consumption, and if it is caused by frame-relay payload compression, can stop doing the compression or some hardware modules will off-load some types of compression. (Without research, don't know if any are available or would do so for 3745.)

Hi.Sorry for the late reply but I've been out of action so to speak since I last posted. So here is more information:

1. ip cef is configured

2. QoS is configured for some frame-relay sub-ifs, configured by previous administrator and never had any problems with them

3. frame relay compression is enabled for 1 frame relay sub-if

Output of show interface summary:

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

---------------------------------------------------------------------

* FastEthernet0/0 0 505245 0 0 3466000 498 949000 448 0

* Serial0/0 0 688 0 2753634 15000 51 0 0 0

* FastEthernet0/1 0 438139 0 0 1114000 480 2472000 524 0

* Virtual-Access1 0 0 0 0 0 0 0 0 0

But something Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

---------------------------------------------------------------------

* FastEthernet0/0 0 505245 0 0 3466000 498 949000 448 0

* Serial0/0 0 688 0 2753634 15000 51 0 0 0

* FastEthernet0/1 0 438139 0 0 1114000 480 2472000 524 0

* Virtual-Access1 0 0 0 0 0 0 0 0 0

IQD seems very high on serial and fastethernet interfaces.

But I've been thinking of something else also. On the same router we have several subifs on Fastethernet 0/1 interface that we use to connect to several remote locations. Could the problem be in multiple subifs on FastEthernet and Serial(for frame relay) interfaces. There are 7 subifs on each interface.Is that perhaps too much subinterfaces?

paolo bevilacqua
Hall of Fame
Hall of Fame

Please send output of "show interface summary" at the time high CPU occurs.

Here it is. The one here is about 15 minutes newer than the one above.

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

---------------------------------------------------------------------

* FastEthernet0/0 1 505245 0 0 3530000 534 1283000 504 0

* Serial0/0 0 688 0 2753634 14000 51 1000 2 0

* FastEthernet0/1 35 439826 0 0 2249000 631 1948000 590 0

* Virtual-Access1 0 0 0 0 0 0 0 0 0

The high processor time is almost constant through the day but it doesn't seem to affect network performance at this moment.

Hi,

doesn't seem like you have a very high traffic but you are experiencing many input drops. S0/0 appears to be severely congested with only 14 kbps on output. Do you have FR shaping on it ?

This router needs to be looked interactively, something doesn't seem right.

Do you have CEF enabled ?

Yes ip cef is enabled. The serial interface isn't active as much. It's main purpose is as backup to some remote sites(that we usually have access to over fastethernet sub interfaces) over frame relay and it will be phased out in the near future.

Ok, then it's a FE to FE issue. Please monitor if drops are increasing, it can be a buffers issue.

I have been following the situation for some time with following results. I have entered the show interface summary command and have seen that sometimes when there are many packets in input queue FastEthernet 0/1 that there are dropped packets. Here are the outputs of 3 consecutive show interface summary commands:

*: interface is up

IHQ: pkts in input hold queue IQD: pkts dropped from input queue

OHQ: pkts in output hold queue OQD: pkts dropped from output queue

RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec)

TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec)

TRTL: throttle count

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

---------------------------------------------------------------------

* FastEthernet0/0 2 505245 0 0 1140000 153 715000 137 0

* Serial0/0 3 688 0 2753634 14000 51 0 0 0

* FastEthernet0/1 76 471420 0 0 406000 248 2855000 393 0

* Virtual-Access1 0 0 0 0 0 0 0 0 0

NOTE:No separate counters are maintained for subinterfaces

Hence Details of subinterface are not shown

*: interface is up

IHQ: pkts in input hold queue IQD: pkts dropped from input queue

OHQ: pkts in output hold queue OQD: pkts dropped from output queue

RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec)

TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec)

TRTL: throttle count

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

---------------------------------------------------------------------

* FastEthernet0/0 1 505245 0 0 1330000 171 708000 145 0

* Serial0/0 0 688 0 2753634 15000 52 0 0 0

* FastEthernet0/1 18 471429 0 0 428000 313 4353000 529 0

* Virtual-Access1 0 0 0 0 0 0 0 0 0

NOTE:No separate counters are maintained for subinterfaces

Hence Details of subinterface are not shown

*: interface is up

IHQ: pkts in input hold queue IQD: pkts dropped from input queue

OHQ: pkts in output hold queue OQD: pkts dropped from output queue

RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec)

TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec)

TRTL: throttle count

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

---------------------------------------------------------------------

* FastEthernet0/0 0 505245 0 0 1330000 171 708000 145 0

* Serial0/0 1 688 0 2753634 15000 52 0 0 0

* FastEthernet0/1 4 471433 0 0 428000 313 4353000 529 0

* Virtual-Access1 0 0 0 0 0 0 0 0 0

NOTE:No separate counters are maintained for subinterfaces

Hence Details of subinterface are not shown

Any advice if it is a buffers issue? What could I check to confirm it?

You could review "Troubleshooting Input Queue Drops and Output Queue Drops", http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml#topic2

You might also try increasing (e.g. doubling) your ethernet inbound queue sizes.

One other thing I noticed. I have been pinging the router at random intervals and noticed that sometime the ping reply is around 300 ms and that drops happen on FE0/1 at those times.

Could that be also caused by queue size?

I'll try to double the input size to 150 with the hold-queue command as the default for the interface is 75 and see if it will resolve anything.

Possibly the delay in pings is caused by the box being busy doing whatever also is causing the input queue to fill. Routers respond to pings when they get around to it.

Increasing any FIFO queue also adds latency. So we're trying to reduce drops without dramatically increasing latency. Increasing the queue size by 75 would add roughly about 9 or 10 ms, for 1500 byte size packets at 100 Mbps. So, best to try a bit and see what happens.

Sorry for a very late reply. I did increase the input queue by 75 but I'm still seeing input queue drops. In fact the CPU usage has gone up about 3 percent in this time also with no configuration changes.

And from time to time I also see this error in the logs: %ERROR: delta cannot be less than 0.

I think I may know what the problem is though.The regional offices are generating more and more traffic toward our central site.I'll try to split them over 2 routers and see what happens.

If you have anymore advice I'll try them before splitting the traffic so please post them.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco