Extreme High CPU peaks

mkoopsen · ‎08-06-2003

Hello,

In our network we have a 7507 acting as a border router.

Two weeks ago the CPU from time to time goes skyhigh for some time (between 1 and 60 minutes). The only other thing we see at that moment is a slight increase of traffic going to the servers behind the router. the increase is only 3 to 4 Mbit.

I have been breaking my head over this problem and did extensive testing but i can not find a reason why the CPU goes skyhigh.

Is there somebody who has seen this problem and knows how to fix it?

Regards,

Martijn Koopsen

P.S. if you want some extra information, please let me know.

hbaerten · ‎08-07-2003

Some thoughts:

- could be a DoS attack; are you seeing anything suspicious, e.g. do you have acl's that get hit massively?

Two weeks ago, that's around the time the Cisco and MS vulnerabilities were announced? Could be coincidence.

- is the router doing encryption, compression, PBR without CEF, OSPF, other CPU intensive things?

- which processes are using the CPU? i.e. do a "show proc cpu"

regards

Herbert

mkoopsen · ‎08-07-2003

Herbert,

I am also afraid it is a DoS but i can not find any evidence. Looking at the hits on acl's i don't see massive hits.

I know about the vulnerability but according to the documentation our IO is not vulnerable. Also i do not see any hits on the acl mentioned in the workaround.

I have CEF configured and it does not do any strange things other than BGP routing.

The proces that consumes the most cpu is IP Input.

It is making my heard spin around.

spremkumar · ‎08-07-2003

hi

we hv faced the same probs with one of our 7513,few weeks before.chcek the bgp summary status when the utilisation is high.

we were getting huge amount of prefixes which the router wasnt able to process which made the cpu to reach its peak even to 100%.

After sorting out this BGP prefix issue we hve seen our CPU reaching 40 to 55 % all the time which got sorted out after configuring CEF(recomended by the vendor) .

Regds

Prem

ruwhite · ‎08-07-2003

Yes, we need extra information. :-)

First, we need to know if this is a process level problem, or an interrupt level problem. Do a show prc cpu, and look at the first couple of lines. One of those will say the proc util is x%/y%. What are those two numbers? If the first number is high, and the second low, then you need to look through the list of processes following, and note the top 5 or 6 processed in terms of cpu time. Post the information about the top 5 cpu users, if you find this to be the case.

If the second number is high as well as the first number, the processor is busy in interrupt context, which only comes down to two things: packet switching and error correction. Take a look at show align, and see if you see a _lot_ of alignment errors or spurious accesses. If you see a _lot_ of either of these, then post that output. If not, then you're just switching a lot of traffic through the router.... Turning on dcef, if the box is capable of doing so (and all of the line cards) might help.

Russ.W

mkoopsen · ‎08-07-2003

All,

At this moment the router is doing his job quit well. No problems at the time. As soon as the CPU is going through the roof i will take a look and let you know.

Regards,

Martijn

rjrech · ‎08-08-2003

Are the border routers taking a large number of bgp routes? If so and you have an SNMP manager pulling the routing table {such as OpenView} it will cause the router to try to sort all the BGP routes and send them back to answer the SNMP query. You may want to create an SNMP view to allow the manager to pull the cpu, buffer, memory and interface stats but not the routing table. This will definetly bury a router that is holding an internet routing table

ruwhite · ‎08-09-2003

If you post more information, send me an email so I'll make certain I look at it. I don't normally see the architecture questions through my filters.

Russ.W

riw@cisco.com