Router CPU and Mem pegged 98% CPU

jburk · ‎12-31-2008

I have a 2801SEC/K9 running 12.4.22T Advanced Security code. I have a 3 T1 multilink configured. It was working fine until the other day I start getting calls about the internet being down, so I send a tech over and he reboots it and it is fixed for a while, then the problem comes back.

So I look at it and the CPU and Memory are pegged, lots of (fragment) processes and the biggest user of CPU is the "IP Input" process.

So I thought it was a router problem and replaced the 2801 with a 3620 I had that was running Firewall code. It too has the same problem - CPU maxes out quickly, even when I only inspect one protocol, like http, outbound.

If I shut down the multilink interface, the CPU usage drops to like 20% immediately.

Also, when the 2801 CPU gets maxed, I keep getting (%LINK3-UPDOWN - Interface FastEthernet 0/1 state changed to up) repeatedly every few seconds.

Anyone know how to troubleshoot this? Is it a worm or DoS attack?

ohassairi · ‎12-31-2008

try to capture the traffic on multilink port using a sniffer. then you can see what type of traffic is it (legal or illegal)

then you can use access-lists to filter unwanted traffic. or may be you will discover a virus some where....

jburk · ‎12-31-2008

The multilink is 3 T1's - how do I sniff that?

I have a deny any any statement at the end of my inbound ACL

I am also seeing this message

%FW-4-ALERT_ON: getting aggressive, count (6/500) current 1-min rate: 501

jburk · ‎01-01-2009

here is some of the memory error

*Dec 30 03:53:19.009: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for NA

T Port Range. No memory available

-Process= "Chunk Manager", ipl= 3, pid= 1

-Traceback= 0x6091F438 0x602187B4

*Dec 30 03:53:24.637: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed

from 0x602197E4, alignment 8

Pool: Processor Free: 184804 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 58

-Traceback= 0x6091F438 0x601FBB88 0x60200C98 0x602197EC 0x60218A54 0x6021AF70 0x

602188F8 0x60F09FD0 0x60F0B2F8 0x60F0BE3C 0x60F0E074 0x60F09588 0x60D3EEC8 0x60D

3D654 0x60D3D924 0x60D3D9E0

*Dec 30 03:53:29.017: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for NA

T Port Range. No memory available

-Process= "Chunk Manager", ipl= 3, pid= 1

-Traceback= 0x6091F438 0x602187B4

System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

PLD version 0x10

GIO ASIC version 0x127

c2801 platform with 262144 Kbytes of main memory

Main memory is configured to 64 bit mode with parity disabled

Goutam Sanyal · ‎01-02-2009

Hello,

As I understand as per you output of error massage please find the bellow (as per Cisco)

%FW-4-ALERT_ON: [chars], count ([dec]/[dec]) current 1-min rate: [dec]

Either the max-incomplete high threshold of half-open connections or the new connection initiation rate has been exceeded. This error message indicates that an unusually high rate of new connections is coming through the firewall, and a DOS attack may be in progress. This message is issued only when the max-incomplete high threshold is crossed.

Recommended Action: This message is for informational purposed only, but it may indicate a security problem.

Related documents- No specific documents apply to this error message.

%SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for [chars]. No memory available

There is not enough processor memory left to expand this chunk pool.

Recommended Action: Copy the error message exactly as it appears on the console or in the system log, contact your Cisco technical support representative, and provide the representative with the gathered information.