Router CPU and Mem pegged 98% CPU

Unanswered Question

I have a 2801SEC/K9 running 12.4.22T Advanced Security code. I have a 3 T1 multilink configured. It was working fine until the other day I start getting calls about the internet being down, so I send a tech over and he reboots it and it is fixed for a while, then the problem comes back.

So I look at it and the CPU and Memory are pegged, lots of (fragment) processes and the biggest user of CPU is the "IP Input" process.

So I thought it was a router problem and replaced the 2801 with a 3620 I had that was running Firewall code. It too has the same problem - CPU maxes out quickly, even when I only inspect one protocol, like http, outbound.

If I shut down the multilink interface, the CPU usage drops to like 20% immediately.

Also, when the 2801 CPU gets maxed, I keep getting (%LINK3-UPDOWN - Interface FastEthernet 0/1 state changed to up) repeatedly every few seconds.

Anyone know how to troubleshoot this? Is it a worm or DoS attack?

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
ohassairi Wed, 12/31/2008 - 21:49

try to capture the traffic on multilink port using a sniffer. then you can see what type of traffic is it (legal or illegal)

then you can use access-lists to filter unwanted traffic. or may be you will discover a virus some where....

here is some of the memory error

*Dec 30 03:53:19.009: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for NA

T Port Range. No memory available

-Process= "Chunk Manager", ipl= 3, pid= 1

-Traceback= 0x6091F438 0x602187B4

*Dec 30 03:53:24.637: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed

from 0x602197E4, alignment 8

Pool: Processor Free: 184804 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 58

-Traceback= 0x6091F438 0x601FBB88 0x60200C98 0x602197EC 0x60218A54 0x6021AF70 0x

602188F8 0x60F09FD0 0x60F0B2F8 0x60F0BE3C 0x60F0E074 0x60F09588 0x60D3EEC8 0x60D

3D654 0x60D3D924 0x60D3D9E0

*Dec 30 03:53:29.017: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for NA

T Port Range. No memory available

-Process= "Chunk Manager", ipl= 3, pid= 1

-Traceback= 0x6091F438 0x602187B4

System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 2006 by cisco Systems, Inc.

PLD version 0x10

GIO ASIC version 0x127

c2801 platform with 262144 Kbytes of main memory

Main memory is configured to 64 bit mode with parity disabled

Goutam Sanyal Fri, 01/02/2009 - 00:19

Hello,

As I understand as per you output of error massage please find the bellow (as per Cisco)

%FW-4-ALERT_ON: [chars], count ([dec]/[dec]) current 1-min rate: [dec]

Either the max-incomplete high threshold of half-open connections or the new connection initiation rate has been exceeded. This error message indicates that an unusually high rate of new connections is coming through the firewall, and a DOS attack may be in progress. This message is issued only when the max-incomplete high threshold is crossed.

Recommended Action: This message is for informational purposed only, but it may indicate a security problem.

Related documents- No specific documents apply to this error message.

%SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for [chars]. No memory available

There is not enough processor memory left to expand this chunk pool.

Recommended Action: Copy the error message exactly as it appears on the console or in the system log, contact your Cisco technical support representative, and provide the representative with the gathered information.

Related documents-

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a6f3a.shtml

%SYS-2-MALLOCFAIL: Memory allocation of [dec] bytes failed from [hex], alignment [dec] \nPool: [chars] Free: [dec] Cause: [chars] Alternate Pool: [chars] Free: [dec] Cause: [chars] \n

The requested memory allocation is not available from the specified memory pool. The router memory has been exhausted or fragmented. This condition may be caused by the current system configuration, the network environment, or a software error.

Recommended Action: Check the minimum memory requirements for your system configuration. If your system meets those requirements, this condition is probably caused by a software failure. To take advantage of recent fixes, upgrade your system to the latest Cisco IOS software release in your release train.

Or can visit

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a6f3a.shtml

http://www.cisco.com/en/US/products/hw/iad/ps397/products_tech_note09186a00800a7b85.shtml

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a0080110d68.shtml

%SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for [chars]. No memory available

There is not enough processor memory left to expand this chunk pool.

Recommended Action: Copy the error message exactly as it appears on the console or in the system log, contact your Cisco technical support representative, and provide the representative with the gathered information.

Or you can visit

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a6f3a.shtml

Also requesting you to post the log file.

Thanks & Regards

Goutam

please rate if its helps you

Actions

This Discussion