Solved: SYS-2-MALLOCFAIL error

noobieee7 · ‎01-31-2010

Hi All,

I am having this error every few days and I have to reboot the router to resolve it. Any idea what might have caused it?

Below is the error.

000014: *Jan 30 02:02:14.531 GMT: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x6021A5C4, alignment 16
Pool: Processor Free: 244108 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "ARP Input", ipl= 3, pid= 10
-Traceback= 0x608DD040 0x601FBB28 0x60200C38 0x6021A5CC 0x60245E38 0x602466D8 0x60246960 0x60A63A3C 0x60A69720 0x60A6D4A4 0x60A6DB3C 0x60696374 0x60A9529C 0x60C0F518 0x60C0FB24 0x60C10614
000015: *Jan 30 02:02:14.535 GMT: %FIB-3-NOMEM: Malloc Failure, disabling CEF
-Traceback= 0x608DD040 0x60A63AA4 0x60A69720 0x60A6D4A4 0x60A6DB3C 0x60696374 0x60A9529C 0x60C0F518 0x60C0FB24 0x60C10614 0x606A3AF0

Reza Sharifi · ‎01-31-2010

Hi,

Is this router facing the Internet?

How many routes are in your route table?

It seems that you are running out of memory and the router disables CEF.

Can you change "ip route 0.0.0.0 0.0.0.0 FastEthernet0/1" to "ip route 0.0.0.0 0.0.0.0 IP address of next hop" instead and see if it happens again?

HTH

Reza

View solution in original post

Ganesh Hariharan · ‎01-31-2010

Hi,

This device has recorded a %SYS-2-MALLOCFAIL: error message.

An attempt to allocate memory has failed. Depending on the criticality of the data to be stored, CEF operation may be impacted.
A memory shortage has caused an internal software error. CEF switching has been disabled on the specified slot.

This error indicates a memory allocation failureA memory allocation failure occurs in one of these situations:

1. The current system configuration, network environment, or possibly a software error may have exhausted or fragmented the router's memory.

2. When the router uses up all available memory (temporarily or permanently)
3. When the memory is fragmented into such small pieces that the router cannot find a usable block. This issue can occur with the processor memory or with the packet memory.

This error can occur due to a variety of reasons that include:

    * Incorrect configuration
    * Too many features enabled for a small amount of memory
    * A single process that consumes more memory

Can you share the output of show process cpu and show memory

HTH

Ganesh.H

View solution in original post

marikakis · ‎01-31-2010

I think Reza's suggestion is good anyway. The output posted in the original post includes the phrase Process= "ARP Input", so maybe you could also check if for some reason (e.g. attack or virus/malignant software in a pc) your router receives an abnormally high number of ARP requests.

Edit: Just a clarification. My additional suggestion makes sense if problem persists after following Reza's suggestion. See the following document:

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a00800ef7b2.shtml#background

"This kind of default route, especially if it is used by a lot of packets to many different destination subnets, can cause high processor utilization and a very large ARP cache (along with attendant memory allocation failures)."

View solution in original post

noobieee7 · ‎01-31-2010

To add on,

I am using ciso 1841

IOS:c1841-ipbase-mz.124-3i.bin

Below is the config

Current configuration : 3494 bytes
version 12.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers

security authentication failure rate 3 log
security passwords min-length 6
logging buffered 4096 debugging
logging console critical

no aaa new-model
resource policy
clock timezone GMT 8
mmi polling-interval 60
no mmi auto-configure
no mmi pvc
mmi snmp-timeout 180
ip subnet-zero
no ip source-route
ip cef

ip tcp synwait-time 10
ip dhcp excluded-address 10.10.10.1 10.10.10.99
ip dhcp excluded-address 10.60.178.1 10.60.178.100
ip dhcp excluded-address 10.60.178.203 10.60.178.204
ip dhcp excluded-address 172.27.1.1 172.27.1.199

no ip bootp server
ip name-server 203.116.1.78
ip name-server 203.116.254.150
ip name-server 202.156.1.58

interface Loopback0
no ip address

interface FastEthernet0/0
description LAN
ip address 172.27.1.1 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
ip route-cache flow
duplex auto
speed auto
no mop enabled

interface FastEthernet0/1
description WAN
ip address xxx.xxx.xxx.xxx 255.255.255.252
no ip unreachables
no ip proxy-arp
ip nat outside
ip route-cache flow
duplex auto
speed auto
no mop enabled

ip classless
ip route 0.0.0.0 0.0.0.0 FastEthernet0/1
ip route 192.168.1.0 255.255.255.0 172.27.1.253
ip http server
ip http authentication local
ip http timeout-policy idle 60 life 86400 requests 10000
ip nat inside source list 1 interface FastEthernet0/1 overload
ip nat inside source static 172.27.1.149 xxx.xxx.xxx.xxx
ip nat inside source static 172.27.1.251 xxx.xxx.xxx.xxx

logging trap debugging

control-plane

Reza Sharifi · ‎01-31-2010

Hi,

Is this router facing the Internet?

How many routes are in your route table?

It seems that you are running out of memory and the router disables CEF.

Can you change "ip route 0.0.0.0 0.0.0.0 FastEthernet0/1" to "ip route 0.0.0.0 0.0.0.0 IP address of next hop" instead and see if it happens again?

HTH

Reza

marikakis · ‎01-31-2010

I think Reza's suggestion is good anyway. The output posted in the original post includes the phrase Process= "ARP Input", so maybe you could also check if for some reason (e.g. attack or virus/malignant software in a pc) your router receives an abnormally high number of ARP requests.

Edit: Just a clarification. My additional suggestion makes sense if problem persists after following Reza's suggestion. See the following document:

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a00800ef7b2.shtml#background

"This kind of default route, especially if it is used by a lot of packets to many different destination subnets, can cause high processor utilization and a very large ARP cache (along with attendant memory allocation failures)."

Ganesh Hariharan · ‎01-31-2010