Cisco 1841 Crashing / Running Out of Memory (maybe)

Unanswered Question

I have a Cisco 1841 that has 8 VPN's running through it (it is not the tunnel endpoints though), it also provides internet access for the main office through a 10 Mb leased line.


It keeps crashing every few days with this appearing in the Syslog :


2008-10-06 09:00:14 Local7.Critical 212.240.x.x 50: Pool: Processor Free: 206548 Cause: Memory fragmentation

2008-10-06 09:00:14 Local7.Critical 212.240.x.x 51: Alternate Pool: None Free: 0 Cause: No Alternate pool

2008-10-06 09:00:14 Local7.Critical 212.240.x.x 52:

2008-10-06 09:00:14 Local7.Critical 212.240.x.x 53: -Process= "ARP Input", ipl= 3, pid= 10

2008-10-06 09:00:14 Local7.Critical 212.240.x.x 54: -Traceback= 0x608DD040 0x601FBB28 0x60200C38 0x6021A5CC 0x60245E38 0x602466D8 0x60246960 0x60A63A3C 0x60A69720 0x60A6D4A4 0x60A6DB3C 0x60696374 0x60A9529C 0x60C0F518 0x60C0FB24 0x60C10614

2008-10-06 09:00:14 Local7.Critical 212.240.x.x 49: 000046: *Oct 6 08:03:50: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x6021A5C4, alignment 16

2008-10-06 09:00:15 Local7.Error 212.240.x.x 55: 000047: *Oct 6 08:03:50: %FIB-3-NOMEM: Malloc Failure, disabling CEF

2008-10-06 09:00:15 Local7.Error 212.240.x.x 56: -Traceback= 0x608DD040 0x60A63AA4 0x60A69720 0x60A6D4A4 0x60A6DB3C 0x60696374 0x60A9529C 0x60C0F518 0x60C0FB24 0x60C10614 0x606A3AF0

2008-10-06 09:05:29 Local7.Error 212.240.x.x 61: 000052: *Oct 6 08:09:02: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to up


From what I have read, this error normally indicates that the Processor pool is running out of memory - I have tried 'memory-size iomem 5' but it still keeps crashing.


I have also swapped the router with a different one (1812) and upgraded the firmware - still the same result. If i disable CEF it lasts a week before it crashes, this time with nothing in the syslog.


I am now looking to upgrade the internal memory - will this fix the problem ?


Thanks

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4 (2 ratings)
Loading.
satish_zanjurne Wed, 10/08/2008 - 01:43
User Badges:
  • Silver, 250 points or more


Hi


Explanation: This device has recorded a %SYS-2-MALLOCFAIL: error message.

This error indicates a memory allocation failure. Memory allocation of [dec] bytes

failed from [hex], alignment [dec] Pool: [chars] Free: [dec] Cause: [chars] Alternate

Pool: [chars] Free: [dec] Cause: [chars] The requested memory allocation is not

available from the specified memory pool. A memory allocation failure occurs in

one of these situations:

- The current system configuration, network environment, or possibly a software

error may have exhausted or fragmented the router's memory

- When the router uses up all available memory (temporarily or permanently)

- When the memory is fragmented into such small pieces that the router cannot

find a usable block. This issue can occur with the processor memory or with the

packet memory. This error can occur due to a variety of reasons that include:

* Incorrect configuration

* Too many features enabled for a small amount of memory

* A single process that consumes more memory

Recommended Action: Begin to troubleshoot with a check on the process that the

error message mentions. Submit the output of show process cpu and show memory

commands to Output Interpreter to check whether the process consumes more memory.

If you are unable to reproduce the error message, you can ignore the error message.

If the error message recurs, review the Bug Toolkit to locate any software bugs

that can cause this error.


HTH...rate if helpful..

Thanks for the reply, the configuration is really basic, and there are no advanced features enabled really. I have also tried two different routers with two firmwares so it's unlikely to be a bug ?


Here's th config, is there anything that sticks out ?



Current configuration : 3354 bytes

!

version 12.4

service tcp-keepalives-in

service tcp-keepalives-out

service timestamps debug datetime msec localtime show-timezone

service timestamps log datetime localtime

service password-encryption

service sequence-numbers

!

hostname x

!

boot-start-marker

boot-end-marker

!

logging buffered 51200 warnings

enable secret 5 X

!

no aaa new-model

!

resource policy

!

memory-size iomem 5

mmi polling-interval 60

no mmi auto-configure

no mmi pvc

mmi snmp-timeout 180

ip subnet-zero

ip cef

!

!

!

!

no ip bootp server

no ip domain lookup

ip domain name X

!


!

!


!

!

interface FastEthernet0/0

description External Connection to THUS

bandwidth 81920

ip address 193.195.x.x 255.255.255.252

ip access-group 100 in

no ip proxy-arp

ip route-cache flow

speed 10

full-duplex

!

interface FastEthernet0/1

description Internal - Connection to Sonicwall

ip address 212.240.x.x 255.255.255.248

no ip proxy-arp

ip route-cache flow

speed 100

full-duplex

!

ip classless

ip route 0.0.0.0 0.0.0.0 FastEthernet0/0

ip flow-export source FastEthernet0/0

ip flow-export version 5

ip flow-export destination 212.240.x.x 9996

!

ip http server

ip http access-class 23

ip http authentication local

ip http timeout-policy idle 60 life 86400 requests 10000

!

logging trap debugging

logging 212.240.202.202

access-list 23 permit 10.10.10.0 0.0.0.7

access-list 100 permit tcp host 83.104.x.x host 193.195.x.x eq telnet

access-list 100 permit tcp host 83.104.x.x host 193.195.x.x eq 22

access-list 100 permit tcp host 83.104.x.x host 193.195.x.x eq www

access-list 100 permit tcp host 83.104.x.x host 193.195.x.x eq 443

access-list 100 permit tcp host 83.104.x.x host 193.195.x.x eq cmd

access-list 100 deny tcp any host 193.195.x.x eq telnet

access-list 100 deny tcp any host 193.195.x.x eq 22

access-list 100 deny tcp any host 193.195.x.x eq www

access-list 100 deny tcp any host 193.195.x.x eq 443

access-list 100 deny tcp any host 193.195.x.x eq cmd

access-list 100 deny udp any host 193.195.x.x eq snmp

access-list 100 deny tcp any any range 6881 6999

access-list 100 permit ip any any


no cdp run

!

control-plane

!

banner login ^C

Authorized Access Only!


If you are not authorized to use this system please disconnect now.


All access attempts are logged. ^C

!

line con 0

login local

transport output telnet

line aux 0

login local

transport output telnet

line vty 0 4

access-class 101 in

privilege level 15

login local

transport input telnet

line vty 5 15

access-class 102 in

privilege level 15

login local

transport input telnet

!

end



Maybe I just need to upgrade the memory ?

netwalkr1 Wed, 10/08/2008 - 06:31
User Badges:

Change the default route to the next hop IP address. When you use a default route next hop as the interface on an Ethernet environment it will populate the ARP table with each destination. This consumes the routers RAM.

For some reason, the number of buffer elements on the free list is way over the max, could this be related somehow ?





#show buffers

Buffer elements:

1117 in free list (500 max allowed)

21421 hits, 0 misses, 1119 created


Public buffer pools:

Small buffers, 104 bytes (total 50, permanent 50):

49 in free list (20 min, 150 max allowed)

11963 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

Middle buffers, 600 bytes (total 25, permanent 25):

23 in free list (10 min, 150 max allowed)

960 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

Big buffers, 1536 bytes (total 50, permanent 50):

49 in free list (5 min, 150 max allowed)

2608 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

VeryBig buffers, 4520 bytes (total 10, permanent 10):

10 in free list (0 min, 100 max allowed)

0 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

Large buffers, 5024 bytes (total 0, permanent 0):

0 in free list (0 min, 10 max allowed)

0 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

Huge buffers, 18024 bytes (total 0, permanent 0):

0 in free list (0 min, 4 max allowed)

0 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)


Header pools:

Header buffers, 0 bytes (total 768, permanent 768):

256 in free list (128 min, 1024 max allowed)

512 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

512 max cache size, 512 in cache

29 hits in cache, 0 misses in cache


Particle Clones:

1024 clones, 0 hits, 0 misses


Public particle pools:

F/S buffers, 256 bytes (total 768, permanent 768):

256 in free list (128 min, 1024 max allowed)

512 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

512 max cache size, 512 in cache

0 hits in cache, 0 misses in cache

Normal buffers, 1548 bytes (total 768, permanent 768):

768 in free list (128 min, 1024 max allowed)

0 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)


Private particle pools:

FastEthernet0/0 buffers, 1552 bytes (total 512, permanent 512):

0 in free list (0 min, 512 max allowed)

512 hits, 0 fallbacks

512 max cache size, 256 in cache

1703973 hits in cache, 0 misses in cache

FastEthernet0/1 buffers, 1552 bytes (total 512, permanent 512):

0 in free list (0 min, 512 max allowed)

512 hits, 0 fallbacks

512 max cache size, 256 in cache

2123246 hits in cache, 0 misses in cache

Thanks, I have read that guide before but did not change anything since the buffers are, like you say, usually empty.


Yes the router is in production, but it is used mainly for routing CCTV video so the traffic is very bursty - normally the link is barely utilised but sometimes jumps up to 10 Mb very quickly depending on how many camera's are utilized.


I guess I will open a service case with Cisco since the router is still crashing, I don't know why when usually the 10 Mb connection is usually using less than 1Mb of bandwidth.

Actions

This Discussion