memory fragmentation error

Unanswered Question
Jun 16th, 2011

Hi recently I had an issue with two flexwan cards on a cisco 6509 reporting memory issues at the exact same time. On the 6509's log I notice some memory allocation messages related to the flexwan on ports 8/1 and 9/1, it complaints that it can't allocate memory and then disables Dcef.

The log on the flexwan says

"Memory allocation of 65536 bytes failed from 0x60197FA0, alignment 16

Pool: Processor Free: 800304 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool"

What exactly does this mean, it seems like at the time I had 800304 bytes free which was enough for the requested 65536 bytes. I'm assuming is requesting Processor memory and not I/O memory so those this mean even though I have 800304 bytes free its fragmented memory and therefore can't be allocated? If so what can I do to fix this issue.

The current memory for the flexwan is:

FlexWAN-8/1>sh memory sum

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)

Processor   6124B260    98258336    19751320    78507016      788768     1567384

      I/O   20000000    16777216    16544104      233112      230448      230264

The I/O pool concerns me it seems kind of low if there anything I can do without adding more ram to allocate more memory to the I/O pool?

The flexwan log shows:

SLOT 8/1: Jun 15 19:29:39: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed

from 0x60197FA0, alignment 16

Pool: Processor  Free: 800304  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "CEF LC IPC Background", ipl= 0, pid= 44

-Traceback= 6017FA60 60182448 60197FA8 6049D940 6047CEDC 60481698 60488404 60498AC8 6

0491054 60491484 60491760 604923E0

SLOT 8/1: Jun 15 19:29:39: %FIB-3-NOMEM: Malloc Failure, disabling DCEF on linecard

Cisco Internetwork Operating System Software

IOS (tm) cwlc Software (cwpa-DW-M), Version 12.2(18)SXF14, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2008 by cisco Systems, Inc.

Compiled Thu 08-May-08 00:57 by kellythw

Image text-base: 0x60011038, data-base: 0x6099C000

ROM: System Bootstrap, Version 12.1(1r)EX, RELEASE SOFTWARE (fc1)

ROM: cwlc Software (cwpa-DW-M), Version 12.2(18)SXF14, RELEASE SOFTWARE (fc1)

FlexWAN-8/1 uptime is 36 weeks, 21 hours, 53 minutes

System returned to ROM by power-on

System restarted at 12:44:50 edt Wed Oct 6 2010

Running default software

cisco FLEXWAN (R7000) processor (revision 0xFF) with 114688K/16384K bytes of memory.

R7000 CPU at 262Mhz, Implementation 0x27, Rev 2.1, 256KB L2 Cache

Last reset from power-on

1 FastEthernet/IEEE 802.3 interface

2 Serial network interfaces

Configuration register is 0x101

thanks for your time, paul

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
farrell.da Thu, 06/16/2011 - 09:57

Can you also change the IOMEM size to allocate more RAM to IO?

David.

skarthic Thu, 06/16/2011 - 12:52

Paul,

The router is in a low memory condition due to memory fragmentation. The fragmentation is affecting the Process pool here.

FlexWAN-8/1>sh memory sum

                      Head             Total(b)    

Processor   6124B260    98258336

Used(b)     19751320  => Amt of memory currently used up

Free(b)      78507016  => Amt of memory free currently (78M)

Lowest(b)      788768  => The lowest amt of  free memory the router has ever had

Largest(b)    1567384  => The largest amount of free contigous memory currently available.(1.5M)

Though 78 Meg of mem is free, for a process to be allocated which would need contigous memory we only have 1.5 M due a fragmented memory. Hence Memory allocation fails.

You can collect the following o/ps and contact TAC..

    • show memory free
    • show memory summary
    • show processor memory
    • show memory frag
    • show tech

The IO Pool looks fine

Regards,

Subramaniya Karthic

paul amaral Fri, 06/17/2011 - 09:35

i thanks all for the responses especially Subramaniya, however today i had another issue and this time i looked at it closely and i think the issue is one of running low on memory and not so much a memory fragmentation issue although its possible its both.

below is the error log from the flexwan and the sh mem, you will notice that a task called VIP Encap IPC" is asking for 65536 bytes but only 76352 bytes from the processor pool. Unlike i previously thought the problem seems to be with this pool that has 98 megs total and not the I/O as i previously thought. So for whatever reason, maybe because i have more customers frame-relay/ppp/MLPPP with interleave on the card is running out of memory. I think the solution is going to be a new card or more ram. i would apriciate any thoughts on this.

thanks paul

-Process= "VIP Encap IPC Background", ipl= 0, pid= 31 -Traceback= 6017FA60 601849B8 6031FEB4 6032017C 603BF524 603BF8E0 SLOT 13/1: Jun 17 07:52:48: %SYS-2-MALLOCFAIL: Memory allocation of 65592 bytes failed from 0x6031FEAC, a lignment 32

Pool: Processor Free: 72192 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "VIP Encap IPC Background", ipl= 0, pid= 31 -Traceback= 6017FA60 601849B8 6031FEB4 6032017C 603BF524 603BF8E0 SLOT 13/1: Jun 17 07:53:37: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x60197FA0, a lignment 16

Pool: Processor Free: 76352 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "CEF LC IPC Background", ipl= 0, pid= 74 -Traceback= 6017FA60 60182448 60197FA8 6049D940 6047CEDC 60481698 60488404

60498AC8 60491054 60491484 604

91760 604923E0

SLOT 13/1: Jun 17 07:53:37: %FIB-3-NOMEM: Malloc Failure, disabling DCEF on linecard FlexWAN-13/1> sh mem FlexWAN-13/1> sh memory

               Head   Total(b)     Used(b)     Free(b)   Lowest(b)

Largest(b)

Processor   6124B260   98258336   21595032   76663304       64576  

68952160

     I/O   20000000   16777216   16515952     261264     258992    

243224

paul amaral Wed, 12/14/2011 - 12:13

this tunrned out to be related to the flexwan card i was using running out of memory. it created all sorts of wierd things that you can search on my posts. I replaced it with an enhanced flexwan with more memory which fixed the issue.

Joseph W. Doherty Thu, 12/15/2011 - 06:48

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

BTW, as I understand it, Cisco routers do not do memory garbage collection, so low memory situations will often lead to memory fragmentation, which then leads to all kinds of "bad stuff".  Further, once memory is fragmented, the only method to temporarily fix it is to reload.  As you've noted, the "solution" is sufficient memory to avoid a low memory situation.

paul amaral Thu, 12/15/2011 - 07:13

That’s correct, we were seeing all kinds of weird stuff, frame-relay circuits would show as up/up but stoped passing lmi/traffic, (switch them to PPP and they would work) cef stopped on the flexwans, upon reboot the flexwan cards would sometimes crash etc etc.

Actions

This Discussion