memory fragmentation error

paul amaral · ‎06-16-2011

Hi recently I had an issue with two flexwan cards on a cisco 6509 reporting memory issues at the exact same time. On the 6509's log I notice some memory allocation messages related to the flexwan on ports 8/1 and 9/1, it complaints that it can't allocate memory and then disables Dcef.

The log on the flexwan says

"Memory allocation of 65536 bytes failed from 0x60197FA0, alignment 16

Pool: Processor Free: 800304 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool"

What exactly does this mean, it seems like at the time I had 800304 bytes free which was enough for the requested 65536 bytes. I'm assuming is requesting Processor memory and not I/O memory so those this mean even though I have 800304 bytes free its fragmented memory and therefore can't be allocated? If so what can I do to fix this issue.

The current memory for the flexwan is:

FlexWAN-8/1>sh memory sum

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)

Processor 6124B260 98258336 19751320 78507016 788768 1567384

I/O 20000000 16777216 16544104 233112 230448 230264

The I/O pool concerns me it seems kind of low if there anything I can do without adding more ram to allocate more memory to the I/O pool?

The flexwan log shows:

SLOT 8/1: Jun 15 19:29:39: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed

from 0x60197FA0, alignment 16

Pool: Processor Free: 800304 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "CEF LC IPC Background", ipl= 0, pid= 44

-Traceback= 6017FA60 60182448 60197FA8 6049D940 6047CEDC 60481698 60488404 60498AC8 6

0491054 60491484 60491760 604923E0

SLOT 8/1: Jun 15 19:29:39: %FIB-3-NOMEM: Malloc Failure, disabling DCEF on linecard

Cisco Internetwork Operating System Software

IOS (tm) cwlc Software (cwpa-DW-M), Version 12.2(18)SXF14, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Thu 08-May-08 00:57 by kellythw

Image text-base: 0x60011038, data-base: 0x6099C000

ROM: System Bootstrap, Version 12.1(1r)EX, RELEASE SOFTWARE (fc1)

ROM: cwlc Software (cwpa-DW-M), Version 12.2(18)SXF14, RELEASE SOFTWARE (fc1)

FlexWAN-8/1 uptime is 36 weeks, 21 hours, 53 minutes

System returned to ROM by power-on

System restarted at 12:44:50 edt Wed Oct 6 2010

Running default software

cisco FLEXWAN (R7000) processor (revision 0xFF) with 114688K/16384K bytes of memory.

R7000 CPU at 262Mhz, Implementation 0x27, Rev 2.1, 256KB L2 Cache

Last reset from power-on

1 FastEthernet/IEEE 802.3 interface

2 Serial network interfaces

Configuration register is 0x101

thanks for your time, paul

farrell.da · ‎06-16-2011

Hello Paul,

This could be due to that fact that although there is free memory, the memory is so fragmented that there isn't a big enough chunk of memory available for the process that requires it.

This link might help;

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a6f3a.shtml#tshoot4

Regards,

David.

farrell.da · ‎06-16-2011

Can you also change the IOMEM size to allocate more RAM to IO?

David.

skarthic · ‎06-16-2011

Paul,

The router is in a low memory condition due to memory fragmentation. The fragmentation is affecting the Process pool here.

FlexWAN-8/1>sh memory sum

Head Total(b)

Processor 6124B260 98258336

Used(b) 19751320 => Amt of memory currently used up

Free(b) 78507016 => Amt of memory free currently (78M)

Lowest(b) 788768 => The lowest amt of free memory the router has ever had

Largest(b) 1567384 => The largest amount of free contigous memory currently available.(1.5M)

Though 78 Meg of mem is free, for a process to be allocated which would need contigous memory we only have 1.5 M due a fragmented memory. Hence Memory allocation fails.

You can collect the following o/ps and contact TAC..

show memory free
show memory summary
show processor memory
show memory frag
show tech

The IO Pool looks fine

Regards,

Subramaniya Karthic

paul amaral · ‎06-17-2011

i thanks all for the responses especially Subramaniya, however today i had another issue and this time i looked at it closely and i think the issue is one of running low on memory and not so much a memory fragmentation issue although its possible its both.

below is the error log from the flexwan and the sh mem, you will notice that a task called VIP Encap IPC" is asking for 65536 bytes but only 76352 bytes from the processor pool. Unlike i previously thought the problem seems to be with this pool that has 98 megs total and not the I/O as i previously thought. So for whatever reason, maybe because i have more customers frame-relay/ppp/MLPPP with interleave on the card is running out of memory. I think the solution is going to be a new card or more ram. i would apriciate any thoughts on this.

thanks paul

-Process= "VIP Encap IPC Background", ipl= 0, pid= 31 -Traceback= 6017FA60 601849B8 6031FEB4 6032017C 603BF524 603BF8E0 SLOT 13/1: Jun 17 07:52:48: %SYS-2-MALLOCFAIL: Memory allocation of 65592 bytes failed from 0x6031FEAC, a lignment 32

Pool: Processor Free: 72192 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "VIP Encap IPC Background", ipl= 0, pid= 31 -Traceback= 6017FA60 601849B8 6031FEB4 6032017C 603BF524 603BF8E0 SLOT 13/1: Jun 17 07:53:37: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x60197FA0, a lignment 16

Pool: Processor Free: 76352 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "CEF LC IPC Background", ipl= 0, pid= 74 -Traceback= 6017FA60 60182448 60197FA8 6049D940 6047CEDC 60481698 60488404

60498AC8 60491054 60491484 604

91760 604923E0

SLOT 13/1: Jun 17 07:53:37: %FIB-3-NOMEM: Malloc Failure, disabling DCEF on linecard FlexWAN-13/1> sh mem FlexWAN-13/1> sh memory

Head Total(b) Used(b) Free(b) Lowest(b)

Largest(b)

Processor 6124B260 98258336 21595032 76663304 64576

68952160

I/O 20000000 16777216 16515952 261264 258992

243224

paul amaral · ‎12-14-2011

this tunrned out to be related to the flexwan card i was using running out of memory. it created all sorts of wierd things that you can search on my posts. I replaced it with an enhanced flexwan with more memory which fixed the issue.

Joseph W. Doherty · ‎12-15-2011

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

BTW, as I understand it, Cisco routers do not do memory garbage collection, so low memory situations will often lead to memory fragmentation, which then leads to all kinds of "bad stuff". Further, once memory is fragmented, the only method to temporarily fix it is to reload. As you've noted, the "solution" is sufficient memory to avoid a low memory situation.

paul amaral · ‎12-15-2011

That’s correct, we were seeing all kinds of weird stuff, frame-relay circuits would show as up/up but stoped passing lmi/traffic, (switch them to PPP and they would work) cef stopped on the flexwans, upon reboot the flexwan cards would sometimes crash etc etc.