06-16-2011 07:57 AM - edited 03-04-2019 12:44 PM
Hi recently I had an issue with two flexwan cards on a cisco 6509 reporting memory issues at the exact same time. On the 6509's log I notice some memory allocation messages related to the flexwan on ports 8/1 and 9/1, it complaints that it can't allocate memory and then disables Dcef.
The log on the flexwan says
"Memory allocation of 65536 bytes failed from 0x60197FA0, alignment 16
Pool: Processor Free: 800304 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool"
What exactly does this mean, it seems like at the time I had 800304 bytes free which was enough for the requested 65536 bytes. I'm assuming is requesting Processor memory and not I/O memory so those this mean even though I have 800304 bytes free its fragmented memory and therefore can't be allocated? If so what can I do to fix this issue.
The current memory for the flexwan is:
FlexWAN-8/1>sh memory sum
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 6124B260 98258336 19751320 78507016 788768 1567384
I/O 20000000 16777216 16544104 233112 230448 230264
The I/O pool concerns me it seems kind of low if there anything I can do without adding more ram to allocate more memory to the I/O pool?
The flexwan log shows:
SLOT 8/1: Jun 15 19:29:39: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed
from 0x60197FA0, alignment 16
Pool: Processor Free: 800304 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "CEF LC IPC Background", ipl= 0, pid= 44
-Traceback= 6017FA60 60182448 60197FA8 6049D940 6047CEDC 60481698 60488404 60498AC8 6
0491054 60491484 60491760 604923E0
SLOT 8/1: Jun 15 19:29:39: %FIB-3-NOMEM: Malloc Failure, disabling DCEF on linecard
Cisco Internetwork Operating System Software
IOS (tm) cwlc Software (cwpa-DW-M), Version 12.2(18)SXF14, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by cisco Systems, Inc.
Compiled Thu 08-May-08 00:57 by kellythw
Image text-base: 0x60011038, data-base: 0x6099C000
ROM: System Bootstrap, Version 12.1(1r)EX, RELEASE SOFTWARE (fc1)
ROM: cwlc Software (cwpa-DW-M), Version 12.2(18)SXF14, RELEASE SOFTWARE (fc1)
FlexWAN-8/1 uptime is 36 weeks, 21 hours, 53 minutes
System returned to ROM by power-on
System restarted at 12:44:50 edt Wed Oct 6 2010
Running default software
cisco FLEXWAN (R7000) processor (revision 0xFF) with 114688K/16384K bytes of memory.
R7000 CPU at 262Mhz, Implementation 0x27, Rev 2.1, 256KB L2 Cache
Last reset from power-on
1 FastEthernet/IEEE 802.3 interface
2 Serial network interfaces
Configuration register is 0x101
thanks for your time, paul
06-16-2011 09:54 AM
Hello Paul,
This could be due to that fact that although there is free memory, the memory is so fragmented that there isn't a big enough chunk of memory available for the process that requires it.
This link might help;
Regards,
David.
06-16-2011 09:57 AM
Can you also change the IOMEM size to allocate more RAM to IO?
David.
06-16-2011 12:52 PM
Paul,
The router is in a low memory condition due to memory fragmentation. The fragmentation is affecting the Process pool here.
FlexWAN-8/1>sh memory sum
Head Total(b)
Processor 6124B260 98258336
Used(b) 19751320 => Amt of memory currently used up
Free(b) 78507016 => Amt of memory free currently (78M)
Lowest(b) 788768 => The lowest amt of free memory the router has ever had
Largest(b) 1567384 => The largest amount of free contigous memory currently available.(1.5M)
Though 78 Meg of mem is free, for a process to be allocated which would need contigous memory we only have 1.5 M due a fragmented memory. Hence Memory allocation fails.
You can collect the following o/ps and contact TAC..
The IO Pool looks fine
Regards,
Subramaniya Karthic
06-17-2011 09:35 AM
i thanks all for the responses especially Subramaniya, however today i had another issue and this time i looked at it closely and i think the issue is one of running low on memory and not so much a memory fragmentation issue although its possible its both.
below is the error log from the flexwan and the sh mem, you will notice that a task called VIP Encap IPC" is asking for 65536 bytes but only 76352 bytes from the processor pool. Unlike i previously thought the problem seems to be with this pool that has 98 megs total and not the I/O as i previously thought. So for whatever reason, maybe because i have more customers frame-relay/ppp/MLPPP with interleave on the card is running out of memory. I think the solution is going to be a new card or more ram. i would apriciate any thoughts on this.
thanks paul
-Process= "VIP Encap IPC Background", ipl= 0, pid= 31 -Traceback= 6017FA60 601849B8 6031FEB4 6032017C 603BF524 603BF8E0 SLOT 13/1: Jun 17 07:52:48: %SYS-2-MALLOCFAIL: Memory allocation of 65592 bytes failed from 0x6031FEAC, a lignment 32
Pool: Processor Free: 72192 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "VIP Encap IPC Background", ipl= 0, pid= 31 -Traceback= 6017FA60 601849B8 6031FEB4 6032017C 603BF524 603BF8E0 SLOT 13/1: Jun 17 07:53:37: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x60197FA0, a lignment 16
Pool: Processor Free: 76352 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "CEF LC IPC Background", ipl= 0, pid= 74 -Traceback= 6017FA60 60182448 60197FA8 6049D940 6047CEDC 60481698 60488404
60498AC8 60491054 60491484 604
91760 604923E0
SLOT 13/1: Jun 17 07:53:37: %FIB-3-NOMEM: Malloc Failure, disabling DCEF on linecard FlexWAN-13/1> sh mem FlexWAN-13/1> sh memory
Head Total(b) Used(b) Free(b) Lowest(b)
Largest(b)
Processor 6124B260 98258336 21595032 76663304 64576
68952160
I/O 20000000 16777216 16515952 261264 258992
243224
12-14-2011 12:13 PM
this tunrned out to be related to the flexwan card i was using running out of memory. it created all sorts of wierd things that you can search on my posts. I replaced it with an enhanced flexwan with more memory which fixed the issue.
12-15-2011 06:48 AM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
BTW, as I understand it, Cisco routers do not do memory garbage collection, so low memory situations will often lead to memory fragmentation, which then leads to all kinds of "bad stuff". Further, once memory is fragmented, the only method to temporarily fix it is to reload. As you've noted, the "solution" is sufficient memory to avoid a low memory situation.
12-15-2011 07:13 AM
That’s correct, we were seeing all kinds of weird stuff, frame-relay circuits would show as up/up but stoped passing lmi/traffic, (switch them to PPP and they would work) cef stopped on the flexwans, upon reboot the flexwan cards would sometimes crash etc etc.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide