User receives the %SYS-2-MALLOCFAIL messages due to a memory leak or fragmentation issue

Document

Wed, 11/18/2009 - 18:25
Jun 22nd, 2009
User Badges:
  • Gold, 750 points or more

Core issue


This message indicates that the process is unable to find a large enough block of contiguous memory. The IP Input process attempts to get 1028 bytes from the processor pool of memory, as shown in this example:

%SYS-2-MALLOCFAIL: Memory allocation of 1028 bytes failed from 0x601617A4, 
pool Processor, alignment 0 -Process= "IP Input", ipl= 2, pid= 21

These are the causes of these messages:

  • The memory in the router is insufficient to support the Cisco IOS® Software or the size of the routing table.
  • A memory leak occurs.
  • The memory becomes fragmented.


Note: The Blaster and Nachi Worms can lead to memory problems.


Resolution


If the %SYS-2-MALLOCFAIL messages are logged, perform these steps:


  1. Verify that the router has enough DRAM to support the Cisco IOS Software by issuing the show version command, as shown in this example:
    Router#show version Cisco Internetwork Operating System Software IOS (tm) C2600 Software (C2600-I-M), Version 12.2(8)T5, RELEASE SOFTWARE (fc1)
    TAC Support:  Copyright (c) 1986-2002 by Cisco Systems, Inc. Compiled Fri 21-Jun-02 08:50 by ccai Image text-base: 0x80008074, data-base: 0x80A2BD40  ROM: System Bootstrap, Version 12.2(7r) [cmong 7r], RELEASE SOFTWARE (fc1)  morannon uptime is 5 days, 1 hour, 27 minutes System returned to ROM by power-on System image file is "flash:c2600-i-mz.122-8.T5.bin"  Cisco 2620XM (MPC860P) processor (revision 0x100) with 28672K/4096K bytes of memory.
    Processor board ID JAD071008B1 (1020186390)

    The items in bold show that the router is running Cisco IOS Software release 12.2(8)T5 with the I feature set (C2600-I-M). In addition, it has 32 MB of DRAM (28672K/4096K).

  2. Verify the minimum memory requirements for Cisco IOS Software, such as the platform, train and feature set. To check the minimum memory requirements for Cisco IOS Software, perform show version analysis by cutting and pasting the command output in the Output Interpreter tool. Follow the link provided in the Cisco IOS Image Software Advisor-IOS Image Name section of the analysis output.
  3. Verify that the minimum recommended amount of DRAM is installed on the router. A large routing table requires more than the minimum DRAM requirements.


    For specific information on DRAM memory requirements when using Border Gateway Protocol (BGP), refer to Achieve Optimal Routing and Reduce BGP Memory Consumption.

    The Cisco series 2600, 3600 and 3700 routers require a minimum amount of I/O memory to support certain interface processors.

    If the amount of installed memory is less than the recommended amount, perform one of these options:


    • Add memory.
      • Go to the Cisco IOS Software version supported with the current amount of memory.

    • Some applications have features, such as the User Tracking (UT) Discovery feature of CiscoWorks, which can result in low memory conditions unless the ip cef command is issued.

    • Memory allocation failures can be caused by a memory leak bug or memory fragmentation. In this case, analyze the output of the show memory command by pasting the output into the Output Interpreter tool.

    • To determine if fragmentation is occurring, compare the Largest and Free fields by issuing the show memory [summary] command.


      Fragmentation is taking place if the number in the Largest field is much smaller than the number in the Free field. This is because the Largest field indicates the largest contiguous free memory block and it should normally be close to the free memory, as shown in this example:


      Router#show memory summary           Head   Total(b) Used(b) Free(b)  Lowest(b) Largest(b)
      Processor C0E48  13885880 1615712 12270168 12077808  12077808
      I/O       E00000 2097152  398396  1698756  1698756   1698588

      This is a brief description of the fields:

      • Total is the total memory allocated to the processor or I/O memory. This does not include the amount of memory taken up by the Cisco IOS Software.
      • Used is the amount of memory being used at the time the show memory [summary] command is issued.
      • Free is the amount of available free memory at the time the show memory [summary] command is issued.
      • Lowest is the lowest amount of memory available since the last reload.
      • Largest is the largest amount of free contiguous memory at the time the show memory [summary] command is issued. This should normally be close to the free memory. A small number compared to the free memory indicates fragmentation.

    • To determine if a memory leak is occurring, capture the output of the show memory [summary] command several times at regular intervals. The intervals depend on the length of time it takes for the memory allocation failures to appear. If the router begins to display the errors after four days, then one or two captures per day is sufficient to establish a pattern.


      If the free memory steadily decreases, a memory leak may be occurring.

      A memory leak occurs when a process takes and uses memory, but does not release the memory back to the system. To determine the offending process, issue the show processes memory command and perform these steps:

      1. To determine which process is not freeing memory back to the system, capture the show processes memory command output several times at regular intervals.
      2. The two counters used for this capture are Freed and Holding. If the Holding counter for a process increases, but the Freed counter does not increase, that process may be the cause of the memory leak.
      3. Once the process is identified, search for a memory leak issue by referring to the Bug Toolkit. This issue relates to the process that affects the Cisco IOS Software currently installed on the router.


    Note: The Blaster and Nachi Worms can lead to memory problems.


    To determine if your problem could be related to these worms, refer to these documents:


    For more information on solving memory issues, refer to Troubleshooting Memory Problems.

    /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman"; mso-ansi-language:#0400; mso-fareast-language:#0400; mso-bidi-language:#0400;}
    Loading.

    Actions

    This Document

    Related Content