Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

5500 rsm low memory,how could this happened?

this morning,our 55 rsm perform dumping.

below are the related info,thanks if anyone could find the reason and tell me how to handle it!!!!!

5500#sh ver

Cisco Internetwork Operating System Software

IOS (tm) C5RSM Software (C5RSM-ISV-M), Version 12.0(17), RELEASE SOFTWARE (fc1)

Copyright (c) 1986-2001 by cisco Systems, Inc.

Compiled Tue 17-Apr-01 03:08 by nmasa

Image text-base: 0x60010930, data-base: 0x60AEC000

ROM: System Bootstrap, Version 11.2(17523) [mohsen 102], INTERIM SOFTWARE

BOOTFLASH: C5RSM Software (C5RSM-ISV-M), Version 11.3(5)T1, RELEASE SOFTWARE (f

c1)

5500 uptime is 10 hours, 44 minutes

System restarted by bus error at PC 0x601683A8, address 0xB0D0BCD at 08:46:57 Be

ijing Fri Aug 1 2003

5500#sh log

Aug 1 10:56:40: %SYS-2-MALLOCFAIL: Memory allocation of 18180 bytes failed from

0x601F6268, pool Processor, alignment 0

-Process= "Pool Manager", ipl= 6, pid= 4

-Traceback= 60225CCC 602270C0 601F6270 60231A78 6022108C 60221078

Aug 1 10:58:01: %SYS-2-MALLOCFAIL: Memory allocation of 1500 bytes failed from

0x60227EE4, pool Processor, alignment 0

-Process= "IP-EIGRP Router", ipl= 0, pid= 76

-Traceback= 60225CCC 60227778 60227EEC 6056FBF4 6057993C 60572A18 60572E48 60573

D50 60576620 60502420 6022108C 60221078

5500#sh mem

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)

Processor 60F53B80 17482880 15986664 1496216 12472 79864

Fast 60F33B80 131072 129112 1960 1960 1916

Thanks a lot

4 REPLIES
Gold

Re: 5500 rsm low memory,how could this happened?

Pool Manager is the process which handles the system buffers shown in "show buffers." If it's trying to allocate 18180, it's probably trying to allocate a huge buffer, which indicates this router is probably terminating some sort of a tunnel, dlsw session, or something like that. The second process is, of course, EIGRP, and it's trying to grab 1500 bytes, which probably means its trying to build a packet for sending out an update, or something along those lines.

The problem isn't that you don't have enough memory to fulfill these requests, it's that the router doesn't have a large enough block of free memory to fulfill this request. The current amount of free memory is 1496216, not much, but it's more than what's being requested, but the largest free block is 79864, which is much smaller. The lowest is 12472. My guess: These attempted allocations occured at the low water mark, when memory was even more fragmented than what you are seeing right now. When you are down close to your lowest, 12472, the largest block size is probably under the 1500 byte mark.

Memory is fragmented moderately bad right now, with only a 79k block largest out of 1.3meg free at the moment.

So.... There could be two problems here. The first is the router doesn't have enougb memory. It looks like you are running an RSM with about 64meg of memory (?), with about 1mb free normally. That's probably not good, I'd want to see more like 6mb free normally. When IOS gets into extremely low memory conditions, it can fragment the memory severely, and never be able to clean the fragmentation up. Since you are seeing moderately bad fragmentation, I'd suspect this as a cause.

The other possibility is that some major network trauma has occured recently, causing memory to drop very low, and fragment heavily. It could be some sort of defect in IOS, since there have been memory fragmentation issues in the past.

My initial prescription, though, for this router, is to take two new memory simms as a start. A possible code upgrade might not be bad, but I would only do that if I were comfortable with some later rev of code.

Russ.W

New Member

Re: 5500 rsm low memory,how could this happened?

Great help,

Thank you very much!

another thing,our NSA asked me to record the process '*dead*' mem usage.see output below:

5500#sh proces mem

Total: 17482880, Used: 15385976, Free: 2096904

PID TTY Allocated Freed Holding Getbufs Retbufs Process

0 0 75548 1852 6684716 0 0 *Init*

0 0 684 527676 684 0 0 *Sched*

0 0 11195672 5030860 168908 771556 0 *Dead*

1 0 268 268 3796 0 0 Load Meter

2 2 18764 17252 15200 0 0 Virtual Exec

3 0 0 0 6796 0 0 Check heaps

4 0 1317904 3192 94908 936684 0 Pool Manager

what is the *dead* process mean and any help info. it can give us in dealing with this problem?

Tks.

New Member

Re: 5500 rsm low memory,how could this happened?

Great help,

Thank you very much!

another thing,our NSA asked me to record the process '*dead*' mem usage.see output below:

5500#sh proces mem

Total: 17482880, Used: 15385976, Free: 2096904

PID TTY Allocated Freed Holding Getbufs Retbufs Process

0 0 75548 1852 6684716 0 0 *Init*

0 0 684 527676 684 0 0 *Sched*

0 0 11195672 5030860 168908 771556 0 *Dead*

1 0 268 268 3796 0 0 Load Meter

2 2 18764 17252 15200 0 0 Virtual Exec

3 0 0 0 6796 0 0 Check heaps

4 0 1317904 3192 94908 936684 0 Pool Manager

what is the *dead* process mean and any help info. it can give us in dealing with this problem?

Tks.

Gold

Re: 5500 rsm low memory,how could this happened?

The *dead* process holds memory that was allocated and owned by some other process which is no longer running. For instance, process A starts, allocates some memory, and passes a reference to that memory to process B. Process A now terminates, leaving the memory "ownerless." When process A stops running, IOS will mark the owner of this memory *dead*. It's important to remember that IOS is a shared memory operating system, all the processes can read and write into the entire memory space (with the exception of some protected memory spaces).

It sounds like you need to pick up a copy of Inside Cisco IOS Software! :-)

In this case, the dead process shows 168908 holding, which isn't too much. It does show quite a bit of free'd, 11195672, which is interesting, and a lot of getbuf's, which means it got buffers from the public buffer pools (those you see when you do a show buffers). It could be that some process was running on your router which ate a lot of memory and buffers, but then was turned off. There's not a lot of ways to track this down, without a lot of knowledge of the actual memory block structure (and even then, it's not really easy, if at all possible).

I would generally rule out *dead* as a part of any problem, unless I could see that some process has been configured then removed, for some reason.

Russ.W

216
Views
0
Helpful
4
Replies
CreatePlease to create content