Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

SYS-2-MALLOCFAIL: Memory allocation of %d bytes failed

Greetings!

We started to receive SYS-2-MALLOCFAIL messages on our CAT3550-12T.

#sh ver | i IOS

IOS (tm) C3550 Software (C3550-I5K2L2Q3-M), Version 12.1(13)EA1c, RELEASE SOFTWARE (fc1)

Messages:

Dec 13 19:11:06: %SYS-2-MALLOCFAIL: Memory allocation of 1680 bytes failed from 0x1627E4, alignment 0

Pool: I/O Free: 9868 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 6

-Traceback= 1A647C 1A7AA0 1627E8 1B389C 1B3AE4 1C70E0

Dec 13 20:43:53: %SYS-2-MALLOCFAIL: Memory allocation of 1680 bytes failed from 0x1627E4, alignment 0

Pool: I/O Free: 11492 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 6

-Traceback= 1A647C 1A7AA0 1627E8 1B389C 1B3AE4 1C70E0

Is that software bug or something else?

Best regards,

Dmitry N. Hramtsov

10 REPLIES
New Member

Re: SYS-2-MALLOCFAIL: Memory allocation of %d bytes failed

Usually not. You need to figure out what led up to this message - topology or config change, etc. I've seen this happen many times when there was a temporary loop. It is much better behavior than crashing the box.

New Member

Re: SYS-2-MALLOCFAIL: Memory allocation of %d bytes failed

It is difficult to say what led up to this message.

Nearest message in log dated about 30 minutes before MALLOCFAIL.

Don't know can this help, but here is more complete log for interesting period:

Dec 13 18:31:39: %OSPF-5-ADJCHG: Process 1, Nbr 10.10.213.126 on Vlan54 from 2WAY to DOWN, Neighbor Down: Dead timer expired

Dec 13 18:31:51: %OSPF-5-ADJCHG: Process 1, Nbr 10.10.213.126 on Vlan54 from DOWN to INIT, Received Hello

Dec 13 18:31:51: %OSPF-5-ADJCHG: Process 1, Nbr 10.10.213.126 on Vlan54 from INIT to 2WAY, 2-Way Received

Dec 13 19:11:06: %SYS-2-MALLOCFAIL: Memory allocation of 1680 bytes failed from 0x1627E4, alignment 0

[...SYS-2-MALLOCFAIL details skipped...]

Dec 13 19:12:33: %OSPF-5-ADJCHG: Process 1, Nbr 10.10.213.126 on Vlan54 from 2WAY to DOWN, Neighbor Down: Dead timer expired

Dec 13 19:13:10: %OSPF-5-ADJCHG: Process 1, Nbr 10.10.213.126 on Vlan54 from DOWN to INIT, Received Hello

Dec 13 19:13:10: %OSPF-5-ADJCHG: Process 1, Nbr 10.10.213.126 on Vlan54 from INIT to 2WAY, 2-Way Received

Dec 13 20:43:53: %SYS-2-MALLOCFAIL: Memory allocation of 1680 bytes failed from 0x1627E4, alignment 0

[...SYS-2-MALLOCFAIL details skipped...]

Next messages dated Dec 14.

New Member

Re: SYS-2-MALLOCFAIL: Memory allocation of %d bytes failed

Could still be caused by a loop in the network. You will need to check that first. Here is a bug that had same symptoms, CSCec13716. Your OSPF does not seem stable as well. Could be due to the same symptom. Try fixing that as well to see if it could be related.

Gold

Re: SYS-2-MALLOCFAIL: Memory allocation of %d bytes failed

I don't think CSCec13716 is related, from looking at the bug (listed as unreproducable). It sounds like you just have a memory fragmentation problem in the input/output pool of memory, so that's where I would attack the problem first. Most likely, you've gotten a lot of receive or through traffic in small enough packet sizes that you're still holding for some reason to fragment the i/o memory.

So, what I'd most likely do here is to start looking at show buffers old and show buffers input-interface to see if I could figure out what sorts of packets are being held in memory for a long time, then figuring out what I could do about it--are they all being received? Is there something throwing a lot of traffic at the router (a virus or an attacker)?

Beyond this, I would restart the box so the memory fragmentation problem goes away, and I could start with a clean slate. This may just be a residue of some earlier virus, attack, or network condition which no longer exists.

:-)

Russ.W

New Member

Re: SYS-2-MALLOCFAIL: Memory allocation of %d bytes failed

Hello Russ,

> It sounds like you just have a memory fragmentation problem

Agree with you, but AFAIK it should not happens. Is it possible that this problem appears because of incorrect software behavior?

I tried "show buffers" as you say.

As far as I see there is no "old" buffers. May be they was at the moment when MALLOCFAIL occures, who knows.

#show buffers old

Header DataArea Pool Rcnt Size Link Enc Flags Input Output

"show buffers failures" returns interesting results:

#show buffers failures

Caller Pool Size When

0x2AF13C Big 1524 1d01h

0x2AF13C Big 1524 1d01h

0x2AF13C Big 1524 1d01h

0x2AF13C Big 1524 1d01h

0x2AF13C Big 1524 1d01h

0x2AF13C Big 1524 1d01h

0x2AF13C Big 1524 1d01h

0x2AF13C Big 1524 1d01h

0x2AF13C Big 1524 1d01h

0x2AF13C Big 1524 1d01h

Best regards,

Dmitry N. Hramtsov

Gold

Re: SYS-2-MALLOCFAIL: Memory allocation of %d bytes failed

Then the network condition that caused this doesn't exist any longer.... It could have been any number of things, but it's going to be impossible to track down now that it's gone.

:-)

Russ.W

New Member

Re: SYS-2-MALLOCFAIL: Memory allocation of %d bytes failed

Like I said, I have seen many, many times that this was caused by a temporary loop. These malloc errors could be caused by anything. A detailed technical reason can be given but will not provide an answer as to why??? The bug I had mentioned was not a root cause. Did you check the U-comments? It turned out to be caused by a loop.

Gold

Re: SYS-2-MALLOCFAIL: Memory allocation of %d bytes failed

It could have been, or a code red attack, or some ddos attack on the router itself, or a number of other things. At this point, I'd say reload the router to clear the fragmentation, and watch the router for a week or two to see what the fragmentation level look like.

If you see a serious upswing in the fragmentation level in i/o memory over a very short period of time, then I would look back at any other logs and see what other events have occurred around the same time as the increase in fragmentation--routing changes, etc. If you see a small, but constantly climbing fragmentation over a longer period of time, I would be more likely to suspect a defect (but not necessarily).

So, what we really need here is more information--reload the router, and watch.

:-)

Russ.W

New Member

Re: SYS-2-MALLOCFAIL: Memory allocation of %d bytes failed

Hello,

Interesting post.

I had got the same phenomen on customer site one week ago.

Here are the traces found on 4 3550.

(version : IOS (tm) C3550 Software (C3550-I9K2L2Q3-M), Version 12.1(14)EA1a )

Dec 8 15:03:18.215: %SYS-2-MALLOCFAIL: Memory allocation of 1680 bytes failed from 0x164AD4, alignment 0

Pool: I/O Free: 2712 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 5

-Traceback= 1ABD2C 1AD358 164AD8 1B933C 1B9584 1CCCD8

After investigation, it's sound like a tempory loop like says ruwhite.

Are you using RSTP ?

Regards.

New Member

I had familiar problem at

I had familiar problem at edge router c851. We found loops at client network.

%SYS-2-MALLOCFAIL: Memory allocation

2490
Views
0
Helpful
10
Replies
CreatePlease login to create content