Memory fail.

examples20001 · ‎09-12-2006

Hi All

I saw the below %SYS-2-MALLOCFAIL log in my Catalyst switch for 30 mins.

After that there is no messages at all for past 1 week and switch is running good and processor memory usage is around 7% as of now.

Why this message got diaplayed at that time? Is it because of some virus attack or some other thing?

Sep 5 16:36:33 XXX: %SYS-2-MALLOCFAIL: Memory allocation of 1680 bytes failed f

rom 0x152778, pool I/O, alignment 0

-Process= "Pool Manager", ipl= 0, pid= 5

-Traceback= 192E90 194338 15277C 19FDCC 1B2D3C

show version:

Cisco Internetwork Operating System Software

IOS (tm) C3550 Software (C3550-I5Q3L2-M), Version 12.1(11)EA1, RELEASE SOFTWARE

(fc1)

Compiled Wed 28-Aug-02 10:03 by antonino

Image text-base: 0x00003000, data-base: 0x0071D658

ROM: Bootstrap program is C3550 boot loader

AXX00 uptime is 39 weeks, 4 days, 18 hours, 47 minutes

System returned to ROM by power-on

System restarted at 20:20:28 XXX Wed Dec 7 2005

System image file is "flash:c3550-i5q3l2-mz.121-11.EA1/c3550-i5q3l2-mz.121-11.EA

1.bin"

cisco WS-C3550-24 (PowerPC) processor (revision G0) with 65526K/8192K bytes of m

emory.

Processor board ID CAT0639X0Q7

Last reset from warm-reset

Bridging software.

Running Layer2/3 Switching Image

Ethernet-controller 1 has 12 Fast Ethernet/IEEE 802.3 interfaces

Ethernet-controller 2 has 12 Fast Ethernet/IEEE 802.3 interfaces

Ethernet-controller 3 has 1 Gigabit Ethernet/IEEE 802.3 interface

Ethernet-controller 4 has 1 Gigabit Ethernet/IEEE 802.3 interface

24 FastEthernet/IEEE 802.3 interface(s)

2 Gigabit Ethernet/IEEE 802.3 interface(s)

The password-recovery mechanism is enabled.

384K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address: 00:0A:F4:F3:6C:00

Motherboard assembly number: 73-5700-09

Power supply part number: 34-0966-02

Motherboard serial number: CAT063804PQ

Power supply serial number: DCA06350RD3

Model revision number: G0

Motherboard revision number: A0

Model number: WS-C3550-24-EMI

System serial number: CAT0639X0Q7

Configuration register is 0x10F

leighharrison · ‎09-12-2006

Hey there,

This could have happened for a number of reasons. You would get a SYS-2-MALLOCFAIL if your switch has temporarily or permanently used all available memory. Or if the switch has fragmented all of the memory down in to such small peices that it can't find a chunk big enough for its needs. Or it could be a memory leak, which would be a software bug. The traceback sometime points to a bug.

You could do a "show proc mem" and have a look to see what memory is free and if any processes are hogging the memory.

I have had this before and it was a virus. You could see that the main user of memory was the IP Packets. But this was on a router, on a 3550, you would expect the switching to take place so fast that the packets are not in memory for too long.

So, have a look at the "show proc mem" and see if anything jumps out - or post it if you like. If it's memory fragmentation, then a reboot could sort. If it's a software bug, then you should find it on the bug tracker.

You could also try turning on nbar protocol-discovery to see what protocols are hogging the interfaces - that may also help in the hunt for a virus.

Hope that helps,

LH

examples20001 · ‎09-12-2006

Hello Harrison,

Thank you very much for the reply.

Top 3 processes that are holding less than 1 MB of memory are:

'HMATM Learn proc' is holding 65408 bytes

'VFM/VQM Merge Pr' is holding 62320 bytes

'Entity MIB API' is holding 31064 bytes

What are these 3 processes used for? Is these processes are the showing problem?

show process memory:

Total: 55052848, Used: 4314664, Free: 50738184

Show memory dead show lot of Virtual Exe details. Is it because of some virus attack to switch interface?

Attached file is the details of show memory; show process memory; show memory dead.

leighharrison · ‎09-14-2006

Hi there,

This doesn't seem too bad, there is a good amount of free memory - 90%.

Is the box doing much routing? If so, is CEF turned on?

I'd suggest rebooting it and having a look at the memory usage then, so you can build up an idea of what is usual for your network.

I would also use snmp to track/graph free memory, so that you can see what it dips to when the messages appear.

I'm pretty sure the OID is 1.3.6.1.4.1.9.2.1.8.0&1.3.6.1.4.1.9.2.1.8.0

Hope that helps,

LH

Please rate all posts

examples20001 · ‎09-18-2006

Hi,

Thank you very much for the reply.

I rebooted the switch and now the processor memory is 13% usage.

I`ll be monitoring the device for some perioed of time and will check which processor is showing problem in period of time.

I`ll try with snmp to track/graph free memory.

Thank you.