12.2(58)SE2 on a 2960 cause memory leaks

pasztor.richard · ‎09-07-2011

Hi all,

I had a very disappointed experience today during the planned upgrade on my 2960 to IOS version 12.2(58)SE2 after business hours. We have a small office with 3x 2960 and a couple of 2950. There is a single 2960 called as the "central" as its the STP root, and most other switches are directly connected to the central. The remaining switches are conected to one of the switches conneting to the central.

I tested the aforementioned IOS on 2x 2960 las t week, but did not upgrade the central one. The 2x switches ran fine for a week, so I decided to upgrade the central one: basically it was a disaster. After the restart, the central became immediately unavailable via SSH (though admin IP pingable). I connected to the console, and realized that even console does not work, as I received the error message: "Low memory", and the console session could not be started. I restarted the switch once again (had to cut the power) and this time I managed to connect via the serial: the free memory was approx 2 MB, though its was over 5 MB if running 12.2(55)SE1. After 1-2 mins, free memory decreased down to 1.8 megs. SSH was still not functioning. At this time, fearing of a sudden memory leak during the night, I had to rollback to the previous version.

My config is quite simple: RSTP root, DHCP snooping, 1 LACP etherchannel, only 5x VLANs with VTP server, MOTD and BANNER (yeah I know its a risky feature since the previous IOS) I wonder what can be the problem here?

I have attached the running config of the central 2960 (removed some sensitive entries)

Somasundaram Jayaraman · ‎09-07-2011

Hi,

There is a bug raised for this issue and it is in assigned to a Development engineer now.

As of now i cannot share the bug id with you, since it is internal to cisco.

Symptomps:

===========

When booting a 2960 with 64Mb of DRAM a low memory error can be observed under certain 

circumstances when running 12.2(58)SE or later

This will prevent acess to the console/ssh/telnet of the switch.

++ As a workaround we need to downgrade the IOS until the bug is fixed.

Hope this helps.

Cheers

Somu

Rate helpful posts

Cedric DELAUNAY · ‎09-19-2011

Hi,

Is there any news about this bug ?

Same problem with our 2960s.

We have more than 180 and some of them can't be managed with telnet.

Which release can we downgrade and be sure do not have any problems ?

12.2.58SE1 ? 12.2.55SE4 ?

Thanks

Alfredo Bosca Bataller · ‎05-19-2015

I have the same issue. And my current IOS is: 12.2(35)SE5. What advice would give me? I can to upgrade the IOS to the last stable: c2960-lanbasek9-mz.150-2.SE7.bin

show memory debug leaks summary

Adding blocks for GD...

Processor memory

Alloc PC Size Blocks Bytes What

0x3D1370 0000000040 0000000001 0000000040 hulc_flash init

0x3D13E4 0000001512 0000000001 0000001512 hulc_flash init

0x3D159C 0000000040 0000000001 0000000040 hulc_flash init

0x669FE0 0000000024 0000000014 0000000336 Init

0x669FE0 0000000056 0000000004 0000000224 Init

0x669FE0 0000000080 0000000002 0000000160 Init

0x669FE0 0000000120 0000000002 0000000240 Init

0x7380B0 0000000024 0000000001 0000000024 Init

0x7380D8 0000000040 0000000001 0000000040 Init

0x7380F4 0000000024 0000000001 0000000024 Init

0x73811C 0000000040 0000000001 0000000040 Init

0x96C878 0000000024 0000000001 0000000024 Init

0x96F0C8 0000000024 0000000001 0000000024 Init

0x96F0C8 0000000200 0000000001 0000000200 Init

Thanks,

http://networkingcontrol.wordpress.com/ #CCNP CSCO11962956

Leo Laohoo · ‎09-07-2011

I've got the same issue with 15.0(1)SE. Mostly affecting the WS-C2960-48-PST. Currently got a TAC Case. Don't know where it is now.

pasztor.richard · ‎09-08-2011

Thanks for the update. I think I'll wait one more release to check this. Strange though, all the 3x 2960 are the same model 2960-24TC-L, and the 2 non-central ones are not experiencing such issue.

Leo Laohoo · ‎09-09-2011

Not happy with the 15.0 IOS. ~~I have good runs using 12.2(58)SE1 IOS.~~ No issues so far.

pasztor.richard · ‎09-09-2011

Its the 1st release of a new major version: I wasnt brave enough to install it on any production equipment.

Leo Laohoo · ‎09-19-2011

Good on you. The 15.0 IOS has a major memory leak even before the config gets loaded. This is happening across the board and not just on the 2960.

I recommend anyone reading this post to stay away from the 15.0 like a plague.

ankugarg · ‎09-20-2011

HI Leolaohoo,

I think its not that bad ...I was of the opinion that memory leak is specific to 2960s...Can you please share the memory leak..I am kinda sure that 3ks won't have memory leak without any config

Leo Laohoo · ‎09-20-2011

I value second opinions, Ankur. Maybe you can make some sense ...

The TAC engineer successfully recreated the issue when loaded 15.0(1) IOS into a 2960 in his first attempt.

As soon as I upgraded to 15.01(SE) (c2960-lanbasek9-mz.150-1.SE) the switch memory went very low, this is without applying the configuration as per your network. Below is the output taken from my lab switch. I would suggest to downgrade the IOS Version image. We are getting in touch with the development team regarding this issue.

Switch#sh processes memory sorted

Processor Pool Total: 21341260 Used: 18412492 Free: 2928768 (Hardly 3MB left, in your case it was 2MB)

I/O Pool Total: 4194304 Used: 2484948 Free: 1709356

Driver te Pool Total: 1048576 Used: 40 Free: 1048536

PID TTY Allocated Freed Holding Getbufs Retbufs Process

0 0 24228980 6113380 15373936 0 0 *Init*

0 0 17229304 16069812 1172040 5380179 2002472 *Dead*

According to the TAC engineer, he has seen some cases of 3750's coming their desk.

Cisco Bug ID: CSCts52797

aranyushkin · ‎10-01-2011

I'm also getting the same problem with WS-C2960-48PST-L. It even can't start CLI and gives a message "

%% Low on memory; try again later" for any keybord input.

glen.grant · ‎10-01-2011

Cisco seems to have a real issue with this . We saw this even on earlier versions of the 3750 where you could not telnet to the box at times . Reload was the only fix.

smogra · ‎10-01-2011

The bug has been added as a open caveat in release notes:

CSCts52797 (Catalyst 2960)

A Catalyst 2960 with 64Mb of DRAM might display low memory on the console after you upgrade the switch to 12.2(58)SE or later.

The workaround is to limit the memory that is used by different features on the switch if this release is required. You can reduce memory usage by minimizing the number of trunk ports and VLANs in use on the switch.

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/release/15.0_1_se/release/notes/OL25301.html#wp1044235

--- Sweta

Please rate helpful posts.

Leo Laohoo · ‎10-02-2011

The workaround is to limit the memory that is used by different features  on the switch if this release is required. You can reduce memory usage  by minimizing the number of trunk ports and VLANs in use on the switch.

Hi Sweta,

This ain't going to work. All my switch has only ONE (1) trunk port and 5 VLANs allowed on the trunk port.

I'll say it once and I'll say it again, the 15.0(1)SE IOS is extremely buggy. I would recommend this version to be pulled out from the website as well as recommend anyone to treat this version like a plague.