Solved: Catalyst 2960 go crazy with 300++ days uptime

minko_at_work · ‎07-12-2009

We have a bunch of cisco catalyst 2960 switches (WS-C2960G-24TC-L, WS-C2960-48TT-L) with various IOS 12.2(25)SEE3, 12.2(35)SE1 with uptime 300 days and more that have gone mad: they answers very slow by snmp (or just doesn't answer), may not answer from ping, http almost don't work (it may allow you to log in but after that just hangs) and finally cisco don't allow you to login via telnet!!! It just prompts you for username and after you enter first char of it it just closes the session. There are !no errors! in the cisco log and after reboot everything is working ok. What is going on?

Leo Laohoo · ‎07-13-2009

Firstly, you can pump the IOS anytime during the day and set a reboot/reload during the night or low traffic. I'd recommend a Change Control.

Next, "guarantee". Ain't no guarantee in place. This is why I am not recommending any versions. Choose a version of your liking and read the Release Notes.

I've been in organizations where people just configure and rack appliances without checking if the firmware running is the standard set by the company. In the end, it's people like you who has to stick their neck out to fix careless implementations like this.

Just to give you an idea what I'm using for my switches. I'm standardizing 2960/3560/3750 IOS to 12.2(46)SE and bootstrap is 12.2(44)SE5. I know you can't get this bootstrap version on the IOS but I upgrade to the newer 12.2(50) IOS for the bootstrap upgrade and downgrade the IOS.

Hope this helps.

View solution in original post

Leo Laohoo · ‎07-12-2009

Sounds like IOS bugs. Don't you want to consider upgrading the IOS? They seem pretty old.

minko_at_work · ‎07-13-2009

The problem is that we have several dozens of cisco catalysts. And we don't have the permission to perform ios upgrade during worktime. And finally, where is the guarantee that this the IOS problem and the situation doesn't happen again? We have to be sure that this is ios bug and the bug is fixed. Please, help!

iyde · ‎07-13-2009

Could be a memory leaking issue. Try checking the "show mem stat" output regularly and see if the free mem and the max mem is decreasing.

HTH

minko_at_work · ‎07-13-2009

We have checked free mem value and it seems to be just fine:

Processor Pool Total: 42293608 Used: 6922172 Free: 35371436

I/O Pool Total: 4186112 Used: 1654528 Free: 2531584

CPU usage also normal: 12-15%

To observe a long-time changes we need another year of observations and anyway it won't fix IOS problems. I don't believe that there were no such problems with cisco catalyst before.

We just need to make descision now.

Leo Laohoo · ‎07-13-2009

Firstly, you can pump the IOS anytime during the day and set a reboot/reload during the night or low traffic. I'd recommend a Change Control.

Next, "guarantee". Ain't no guarantee in place. This is why I am not recommending any versions. Choose a version of your liking and read the Release Notes.

I've been in organizations where people just configure and rack appliances without checking if the firmware running is the standard set by the company. In the end, it's people like you who has to stick their neck out to fix careless implementations like this.

Just to give you an idea what I'm using for my switches. I'm standardizing 2960/3560/3750 IOS to 12.2(46)SE and bootstrap is 12.2(44)SE5. I know you can't get this bootstrap version on the IOS but I upgrade to the newer 12.2(50) IOS for the bootstrap upgrade and downgrade the IOS.

Hope this helps.

hobbe · ‎07-14-2009

ok lets make this an example case.

1) choose an ios you think will work for you.

2) test all your configurations for bugs and faliures on the spare equipment you ofcourse have incase of an emergency.

since you have WS-C2960G-24TC-L and WS-C2960-48TT-L you ofcourse have atleast one of each laying around incase a powersupply breaks down and so on.

3) when you have tested and checked and it works just fine, copy the IOS to the flash of the devices, I personally only use the .BIN file and dont bother with the http version. less code = less bugs and problems. prepare the equipment and make any neccesary changes.

4) anounce that a network outage will occur at day X at time Y "to whom it may cencern" and that this is due to service upgrade in the network. do this some time in advance.

5) stay at workt that day and do the upgrade offworking hours. (set boot system variable and reload the switches)

check to make shure things are coming up and working.

6) Sit down and plan a contingency plan and the next upgrade of the switches.

if an apropriate amount of time passes or a bug arises

go to 1

nobody bothers you when you need to reload the servers after service packs and so on right? or service the car?

Same thing with the network, it also needs some tender care and love.

if it does not get it it will degrade and break down at the worst possible time.

PS. Dont use telnet. use ssh.

There are some nice things you can do with a computer and some usb-multiserial interfaces and a computer that you can always keep in control via the serial interface of the switches.

Good luck.

HTH

Leo Laohoo · ‎07-14-2009

Thanks for the ratings.