unstable iOS on 3750's

[keithsauer507]

We've had issues with our 9 switch 3750 stack crashing on 12.2(58)SE2 so tac said to upgrade to 15.0(2)SE4. We did that upgrade today and while at first it seemed to take, the stack eventually froze and after a few reboots we just couldn't log in via ssh or console. Console would complain about memory.

So we eventually broke the stack and one by one downgraded to 12.2(55)SE8. Was a painful process as the last 3 switches are 3750X POE so they had to do the microcode update again. We also took out a 3750v2 48 port since we consolidated some things.

So now all seems working with 3 3750X PoE and 5 3750v2 on 12.2(55)SE8. Think that was a good idea downgrading to that version of IOS?

Sent from Cisco Technical Support iPhone App

[Jose Solano]

Hi,

This may be related to the bug CSCuf32893 you can check it out.


Sent from Cisco Technical Support iPhone App

[keithsauer507]

That post sounds logical. I did update to 15.0.2(SE4) on a small stack of 3 switches (2 3750v2 and 1 3750X PoE) and that went fine.

But our big stack hated 15... We couldn't even connect to a remote office via one of its ports connected to an ME3400. Rolling back to 12.2.55(SE8) appears to resume stability and we can now talk to the remote branch office.

I just want to make sure 12.2.55(SE8) is a good version. 12.2.58 (I forget the SE#) would crash and reboot the core sw stack if there was a network vulnerability scan done or a stress test done for our new VoIP phone system we are rolling out.

Sent from Cisco Technical Support iPhone App

[keithsauer507]

Well here's what's in the stack today and keep in mind we removed switch 4 (a 48 port 10/100) because we consolidated a lot of stuff onto switch 7,8,9 PoE using the new VoIP phones pass through.

*    1 52    WS-C3750G-48TS     12.2(55)SE8           C3750-IPSERVICESK9-M

     2 52    WS-C3750V2-48TS    12.2(55)SE8           C3750-IPSERVICESK9-M

     3 52    WS-C3750V2-48TS    12.2(55)SE8           C3750-IPSERVICESK9-M

     5 52    WS-C3750V2-48TS    12.2(55)SE8           C3750-IPSERVICESK9-M

     6 52    WS-C3750V2-48TS    12.2(55)SE8           C3750-IPSERVICESK9-M

     7 54    WS-C3750X-48P      12.2(55)SE8           C3750E-UNIVERSALK9-M

     8 54    WS-C3750X-48P      12.2(55)SE8           C3750E-UNIVERSALK9-M

     9 30    WS-C3750X-24P      12.2(55)SE8           C3750E-UNIVERSALK9-M

WS-C3750V2-48TS-S Version ID V01 is the first 5 then WS-C3750X-48P-E ipservices V04 are the last switches.

So on 15.0(2)SE4, this core switch stack drops connection to a remote office and does not allow local console or SSH login.  Backing it down to 12.2(55)SE8 seems to fix all our problems.  The original cause for upgrade was because we were on 12.2(58)SE2, and the core switch stack would crash and reboot during network vulnerability scans and also when our new Voip phone system vendor wanted to do a stress test.  Crashing the core in the middle of the day is not fun, especially since its about 6 1/2 minutes to reboot the stack.  Things not quick that's for sure.

TAC told us to upgrade to 15.0(2)SE4 to resolve those crashes because of some HTTP / HTTPs DoS vulnerability that caused the stack to reboot.  Well that was bad advice, and they had the entire show tech-support dumps from those crashes.  If our switch stack can't handle 15 train, then why would they suggest it?

I have a smaller stack that is working fine on 15, but its only these 3 switches:

*    1 52    WS-C3750V2-48TS    15.0(2)SE4            C3750-IPSERVICESK9-M

     2 26    WS-C3750V2-24TS    15.0(2)SE4            C3750-IPSERVICESK9-M

     3 54    WS-C3750X-48P      15.0(2)SE4            C3750E-UNIVERSALK9-M

Here is the memory summary of the core stack 8 switches running 12.2(55)SE8

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)

Processor    4338050    68455988    46004212    22451776    20181964    16768896

      I/O    6400000    12582912     8741984     3840928     2741700     2852672

Driver te    2E00000     1048576          44     1048532     1048532     1048532

Here is the memory summary of the 3 switch stack running 15.0(2)SE4

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)

Processor    4DC8CD4    56827872    42337916    14489956    12333556    12574356

      I/O    6400000    12582912     9397552     3185360     2597428     3183248

Driver te    3600000     1048576          44     1048532     1048532     1048532

So less memory free on the small stack running 15 train.  I can't imagine that running 9 switches in a stack plus extra config that makes it the center of our routing universe, that plain old 3750v2's could handle it.

So my question is, is 12.2.55(SE8) a good release?

[Leo Laohoo · Accepted solution]

So my question is, is 12.2.55(SE8) a good release?

One of the best releases of IOS started from 12.2(55)SE6 to SE8.  I strongly recommend 12.2(55)SE8 as it's very stable.  All my 3750/G/E/X are running this version without any issues or even crashes.

15.0(2)SE4 is also not bad.  The rest are plain rubbish. 

[keithsauer507]

Thank you Leo. That's what I suspected and I'm glad your able to confirm based on your experiences.

I have a small stack with only 2 3750v2's and 1 3750X, and so far nothing wrong with 15 but for our larger stack of 6 3750v2's and 3 3750X (9 switch total), I just don't think it could handle 15 due to memory. Not sure why not only SSH and Console broke but so did WAN. Only thing specific to the WAN is the one port specified speed 10 and it had an IP address. The switch stack has routing tables on it too. Maybe something changed in 15.X.

Anyway with the 12.2(55)SE8 it seems stable so far. We also disabled http and https server since why have that running if we use CLI all the time.

Sent from Cisco Technical Support iPhone App

[Leo Laohoo]

Thanks for taking the time to rate our posts, Keith.