WS-C3750X-48P Switch stack crash.

Nikos Nicolaides · ‎02-06-2014

Hello,

I have a stack of 9x3750X switches and one of them has just crashed. Can anyone tell me what the problem might be?

6 15:22:24.608: %SWITCH_QOS_TB-5-TRUST_DEVICE_DETECTED: cisco-phone detected on port Gi4/0/39, port's configured trust state is now operational.

6 15:22:27.847: %PLATFORM-1-CRASHED: System previously crashed with the following message: (Switch-4)

6 15:22:27.847: %PLATFORM-1-CRASHED: Cisco IOS Software, C3750E Software (C3750E-IPBASEK9NPE-M), Version 12.2(55)SE1, RELEASE SOFTWARE (fc1) (GB-LON-

6 15:22:27.847: %PLATFORM-1-CRASHED: Technical Support: http://www.cisco.com/techsupport (Switch-4)

6 15:22:27.847: %PLATFORM-1-CRASHED: Compiled Thu 02-Dec-10 06:08 by prod_rel_team (Switch-4)

6 15:22:27.847: %PLATFORM-1-CRASHED: (Switch-4)

6 15:22:27.847: %PLATFORM-1-CRASHED: Data TLB Miss Exception (0x1100)! (Switch-4)

6 15:22:27.847: %PLATFORM-1-CRASHED: (Switch-4)

6 15:22:27.847: %PLATFORM-1-CRASHED: SRR0 = 0x0119D754 SRR1 = 0x00029230 SRR2 = 0x01AB3880 SRR3 = 0x00029230 (Switch-4)

6 15:22:27.847: %PLATFORM-1-CRASHED: ESR = 0x00000000 DEAR = 0x94220004 TSR = 0x84000000 DBSR = 0x00000000 (Switch-4)

6 15:22:27.847: %PLATFORM-1-CRASHED: (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: CPU Register Context: (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: Vector = 0x00001100 PC = 0x0119D754 MSR = 0x00029230 CR = 0x50000055 (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: LR = 0x011D0014 CTR = 0x011CFFE8 XER = 0xE000001E (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: R0 = 0x2F800000 R1 = 0x0593EAD8 R2 = 0x00000000 R3 = 0x9421FFE8 (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: R4 = 0x05165C14 R5 = 0x04F5AB48 R6 = 0x04F5AB48 R7 = 0x0593EB0C (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: R8 = 0x0593EB68 R9 = 0x034A0ED4 R10 = 0xFFFFFFF5 R11 = 0x00000000 (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: R12 = 0x00000040 R13 = 0x0B98279C R14 = 0x01199390 R15 = 0x00000000 (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: R16 = 0x0E7AE16E R17 = 0x00000000 R18 = 0x00000001 R19 = 0x00000076 (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: R20 = 0x00000000 R21 = 0x00000001 R22 = 0x00000000 R23 = 0x00000000 (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: R24 = 0x00000FFF R25 = 0x04F5AB48 R26 = 0x00000000 R27 = 0x0593EB68 (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: R28 = 0x05165C14 R29 = 0x00000FFF R30 = 0x0593EB68 R31 = 0x0593EB0C (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: Stack trace: (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: PC = 0x0119D754, SP = 0x0593EAD8 (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: Frame 00: SP = 0x0593EAE0 PC = 0x011E21FC (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: Frame 01: SP = 0x0593EAF0 PC = 0x011A0BC0 (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: Frame 02: SP = 0x0593EB60 PC = 0x011E8A40 (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: Frame 03: SP = 0x0593EB98 PC = 0x01198340 (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: Frame 04: SP = 0x0593EBB0 PC = 0x01199420 (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: Frame 05: SP = 0x0593EBB8 PC = 0x01EF1968 (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: Frame 06: SP = 0x00000000 PC = 0x01EE83D8 (Switch-4)

6 15:22:27.855: %PLATFORM-1-CRASHED: (Switch-4)

6 15:22:34.363: %SWITCH_QOS_TB-5-TRUST_DEVICE_DETECTED: cisco-phone detected on port Gi4/0/6, port's configured trust state is now operational.

6 15:22:35.362: %SWITCH_QOS_TB-5-TRUST_DEVICE_DETECTED: cisco-phone detected on port Gi4/0/6, port's configured trust state is now operational.

6 15:22:35.387: %SWITCH_QOS_TB-5-TRUST_DEVICE_DETECTED: cisco-phone detected on port Gi4/0/6, port's configured trust state is now operational.

TIA,

Nicos Nicolaides

TIA, Nicos Nicolaides

Tinya Cheng · ‎02-06-2014

Hi Nicos,

Majority of folks on this forum will not have the tools to decode the tracebacks for you, which is necessary to see what caused the crash.

For RCA of this issue, I suggest opening a case with TAC and including the above logs, the crashfiles, and a "show tech".

Joseph W. Doherty · ‎02-07-2014

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Like Tinya, I too would suggest such analysis is really something for TAC. However, as you're running (55)SE1, I would suggest upgrading to (55)SE8.

Leo Laohoo · ‎02-07-2014

Hmmmmm ... more than 1 year uptime. This looks like a memory hog- or memory allocation caused crashed.

If you have the time to upgrade your 3750 stack, use 12.2(55)SE8. The current IOS, although slightly stable, is not yet fully "mature". The "55" IOS started getting stable after the release of 12.2(55)SE5.