%PLATFORM-1-CRASHED on 3750X stack

Steve Prescott · ‎03-03-2014

I have a stack of two 3750X's (WS-C3750X-24P-S) running c3750e-universalk9-mz.150-1.SE3.bin.

One of them (the master at the time) crashed on 27/28th Feb & appears to have kept restarting and crashing again.

Here is the syslog...

Feb 28 04:11:28.770: %PLATFORM-1-CRASHED: System previously crashed with the following message: (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.770: %PLATFORM-1-CRASHED: Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.0(1)SE3, RELEASE SOFTWARE (fc1) (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.770: %PLATFORM-1-CRASHED: Technical Support: http://www.cisco.com/techsupport (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.770: %PLATFORM-1-CRASHED: Compiled Wed 30-May-12 13:41 by prod_rel_team (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.770: %PLATFORM-1-CRASHED: (QG-BLUECP-

Feb 28 04:11:28.770: %PLATFORM-1-CRASHED: ESR: 0x00000000 (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: SRR0: 0x01B97A9C SRR1: 0x00029230 SRR2: 0x01B80DAC SRR3: 0x00029230 (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: BESRA: 0x00000000, BESRB: 0x00000000, BEAR: 0x00000000 (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: CPU Register Context: (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Vector = 0x00000200 PC = 0x01B80DAC MSR = 0x00029230 CR = 0x30009005 (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: LR = 0x01B80C90

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: R0 = 0x00000000 R1 = 0x06341CC8 R2 = 0x00000000 R3 = 0x00000001 (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: R4 = 0x045A18E8 R5 = 0x00008002 R6 = 0x00000000 R7 = 0x0F000000 (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: R8 = 0x00000000 R9 = 0x01B7AB64 R10 = 0x00000000 R11 = 0x00178298 (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: R12 = 0x30000003 R13 = 0x0D98279C R14 = 0x00000000 R15 = 0x00000000 (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: R16 = 0x01E8078C R17 = 0x03C00000 R18 = 0x00000000 R19 = 0x00000000 (QG-BLUECP-3750X-01-1)

Feb 28 04:11

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: R24 = 0x045A1BB0 R25 = 0x045A1DCC R26 = 0x00000350 R27 = 0x045A18E8 (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: R28 = 0x00008002 R29 = 0x00000001 R30 = 0x045A18E8 R31 = 0x0ED0FAA8 (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Stack trace: (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: PC = 0x01B80DAC, SP = 0x06341CC8 (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 00: SP = 0x06341CE0 PC = 0x045A18E8 (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 01: SP =

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 03: SP = 0x06341DF0 PC = 0x01E80790 (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 04: SP = 0x06341E10 PC = 0x01E80E5C (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 05: SP = 0x06341E58 PC = 0x01E7DB4C (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 06: SP = 0x06341EC0 PC = 0x022DF73C (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 07: SP = 0x06341F20 PC = 0x022E00E8 (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 08: SP = 0x06341F40 PC = 0x022E0218 (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 10: SP = 0x00000000 PC = 0x028478FC (QG-BLUECP-3750X-01-1)

Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: (QG-BLUECP-3750X-01-1)

Mar 3 08:38:41.986: %SYS-5-PRIV_AUTH_FAIL: Authentication to privilege level 15 failed by admin on vty0 (10.191.6.1)

The following files were created in flash...

QG-BLUECP-3750X-01-1#dir

Directory of flash:/crashinfo_ext/

475 -rwx 329296 Feb 27 2014 23:41:04 +00:00 crashinfo_ext_1

476 -rwx 260421 Feb 28 2014 03:07:18 +00:00 crashinfo_ext_2

477 -rwx 320129 Feb 28 2014 03:12:33 +00:00 crashinfo_ext_3

478 -rwx 290248 Feb 28 2014 03:16:54 +00:00 crashinfo_ext_4

479 -rwx 284496 Feb 28 2014 03:21:18 +00:00 crashinfo_ext_5

480 -rwx 65 Feb 28 2014 03:25:34 +00:00 crashinfo_ext_6

481 -rwx 276767 Feb 28 2014 03:29:52 +00:00 crashinfo_ext_7

482 -rwx 271592 Feb 28 2014 03:34:18 +00:00 crashinfo_ext_8

483 -rwx 65 Feb 28 2014 03:39:26 +00:00 crashinfo_ext_9

484 -rwx 65 Feb 28 2014 03:43:49 +00:00 crashinfo_ext_10

485 -rwx 65 Feb 28 2014 03:48:14 +00:00 crashinfo_ext_11

486 -rwx 65 Feb 28 2014 03:52:33 +00:00 crashinfo_ext_12

487 -rwx 65 Feb 28 2014 03:56:47 +00:00 crashinfo_ext_13

488 -rwx 65 Feb 28 2014 04:01:29 +00:00 crashinfo_ext_14

489 -rwx 65 Feb 28 2014 04:06:30 +00:00 crashinfo_ext_15

57671680 bytes total (13599744 bytes free)

QG-BLUECP-3750X-01-1#cd ..

QG-BLUECP-3750X-01-1#cd crashinfo

QG-BLUECP-3750X-01-1#dir

Directory of flash:/crashinfo/

491 -rwx 1761 Mar 1 1993 00:00:08 +00:00 crashinfo_1

492 -rwx 1446 Mar 1 1993 00:00:08 +00:00 crashinfo_2

493 -rwx 1761 Mar 1 1993 00:00:08 +00:00 crashinfo_3

494 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_4

495 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_5

496 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_6

497 -rwx 1761 Mar 1 1993 00:00:08 +00:00 crashinfo_7

498 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_8

499 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_9

500 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_10

501 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_11

502 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_12

503 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_13

504 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_14

505 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_15

57671680 bytes total (13599744 bytes free)

The second switch took over as the stack master as expected...

QG-BLUECP-3750X-01#sh switch

Switch/Stack Mac Address : 4403.a727.4980

H/W Current

Switch# Role Mac Address Priority Version State

----------------------------------------------------------

1 Member 6c41.6a05.c780 15 1 Ready

*2 Master 4403.a727.4980 1 1 Ready

I have attached the crashdump files from flash1 that I listed above.

Is it possible for anyone to decode why the switch crashed both initially and then repeatedly when trying to come back up.

btw, It has been up for three days now since these events, but I would like to get to the root cause.

Many thanks

Steve

Jose Solano · ‎03-03-2014

Hi,

It will be hard for someone on this forum to have the proper tools to the decode the crash files, that said my suggestion will be to reach out TAC for the proper assistance.

Regards,