WS-C3750G-12S stack problems can't recover switches even in standalone mode (no stack)

Unanswered Question
Oct 2nd, 2014
User Badges:

I have (2) WS-C3750G-12S switches that I am reutilizing (extras IT had available) that were both initially at fw 12.2(25)SEC configured in a stack. Switch 1 appeared to have a problem and would not boot after a power outage (Unexpected exception to CPUvector 1100, PC = 44EC5C

-Traceback= 44EC5C 4536DC 459918 45E998 45EBDC 79FC48 799E8C) and is now showing checksum errors... (I was able to get the system up at one point (see below). I don't know how, but was able to upgrade the fw to 12.2(55)SE6).

 

Currently switch 1 is getting checksum errors and is reloading without ever actually completing the boot process - both connected in the stack and standalone, no differences. The output from this is below.


  ...      cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706

 

Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(55)SE6, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Mon 23-Jul-12 13:22 by prod_rel_team
Image text-base: 0x01000000, data-base: 0x02F00000

Initializing flashfs...

flashfs[1]: 79 files, 6 directories
flashfs[1]: 0 orphaned files, 0 orphaned directories
flashfs[1]: Total bytes: 15998976
flashfs[1]: Bytes used: 13567488
flashfs[1]: Bytes available: 2431488
flashfs[1]: flashfs fsck took 2 seconds.
flashfs[1]: Initialization complete....done Initializing flashfs.
Checking for Bootloader upgrade.. not needed

Code checksum failed--memory corruption likely

%Software-forced reload

Preparing to dump core...

 00:00:09 UTC Mon Mar 1 1993: Unexpected exception to CPUvector 2000, PC = 1BD2A98
-Traceback= 1BD2A98 1BD2A98 2AF6BF0 1BD4A68 1BCB4E0


=== Flushing messages (00:00:09 UTC Mon Mar 1 1993) ===

Buffered messages:
Queued messages:
Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(55)SE6, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Mon 23-Jul-12 13:22 by prod_rel_team

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x02B032B4  SRR1 = 0x00029230  SRR2 = 0x01BD2A98  SRR3 = 0x00029230
ESR = 0x00000000  DEAR = 0x00000000  TSR = 0x84000000  DBSR = 0x10000000

CPU Register Context:
Vector = 0x00002000  PC = 0x01BD2A98  MSR = 0x00029230  CR = 0x20000008
LR = 0x01BD2A98  CTR = 0x01BCD74C  XER = 0x6000006A
R0 = 0x01BD2A98  R1 = 0x04371340  R2 = 0x00000000  R3 = 0x00000000
R4 = 0x02B032B4  R5 = 0x00029230  R6 = 0x00029230  R7 = 0xBEEFCAFE
R8 = 0x00000000  R9 = 0x03EC0000  R10 = 0x03DD3180  R11 = 0x03DD3090
R12 = 0x000024C4  R13 = 0x00110000  R14 = 0x02AF6BDC  R15 = 0x00000000
R16 = 0x00000000  R17 = 0x00000000  R18 = 0x00000000  R19 = 0x00000000
R20 = 0x00000000  R21 = 0x00000000  R22 = 0x00000000  R23 = 0x00000000
R24 = 0x00000000  R25 = 0x00000000  R26 = 0x00000000  R27 = 0x00000000
R28 = 0x00000000  R29 = 0x02DD9C08  R30 = 0x06A14953  R31 = 0x00000003

Stack trace:
PC = 0x01BD2A98, SP = 0x04371340
Frame 00: SP = 0x04371350    PC = 0x01BD2A98
Frame 01: SP = 0x04371370    PC = 0x01BD5D94
Frame 02: SP = 0x04371388    PC = 0x02AF6BF0
Frame 03: SP = 0x04371390    PC = 0x01BD4A68
Frame 04: SP = 0x00000000    PC = 0x01BCB4E0


Switch uptime is 11 seconds

cisco WS-C3750G-12S (PowerPC405) processor (revision M0) with 131072K bytes of memory.
Processor board ID CAT0926X07J

Ok, the memory might be on it's way out on switch 1 - but I can get into the bootloader prompt but that is as far as I seem to be able to get. Anything I look at or change, produces the same result.

 

The big issue I have now is that I have no way to access switch 2, even if I disconnect the switch from the stack and run it standalone. When I apply power, the Syst LED flashes a few times and then goes solid. Nothing is written out of the console port.

 

I had a failed xmodem transfer attempt to put another firmware (.bin) file on the switch that failed due to insufficient memory earlier and had taken the switch out of the stack, connected the stack cables between ports 1 and 2 (a 1 switch stack) and that let me get into the switch 2 and set an IP address etc. (switch# setup). After I did that, I was able to power the switch down, disconnect the cables, boot the switch (successfully) then connect switch 1 again and power it up. Everything appeared to be OK, they saw that they were a stack and assigned stack numbers as expected. This is when I was able to update the firmware (archive download-sw /overwrite /reload....) which updated both switches in the stack, however it gave me a message that it skipped the reload because changes had been made. I saved the configuration and rebooted the switch and that is when I got back into the boot loop on switch1 with switch 2 being unresponsive.

 

Can anyone help me understand how I can gain access to switch 2 again? Previously, I could get to the bootloader prompt on switch 2, but now, nothing.

 

Also, not that I think it really matters, but it may... I have removed all of the SFPs - mixture of 1000F (fiber)and 1000T (copper) - all Cisco. The two switches have different bootloader versions (12.2(44)xxx (not sure) on switch 1 and 12.2(25r)SEB1 on switch 2.

 

Any insight would be helpful.

 

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Leo Laohoo Thu, 10/02/2014 - 16:13
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

%Software-forced reload

Try using 12.2(55)SE9.

Robert Ulmer Thu, 10/02/2014 - 16:25
User Badges:

Leo,

 

Thank you for replying.

 

Sending that with xmodem (.bin file only, not .tar which has web pages which are desirable to non-IT folks like me...) as your reply came in. Needed someone in IT w/access to service contract info to download. SE6 was most recent I could get without the service contract number.

 

Fingers crossed.

Robert Ulmer Thu, 10/02/2014 - 18:22
User Badges:

Leo,

There was a glimmer of hope that was quickly dashed.

 

Switch 1 (now disconnected from stack with cable between stack port 1 and 2) booted, ran for 30 -120 seconds then got the same %Software-forced reload.

 

I had something going on in there so I have formatted the flash: file system from the bootloader prompt. I'm sending xmodem xfer again to see if there might have been some random 'ghost' due to numerous power outages / crashes etc..

 

I'll keep trying for now.

Actions

This Discussion